File processing method and device and medium

Abstract

The invention discloses a file processing method and device and a medium, relates to the technical field of computers, is applied to a server, and comprises the following steps: generating a fragmentation scheme aiming at an uploaded file by utilizing a dynamically extensible micro-service cluster and considering the resource allocation and utilization condition of the micro-service cluster; from the perspective of concurrency, the speed of uploading the fragmented data by the client is improved by utilizing the micro-service cluster, so that resources of the server are effectively utilized, and the file uploading speed and the file uploading error-tolerant rate are greatly improved; furthermore, a parallel compression scheme is established based on the constructed dynamic dictionary, and persistence is performed on the dynamically constructed dictionary, so that a good compression rate can be ensured to reduce the occupation of storage resources while the compression / decompression speed is increased; in general, the efficiency of file transmission and compression tasks can be greatly improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a file processing method, device, and medium. Background Art

[0002] In today's digital age, with the continuous growth of data interaction and storage requirements, the file upload function has become an indispensable part of many applications. Traditional large file fragmentation usually involves direct upload or serial upload of fragments, and involves compressed storage of the uploaded file. Usually, a certain compression algorithm is used to compress the entire file and then store it on a specified file server. However, this method has several disadvantages: one is that the upload speed is slow, and after a direct upload fails, the entire file needs to be re-uploaded, resulting in a very poor functional experience; the other is that the compression speed and compression ratio are average, and there is room for optimization. Summary of the Invention

[0003] This application provides a file processing method, device, and medium to at least solve the problem of low efficiency in file transmission and compression tasks in related technologies.

[0004] In a first aspect, this application provides a file processing method, which is applied to a server and includes:

[0005] When a client executes a file upload task through a fragmented upload execution method, detect the resource allocation and utilization situation of the microservice cluster, and generate a corresponding fragmentation scheme for the file upload task according to the resource allocation and utilization situation; the microservice cluster is a dynamically scalable cluster;

[0006] Obtain the fragmented data determined and concurrently transmitted at one time by the client based on the fragmentation scheme, and distribute the fragmented data to the microservice cluster based on a preset routing rule;

[0007] Based on the dynamic dictionary in the shared cache, use the microservice cluster to start multiple threads to concurrently compress the fragmented data to obtain fragmented compressed data;

[0008] After the multiple threads finish execution, persist the dynamic dictionary to obtain a fragmented dictionary file for decompressing the fragmented compressed data according to the fragmented dictionary file.

[0009] In a second aspect, this application also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of any of the above file processing methods when executing the computer program.

[0010] In a third aspect, this application also provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above file processing methods are implemented.

[0011] Fourthly, the present application also provides a computer program product, including a computer program, which when executed by a processor, implements the steps of any of the above file processing methods.

[0012] Beneficial effects: Currently, the miniaturization, containerization, and clustering of microservices are the mainstream trends. Through the present application, for the file upload task of sharded upload, from the perspective of resource optimization, the resource allocation and utilization of the microservice cluster are fully considered, and a sharding scheme for the uploaded file is generated; at the same time, from the perspective of concurrency, the microservice cluster is utilized to improve the speed of the client to upload sharded data. In this way, it can ensure that the upload task is completed reliably and quickly, not only effectively utilizing the resources of the server, but also greatly improving the speed of file upload and the fault tolerance rate of file upload. Further, a parallel compression scheme is established based on the constructed dynamic dictionary, and the dynamically constructed dictionary is persisted. It can improve the compression / decompression speed while ensuring a good compression rate to reduce the occupation of storage resources, and at the same time, the dictionary persistence provides good search efficiency for decompression. Generally speaking, the present invention can greatly improve the efficiency of file transmission and compression tasks. Description of the Drawings

[0013] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0014] Figure 1 It is a flowchart of a file processing method disclosed in the present application;

[0015] Figure 2 It is a schematic diagram of a complete request and response process of file processing disclosed in the present application;

[0016] Figure 3 It is a schematic diagram of the structure of a file processing device disclosed in the present application. Detailed Embodiments

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0018] It should be noted that in the description of this application, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] In today's digital age, with the continuous growth of data interaction and storage requirements, the file upload function has become an indispensable part of many applications. When using various applications, users often have the need to share files such as pictures, documents, and videos. For example, sharing photos on social platforms and uploading work documents in office systems, which requires applications to have an efficient and stable file upload function.

[0020] With the development of cloud computing and big data technologies, the way of storing and processing data has changed significantly. Enterprises need to upload a large amount of file data to the cloud for storage and analysis, which poses new challenges to the speed, reliability, and compressed storage capacity of file uploads.

[0021] Currently, many applications limit the size of files that users can upload. However, with the continuous evolution of digital business requirements and the development of communication technologies, the size of files uploaded by clients has an inevitable tendency to expand. Moreover, with the widespread and in-depth application of the microservices design concept of server programs and the rise of containerized deployment of cloud platform services, SaaS services are becoming more and more miniaturized, clustered, and elastic. The mode of a single service handling file upload requests can no longer meet the requirements of users for program performance.

[0022] The following are the optimization solutions proposed for the file upload scenario currently:

[0023] 1) After the server receives the file upload request from the client, it performs verification through an interceptor. If it passes, it allocates connection nodes. The server receives the file uploaded by the client and the file detection request, uses the microservice file check function to obtain the uploaded file information and check the upload file status. When the uploaded file does not exist, the client calculates the number of stream transfer objects based on the number of hardware-available startable threads, performs binary data segmentation based on the un-successfully uploaded file to obtain multiple shard files, assigns index values, calculates the MD5 (Message-Digest Algorithm 5) value, and uploads it to the server through the stream transfer object for comparison to determine the upload result of the shard file. When the upload of the shard file is completed, the uploaded completed shard files are merged, and the MD5 value of the merged file is calculated to verify the integrity of the uploaded file, effectively improving the file upload efficiency.

[0024] 2) Read the data to be compressed from the storage device through parallel I / O (Input / Output), evenly divide the data to be compressed into N data blocks, enable multiple processes, each process reads the corresponding data block and performs parallel compression encoding on the data block using the multi-threaded method, and finally merge the data after compression encoding by each process and write it into the final output file.

[0025] The above optimization scheme has the following disadvantages:

[0026] First of all, for the optimization scheme of sharded upload of large files, starting from the connection overhead of HTTP (Hyper Text Transfer Protocol) / HTTPS (Hyper Text Transfer Protocol over Secure Socket Layer), use TCP (Transmission Control Protocol) connection to replace HTTP / HTTPS connection. This scheme has poor generality in using the TCP protocol to process file uploads and cannot cover most file upload scenarios; in addition, this scheme has limited performance improvement and fails to utilize the microservice cluster to improve the upload speed from the perspective of concurrency.

[0027] Secondly, file data upload compression is usually serial compression, and the compression task is a resource-intensive task. When the file size is very large, serial compression will cause a large time overhead. Currently, the dictionaries constructed for the concurrent compression mode of file data cannot be shared between threads and there is no dynamic update mechanism, so it is impossible to obtain ideal compression speed and compression ratio. In addition, since the dynamically constructed dictionary references the indexes pointed to by the duplicate data of other segments, the dictionary needs to be persisted (and good search efficiency needs to be provided during decompression), otherwise the index pointer will be lost due to dictionary loss and the original file data cannot be decompressed. Currently, there is no dictionary persistence scheme, and it is difficult to solve this problem.

[0028] For this reason, the present application provides a file processing scheme, which can greatly improve the efficiency of file transmission and compression tasks. In order to enable those skilled in the art of this technology to better understand the solution of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Combined with the specific application environment architecture or specific hardware architecture on which the execution of the file processing method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0030] The embodiment of the present application provides a file processing method. Combined with the execution process of the file processing method, see Figure 1As shown in the figure, a method is described in detail, which includes:

[0031] Step S11: When the client executes a file upload task through the sharding upload execution method, detect the resource allocation and utilization situation of the microservice cluster, and generate a corresponding sharding scheme for the file upload task according to the resource allocation and utilization situation; the microservice cluster is a dynamically scalable cluster.

[0032] In the embodiment of the present application, a cluster performance monitoring module, a sharding module, and a sharding routing module are set on the server side. For a file upload task with a large file size that needs to upload the file in shards, first use the cluster performance monitoring module to detect the resource allocation and utilization situation of the microservice cluster to obtain the resource occupancy status of the microservice cluster in real time; then, according to the resource allocation and utilization situation, use the sharding module to combine the resource load balancing algorithm and the sharding technology to analyze and generate a corresponding sharding scheme for the client; finally, after the client shards the file according to the sharding scheme and obtains the sharded data, the server transmits the sharded data to each microservice through the sharding routing module.

[0033] It should be noted that the microservice cluster is a dynamically scalable file processing service cluster. The microservice cluster consists of multiple microservices, and these microservices are deployed on a group of physical or virtual nodes to form an overall system that can work together. Each microservice is responsible for an independent functional module, and the microservices interact through a lightweight communication mechanism. Each microservice in the cluster has the ability to be independently deployed and can dynamically adjust resource allocation according to requirements. This architecture design aims to improve the scalability, flexibility, and reliability of the system, while reducing the risk of single-point failures.

[0034] In a specific implementation manner, in order to implement a resource-fully utilized and efficient file processing scheme based on a dynamically scalable file processing service cluster, for a file upload task with sharding upload, the process of generating a corresponding sharding scheme for the file upload task according to the resource allocation and utilization situation of the microservice cluster includes the following steps:

[0035] Determine the processor resource situation of each microservice in the microservice cluster, and determine the first number of sharding tasks that each microservice can execute according to the processor resource situation;

[0036] Based on the first number of sharding tasks, determine the second number of sharding tasks that is the total number of sharding tasks for the file upload task, and allocate a sharding routing identifier to the sharded data;

[0037] Generate a sharding scheme according to the second number of sharding tasks and the sharding routing identifier, and send the sharding scheme to the client.

[0038] In the embodiments of the present application, the server analyzes the resource allocation and utilization situation of the current microservice cluster to plan a sharding scheme for the file upload task, including the number of sharding tasks and the sharding routing identifiers of the sharded data. According to the number of sharding tasks, the splitting quantity for splitting the file upload task can be determined, as well as the quantity of sharded data received by each microservice; according to the sharding routing identifiers, the sharded data can be distributed to the microservice cluster after executing the corresponding routing rules.

[0039] It can be understood that the number of sharding tasks represents the concurrency of the corresponding file upload and compression. Combining theoretical and actual experimental data, generally, the important factors affecting the upload and compression tasks are the network environment and the performance of hardware resources, such as the network transmission rate and the processor resource situation. Since the network connection conditions between the microservice cluster and the gateway are usually relatively consistent and generally are local area networks, the impact on the load balancing of the sharding tasks is relatively small. Correspondingly, the processor resource situation is the key factor affecting the compression task. Therefore, it is necessary to split the sharding tasks in combination with the processor resource situation.

[0040] Exemplarily, the processor resource situation is described by taking the distribution of CPU (Central Processing Unit) resources as an example. The total number of shards for the file upload task and the number of execution tasks for each microservice are calculated according to parameters such as the single-core compression performance ratio of the CPU hardware resources of a single microservice, the number of CPU cores, and the average utilization rate of the CPU of this microservice within a period of time.

[0041] Specifically, the number of sharding tasks received by each microservice is: ;

[0042] C represents the single-core compression performance ratio of the CPU hardware resources of a single microservice (selecting the single-core performance of a certain common model CPU as the benchmark, and the performance ratio is determined according to the test results of the compression algorithm); w is the number of CPU cores of a single microservice; p is the average utilization rate of the CPU of this microservice in the recent period of time.

[0043] Furthermore, the total number of shards for the file upload task is: .

[0044] Meanwhile, the server will allocate sharding routing identifiers for the sharded data according to the total number of second sharding tasks. The client splits the large file according to the sharding routing identifiers and the number of second sharding tasks in the sharding scheme returned by the server, and enables multi-threading to concurrently transmit the sharded data to the server at one time.

[0045] Step S12: Obtain the sharded data determined by the client based on the sharding scheme and concurrently transmitted at one time, and distribute the sharded data to the microservice cluster based on the preset routing rules.

[0046] In the embodiments of the present application, the client slices a large file according to the slice routing identifier and the second slice task number in the slice scheme returned by the server, and enables multi-threading to concurrently transmit the sliced data to the server at one time. After receiving the sliced data, the server distributes the sliced data to each microservice for execution according to the routing rules.

[0047] In a specific implementation, a load balancing module is also set in the server, which is responsible for routing slice requests according to the number of slice tasks executable by each microservice. After obtaining the sliced data, the slice routing identifier is determined, and a task number is assigned to the sliced data. Specifically, the routing process is that the load balancing module creates a consumption task queue based on the sliced data. It can be known that the length of the consumption queue is equal to the second slice task number, that is: ; where N is the second slice task number and T is the slice task number of a single microservice. The consumption task queue should record information such as the slice routing identifier (i.e., the microservice identifier to which it belongs) and the task number of the sliced data, so as to distribute the sliced data to the microservice cluster according to the slice routing identifier and the task number.

[0048] It should be noted that since the entire file needs to be re-uploaded after the current file upload fails, the functional experience is very poor. Therefore, in view of this problem, in the embodiments of the present application, when an exception occurs during the execution of the sliced data, if it is determined that the microservice to which the current sliced data belongs is available, the task is continued to be distributed to the corresponding microservice. In this way, the file upload speed and the fault tolerance rate of the file upload system are greatly improved.

[0049] Step S13: Based on the dynamic dictionary in the shared cache, use the microservice cluster to start multi-threading to concurrently compress the sliced data to obtain sliced compressed data.

[0050] In the embodiments of the present application, a dynamic dictionary is constructed in the shared cache. The dynamic dictionary introduces a dynamic update mechanism and can be shared among multiple threads at the same time. A parallel compression scheme is established based on the dynamic dictionary built in the shared cache to achieve efficient compression and decompression of data.

[0051] In the embodiments of the present application, the microservice cluster provides a compression service. For the received sliced data, multi-threading is enabled to concurrently compress the sliced data. Specifically, the number of multi-threads is determined according to the number of available core threads of the file server system; the sliced data is divided into block data equal to the number of multi-threads, and the corresponding block data is read and compressed concurrently by multi-threading. That is, the number of multi-threads W = the number of available core threads of the system. Correspondingly, the file data is divided into equal blocks according to the number of concurrent threads, and all threads concurrently read and compress the corresponding block data. For example, for N sliced data, each sliced data is further divided into W blocks, and these blocks are concurrently processed by W threads for reading and compression.

[0052] Step S14: After the multi-threaded execution is completed, persist the dynamic dictionary to obtain a sharded dictionary file, so as to decompress the sharded compressed data according to the sharded dictionary file.

[0053] It should be noted that the dynamic dictionary is based on a dynamic update mechanism and references the indexes pointed to by the duplicate data of other segments. Therefore, after the multi-threaded execution is completed, the dictionary needs to be persisted, otherwise the index pointer will be lost due to the loss of the dictionary, and the original file data cannot be decompressed.

[0054] In the embodiment of the present application, when the multi-threaded execution is completed, it represents the end of the scanning and compression of the corresponding sharded data. At this time, each thread outputs the sharded data to the file storage server in sequence, and at the same time persists the dynamic dictionary to obtain a sharded dictionary file. The sharded dictionary file will be stored together with the sharded compressed data according to a certain naming rule, which is convenient for reading the file content and use during decompression.

[0055] Beneficial effects: Currently, the miniaturization, containerization, and clustering of microservices are the mainstream trends. Through the present application, for the file upload task of sharded upload, from the perspective of resource optimization, the resource allocation and utilization of the microservice cluster are fully considered, and a sharding scheme for the uploaded file is generated; at the same time, from the perspective of concurrency, the microservice cluster is used to improve the speed of the client uploading sharded data. In this way, it can ensure that the upload task is completed reliably and quickly, not only effectively utilizing the resources of the server, but also greatly improving the speed of file upload and the fault tolerance rate of file upload. Further, a parallel compression scheme is established based on the constructed dynamic dictionary, and the dynamically constructed dictionary is persisted. It can improve the compression / decompression speed while ensuring a good compression rate to reduce the occupation of storage resources, and at the same time, the dictionary persistence provides good search efficiency for decompression. Generally speaking, the present invention can greatly improve the efficiency of file transmission and compression tasks.

[0056] Based on the above embodiment, in a feasible implementation manner, the client sends a file upload task request to the server, and the server judges the execution mode of the file upload task according to the preset threshold and the task request parameters carried in the file upload task request. Specifically, the following steps may further be included:

[0057] Obtain the file upload task request sent by the client, and determine the verification value of the uploaded file in the file upload task according to the file upload task request;

[0058] Compare the verification value of the uploaded file with the verification value of the target file already existing in the file server to judge the execution mode of the file upload task;

[0059] When the verification value of the uploaded file is the same as the verification value of the target file, it is determined that the execution mode is to submit the file name;

[0060] When the check value of the uploaded file is different from the check value of the target file, if the file size of the uploaded file does not exceed the preset threshold, it is determined that the execution method is direct upload;

[0061] When the check value of the uploaded file is different from the check value of the target file, if the file size of the uploaded file exceeds the preset threshold, it is determined that the execution method is piecemeal upload.

[0062] In the embodiments of the present application, the execution methods of the file upload task are divided into submitting the file name, direct upload, and piecemeal upload. When executing the file upload task, the client will first send the file information to the server, including the file name, file size, check value, etc., and the server returns a task executable identifier according to the file information.

[0063] Among them, the check value is used to ensure the integrity and consistency of information transmission. In a feasible implementation, the check value is illustrated by taking the MD5 (Message-Digest Algorithm 5) value as an example. After the server receives the file information sent by the client, when the file MD5 value is the same as the file MD5 value already existing in the file server, the execution method is to submit the file name, and a file upload unnecessary identifier is returned. At this time, data can be transmitted in seconds, that is, there is no need to upload the file. The client only needs to submit the file name, etc., and at the same time the server links the existing file address to the file upload task. Otherwise, the server judges whether the file size exceeds the threshold setting. If it does not exceed, it is considered that the file is a small file, and the execution method is direct upload, and a direct upload identifier is returned, and the server receives the uploaded file. Otherwise, it is considered that the file size is too large, and the execution method is piecemeal upload. At this time, the server will analyze the resource distribution and availability of the microservice cluster of the current server, and plan a piecemeal scheme, including the number of piecemeal tasks and the piecemeal routing identifier.

[0064] It should be noted that for the threshold setting used by the server to judge the file size to determine whether to perform piecemeal upload, it can be determined according to the network environment of the file server, the performance of the server hardware resources, and combined with the specific business scenario. For example, when the network rate is fast and the resources such as the server CPU are strong, the preset threshold can be appropriately increased, and vice versa. In this way, the performance difference between piecemeal upload and direct upload can be balanced, so as to determine the best way to execute the upload task.

[0065] Based on the above embodiments, the construction process of the dynamic dictionary is described, and the compression algorithm principle in this embodiment is implemented based on the dynamic dictionary. It should be noted that the dynamic dictionary is a typical data structure, which includes a combination of a non-persistent hash table and a binary tree structure, and a persistent array structure; specifically, the construction process of the dynamic dictionary includes the following steps:

[0066] Use a hash table to store the first characters of the sharded data, and point the hash table to the address in the binary tree corresponding to the first character;

[0067] Use each node in the binary tree to correspondingly store each text character in the sharded data and the index value corresponding to the text character;

[0068] Use an array structure to store the sharded data, and set an array subscript for each node of the array structure; wherein, the array subscript corresponds one-to-one with the index value in the binary tree.

[0069] In the embodiment of the present application, the non-persistent hash table stores the first characters of the repeated data segments to be compressed in the sharded data, and points to the address of the binary tree structure starting with the corresponding first character; each node of the binary tree contains a text character and an index value; the array contains the complete text string, each node contains an array subscript, which corresponds one-to-one with the index value of the node in the binary tree.

[0070] It should be noted that the principle of the compression algorithm used in the embodiment of the present application is: set the sliding window size, and sequentially scan the sharded data based on the sliding window size to find the repeated parts of the currently scanned data in the dynamic dictionary and the sliding window area, extract the repeated data segments in the sharded data, and store them in the dynamic dictionary of the microservice process (if they exist in the shared cache, there is no need to extract). An index value will be assigned to this repeated data segment in the dynamic dictionary. This index value is a numbered value generated by progressively numbering the repeated data segments using a preset numbering algorithm. Among them, the numbered value is shared among multiple threads and is designed to be thread-safe. Use this index value to replace the repeated data (if it is matched in the sliding window, the repeated data in the sliding window will also be replaced). Finally, construct a binary tuple using the index value and the data length of the repeated data segment corresponding to the index value. When all the sharded data is scanned, determine the sharded compressed data based on the binary tuple. That is, sequentially execute the above process until the scanning is completed, and the finally obtained sharded compressed data contains the index value and the matching data length.

[0071] It is worth noting that during the process of scanning the sharded data, there is an inclusion relationship between the currently scanned data segment and the existing data segment in the dynamic dictionary. At this time, point the index value in the binary tree to the target node, and correspondingly update the array subscript; wherein, the text character stored in the target node can determine the data segment with the longer data length among the currently scanned data segment and the existing data segment. At this time, the index value in the binary tree points to the node of the longer segment, and the content in the corresponding array subscript is updated to the longer segment. The obtained sharded compressed data is the binary tuple containing the index value and the data length obtained after the corresponding replacement.

[0072] The following uses a specific example to illustrate the construction process of the dynamic dictionary. Suppose a series of duplicate strings to be compressed, AB, ABCD, ABE, and ABCG, are found in the original data during the scanning process. Then the corresponding dynamic dictionary construction process is as follows:

[0073] 1) Create a binary tree with node A as the root node, and then store the character A and its corresponding root node address in the hash table;

[0074] 2) Build node B of the binary tree, that is, extend a child node B downward from the root node A to form the path A - B, corresponding to the complete string AB. This node contains the index value 0 of AB, and at the same time, the string AB is stored at the 0th position of the persistent array;

[0075] 3) Build nodes C and D of the binary tree. Starting from the root node A, continue to extend child nodes C and D along the existing path A - B to form the path A - B - C - D, corresponding to the string ABCD. Since ABCD contains AB but is a longer string, node D contains the index value 0 of ABCD, and at the same time, the string ABCD is stored at the 0th position of the persistent array (overwriting the original AB number, updating the 0th position of the persistent array, and replacing the content with ABCD); in this way, longer strings can be used to replace shorter strings preferentially to further compress the data volume;

[0076] 4) Build node E of the binary tree. Fork from the path A→B of the root node and add a new child node E to form the path A→B→E, corresponding to the string ABE. Among them, node E contains the index value 1 of ABE, and at the same time, the string ABE is stored at the 1st position of the persistent array;

[0077] 5) Build node G of the binary tree. Fork from the path A→B→C of the root node and add a new child node G to form the path A→B→C→G, corresponding to the string ABCG. Among them, node G contains the index value 2 of ABCG, and at the same time, the string ABCG is stored at the 2nd position of the persistent array.

[0078] It can be seen that through the above steps, the achieved compression effect is to use the index value and the corresponding data length to replace the complete string. For example, ABE is replaced by 1, and ABCG is replaced by 2. It can be understood that the system can process the decompression process of the fragmented compressed data in the same way.

[0079] In a specific implementation, after the multi-threading execution is completed, the shard compressed data is output to the file storage server according to the execution order of the multi-threading; meanwhile, the data in the array structure in the dynamic dictionary is persisted into a shard dictionary file, and the data in the hash table and the binary tree is destroyed in the memory after being used. The shard dictionary file and the shard compressed data are stored together based on a preset naming rule, so as to decompress the shard compressed data according to the shard dictionary file.

[0080] Specifically, the decompression process only needs to preload the shard dictionary file, and perform content replacement according to the index value and the matching length when sequentially scanning the shard compressed file. That is, during decompression, the original string is taken out from the array according to the number to restore the data.

[0081] It can be seen that the compression algorithm principle implemented based on the dynamic dictionary in the embodiment of the present application extracts the matched repeated data segments into the dictionary, and the original content will be completely replaced, and the effective range of the dictionary can cover the entire shard file. Compared with the current compression algorithm, it can greatly improve the compression ratio of the data. Adopting a data structure with both a binary tree and an array can effectively reduce the time complexity during compression and decompression. The tree nodes are shared by strings with the same prefix, avoiding repeated storage and reducing redundancy. At the same time, when adding a new string, only the branches of the tree need to be extended, without reconstructing the structure, which is convenient for dynamic expansion. In this way, high-efficient compression and decompression speeds are ensured.

[0082] In a feasible implementation, in order to further compress the data and improve the efficiency of the compression task, the recoding method is dynamically selected according to a preset coding optimization strategy, and the shard compressed data is recoded by the recoding method. Exemplarily, for the compressed data, the recoding method using the Huffman algorithm can be selected for recoding to further compress the data.

[0083] In addition, in another feasible implementation, in order to achieve security and privacy enhancement, for the dynamic dictionary, the data segment can be homomorphically encrypted during the dictionary storage stage to ensure that the compressed ciphertext can still be indexed and replaced, and support direct processing by the server without decompression. For example, using an encryption algorithm, the hash value of the encrypted data segment can be used for matching, but the content cannot be decrypted. In this way, the privacy computing requirements can be met, and it is applicable to some sensitive data scenarios, such as medical and financial fields. Or, an anti-tampering design is carried out for the dynamic dictionary. By attaching a digital signature to each dictionary entry, any tampering will cause the root hash value to change, preventing man-in-the-middle attacks or data tampering and enhancing the system security. At the same time, in order to achieve hardware acceleration and energy efficiency optimization, a dedicated logic circuit can also be designed for the above implementation process to realize parallel matching and updating operations of the dictionary, which is applicable to edge computing devices.

[0084] Such as Figure 2Shown below is a complete request and response process implemented by integrating the foregoing embodiments. Specifically, the client sends a file upload task request; the server determines the upload method. If the execution method selects the sharding upload method, the server generates a sharding scheme according to the sharding strategy and returns the scheme to the client; then the client shards the file, numbers the shards, and transfers the sharded data to the server in parallel; the server distributes the sharded data to each microservice for execution according to the routing rules; the microservice is responsible for receiving the sharded data and starting multiple threads for parallel compression, and stores the compressed sharded data in the file server after the parallel compression is completed. It can be seen that through the above process, a resource-adaptive dynamic acceleration upload technology and a parallel compression technology based on a dynamic dictionary are realized. The resource occupancy status of the microservice cluster is obtained in real time, and the file is transferred and compressed in parallel by combining the resource load balancing algorithm, sharding technology, parallel compression technology, and compression algorithm, maximizing the speed of file upload and compression. A file processing service cluster based on dynamic expansion is implemented, and an efficient file parallel upload and compression processing scheme that makes full use of resources is realized.

[0085] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0086] An embodiment of the present application also provides a file processing device, which is applied to the server. Refer to Figure 3 As shown, the device includes:

[0087] A sharding scheme generation module 11, configured to detect the resource allocation and utilization situation of the microservice cluster when the client executes a file upload task in the sharding upload execution method, and generate a corresponding sharding scheme for the file upload task according to the resource allocation and utilization situation; the microservice cluster is a dynamically scalable cluster;

[0088] A sharded data forwarding module 12, configured to obtain the sharded data determined and concurrently transmitted at one time by the client based on the sharding scheme, and distribute the sharded data to the microservice cluster based on a preset routing rule;

[0089] A data compression module 13, configured to use the microservice cluster to start multiple threads to perform concurrent compression on the sharded data based on the dynamic dictionary in the shared cache to obtain sharded compressed data;

[0090] A data decompression module 14, configured to, after the multi-threaded execution is completed, persist the dynamic dictionary to obtain a sharded dictionary file, so as to decompress the sharded compressed data according to the sharded dictionary file.

[0091] For the description of the features in the corresponding embodiment of the file processing device, reference may be made to the relevant description in the corresponding embodiment of the file processing method, which will not be elaborated here one by one.

[0092] Advantageous effects: Currently, the miniaturization, containerization, and clustering of microservices are the mainstream trends. Through this application, for the file upload task of sharded upload, from the perspective of resource optimization, the resource allocation and utilization of the microservice cluster are fully considered, and a sharding scheme for the uploaded file is generated. At the same time, from the perspective of concurrency, the microservice cluster is used to improve the speed of the client uploading sharded data. In this way, it can ensure that the upload task is completed reliably and quickly, not only effectively utilizing the resources of the server, but also greatly improving the speed of file upload and the fault tolerance rate of file upload. Further, a parallel compression scheme is established based on the constructed dynamic dictionary, and the dynamically constructed dictionary is persisted. It can improve the compression / decompression speed while ensuring a good compression rate to reduce the occupancy of storage resources, and at the same time, the dictionary persistence provides good search efficiency for decompression. Generally speaking, the present invention can greatly improve the efficiency of file transmission and compression tasks.

[0093] In a specific embodiment, the file processing device further includes:

[0094] A file checksum acquisition module, configured to acquire a file upload task request sent by a client, and determine the checksum of the uploaded file in the file upload task according to the file upload task request;

[0095] An execution mode determination module, configured to compare the checksum of the uploaded file with the checksum of the target file already existing in the file server to determine the execution mode of the file upload task; when the checksum of the uploaded file is the same as the checksum of the target file, it is determined that the execution mode is to submit the file name; when the checksum of the uploaded file is different from the checksum of the target file, if the file size of the uploaded file does not exceed a preset threshold, it is determined that the execution mode is direct upload; when the checksum of the uploaded file is different from the checksum of the target file, if the file size of the uploaded file exceeds the preset threshold, it is determined that the execution mode is sharded upload.

[0096] A first execution unit, configured to, when the execution mode is to submit the file name, acquire the file name of the uploaded file submitted by the client, and link the file address of the target file to the file upload task;

[0097] A second execution unit, configured to, when the execution mode is direct upload, return identification information that can be directly uploaded to the client and receive the uploaded file;

[0098] A third execution unit, configured to, when the execution mode is sharded upload, trigger the step of detecting the resource allocation and utilization of the microservice cluster.

[0099] A threshold adjustment module, configured to adjust the size of a preset threshold based on the current business scenario of the file server, according to the network environment and hardware resource performance of the file server.

[0100] A dynamic dictionary construction module, configured to:

[0101] Store the first characters of the sharded data in a hash table, and point the hash table to the address corresponding to the first character in the binary tree;

[0102] Use each node in the binary tree to correspondingly store each text character in the sharded data and the index value corresponding to the text character;

[0103] Store the sharded data using an array structure, and set an array subscript for each node of the array structure; wherein, the array subscripts and the index values in the binary tree correspond one by one.

[0104] A recoding module, configured to dynamically select a recoding method according to a preset coding optimization strategy, and recode the sharded compressed data through the recoding method.

[0105] In a specific implementation manner, the sharding scheme generation module 11 is specifically configured to:

[0106] Determine the processor resource situation of each microservice in the microservice cluster, and determine the number of first sharding tasks that each microservice can execute according to the processor resource situation;

[0107] Determine a second number of sharding tasks as the total number of sharding tasks for the file upload task based on the number of first sharding tasks, and allocate a sharding routing identifier to the sharded data;

[0108] Generate a sharding scheme according to the second number of sharding tasks and the sharding routing identifier, and send the sharding scheme to the client.

[0109] In a specific implementation manner, the sharded data forwarding module 12 is specifically configured to:

[0110] Obtain the sharded data determined and transmitted concurrently at one time by the client based on the sharding scheme, and determine the sharding routing identifier and the task number of the sharded data according to the sharded data;

[0111] Create a consumption task queue based on the sharded data;

[0112] Use the consumption task queue to record the sharding routing identifier and the task number, so as to distribute the sharded data to the microservice cluster according to the sharding routing identifier and the task number; wherein, the length of the consumption task queue is equal to the second number of sharding tasks.

[0113] In a specific implementation manner, the sharded data forwarding module 12 further includes:

[0114] An abnormal data execution unit, which is used to determine whether the target microservice to which the current shard data belongs is available if the execution of the current shard data is abnormal, and continue to distribute the current shard data to the target microservice when the target microservice is available.

[0115] In a specific implementation, the data compression module 13 includes:

[0116] A thread number determination module, which is used to determine the number of multi-threads according to the number of system available core threads of the file server; divide the shard data into block data equal to the number of multi-threads, and use multi-threads to concurrently read and compress the corresponding block data.

[0117] An inclusion relationship processing module, which is used to point the index value in the binary tree to the target node and update the array subscript accordingly when there is an inclusion relationship between the currently scanned data segment and the existing data segments in the dynamic dictionary; the text characters stored in the target node can determine the data segment with the longer data length among the current data segment and the existing data segments.

[0118] In a specific implementation, the data compression module 13 is specifically used for:

[0119] Set the sliding window size, and sequentially scan the shard data based on the sliding window size to obtain the repeated part of the dynamic dictionary and the sliding window area, and obtain the repeated data segments in the shard data;

[0120] Use a preset numbering algorithm to progressively number the repeated data segments, and set the generated number value as the index value of the repeated data segments; the number value is shared among multi-threads;

[0121] Replace the repeated data segments with the index values, and construct a binary tuple using the index values and the data lengths of the repeated data segments corresponding to the index values. After all the shard data is scanned, determine the shard compressed data based on the binary tuple.

[0122] In a specific implementation, the data decompression module 14 is specifically used for:

[0123] After the multi-threads finish execution, output the shard compressed data to the file storage server in the execution order of the multi-threads;

[0124] Persist the data in the array structure into a shard dictionary file, and destroy the data in the hash table and the binary tree;

[0125] Store the shard dictionary file and the shard compressed data together based on a preset naming rule, so as to decompress the shard compressed data according to the shard dictionary file.

[0126] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above-described embodiments of the file processing method.

[0127] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above-described embodiments of the file processing method when running.

[0128] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media that can store computer programs such as USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs.

[0129] An embodiment of the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-described embodiments of the file processing method.

[0130] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-described embodiments of the file processing method.

[0131] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0132] The above has introduced in detail a file processing method, device, and medium provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A file processing method, characterized in that, Applied to the server side, including: When the client executes a file upload task through the sharded upload execution method, detect the resource allocation and utilization situation of the microservice cluster, and generate a corresponding sharding scheme for the file upload task according to the resource allocation and utilization situation; the microservice cluster is a dynamically scalable cluster; Obtain the sharded data determined and concurrently transmitted at one time by the client based on the sharding scheme, and distribute the sharded data to the microservice cluster based on a preset routing rule; Based on the dynamic dictionary in the shared cache, use the microservice cluster to start multiple threads to concurrently compress the sharded data to obtain sharded compressed data; After the execution of the multiple threads is completed, persist the dynamic dictionary to obtain a sharded dictionary file for decompressing the sharded compressed data according to the sharded dictionary file.

2. The document processing method according to claim 1, wherein Also includes: Obtain the file upload task request sent by the client, and determine the check value of the uploaded file in the file upload task according to the file upload task request; Compare the check value of the uploaded file with the check value of the target file existing in the file server to judge the execution method of the file upload task; When the check value of the uploaded file is the same as the check value of the target file, determine that the execution method is to submit the file name; When the check value of the uploaded file is different from the check value of the target file, if the file size of the uploaded file does not exceed the preset threshold, determine that the execution method is direct upload; When the check value of the uploaded file is different from the check value of the target file, if the file size of the uploaded file exceeds the preset threshold, determine that the execution method is sharded upload.

3. The document processing method according to claim 2, wherein Also includes: When the execution method is to submit the file name, obtain the file name of the uploaded file submitted by the client, and link the file address of the target file to the file upload task; When the execution method is direct upload, return the identification information that can be directly uploaded to the client and receive the uploaded file; When the execution method is sharded upload, trigger the step of detecting the resource allocation and utilization situation of the microservice cluster.

4. The document processing method according to claim 2, characterized in that Also includes: Based on the current business scenario of the file server, adjust the size of the preset threshold according to the network environment and hardware resource performance of the file server.

5. The document processing method according to claim 1, wherein The detecting the resource allocation and utilization situation of the microservice cluster and generating a corresponding sharding scheme for the file upload task according to the resource allocation and utilization situation includes: Determine the processor resource situation of each microservice in the microservice cluster, and determine the first number of sharding tasks that each microservice can execute according to the processor resource situation; Based on the first number of sharding tasks, determine the second number of sharding tasks that is the total number of shards of the file upload task, and allocate a sharding routing identifier for the sharded data; Generate the sharding scheme according to the second number of sharding tasks and the sharding routing identifier, and send the sharding scheme to the client.

6. The document processing method according to claim 5, wherein Obtain the shard data determined and concurrently transmitted at one time by the client based on the shard scheme, and distribute the shard data to the microservice cluster based on a preset routing rule, including: Obtain the shard data determined and concurrently transmitted at one time by the client based on the shard scheme, and determine the shard routing identifier and the task number of the shard data according to the shard data; Create a consumption task queue based on the shard data; Use the consumption task queue to record the shard routing identifier and the task number, so as to distribute the shard data to the microservice cluster according to the shard routing identifier and the task number; wherein, the length of the consumption task queue is equal to the number of the second shard tasks.

7. The document processing method according to claim 6, characterized in that During the process of distributing the shard data to the microservice cluster according to the shard routing identifier and the task number, it further includes: If the current shard data execution is abnormal, determine whether the target microservice to which the current shard data belongs is available. When the target microservice is available, continue to distribute the current shard data to the target microservice.

8. The document processing method according to claim 1, wherein The microservice cluster is used to start multiple threads to concurrently compress the shard data, including: Determine the number of the multiple threads according to the number of available core threads of the file server system; Divide the shard data into block data equal to the number of the multiple threads, and use the multiple threads to concurrently read and compress the corresponding block data.

9. The document processing method according to claim 1, characterized in that The dynamic dictionary includes a non-persistent combined structure of a hash table and a binary tree, and a persistent array structure; Among them, the construction process of the dynamic dictionary includes: Use the hash table to store the first character of the shard data, and point the hash table to the address corresponding to the first character in the binary tree; Use each node in the binary tree to correspondingly store each text character in the shard data and the index value corresponding to the text character; Use the array structure to store the shard data, and set an array subscript for each node of the array structure; wherein, the array subscript corresponds to the index value in the binary tree one by one.

10. The document processing method according to claim 9, characterized in that, Based on the dynamic dictionary in the shared cache, the microservice cluster is used to start multiple threads to concurrently compress the shard data to obtain shard compressed data, including: Set a sliding window size, and sequentially scan the shard data based on the sliding window size to obtain the repeated part of the dynamic dictionary and the sliding window area, and obtain the repeated data segment in the shard data; Use a preset numbering algorithm to progressively number the repeated data segment, and set the generated number value as the index value of the repeated data segment; the number value is shared among the multiple threads; Use the index value to replace the repeated data segment, and construct a binary tuple using the index value and the data length of the repeated data segment corresponding to the index value. After all the shard data is scanned, determine the shard compressed data based on the binary tuple.

11. The document processing method according to claim 10, wherein It also includes: When there is an inclusion relationship between the currently scanned data segment and the existing data segments in the dynamic dictionary, the index value in the binary tree is pointed to the target node, and the array subscript is updated accordingly; wherein, the text characters stored in the target node can determine the data segment with the longer data length among the currently scanned data segment and the existing data segments.

12. The document processing method according to claim 9, wherein After the multi-threading is completed, the dynamic dictionary is persisted to obtain a sharded dictionary file, so as to decompress the sharded compressed data according to the sharded dictionary file, including: After the multi-threading is completed, the sharded compressed data is output to the file storage server in the execution order of the multi-threading; The data in the array structure is persisted as the sharded dictionary file, and the data in the hash table and the binary tree is destroyed; Based on a preset naming rule, the sharded dictionary file and the sharded compressed data are stored together, so as to decompress the sharded compressed data according to the sharded dictionary file.

13. The document processing method according to any one of claims 1 to 12, characterized in that, After using the microservice cluster to start multi-threading to concurrently compress the sharded data to obtain sharded compressed data, it further includes: Dynamically select a re-encoding method according to a preset encoding optimization strategy, and re-encode the sharded compressed data through the re-encoding method.

14. An electronic device, characterized in that, It includes: A memory for storing a computer program; A processor for implementing the steps of the file processing method according to any one of claims 1 to 13 when executing the computer program.

15. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of the file processing method according to any one of claims 1 to 13 when executed by a processor.

Images