File processing method and device

Abstract

An embodiment of the present invention provides a file processing method and device, applicable to a distributed cluster including multiple servers. The method comprises: a first server determines, according to a file processing configuration, each subtask corresponding to each fragment of a file to be processed; the first server sends each subtask to a third server and writes the number of each subtask to the third server; the third server receives each subtask and broadcasts each subtask to each second server; each second server processes its own subtask in parallel according to the processing logic in the file processing configuration and sends the processing results to the third server; the third server determines that the number of received processing results equals the number of each subtask, and the processing of the file to be processed is complete. Using the above method, the fragments of the file to be processed can be processed simultaneously by multiple threads, and time and labor costs can be reduced by setting the file processing configuration.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a file processing method and device. Background Art

[0002] With the development of computer technology, more and more technologies are being applied in the financial sector. Traditional finance is gradually shifting towards Fintech. However, the security and real-time requirements of the financial industry also place higher demands on technology. In other words, although the application of computer technology has greatly accelerated the speed and accuracy of business processing and freed up a lot of manpower and material resources, as the industry develops, existing technologies are no longer able to meet business processing needs.

[0003] In the prior art, business processing is performed by a single server, with staff writing the corresponding processing logic and executing its associated logic. For example, by loading a client's financial information file locally on the server, the server executes the processing logic according to its associated logic, thereby processing the client's financial information file locally on the server. While this processing method speeds up processing and improves accuracy compared to traditional methods, due to rapid socioeconomic development, people's financial awareness has deepened, and those interested in financial management are no longer limited to a specific group; for example, young people, people in certain occupations, or people in a certain region, people of all ages, industries, and regions have some understanding of financial products. As a result, the base of the financial management population has expanded, and the sales volume of financial products has increased. Consequently, compiling customer financial information has become a relatively arduous task, and the existing processing methods are no longer able to meet the demands of today's business volume.

[0004] Therefore, there is an urgent need for a file processing method and device that can speed up file processing and reduce time and labor costs. Summary of the Invention

[0005] The embodiments of the present invention provide a file processing method and device, which can speed up file processing and reduce time and labor costs.

[0006] In a first aspect, an embodiment of the present invention provides a file processing method, which is applicable to a distributed cluster including multiple servers; the method includes: the first server determines the subtasks corresponding to each shard of the file to be processed according to the file processing configuration; the first server sends each subtask to a third server, and writes the number of each subtask to the third server; the third server receives the subtasks and broadcasts the subtasks to the second servers; the second servers process their respective subtasks in parallel according to the processing logic in the file processing configuration, and send the processing results to the third server; the third server determines that the number of the received processing results is equal to the number of subtasks, and then the processing of the file to be processed is completed.

[0007] Using the above method, the first server determines the subtasks corresponding to the slices of the file to be processed according to the file processing configuration, and further sends each subtask to the third server. In this way, the third server manages each subtask, which is convenient for the subsequent second server to obtain each subtask and process it. The third server is an atomic server, which can ensure that the task is terminated after all the subtasks of the file to be processed are processed, thereby ensuring the integrity and accuracy of the processing results of the file to be processed. Furthermore, the third server broadcasts so that the second server obtains the subtasks in sequence, so that the second server can obtain the corresponding slice to be processed according to each subtask, and processes the slice to be processed. In this way, compared with the prior art in which one server processes the file to be processed; the present application realizes multi-threading through multiple second servers to process the file to be processed separately at the same time, which can speed up the processing speed and reduce the processing pressure of the second server. In addition, by setting the file processing configuration, time cost and labor cost can also be reduced.

[0008] In one possible design, the first server determines the subtasks corresponding to the slices of the file to be processed according to the file processing configuration, including: the first server obtains the file to be processed according to the file path in the file processing configuration; the first server determines the positions of the slices of the file to be processed according to the slice rules in the file processing configuration; the first server determines the subtasks based on the positions of the slices.

[0009] Using the above method, the first server obtains the file to be processed according to the file path in the file processing configuration, and further determines the position of each shard according to the sharding rules in the file processing configuration. Finally, the first server can determine each subtask based on the position of each shard. In this way, the part of each shard in the file to be processed can be determined by the position of each shard in each subtask. Furthermore, after the second server obtains each subtask, it can obtain the corresponding shard to be processed according to each subtask and process the shard to be processed. In this way, the file to be processed can be processed simultaneously by multiple second servers and multiple threads, which can speed up the processing speed and reduce the processing pressure of the second server. In addition, by setting the file processing configuration, time cost and labor cost can also be reduced.

[0010] In one possible design, the first server determines the positions of each fragment of the file to be processed according to the fragmentation rules in the file processing configuration, including: the first server determines the file body of the file to be processed; the first server determines the positions of each fragment of the file body according to the fragmentation threshold in the fragmentation rules; for each fragment position, it is determined in the following manner: starting from the starting position of the fragment, determining whether the current character is a line break at the position that meets the fragmentation threshold; if not, continuing to count until the position of the first line break is used as the end position of the fragment.

[0011] Using the above method, after the first server determines the file body of the file to be processed, it determines the positions of each fragment of the file body according to the fragmentation threshold in the fragmentation rule. In this way, the file body excluding the file header and the file tail is obtained and the file body is fragmented, which can ensure that the content of the fragmented file is the content that needs to be processed, thereby increasing the accuracy of the processing. And because the fragmentation threshold can be determined by factors such as the size of the file to be processed, the size of the file body, and the processing capabilities of the first server and the second server. In this way, each server can perform multi-threaded processing on the file to be processed with the fastest efficiency. If the end position determined by the statistics from the starting position of the fragment according to the fragmentation threshold is not a line break, then continue to count until the first line break is determined, and the position of the line break is used as the end position. In this way, the integrity of the file information in the fragment to be processed can be guaranteed, and the accuracy of the subsequent server's processing of the fragment to be processed can be increased.

[0012] In one possible design, the first server determines the positions of each fragment of the file body according to the fragment threshold in the fragment rule, including: the fragment threshold can be determined by the following formula:

[0013] P=MIN(G,MAX(P / (2*N),L))

[0014] Wherein, P is the sharding threshold; G is the maximum value of the sharding threshold, L is the minimum value of the sharding threshold, and G and L are determined by historical experience and server performance; N is the number of processor cores of the server.

[0015] Using this method, when sharding a file to be processed, the file is sharded according to a sharding threshold. The sharding threshold is the optimal shard size determined based on historical experience and server performance. This speeds up the second server's processing of the shards corresponding to each subtask, saving time and costs.

[0016] In one possible design, the first server determines the file body of the file to be processed, including: the first server determines the file header of the file to be processed starting from the first non-newline character of the file to be processed and according to the number of file header lines in the file processing configuration; the first server determines the file tail of the file to be processed starting from the last non-newline character of the file to be processed and according to the number of file tail lines in the file processing configuration; the first server determines the part of the file to be processed except the file header and the file tail as the file body.

[0017] Using this method, the first server can accurately determine the file body based on the type of file to be processed and the number of header and footer lines configured for each file type in the file processing configuration. This increases the accuracy of the file fragments to be processed and ensures the accuracy of the content of each fragment to be processed. Furthermore, by counting down the first non-newline byte of the file to be processed to determine the end of the file, the workload of the first server in traversing from the file header to the end of the file can be reduced, thereby accelerating the determination of the file body.

[0018] In one possible design, the third server receives the subtasks and broadcasts the subtasks to the second servers, including: the third server receives the subtasks and broadcasts them to the second servers; the second servers sequentially obtain the subtasks from the third server after receiving the broadcast, and obtain the to-be-processed fragments from the to-be-processed files according to the fragment positions in the subtasks; the second servers process their respective subtasks in parallel according to the processing logic in the file processing configuration, and send the processing results to the third server, including: the second servers process their respective to-be-processed fragments in parallel according to the processing logic in the file processing configuration, and send the processing results of the to-be-processed fragments to the third server.

[0019] Using the above method, after receiving each subtask, the third server broadcasts it to each second server, so that after receiving the broadcast, the second server obtains the subtask from the third server in turn, and obtains the to-be-processed fragment from the to-be-processed file according to the fragment position in the subtask. In this way, each second server will not obtain the same subtask, thereby improving the efficiency of the second server in processing subtasks and ensuring the accuracy of the processing results of the to-be-processed file. The second servers process their respective to-be-processed fragments in parallel according to the processing logic in the file processing configuration. The required processing logic can be flexibly set in the file processing configuration, and the processing results of the required to-be-processed file can be obtained according to the processing logic in the file processing configuration, and the second server processes each subtask in parallel to increase the processing speed of the to-be-processed file. The second server sends the processing results to the third server, which is conducive to the unified management of the processing results of each subtask and the confirmation of the atomicity of the processing task by the third server, thereby ensuring the integrity of the processing of the to-be-processed file.

[0020] In one possible design, the second server obtains the to-be-processed slices from the to-be-processed file according to the slice positions in the subtask, including: the second server obtains the to-be-processed slices from the to-be-processed file to the memory of the second server according to the slice positions in the subtask through memory mapping.

[0021] Using the above method, the second server obtains the pending slices based on the slice positions in the subtask through memory mapping, stores and processes the pending slices, reducing the occupied resources of the second server, lowering the processing pressure on the second server, and accelerating the processing speed of the second server. Furthermore, the second server can sequentially obtain partial contents of the pending slices and process these partial contents to obtain a processing result. In this way, the processing result is retained and the corresponding partial content is deleted, ensuring that the memory resources of the second server are always in a state of maximum utilization, reducing the resource usage of the second server, increasing the processing performance of the second server, and further accelerating the processing speed of the second server.

[0022] In a possible design, the file processing configuration is written on a preset template; the preset template is provided with function codes for opening a file, reading a file, and closing a file.

[0023] With the above method, since the file processing configuration is written based on a preset template, and the preset template contains function codes for opening, reading, and closing files, there is no need for staff to write the corresponding function codes separately, reducing the staff's workload and time costs. In addition, due to the complexity of the function codes in this part, this application can not only reduce the staff's workload and time costs through this method; it can also eliminate the situation where files are damaged due to staff making mistakes in writing the function codes for opening or reading files. It can also eliminate the situation where unprocessed files continue to occupy server resources due to staff forgetting to write or writing the wrong function code for closing files.

[0024] In a second aspect, an embodiment of the present invention provides a file processing device applicable to a distributed cluster including multiple servers; the device includes:

[0025] The processing module is used to determine the subtasks corresponding to the slices of the file to be processed according to the file processing configuration;

[0026] a transceiver module, configured to send each subtask to a third server and write the number of each subtask into the third server;

[0027] a transceiver module, configured to receive each subtask and broadcast each subtask to each second server;

[0028] The processing module is further configured to process the respective subtasks in parallel according to the processing logic in the file processing configuration, and send the processing results to the third server;

[0029] The transceiver module is further configured to determine that if the number of the received processing results is equal to the number of each subtask, the processing of the pending file is completed.

[0030] In a third aspect, an embodiment of the present invention further provides a computing device, comprising: a memory for storing program instructions; a processor for calling the program instructions stored in the memory, and executing the method described in the various possible designs of the first aspect according to the obtained program.

[0031] In a fourth aspect, an embodiment of the present invention further provides a computer-readable non-volatile storage medium, comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method described in the various possible designs of the first aspect.

[0032] These implementations or other implementations of the present invention will be more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 A schematic diagram of the architecture of a file processing method provided by an embodiment of the present invention;

[0035] Figure 2 A flowchart of a file processing method provided by an embodiment of the present invention;

[0036] Figure 3 A flowchart of a file processing method provided by an embodiment of the present invention;

[0037] Figure 4 A schematic diagram of a file processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0039] Figure 1A file processing system architecture is provided in an embodiment of the present invention. The first server 102 can be any one of the second servers 104. The first server 102 obtains the file to be processed from the local server or the file server 101. Here, the file to be processed can be obtained from the file server through memory mapping, which can reduce the resource usage of the first server 102. The first server 102 divides the file to be processed into slices and determines the starting and ending positions of each slice to be processed. Based on the positions of each slice, each subtask is determined and sent to each second server 104, so that each second server 104 obtains the slice to be processed according to each subtask for processing. After the first server 102 determines each subtask, it can also send each subtask to a third server, so that each second server 104 obtains each subtask from the third server and then obtains the slice to be processed for processing. In this way, due to the availability and atomicity of the third server in high-concurrency scenarios, the speed of processing the file to be processed and the integrity and accuracy of the results can be guaranteed. Here, the third server can be a server where any program or algorithm with high concurrency and atomicity functions is located, for example, a Redis server, or a server containing Java's high concurrency and atomicity function programs, etc.

[0040] Based on this, the embodiment of the present invention provides a process of a file processing method, such as Figure 2 Shown, including:

[0041] Step 201: The first server determines the subtasks corresponding to the slices of the file to be processed according to the file processing configuration;

[0042] Here, each subtask can include the location of each shard, the file address of the shard to be processed, the identifier of the file to be processed, and so on. The processing logic in the file processing configuration can be a corresponding operation method. For example, if the file to be processed contains information about a company's customers purchasing financial products for a year, the company's annual turnover can be determined by setting the corresponding processing logic for the corresponding operation method. It can also be a corresponding analytical process; for example, determining the customer who spent the most money on financial products in the company that year. The specific processing logic is not limited here.

[0043] Step 202: The first server sends each subtask to a third server and writes the number of each subtask into the third server.

[0044] Here, when the first server sends each subtask to the third server, it writes the number of subtasks, that is, the number of shards to be processed (taskCount), into the third server, so that the third server can determine the execution status of each subtask processed by each second server. Since the database processing speed of the second server is relatively slow, and in a high concurrency scenario, the status of one or more second servers may not be successfully synchronized, the task execution status will not be updated. Therefore, the status of the second server is synchronized by setting up a third server. This prevents the situation where some shards to be processed are not processed before the processing task is completed, or some shards to be processed are processed multiple times, resulting in inaccurate processing results for the files to be processed.

[0045] Step 203: The third server receives each subtask and broadcasts each subtask to each second server;

[0046] Step 204: Each of the second servers processes its own subtask in parallel according to the processing logic in the file processing configuration, and sends the processing results to the third server;

[0047] Here, sending the processing results to the third server ensures that the third server obtains the subtask processing status of the second server and further provides new subtasks to the second server. The third server obtains the status of each subtask of the second server processing the pending file, speeding up the processing speed of the pending file.

[0048] Step 205: The third server determines that the number of the received processing results is equal to the number of each subtask, and the processing of the pending file is completed.

[0049] Here, the third server determines that the number of processing results equals the number of subtasks and confirms that the pending file has been processed. This ensures the integrity of the processing task and prevents the second server from receiving incomplete processing results due to incomplete task processing or network anomalies.

[0050] Using the above method, the first server determines the subtasks corresponding to the slices of the file to be processed according to the file processing configuration, and further sends each subtask to the third server. In this way, the third server manages each subtask, which is convenient for the subsequent second server to obtain each subtask and process it. The third server is an atomic server, which can ensure that the task is terminated after all the subtasks of the file to be processed are processed, thereby ensuring the integrity and accuracy of the processing results of the file to be processed. Furthermore, the third server broadcasts so that the second server obtains the subtasks in sequence, so that the second server can obtain the corresponding slice to be processed according to each subtask, and processes the slice to be processed. In this way, compared with the prior art in which one server processes the file to be processed; the present application realizes multi-threading through multiple second servers to process the file to be processed separately at the same time, which can speed up the processing speed and reduce the processing pressure of the second server. In addition, by setting the file processing configuration, time cost and labor cost can also be reduced.

[0051] An embodiment of the present application provides a file processing method, in which a first server determines, according to a file processing configuration, subtasks corresponding to each shard of a file to be processed, including: the first server obtains the file to be processed according to the file path in the file processing configuration; the first server determines the positions of each shard of the file to be processed according to the sharding rules in the file processing configuration; and the first server determines each subtask based on the positions of each shard.

[0052] Here, the file path in the file processing configuration can be directly input by the user or pre-set; the file path can be the physical address of the file to be processed in the memory of the first server; it can be the address of the file server where the file to be processed is located, and the physical address of the file to be processed in the memory of the file server; it can also be a logical address generated by memory mapping relative to the physical address of the file server memory where the file to be processed is stored. There is no specific restriction on the file path here. The sharding rules in the file processing configuration are rules for sharding the file to be processed, which may include the size of the shard to be processed, the recording method of the shard to be processed, etc., and are not specifically limited. The position of each shard of the file to be processed is the position information of the shard to be processed in the file to be processed, which can be recorded by the starting position of the shard to be processed in the byte of the file to be processed and the ending position in the byte of the file to be processed. For example, if the file to be processed has 3000 bytes, the first segment to be processed starts at the first byte and ends at the 100th byte of the file to be processed; the second segment to be processed starts at the 101st byte and ends at the 200th byte of the file to be processed. By recording the start and end positions of the segments to be processed, a subtask corresponding to the segments to be processed can be generated.

[0053] The embodiment of the present application provides a fragmentation rule in a file processing configuration, wherein the first server determines the fragmentation positions of the file to be processed according to the fragmentation rule in the file processing configuration, including: the first server determines the file body of the file to be processed; the first server determines the fragmentation positions of the file body according to the fragmentation threshold in the fragmentation rule; for each fragmentation position, the position is determined in the following manner: starting from the fragmentation starting position, determining whether the current character is a line break at a position that meets the fragmentation threshold; if not, continuing to count until the position of the first line break is used as the fragmentation end position. In other words, the fragmentation rule in the file processing configuration can first determine the file body of the file to be processed, and then determine the fragmentation positions of the file body according to the fragmentation threshold set in the fragmentation rule. The shard threshold represents the size of each shard to be processed, measured in B. The shard threshold can be determined based on the size of the file to be processed (measured in B) and the number of processor cores N on the server. This can eliminate the possibility of excessive memory consumption when subsequently processing shards that are too large. It can also eliminate the waste of available memory caused by the generation of many small shards due to shards that are too small. Therefore, a maximum G and minimum L value for the size of the shards to be processed are also set, both in B. This ensures that a file to be processed can be divided into an appropriate number of shards for parallel processing, in order to maximize server performance. The final size of the shards to be processed can be determined using the formula P(shard threshold) = MIN(G, MAX(P / (2*N), L)). In the previous example, the file to be processed has a total of 3000 bytes. If the fragment threshold is set to 100 bytes as a fragment to be processed, the first fragment to be processed starts at the first byte of the file to be processed and ends at the 100th byte of the file to be processed; the second fragment to be processed starts at the 101st byte of the file to be processed and ends at the 200th byte of the file to be processed. The first fragment to be processed is bytes 1-100 of the file to be processed, the second fragment to be processed is bytes 101-200 of the file to be processed, and so on, until the 30th fragment to be processed is bytes 2901-3000 of the file to be processed. If the end position determined by the fragment threshold is not a newline character, the count needs to continue until the first newline character. This ensures the integrity of the content of each fragment to be processed. In the previous example, if the first fragment to be processed ends at the 100th byte of the file to be processed, but the 100th byte is not a newline character, the line has not yet ended. Counting must continue until the first newline character is found to determine the end of the line. If the first newline character is the 120th byte of the file to be processed, the first fragment to be processed consists of bytes 1-120 of the file to be processed. The second fragment to be processed consists of bytes 121-220 of the file to be processed.

[0054] The present application also provides a code formula for implementing the file segmentation process, as shown below:

[0055] local filePair List; / / The result set of pending shards of the files to be processed.

[0056] For long point=0;point<=n-1do; / / Start traversing the files to be processed, point indicates the current pointer position of the files to be processed;

[0057] local end = point + P; / / Sharding is performed according to the fixed size P.

[0058] if end>=n-1; / / Judge whether the file has reached its end.

[0059] end=n-1; / / Indicates that the end of the file has been read.

[0060] else;

[0061] end = seekRowEnd(); / / Traverse backward from the end pointer position to find the first line break as the end position of the slice to be processed. The seekRowEnd() method means starting from the specified position, traverse the file to be processed byte by byte until the first line break is found, and return the position value of the line break in the file to be processed;

[0062] end / / The fragmentation process ends.

[0063] Here, the above method can be used to split the file to be processed into pieces.

[0064] local filePair; / / Create a new shard to be processed.

[0065] filePair.start=point; / / Set the starting position of the fragment to be processed.

[0066] filePair.end=end; / / Set the end position of the fragment to be processed.

[0067] filePairList.add(filePair); / / Add the location information of the shard to be processed to the shard result set.

[0068] point=end+1; / / Move the file pointer down one byte and continue fragmenting.

[0069] End;

[0070] Here, the location information of the to-be-processed fragment obtained in the previous fragmentation process is recorded for subsequent processing of the to-be-processed fragment.

[0071] Thus, by adopting this method; obtaining the file body excluding the file header and the file tail, and slicing the file body, it is possible to ensure that the content of the slicing file is the content that needs to be processed, thereby increasing the accuracy of the processing. And because the slicing threshold can be determined by factors such as the size of the file to be processed, the size of the file body, and the processing capabilities of the first server and the second server. In this way, each server can perform multi-threaded processing on the file to be processed with the fastest efficiency. If the end position determined by the statistics from the starting position of the slicing according to the slicing threshold is not a line break, then statistics are continued until the first line break is determined, and the position of the line break is used as the end position. In this way, the integrity of the file information in the slicing to be processed can be guaranteed, and the accuracy of the subsequent server's processing of the slicing to be processed can be increased.

[0072] An embodiment of the present application provides another fragmentation rule in a file processing configuration, in which the first server determines the file body of the file to be processed, including: the first server starts from the first non-newline character of the file to be processed and determines the file header of the file to be processed according to the number of file header lines in the file processing configuration; the first server starts from the last non-newline character of the file to be processed and determines the file tail of the file to be processed according to the number of file tail lines in the file processing configuration; the first server determines the part of the file to be processed except the file header and the file tail as the file body.

[0073] Here, the first non-newline character of the file to be processed marks the beginning of the file header. Therefore, the file header of the file to be processed can be determined starting from the first non-newline character of the file to be processed and according to the number of file header lines in the file processing configuration. The last non-newline character of the file to be processed marks the beginning of the file tail. Therefore, the file tail of the file to be processed can be determined starting from the last non-newline character of the file to be processed and according to the number of file tail lines in the file processing configuration. In this way, the file body of the file to be processed can be determined by the number of file header lines and the number of file tail lines recorded in the fragmentation rules in the file processing configuration. In this way, according to the type of the file to be processed and the configuration of the number of file header lines and the number of file tail lines for each type of file in the file processing configuration, the file body can be accurately determined further according to the number of file header lines and the number of file tail lines. The accuracy of the fragmented files to be processed is increased, and the accuracy of the content of each fragment to be processed is guaranteed.

[0074] An embodiment of the present application provides a file processing method, wherein the third server receives the subtasks and broadcasts the subtasks to the second servers, including: the third server receives the subtasks and broadcasts them to the second servers; the second servers sequentially obtain the subtasks from the third server after receiving the broadcasts, and obtain the fragments to be processed from the files to be processed according to the fragment positions in the subtasks; the second servers process their respective subtasks in parallel according to the processing logic in the file processing configuration, and send the processing results to the third server, including: the second servers process their respective fragments to be processed in parallel according to the processing logic in the file processing configuration, and send the processing results of the fragments to be processed to the third server.

[0075] Here, after the first server obtains the file to be processed and splits it into slices, it determines each subtask based on the slice position of each slice to be processed and sends each subtask to the third server. After receiving each subtask, the third server broadcasts to the second server, notifying each second server to obtain the subtask. Alternatively, the second server can continuously monitor the slice processing status of the first server. When the first server completes the slice processing, the second server proactively obtains the subtask from the third server. Alternatively, the first server and the second server share the processing status. In this way, when the first server completes the slice processing, the second server proactively obtains the subtask from the third server. Furthermore, the second server obtains the corresponding slice to be processed based on the slice position in the obtained subtask. Each second server simultaneously processes the obtained slice to be processed according to the processing logic in the file processing configuration, accelerating the processing speed of the file to be processed. Furthermore, since the second server obtains the slice to be processed corresponding to the subtask based on the slice position, the second server does not store the file to be processed, so the file to be processed does not occupy the memory resources of the second server. This ensures the processing performance of the second server.

[0076] An embodiment of the present application provides a file processing method, wherein the second server obtains a to-be-processed slice from the to-be-processed file according to the location of the slice in the subtask, including: the second server obtains the to-be-processed slice from the to-be-processed file according to the location of the slice in the subtask into the memory of the second server through memory mapping. In other words, the second server obtains the to-be-processed slice corresponding to the subtask through memory mapping, and after processing the to-be-processed slice, only saves the processing result of the to-be-processed slice and then continues to process the next to-be-processed slice, so that there is always only one to-be-processed slice in the second server, which occupies a small amount of memory, ensuring that the resource occupancy rate is always minimized and the performance of the second server is guaranteed.

[0077] Here, according to the embodiment of the above-mentioned sharding process, the embodiment of the present application further provides a method for reading shards to be processed, as shown below:

[0078] The second server loads the shard to be processed into memory through memory mapping. filePair(k) represents the kth shard in the result set of the shard to be processed. Based on the start and end positions of the shard to be processed, the content of the shard to be processed is mapped into virtual memory. Then, through the cache, the shard is loaded into memory in parts with a set number of rows, so that the second server memory always has only the set number of rows of the shard to be processed. When the set number of rows is V, then:

[0079] local rowList; / / The shard row data collection to be processed.

[0080] local row; / / Read the data content of each row.

[0081] local count=0;

[0082] for bytedata in filePair(k).data do; / / bytedata is the content of the fragment filePair(k) to be processed.

[0083] if bytedata == '\n' / / Determine whether the last byte of the content is a newline character.

[0084] rowList.add(row); / / If it is a line break, it means that a line has been read and the content of the read line is added to rowList.

[0085] row.clear(); / / Clear the row and prepare to read the next row of data.

[0086] count++; / / The number of rows read + 1.

[0087] if count == V; / / Judge whether the total number of rows read this time reaches V value.

[0088] callBusinessProcess(rowList); / / Read V pieces of data and call the processing logic corresponding to the file to be processed.

[0089] rowList.clear(); / / Clear the row data that has been read.

[0090] count=0; / / Clear count and restart counting.

[0091] End;

[0092] Else;

[0093] row.append(data); / / If the fragment content to be processed is not the end of the row, it is added to the row content.

[0094] End;

[0095] End;

[0096] Here, when reading the content of the to-be-processed fragment, the present application also provides a configuration table that supports the configuration of illegal characters. Each byte in the read to-be-processed fragment is first checked through the configuration table, so that some illegal characters can be filtered out during the reading of the to-be-processed fragment.

[0097] An embodiment of the present application provides a method for setting a file processing configuration, wherein the file processing configuration is written based on a preset template; the preset template is provided with function codes for opening, reading, and closing files. Here, the preset template is provided with function codes for opening, reading, and closing files. Therefore, there is no need for staff to separately write the corresponding function codes, reducing their workload and time costs. In addition, due to the complexity of this part of the function code, this method of the present application not only reduces staff workload and time costs, but also eliminates the situation where files are corrupted due to staff errors in writing the function codes for opening or reading files. It also eliminates the situation where unprocessed files continue to occupy server resources due to staff forgetting to write or incorrectly writing the function codes for closing files. Here, corresponding configuration items for each type of file to be processed can also be added to the preset template, and fixed parameters for each type of file to be processed can be added to the configuration items, such as the number of header and footer lines of the file to be processed mentioned above. This eliminates the need for staff to separately set the corresponding parameters, reducing their workload and time costs.

[0098] Based on the above process, an embodiment of the present invention provides a process of a file processing method, wherein the third server is described by taking a Redis server as an example. Figure 3 Shown, including:

[0099] Step 301: The first server obtains a file to be processed;

[0100] Step 302: Determine a sharding threshold based on the file size of the file to be processed and the performance of the first server and the second server;

[0101] Step 303: The first server slices the to-be-processed file into slices according to the slice threshold, generates slice positions of the to-be-processed slices, and generates subtasks according to the slice positions of the to-be-processed slices.

[0102] Step 304: Send each subtask to the Redis server;

[0103] Step 305: After receiving each subtask, the Redis server sends a broadcast to the second server;

[0104] Step 306: After receiving the broadcast, the second server obtains the subtask from the Redis server;

[0105] Step 307: The second server obtains the corresponding to-be-processed fragment according to the subtask, processes the to-be-processed fragment according to the processing logic of the to-be-processed fragment, and records the processing result of the to-be-processed fragment;

[0106] Step 308: The second server may share the processing results of each slice to be processed, and one or more second servers may determine the processing result of the file to be processed based on the processing results of each slice to be processed; or the second server may send the processing results of each slice to be processed to the file server, and determine the processing result of the file to be processed in the file server.

[0107] Based on the same concept, an embodiment of the present invention provides a file processing device. Figure 4 A schematic diagram of a file processing device provided by an embodiment of the present invention is shown in FIG. Figure 4 including:

[0108] Processing module 401, for determining subtasks corresponding to each shard of the file to be processed according to the file processing configuration;

[0109] The transceiver module 402 is used to send each subtask to the third server and write the number of each subtask to the third server;

[0110] The transceiver module 402 is further configured to receive each subtask and broadcast each subtask to each second server;

[0111] The processing module 401 is further configured to process the respective subtasks in parallel according to the processing logic in the file processing configuration, and send the processing results to the third server;

[0112] The processing module 401 is further configured to determine that if the number of the received processing results is equal to the number of each subtask, the processing of the to-be-processed file is completed.

[0113] In one possible design, the processing module 401 is specifically used for: the first server obtains the file to be processed according to the file path in the file processing configuration; the first server determines the positions of each slice of the file to be processed according to the slice rules in the file processing configuration; the first server determines each subtask according to the positions of each slice. In one possible design, the first server determines the positions of each slice of the file to be processed according to the slice rules in the file processing configuration, including: the first server determines the file body of the file to be processed; the first server determines the positions of each slice of the file body according to the slice threshold in the slice rule; for each slice position, the position is determined as follows: starting from the starting position of the slice, determining whether the current character is a line break at the position that meets the slice threshold; if not, continuing to count until the position of the first line break is used as the end position of the slice.

[0114] In a possible design, the processing module 401 is specifically configured to: determine the fragmentation threshold by the following formula:

[0115] P=MIN(G,MAX(P / (2*N),L))

[0116] Wherein, P is the sharding threshold; G is the maximum value of the sharding threshold, L is the minimum value of the sharding threshold, and G and L are determined by historical experience and server performance; N is the number of processor cores of the server.

[0117] In one possible design, the processing module 401 is specifically used to: the first server starts from the first non-newline character of the file to be processed, and determines the file header of the file to be processed according to the number of file header lines in the file processing configuration; the first server starts from the last non-newline character of the file to be processed, and determines the file tail of the file to be processed according to the number of file tail lines in the file processing configuration; the first server determines the part of the file to be processed except the file header and the file tail as the file body.

[0118] In one possible design, the transceiver module 402 is specifically used for: the third server receives the subtasks and broadcasts them to the second servers; after receiving the broadcast, the second server obtains the subtasks from the third server in turn, and obtains the to-be-processed fragments from the to-be-processed file according to the fragment position in the subtask; the second servers process their respective subtasks in parallel according to the processing logic in the file processing configuration, and send the processing results to the third server, including: the second servers process their respective to-be-processed fragments in parallel according to the processing logic in the file processing configuration, and send the processing results of the to-be-processed fragments to the third server.

[0119] In a possible design, the transceiver module 402 is specifically used for: the second server obtains the to-be-processed fragment from the to-be-processed file into the memory of the second server according to the fragment position in the subtask through memory mapping.

[0120] In a possible design, the file processing configuration is written on a preset template; the preset template is provided with function codes for opening a file, reading a file, and closing a file.

[0121] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0125] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A file processing method, characterized in that: Applicable to a distributed cluster including multiple servers; the method includes: The first server determines, according to the file processing configuration, each subtask corresponding to each slice of the file to be processed: the first server obtains the file to be processed according to the file path in the file processing configuration; the first server determines the position of each slice of the file to be processed according to the slice rule in the file processing configuration; the first server determines each subtask according to the position of each slice; wherein, the first server determines the position of each slice of the file to be processed according to the slice rule in the file processing configuration, including: the first server determines the file body of the file to be processed; the first server determines the position of each slice of the file body according to the slice threshold in the slice rule; for each slice position, the position is determined in the following manner: starting from the starting position of the slice, determining whether the current character is a line break at the position that meets the slice threshold; if not, continuing to count until the position of the first line break is used as the end position of the slice; wherein, the file processing configuration is written on a preset template; the preset template is provided with function codes for opening a file, reading a file, and closing a file; The first server sends each subtask to a third server and writes the number of each subtask to the third server; the third server is a server including an atomic function program, and is used to synchronize the status of each second server; The third server receives each subtask and broadcasts each subtask to each second server; Each second server obtains each subtask from the third server according to the received broadcast, processes the respective subtasks in parallel according to the processing logic in the file processing configuration, and sends the processing results to the third server; The third server determines that the number of the received processing results is equal to the number of each subtask, and the processing of the pending files is completed.

2. The method according to claim 1, wherein The first server determines the positions of each fragment of the file body according to the fragment threshold in the fragment rule, including: The fragmentation threshold is determined by the following formula: P=MIN(G,MAX(P / (2*N),L)) Wherein, P is the sharding threshold; G is the maximum value of the sharding threshold, L is the minimum value of the sharding threshold, and G and L are determined by historical experience and server performance; N is the number of processor cores of the server.

3. The method according to claim 1, wherein The first server determines the file body of the to-be-processed file, including: The first server determines the file header of the file to be processed starting from the first non-newline character of the file to be processed and according to the number of file header lines in the file processing configuration; The first server determines the end of the file to be processed starting from the last non-newline character of the file to be processed and according to the number of the end line of the file in the file processing configuration; The first server determines the portion of the to-be-processed file excluding the file header and the file tail as the file body.

4. The method according to claim 1, wherein The third server receives each subtask and broadcasts each subtask to each second server, including: The third server receives each subtask and broadcasts it to each second server; after receiving the broadcast, the second server sequentially obtains the subtask from the third server and obtains the to-be-processed fragment from the to-be-processed file according to the fragment position in the subtask; The second servers process their respective subtasks in parallel according to the processing logic in the file processing configuration, and send the processing results to the third server, including: The second servers process their respective to-be-processed fragments in parallel according to the processing logic in the file processing configuration, and send processing results of the to-be-processed fragments to the third server.

5. The method according to claim 4, wherein Obtaining the fragment to be processed from the to-be-processed file according to the fragment position in the subtask includes: The second server obtains the to-be-processed fragment from the to-be-processed file and stores it in the memory of the second server according to the fragment position in the subtask in a memory mapping manner.

6. A file processing device, characterized in that: Applicable to a distributed cluster including multiple servers; the device includes: A processing module is used to determine, according to a file processing configuration, each subtask corresponding to each fragment of a file to be processed; the processing module is specifically used to obtain a file to be processed according to a file path in the file processing configuration; determine the position of each fragment of the file to be processed according to a fragmentation rule in the file processing configuration; and determine each subtask according to the position of each fragment; wherein, when the processing module is used to determine the position of each fragment of the file to be processed according to the fragmentation rule in the file processing configuration: determine the file body of the file to be processed; determine the position of each fragment of the file body according to a fragmentation threshold in the fragmentation rule; for each fragment position, determine in the following manner: start counting from the starting position of the fragment, and determine whether the current character is a line break at a position that meets the fragmentation threshold; if not, continue counting until the position of the first line break is used as the end position of the fragment; wherein, the file processing configuration is written on a preset template; the preset template is provided with function codes for opening a file, reading a file, and closing a file; a transceiver module, configured to send each subtask to a third server and write the number of each subtask into the third server; the third server is a server including an atomic function program, configured to synchronize the status of each second server; a transceiver module, configured to receive each subtask and broadcast each subtask to each second server; The processing module is further configured to obtain each subtask from the third server according to the received broadcast, and then process the respective subtasks in parallel according to the processing logic in the file processing configuration, and send the processing results to the third server; The transceiver module is further configured to determine that if the number of the received processing results is equal to the number of each subtask, the processing of the pending file is completed.

7. A computing device, characterized in that include: memory for storing computer programs; A processor, configured to call a computer program stored in the memory, and execute the method according to any one of claims 1 to 5 according to the obtained program.

8. A computer-readable non-volatile storage medium, characterized in that: The method comprises a computer-readable program, which, when read and executed by a computer, causes the computer to perform the method according to any one of claims 1 to 5.

Images