Multi-threaded data file processing method, device and medium

Abstract

Embodiments of the present disclosure relate to a multi-threaded data file processing method, device, and medium. According to the method, in response to receiving an operation request to perform a processing task on a data file to be processed, a main thread is started, wherein the main thread is configured to perform the following operations after being started: determining the file slice size and the maximum number of idle task threads based on machine performance; generating file slice nodes for the data file to be processed one by one based on the file slice size, for addition to a list of slice nodes to be processed; if it is determined that there are one or more idle task threads in the thread pool, the next one or more file slice nodes to be processed in the list of slice nodes to be processed are respectively assigned to the one or more idle task threads, so as to start the one or more idle task threads to respectively perform the processing tasks on the assigned corresponding file slice nodes. In this way, the processing efficiency of data files can be greatly improved.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of file processing, and more particularly to a multi-threaded data file processing method, device, and medium. Background Art

[0002] Computer-based data collection, preparation, and migration—particularly those involving financial data—often require processing of related data files. For example, data files containing sensitive data must be encrypted and stored on specialized media before being physically delivered to designated data collection points. Furthermore, when storing various data files from different data sources in a database, these files often require specific formatting and processing before they can be stored, as they often come in varying formats (e.g., output formats) and sizes.

[0003] At present, the awk file processing tool under the Linux environment or the text analyzer such as UltraEdit under the Windows environment are usually used to process the data file to be processed. These tools all read the records included in the data file to be processed into the computing device in the order of lines and process them line by line. For example, the awk file processing tool has the following working order: the tool reads the current line record of the data file to be processed, performs field decomposition on the current line record using a separator (default is a space) to obtain multiple fields (also called domains) of the line record, then processes each field to obtain the desired result, and after the current line record is processed, reads the next line record of the data file to be processed and processes the line record in the same way until all lines of the data file to be processed are processed. As can be seen from this, the awk tool is a serial file processing tool and cannot process data files in parallel. Its processing efficiency for large data files is particularly low. Therefore, the file processing methods adopted by these existing file processing tools are inefficient, especially for large files such as GB or TB.

[0004] Therefore, it is necessary to provide a file processing technology that helps to complete the processing of files (especially large data files of GB, TB or higher orders) at a faster speed. Summary of the Invention

[0005] In response to the above problems, the present disclosure provides a multi-threaded data file processing method, device and medium, which can greatly improve the processing efficiency of files (especially large data files).

[0006] According to the first aspect of the present disclosure, a multi-threaded data file processing method is provided, the method comprising: in response to receiving an operation request to perform a processing task on a data file to be processed comprising multiple lines of records, starting a main thread, the main thread being configured to perform the following operations after starting: determining a file slice size and a maximum number of idle task threads based on machine performance; generating file slice nodes one by one for the data file to be processed based on the file slice size, for addition to a list of slice nodes to be processed; if it is determined that there are one or more idle task threads in a thread pool, assigning the next one or more file slice nodes to be processed in the list of slice nodes to be processed to the one or more idle task threads respectively, so as to start the one or more idle task threads to respectively perform the processing task on the assigned corresponding file slice nodes, and each task thread is recycled as an idle task thread by the thread pool after completing the processing task, and the number of the one or more idle task threads is less than or equal to the maximum number of idle task threads.

[0007] According to a second aspect of the present disclosure, a computing device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to execute the multi-threaded data file processing method of the first aspect of the present disclosure.

[0008] In a third aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the multi-threaded data file processing method of the first aspect of the present disclosure.

[0009] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0011] Figure 1 A schematic diagram of a system 100 for implementing a multi-threaded data file processing method according to an embodiment of the present invention is shown.

[0012] Figure 2 FIG2 shows a flowchart of a method 200 executed by a main thread after being started according to an embodiment of the present disclosure.

[0013] Figure 3 A flowchart of a method 300 executed by a service thread after being started according to an embodiment of the present disclosure is shown.

[0014] Figure 4 A flowchart of a method 400 for generating file slice nodes one by one for data files to be processed according to an embodiment of the present disclosure is shown.

[0015] Figure 5 FIG. 5 is a flowchart illustrating a method 500 executed by a merge node thread after being started according to an embodiment of the present disclosure.

[0016] Figure 6 FIG. 6 is a flowchart illustrating a method 600 executed by a check thread after being started according to an embodiment of the present disclosure.

[0017] Figure 7 A block diagram of an electronic device 700 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0018] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0019] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] As mentioned above, currently, the awk file processing tool under Linux environment or the text analyzer such as UltraEdit under Windows environment are usually used to process the data file to be processed. These tools all read the records included in the data file to be processed into the computing device in the order of rows and process them line by line. For example, the awk file processing tool has the following working order: read the current row record of the data file to be processed, perform field decomposition on the current row record using a separator (default is space) to obtain multiple fields (also called domains) of the row record, then process each field to obtain the desired result, and after the current row record is processed, read the next row record of the data file to be processed and process the row record in the same way until all rows of the data file to be processed are processed. Therefore, the file processing methods adopted by these existing file processing tools are inefficient, especially for large files such as GB or TB.

[0021] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, an example embodiment of the present disclosure proposes a multi-threaded data file processing method, the method comprising: in response to receiving an operation request to perform a processing task on a data file to be processed comprising multiple lines of records, starting a main thread, the main thread being configured to perform the following operations after being started: determining the file slice size and the maximum number of idle task threads based on machine performance; generating file slice nodes one by one for the data file to be processed based on the file slice size, for addition to a list of slice nodes to be processed; if it is determined that there are one or more idle task threads in the thread pool, assigning the next one or more file slice nodes to be processed in the list of slice nodes to be processed to the one or more idle task threads respectively, so as to start the one or more idle task threads to perform the processing tasks on the assigned corresponding file slice nodes respectively, and each task thread is recycled as an idle task thread by the thread pool after completing the processing task, and the number of the one or more idle task threads is less than or equal to the maximum number of idle task threads. In the above method, file slice nodes are generated for the data files to be processed with appropriate file slice sizes, and multiple threads collaborate to complete the entire processing process, thereby greatly improving the processing efficiency of files (especially large data files).

[0022] Figure 1 FIG. 1 is a schematic diagram of a system 100 for implementing a multi-threaded data file processing method according to an embodiment of the present invention. Figure 1As shown, the system 100 may include, for example, a computing device 110, a plurality of user terminals 120-1, 120-M to 120-N, and a network 130. The computing device 110 may exchange data with the plurality of user terminals 120-1, 120-M to 120-N via the network 130.

[0023] User terminals 120-1 to 120-N may include, for example, but are not limited to, a user's mobile terminal, tablet device, personal computer, laptop computer, desktop computer, etc., and may be used to send an operation request to the computing device 110 to execute a processing task on a data file to be processed comprising multiple rows of records, thereby starting a corresponding program to execute the multi-threaded data file processing method of the present disclosure. In the present disclosure, the user terminal may be part of the computing device 110 or may be independent of the computing device 110.

[0024] In some embodiments, the computing device 110 may include at least one processor and at least one memory coupled to the at least one processor, the memory storing instructions executable by the at least one processor that, when executed by the at least one processor, perform the multi-threaded data file processing method described below. In some embodiments, when executed by the at least one processor, the instructions run a Java virtual machine, which, upon activation, can implement the multi-threaded data file processing method of the present disclosure. In the present disclosure, the multi-threaded data file processing method is performed by a collaborative division of labor among multiple threads. In some embodiments, the multiple threads used to perform the multi-threaded data file processing method include a main thread and multiple task threads, wherein the main thread is activated in response to the computing device 110 receiving an operation request to perform a processing task on a data file to be processed comprising multiple rows of records. The task thread is activated after the main thread assigns it a file slice node to be processed. In some embodiments, the multiple threads also include a merge node thread to merge the processing results of each file slice node into the corresponding result file. The merge node thread may be activated at the same time as the main thread. In some embodiments, the multiple threads also include a check thread. The check thread can be started at a predefined time, which can be predefined in the program's parameter file (also called a configuration file) to check the running status of at least one of the main thread, the working task thread, and the merge node thread. The working task thread refers to the task thread that is executing the processing task. In some embodiments, the merge node thread can also be configured to check the running status of the check thread. Specifically, in the present disclosure, the main thread is configured to execute the following steps after being started. Figure 2 The method 200 is described in more detail. Each task thread is configured to execute the following after being started. Figure 3 The method 300 is described in more detail. The merge node thread is configured to execute the following after being started. Figure 5 Method 500 is described in more detail. In the present disclosure, the main thread, each task thread and the merge node thread can run in parallel to cooperate and divide the work to complete the corresponding operations. When running, multiple task threads also execute processing tasks on the corresponding file slice nodes assigned to them in parallel, thereby greatly improving the overall parallelism and making full use of the memory of the computing device 110 without occupying temporary disk space. The specific structure of the computing device 110 can be, for example, combined as follows: Figure 7 As stated.

[0025] In some embodiments, the multi-threaded data file processing method according to the embodiment of the present disclosure includes: in response to receiving an operation request to perform a processing task on a data file to be processed including multiple rows of records, starting a main thread, and after being started, the main thread executes the following steps in conjunction with Figure 2 Method 200 is described in more detail. In the present disclosure, the data file to be processed includes multiple rows of data records, and each row of data records can be referred to as a record. Each record may include multiple data associated with multiple attributes, and each attribute can be referred to as a field or domain. For data files, each record is independent of each other. The data file to be processed mainly refers to a large data file, such as a data file of GB level, TB level or larger level. In the present disclosure, after being started, the main thread will dynamically calculate the file slice size and the maximum number of idle task threads, and then divide the data file to be processed in sequence (also called slicing) to generate corresponding file slice nodes. The generated file slice nodes will be added to the list of slice nodes to be processed for assignment to the idle task thread and start the idle task thread for corresponding processing. After being started by the main thread, the task thread executes the following combined Figure 3 Method 300 is described in more detail. In the present disclosure, the processing tasks mentioned above may include, for example, but are not limited to, converting each line of records in the data file to be processed from a first output format to a customized second output format, detecting whether the data file to be processed includes conflicting characters or processing detected conflicting characters, etc.

[0026] In some embodiments, the multi-threaded data file processing method may further include starting a merge node thread while starting the main thread. After the merge node thread is started, the following will be combined Figure 5 Method 500 is described in more detail.

[0027] In some embodiments, the multi-threaded data file processing method may further include starting a check thread at a predefined time when the merge node thread has not yet completed. The check thread is configured to check the running status of at least one of the main thread, the task thread in operation, and the merge node thread after being started.

[0028] In some embodiments, since the Java language is cross-platform, software programs written in the Java language can run on any operating system. Therefore, in order to achieve cross-platform operation of the multi-threaded data file processing method of the present disclosure, the application program associated with the multi-threaded data file processing method can be written in the Java language, so that the multi-threaded data file processing method runs in the Java virtual machine (hereinafter referred to as JVM).

[0029] Figure 2 1 shows a flow chart of a method 200 executed by the main thread after being started according to an embodiment of the present disclosure. Figure 1 The computing device 110 shown is executed, and may also be executed on Figure 7 The method 200 is executed at the electronic device 700. It should be understood that the method 200 may further include additional blocks not shown and / or may omit the blocks shown, and the scope of the present disclosure is not limited in this respect.

[0030] In step 202 , the main thread determines the file slice size and the maximum number of idle task threads based on machine performance.

[0031] In the present disclosure, the file slice size refers to the number of bytes occupied by each file slice obtained by segmenting the data file to be processed, that is, the number of bytes occupied by the multiple lines of records of the data file to be processed that are divided into each file slice. Selecting an appropriate file slice size is very critical for achieving the minimum processing time of the multi-threaded data file processing method of the present disclosure, thereby maximizing the processing efficiency, because selecting an appropriate file slice size can effectively prevent the computing device 110 from pausing due to excessive memory utilization (the main thread has a memory alarm prediction), which leads to a problem of excessively long processing time. On the other hand, it helps prevent the main thread from frequently starting task threads (thread startup is time-consuming) and the merge node thread from frequently writing result files, which leads to an unsatisfactory processing time.

[0032] In some embodiments, the file slice size may be determined based on the maximum free memory maxFreeMem allocated to the Java virtual machine and the maximum free memory utilization rate MaxFreeMemRate of the Java virtual machine. In some implementations, the file slice size may be expressed using the following formula (1):

[0033] sliceBytes=maxFreeMem*MaxFreeMemRate (1)

[0034] In the above formula (1), sliceBytes indicates the file slice size (in bytes), maxFreeMem indicates the maximum free memory allocated to the Java virtual machine, and MaxFreeMemRate indicates the maximum free memory utilization rate of the Java virtual machine.

[0035] The maximum free memory maxFreeMem allocated to the Java virtual machine can be obtained through a Java method well known in the art.

[0036] In some embodiments, the maximum free memory utilization rate MaxFreeMemRate can be determined by iteratively training the reciprocal of the maximum free memory utilization rate based on the sample data file. In such training, the cutoff condition of the iteration may be, for example, that the second time duration spent on processing the to-be-processed file in units of the file slice size determined based on this iteration (the file slice size is determined based on the reciprocal of the maximum free memory utilization rate of this iteration and the above formula (1)) is greater than the first time duration spent on processing the to-be-processed file in units of the file slice size determined based on the previous iteration (the file slice size is determined based on the reciprocal of the maximum free memory utilization rate of the previous iteration and the above formula (1)), and the difference between the second time duration and the first time duration is less than a predetermined threshold. At this time, the reciprocal of the maximum free memory utilization rate obtained in this iteration is the training result obtained in this training, and the maximum free memory utilization rate can be determined based on this reciprocal. In order to ensure the smooth implementation of the training, it is also necessary to select an initial value of the reciprocal of the maximum free memory utilization rate at the beginning of the training (the initial value may be, for example, 64). In addition, the reciprocal of the maximum idle memory utilization of this iteration may be a multiple of the reciprocal of the maximum idle memory utilization of the previous iteration, for example, 2 times thereof. It is worth mentioning that the reciprocal of the maximum idle memory utilization of the first iteration is a multiple of the initial value of the reciprocal of the maximum idle memory utilization, for example, 2 times thereof.

[0037] Of course, in order to improve the accuracy of training, such training can be performed multiple times, and the average of the multiple training results can be taken as the inverse of the maximum free memory utilization obtained by the final training.

[0038] In this disclosure, the maximum number of idle task threads refers to the maximum number of task threads that are not executing processing tasks at any given moment and, therefore, can be assigned to file slice nodes for processing. In some embodiments, the maximum number of idle task threads is the number of CPU cores of the computing device 110. Therefore, determining the maximum number of idle task threads based on machine performance includes obtaining the number of CPU cores of the computing device and using the number of CPU cores as the maximum number of idle task threads.

[0039] In step 204, the main thread generates file slice nodes for the data files to be processed one by one based on the file slice size, so as to be added to the list of slice nodes to be processed.

[0040] In the present disclosure, except for the first file slice node (i.e., the header file slice node) of the data file to be processed, all subsequent file slice nodes of the data file to be processed are generated one by one in sequence, and each file slice node will be added to the list of slice nodes to be processed by the main thread after being generated, so as to wait to be assigned to an available idle task thread in the list of slice nodes to be processed for processing. In addition, in the present disclosure, all generated file slice nodes store multiple lines of records of the data file to be processed that are divided into the file slice nodes, and except for the first file slice node, the number of bytes occupied by these records is the file slice size. In the present disclosure, the first file slice node can be generated according to a predefined number of rows of the header slice node, that is, the number of rows of the multiple rows of records stored in the first file slice node is equal to the predefined number of rows of the header slice node. In the present disclosure, the predefined number of rows of the header slice node can be predefined in the parameter file of the corresponding program (i.e., the configuration file of the corresponding program).

[0041] In some embodiments, the file slice node can be organized according to a certain data structure. Specifically, as shown in Table 1, each file slice node may include a first part for storing multiple lines of records in the data file to be processed that are divided into the file slice node, a second part for storing the task processing results of the multiple lines of records divided into the file slice node, a third part for storing the completion status of the file slice node, and a fourth part for storing a pointer to the next file slice node in the list of slice nodes to be processed. In the present disclosure, the first part can store the multiple lines of records divided into the file slice node in the form of a string array, and the string array of the multiple lines of records is arranged in the order of these records in the data file to be processed. In addition, in order to reduce the number of times the merge node thread writes the result file and save the time overhead of writing the result file, the processing results obtained by performing task processing on the multiple lines of records in each file slice node can be merged into a string and then stored in the second part of the file slice node. In the present disclosure, when a file slice node has not been processed, the completion status stored in its fourth part defaults to incomplete, and once the file slice node is processed by the corresponding task thread, the task thread will update the completion status stored in the third part to completed.

[0042] Table 1

[0043]

[0044] The following will be combined Figure 4Step 204 is further described in more detail.

[0045] In step 206, if the main thread determines that there are one or more idle task threads in the thread pool, the main thread assigns the next one or more file slice nodes to be processed in the list of pending slice nodes to the one or more idle task threads, respectively, to start the one or more idle task threads to perform processing tasks on the assigned corresponding file slice nodes, and each task thread is recycled by the thread pool as an idle task thread after completing the processing task. In the present disclosure, the number of the one or more idle task threads mentioned above is less than or equal to the maximum number of idle task threads determined in step 202.

[0046] According to step 206 and step 204, in the present disclosure, once the main thread finds that there are one or more idle task threads in the thread pool, it will stop executing step 204 and instead execute step 206 to assign the next one or more file slice nodes to be processed in the list of pending slice nodes to the one or more idle task threads for processing. After the assignment is completed, if the main thread finds that there are no idle task threads in the thread pool, it will continue to perform the operation of step 204. That is, if the main thread does not find that there are one or more idle task threads in the thread pool, the main thread will continue to perform the operation of step 204 unless all file slice nodes of the data file to be processed have been generated and added to the list of pending file slice nodes for processing.

[0047] In the present disclosure, each task thread is configured to load the corresponding processing task after being started, and execute the processing task on the assigned file slice node and generate the corresponding processing result. After the task thread completes the processing for the assigned file slice node, it can be recycled by the thread pool for reallocation by the main thread, and the completion status of the assigned file slice node is updated to completed. In the present disclosure, two queues related to multiple task threads can be maintained in the thread pool, one of which is an execution queue for storing the thread identifier of the task thread that is executing the processing task (for example, the thread number of the task thread), and the other queue is an idle queue for storing the thread identifier of the task thread that is idle, that is, not executing the processing task. Since Java stipulates that each thread can only be started once, in the present disclosure, in order to achieve the reuse of task threads, each task thread will be cancelled after completing the corresponding processing task on the assigned file slice node, but the thread identifier of the task thread will be recycled to the idle queue of the thread pool to indicate that the task thread with the thread identifier is available for the main thread to reallocate to execute the processing task again, thereby achieving the reuse of task threads. In the present disclosure, after completing the processing of the assigned file slice node, each task thread will also send a notification to the main thread to inform the main thread that the task thread with the corresponding thread identifier has been recycled into the thread pool for the main thread to allocate again.

[0048] In the present disclosure, at each moment there are at most a maximum number of idle task threads for the main thread to allocate and start, and the maximum number of idle task threads may depend on the number of CPU cores of the computing device 110, for example, the number of CPU cores of the computing device 110 is used as the maximum number of idle task threads. For example, when the main thread is started in response to an operation request to perform a processing task on a data file to be processed, all task threads are not performing tasks, so all task threads are idle, so at this time, the idle task threads including the maximum number of idle task threads are allocated to the file slice node for the main thread to perform the processing task. As mentioned above, after each task thread completes the processing task on the assigned file slice node, it will be recycled by the thread pool for reallocation by the main thread. It can be seen from this that in the present disclosure, at each moment there can be at most a maximum number of idle task threads for task threads to perform processing tasks, thereby further improving the processing efficiency of the data files to be processed.

[0049] Figure 3 1 shows a flow chart of a method 300 executed by a service thread after being started according to an embodiment of the present disclosure. Figure 1 The computing device 110 shown is executed, and may also be executed on Figure 7The method 300 is executed at the electronic device 700. It should be understood that the method 300 may further include additional blocks not shown and / or may omit the blocks shown, and the scope of the present disclosure is not limited in this respect.

[0050] In step 302, the service thread loads the processing task.

[0051] Loading a processing task actually refers to loading the relevant task parameters of the processing task. In the present disclosure, the processing task is a processing task requested by a user through a terminal device to be executed on a data file to be processed that includes multiple lines of records. In the present disclosure, the processing tasks mentioned above may include, but are not limited to, converting each line of records in the data file to be processed from a first output format to a customized second output format, detecting whether the data file to be processed includes conflicting characters and processing the detected conflicting characters, etc.

[0052] In step 304, the business thread performs the processing task on the records included in the assigned corresponding file slice node.

[0053] For example, if the processing task is to convert each row of records in the data file to be processed from a first output format to a customized second output format, the business thread will convert each record included in the corresponding dispatched file slice node from the first output format to the second output format.

[0054] For another example, if the processing task is to detect whether the data file to be processed includes conflicting characters and process the detected conflicting characters, then when the processing task detects conflicting characters in the records included in the corresponding file slice node assigned, the detected conflicting characters will be processed according to the requirements of the processing task, such as deleting the detected conflicting characters from the corresponding records, identifying the records including the conflicting characters with predetermined characters, etc. In the present disclosure, the conflicting characters indicate characters of the encoding including the auxiliary characters (usually separators) to be added to the records in the encoding. The encoding here can refer to encoding in any encoding format.

[0055] In step 306, after completing the processing task for all records included in the assigned corresponding file slice node, the business thread integrates the obtained processing results into a string and writes it into the assigned corresponding file slice node.

[0056] If all records of the file slice stored in the corresponding file slice node have been processed, it means that the business thread has completed the processing task for all records included in the assigned corresponding file slice node. In this patent application, after the processing task is completed for all records included in the assigned corresponding file slice node, the completion status of the assigned corresponding file slice node is also updated to completed.

[0057] In some embodiments, the resulting processing results are consolidated into a single string and written to the second portion of the assigned file slice node. This helps reduce the number of times the merge node thread writes the result file, saving time overhead. Additionally, the completion status can be updated in the third portion of the corresponding file slice node.

[0058] In this disclosure, after each task thread completes processing tasks on the assigned file slice node, it will be recycled by the thread pool for reallocation by the main thread. Furthermore, after each task thread completes processing tasks on the assigned file slice node, it will also notify the main thread that it has been recycled by the thread pool and is therefore available for reallocation.

[0059] Figure 4 FIG. 4 is a flow chart showing a method 400 for generating file slice nodes one by one for a data file to be processed according to an embodiment of the present disclosure. Figure 1 The computing device 110 shown is executed, and may also be executed on Figure 7 The method 400 is executed at the electronic device 700. It should be understood that the method 400 may further include additional blocks not shown and / or may omit the blocks shown, and the scope of the present disclosure is not limited in this respect.

[0060] In step 402, the main thread generates the first file slice node for the data file to be processed according to the predefined number of rows of the head slice node.

[0061] In the present disclosure, the predefined number of rows of head slice nodes may be predefined in a parameter file of a corresponding program (ie, a configuration file of the corresponding program).

[0062] Generating the first file slice node for the data file to be processed according to the predefined number of rows of the header slice node refers to splitting the records of the number of rows of the header slice node starting from the front of the data file to be processed, and generating the first file slice node based on these records. Therefore, the records included in the generated first file slice node are the front multiple rows of records of the data file to be processed, and the total number of rows of these records is the number of rows of the header slice node.

[0063] In step 404 , the main thread determines the number of file slice rows based on the average number of row bytes of the first file slice node generated in step 402 and the file slice size determined in step 202 .

[0064] The average number of row bytes mentioned in step 404 refers to the average number of bytes occupied by each row of records included in the first file slice node.

[0065] In some embodiments, the required number of file slice rows can be obtained by dividing the file slice size by the average row byte number.

[0066] In step 406 , the main thread detects whether the memory usage is less than a predetermined threshold.

[0067] In the present disclosure, the main thread calculates the memory usage in real time during execution, so that when the memory usage is less than or equal to a predetermined threshold (for example, the lower limit of memory usage), it will automatically give up the CPU and wait for the memory usage to rise above the predetermined threshold before continuing to work. This ensures that the occupied memory rate is controllable while maximizing memory utilization. In the present disclosure, when the memory usage is less than or equal to the predetermined threshold, it means that the current memory is insufficient, and therefore the corresponding operation task cannot be completed well. Therefore, it is necessary to wait for the memory usage to rise above the predetermined threshold before continuing to work.

[0068] Specifically, in step 408, if it is detected that the memory occupancy rate is less than a predetermined threshold, subsequent file slice nodes are generated one by one for the data file to be processed according to the number of file slice rows.

[0069] In addition, if it is detected that the memory occupancy rate is less than or equal to the predetermined threshold, the main thread waits until it is detected that the memory occupancy rate rises back to be greater than the predetermined threshold.

[0070] By adopting the above means, the present disclosure can realize parallel processing by task threads based on file slice nodes including records of file slice line numbers, thereby helping to further improve the efficiency of file processing.

[0071] As previously mentioned, in some embodiments, the multi-threaded data file processing method of the present disclosure may further include starting a merge node thread simultaneously with starting the main thread. In the present disclosure, the merge node thread is configured to perform the following operations after being started to sequentially merge the processing results of each slice node of the data file to be processed into the corresponding processed file.

[0072] The merge node thread periodically checks whether the first file slice node in the list of pending slice nodes has been processed. In the present disclosure, if the completion status stored in the third part of the first slice node is completed, it means that the slice node has been processed. At this time, the corresponding processing result has been added to the second part of the first slice node. In the present disclosure, the first file slice node refers to the first file slice node in the list of pending slice nodes.

[0073] In response to determining that the first file slice node has been processed, the merge node thread writes the processing result to the corresponding result file according to the type of the processing result in the first file slice node (i.e., the processing result stored in the second part of the first file slice node). In the present disclosure, the type of processing result may include normal processing results, abnormal processing results, and repair processing results. Therefore, the processing result of each file slice node can be one of normal processing results, abnormal processing results, and repair processing results, and the result file may correspondingly include a normal processing result file, an abnormal processing result file, and a repair processing result file. Therefore, in the present disclosure, if the processing result belongs to a normal processing result, the processing result is written to the corresponding normal processing result file. If the processing result belongs to an abnormal processing result, the processing result is written to the corresponding abnormal processing result file (for example, if a row of records includes a row that does not meet the requirements (for example, its content format is inaccurate), the processing result of the record will be written to the corresponding abnormal processing result file). If the processing result belongs to a repair processing result (for example, the processing result of the conflicting characters belongs to the repair processing result), the processing result is written to the corresponding repair processing result file.

[0074] After the processing result in the first file slice node has been written to the corresponding result file, the first file slice node is deleted from the list of pending slice nodes. In the present disclosure, if the list of pending slice nodes includes a second file slice node, the second file slice node becomes the first file slice node in the list of pending slice nodes, so the merge node thread will merge the processing results of the new first file slice node next time.

[0075] In the present disclosure, the above merging process performed by the merge node thread will continue until all operations to be performed by the main thread are completed and the list of pending slice nodes becomes empty. Therefore, the merge node thread is configured to complete when all operations to be performed by the main thread are completed and the list of pending slice nodes becomes empty. The main thread is configured to complete when the merge node thread completes.

[0076] In some embodiments, in order to improve the robustness and stability of the algorithm, the multi-threaded data file processing method of the present disclosure further provides a check thread, which can be periodically started when the merge node thread has not yet completed. The check thread is configured to check the operation status of the main thread, the task thread in the work, and the merge node thread after being started, and restart the thread when any thread interrupts the operation for more than the corresponding timeout time interval. In the present disclosure, the period for periodically starting the check thread can be pre-defined in the program parameter file (also called configuration file).

[0077] Figure 6FIG. 6 is a flow chart showing a method 600 executed by a check thread after being started according to an embodiment of the present disclosure. Figure 1 The computing device 110 shown is executed, and may also be executed on Figure 7 The method 600 is executed at the electronic device 700. It should be understood that the method 600 may include additional blocks not shown and / or may omit the blocks shown, and the scope of the present disclosure is not limited in this respect.

[0078] In step 602, the checking thread checks whether the main thread has been interrupted for more than a predetermined first timeout period. In the present disclosure, the first timeout period can be set according to actual use requirements, and can be predefined in a parameter file (also called a configuration file) of the program, for example.

[0079] In step 604 , when the checking thread determines that the main thread has been interrupted for more than a predetermined first timeout period, the checking thread restarts the main thread.

[0080] In step 606, the checking thread checks whether the task thread in the work has been interrupted for more than a predetermined second timeout period. In the present disclosure, the second timeout period can be set according to actual use requirements, and it can be predefined in a parameter file (also called a configuration file) of the program, for example.

[0081] In step 608 , when the checking thread determines that the task thread has been interrupted for more than a predetermined second timeout period, the checking thread restarts the task thread.

[0082] In step 610, the check thread checks whether the merge node thread has been interrupted for more than a predetermined third timeout period. In the present disclosure, the third timeout period can be set according to actual use requirements, and can be predefined in a parameter file (also referred to as a configuration file) of the program.

[0083] In step 612, when the checking thread determines that the merge node thread has been interrupted for more than a predetermined third timeout period, the checking thread restarts the merge node thread.

[0084] In the present disclosure, by providing a check thread, accidental interruptions in the process of processing files can be effectively prevented, thereby improving robustness and stability. In some embodiments, during the operation of the check thread, the merge node thread can also be used to check the operation status of the check thread. Specifically, the merge node thread is also configured to check whether the check thread is interrupted, and restart the check thread when it is determined that the check thread has been interrupted for more than a predetermined fourth timeout time interval. For example, the merge node thread may check the check thread each time the processing result in the first file slice node in the list of slice nodes to be processed is written to the corresponding result file and the first file slice node is deleted from the list of slice nodes to be processed.

[0085] In the present disclosure, the check thread may be configured to complete as the merge node thread completes.

[0086] In the present disclosure, the fourth timeout interval may also be set according to actual usage requirements, for example, predefined in a parameter file of a corresponding program.

[0087] Figure 7 Schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device 110 shown can be implemented by an electronic device 700. As shown, the electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded from a storage unit 708 into a random access memory (RAM) 703. In the random access memory 703, various programs and data required for the operation of the electronic device 700 can also be stored. The central processing unit 701, the read-only memory 702, and the random access memory 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0088] Multiple components in electronic device 700 are connected to input / output interface 705, including: input unit 706, such as a keyboard, mouse, microphone, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as a magnetic disk, optical disk, etc.; and communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0089] The various processes and procedures described above, such as methods 200-600, may be performed by the central processing unit 701. For example, in some embodiments, the methods 200-600 may be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 700 via the read-only memory 702 and / or the communication unit 709. When the computer program is loaded into the random access memory 703 and executed by the central processing unit 701, one or more actions of the methods 200-600 described above may be performed.

[0090] The present disclosure relates to methods, apparatuses, systems, electronic devices, computer-readable storage media, and / or computer program products. The computer program products may include computer-readable program instructions for executing various aspects of the present disclosure.

[0091] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0092] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge computing devices. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0093] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0094] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0095] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0096] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0097] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0098] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-threaded data file processing method, comprising: In response to receiving an operation request for executing a processing task on a data file to be processed including multiple rows of records, a main thread is started. When the main thread is started, a merge node thread is also started. When the merge node thread has not yet been completed, a check thread is periodically started. The check thread is configured to perform the following operations after being started: check whether the main thread has been interrupted for more than a predetermined first timeout time interval, and restart the main thread when it is determined that the main thread has been interrupted for more than the first timeout time interval; check whether a task thread in a job has been interrupted for more than a predetermined second timeout time interval, and restart the task thread in the job when it is determined that the task thread in the job has been interrupted for more than the second timeout time interval; check whether the merge node thread has been interrupted for more than a predetermined third timeout time interval, and restart the merge node thread when it is determined that the merge node thread has been interrupted for more than the third timeout time interval. The main thread is configured to perform the following operations after being started: Determine the file slice size and the maximum number of idle task threads based on machine performance; Generating file slice nodes for the data files to be processed one by one based on the file slice sizes, so as to be added to a list of slice nodes to be processed; If it is determined that there are one or more idle task threads in the thread pool, the next one or more file slice nodes to be processed in the list of slice nodes to be processed are respectively assigned to the one or more idle task threads, so as to start the one or more idle task threads to execute the processing tasks on the corresponding assigned file slice nodes respectively. After completing the processing task, each task thread is recycled by the thread pool as an idle task thread, and the number of the one or more idle task threads is less than or equal to the maximum number of idle task threads.

2. The multi-threaded data file processing method according to claim 1, wherein each task thread is configured to perform the following operations after being started: loading the processing task; Executing the processing task on the records included in the assigned corresponding file slice node; After completing the processing tasks for all records included in the assigned corresponding file slice node, the obtained processing results are integrated into a character string and written into the assigned corresponding file slice node.

3. The multi-threaded data file processing method according to claim 1 , wherein generating file slice nodes for the data files to be processed one by one based on the file slice size comprises: Generate a first file slice node for the data file to be processed according to a predefined number of header slice node rows; Determine the number of file slice rows based on the average number of row bytes of the first file slice node and the file slice size; If it is detected that the memory occupancy rate is less than a predetermined threshold, subsequent file slice nodes are generated one by one for the data file to be processed according to the number of file slice rows.

4. The multi-threaded data file processing method according to claim 3, wherein generating file slice nodes for the data files to be processed one by one based on the file slice size further comprises: If it is detected that the memory occupancy rate is less than or equal to the predetermined threshold, the method waits until it is detected that the memory occupancy rate rises back to be greater than the predetermined threshold.

5. The multi-threaded data file processing method according to claim 1, wherein the merge node thread is configured to perform the following operations after being started: Periodically checking whether the first file slice node in the list of slice nodes to be processed has been processed; In response to determining that the first file slice node has been processed, writing the processing result into a corresponding result file according to the type of the processing result in the first file slice node; After the processing result in the first file slice node has been written into the corresponding result file, the first file slice node is deleted from the list of slice nodes to be processed.

6. The multi-threaded data file processing method according to claim 5, wherein the processing result of each file slice node is one of a normal processing result, an abnormal processing result, and a repair processing result, the result file includes a normal processing result file, an abnormal processing result file, and a repair processing result file, and writing the processing result into a corresponding result file according to the type of the processing result in the first file slice node comprises: If the processing result is a normal processing result, the processing result is written into a corresponding normal processing result file; If the processing result is an abnormal processing result, the processing result is written into a corresponding abnormal processing result file; If the processing result is a repair processing result, the processing result is written into a corresponding repair processing result file.

7. The multi-threaded data file processing method according to claim 1, further comprising: During the operation of the check thread, the merge node thread is used to check the operation of the check thread. The merge node thread checks the running status of the check thread by the following operations: the merge node thread checks whether the check thread is interrupted, and restarts the check thread when it is determined that the check thread has been interrupted for more than a predetermined fourth timeout time interval.

8. The multi-threaded data file processing method according to claim 1, wherein the method is executed in a Java virtual machine, and determining the file slice size based on machine performance comprises: The file slice size is determined based on the maximum free memory allocated to the Java virtual machine and the maximum free memory utilization of the Java virtual machine.

9. The multi-threaded data file processing method according to claim 1 , wherein the method is implemented in a Java virtual machine, the machine is a computing device for running the Java virtual machine, and determining the maximum number of idle task threads based on machine performance comprises: The number of CPU cores of the computing device is obtained, and the number of CPU cores is used as the maximum number of idle task threads.

10. The multi-threaded data file processing method according to claim 1, wherein each file slice node includes a first part for storing a plurality of lines of records in the data file to be processed that are divided into the file slice node, a second part for storing task processing results regarding the plurality of lines of records divided into the file slice node, a third part for storing the completion status of the file slice node, and a fourth part for storing a pointer to the next file slice node in the list of slice nodes to be processed, wherein the first part stores the plurality of lines of records divided into the file slice node in the form of a string array.

11. The multi-threaded data file processing method according to claim 1, wherein the processing tasks include: Each line of record in the data file to be processed is converted from a first output format to a customized second output format, and whether the data file to be processed includes conflicting characters is detected and the detected conflicting characters are processed.

12. A computing device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the multi-threaded data file processing method according to any one of claims 1 to 11.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the multi-threaded data file processing method according to any one of claims 1 to 11.

Images