Multi-file data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a multi-file data processing method and device, electronic equipment and a storage medium, which are used for solving the problems of low efficiency and low resource utilization rate of a mode of processing multiple files in a single-program mode. The method comprises the steps of obtaining the file number of to-be-processed files and a preset handle upper limit; dividing the to-be-processed file to each server according to the file number and the handle upper limit; calculating the number of residual handles of each server; calculating the maximum process number of each server according to the residual handle number; and starting the process according to the maximum process quantity, and processing the to-be-processed file by adopting the started process. According to the method, before data processing, the number of files is firstly judged, then judgment is carried out according to system resources, and different cluster servers are clearly started under different numbers through configuration and setting; then concurrent processing of multiple files and full utilization of resources are achieved through file handle calling of multiple servers, and the processing efficiency and the resource utilization rate of the multiple files are improved.

Description

Technical Field

[0001] The present invention relates to the field of file processing technology, and in particular to a method, device, electronic equipment and storage medium for processing data of multiple files. Background Art

[0002] When processing personalized data, due to the format of data processing, there is often a situation where one file represents one data, and sometimes two files represent one data. Therefore, when data processing is required, a large amount of data processing is faced. Although the file size is not large, the number of files is huge. Currently, for multi-file data processing, basically a single program mode is used, and files are processed one by one through a loop reading method. When the number of files to be processed reaches a certain limit, the system will freeze, resulting in a great challenge to the efficiency and stability of data processing. Summary of the invention

[0003] The present invention provides a multi-file data processing method, device, electronic device and storage medium, which are used to solve the technical problems of low efficiency and low resource utilization in the existing single program mode for processing multiple files.

[0004] The present invention provides a multi-file data processing method, comprising:

[0005] Get the number of files to be processed and the preset handle limit;

[0006] Dividing the to-be-processed files onto each server according to the number of files and the handle upper limit;

[0007] Calculating the number of remaining handles of each of the servers;

[0008] Calculate the maximum number of processes of each server according to the number of remaining handles;

[0009] A process is started according to the maximum number of processes, and the to-be-processed file is processed by using the started process.

[0010] Optionally, the step of dividing the to-be-processed files onto each server according to the number of files and the handle upper limit includes:

[0011] Grouping the to-be-processed files according to the number of files and the handle upper limit to obtain a plurality of file groups;

[0012] Distribute each file group to different servers.

[0013] Optionally, the step of calculating the number of remaining handles of each of the servers includes:

[0014] Obtain the number of used file handles of each of the servers;

[0015] The difference between the handle upper limit and the number of used file handles is calculated to obtain the number of remaining handles of each of the servers.

[0016] Optionally, the step of calculating the maximum number of processes of each server according to the number of remaining handles includes:

[0017] Get the number of required handles for each process;

[0018] The maximum number of processes of each server is calculated according to the number of remaining handles and the number of required handles.

[0019] The present invention also provides a multi-file data processing device, comprising:

[0020] A file quantity and handle upper limit acquisition module is used to obtain the file quantity of the files to be processed and the preset handle upper limit;

[0021] A partitioning module, used for partitioning the to-be-processed files onto each server according to the number of files and the handle upper limit;

[0022] A remaining handle number calculation module, used to calculate the remaining handle number of each server;

[0023] A maximum process number calculation module, used to calculate the maximum process number of each server according to the remaining handle number;

[0024] The file processing module is used to start a process according to the maximum number of processes, and use the started process to process the to-be-processed file.

[0025] Optionally, the division module includes:

[0026] A grouping submodule, used for grouping the to-be-processed files according to the number of files and the upper limit of the handle to obtain a plurality of file groups;

[0027] The allocation submodule is used to allocate each file group to different servers respectively.

[0028] Optionally, the remaining handle number calculation module includes:

[0029] A submodule for obtaining the number of used handles, used to obtain the number of used file handles of each of the servers;

[0030] The remaining handle number calculation submodule is used to calculate the difference between the handle upper limit and the number of used file handles to obtain the remaining handle number of each server.

[0031] Optionally, the maximum process number calculation module includes:

[0032] The submodule for obtaining the number of required handles is used to obtain the number of required handles of each process;

[0033] The maximum process number calculation submodule is used to calculate the maximum process number of each server according to the remaining handle number and the required handle number.

[0034] The present invention also provides an electronic device, the device comprising a processor and a memory:

[0035] The memory is used to store program code and transmit the program code to the processor;

[0036] The processor is used to execute the multi-file data processing method as described above according to the instructions in the program code.

[0037] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes, and the program codes are used to execute the multi-file data processing method as described in any one of the above items.

[0038] It can be seen from the above technical scheme that the present invention has the following advantages: the present invention provides a multi-file data processing method, and specifically discloses: obtaining the number of files to be processed and the preset handle upper limit; dividing the files to be processed to each server according to the number of files and the handle upper limit; calculating the number of remaining handles of each server; calculating the maximum number of processes of each server according to the number of remaining handles; starting the process according to the maximum number of processes, and using the started process to process the files to be processed. Before performing data processing, the present invention first determines the number of files, and then determines the system resources. Through configuration and setting, it is clear to start different cluster servers under different numbers; then, through the file handle call of multiple servers, the concurrent processing of multiple files and the full utilization of resources are realized, thereby improving the processing efficiency and resource utilization of multiple files. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A flowchart of a method for processing data of multiple files provided by an embodiment of the present invention;

[0041] Figure 2A flowchart of a method for processing data of multiple files provided by another embodiment of the present invention;

[0042] Figure 3 A structural block diagram of a multi-file data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The embodiments of the present invention provide a method, device, electronic device and storage medium for processing data of multiple files, which are used to solve the technical problems of low efficiency and low resource utilization in the existing single program mode for processing multiple files.

[0044] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] See also Figure 1 , Figure 1 A flowchart of a method for processing data of multiple files provided by an embodiment of the present invention.

[0046] The present invention provides a method for processing data of multiple files, which may specifically include the following steps:

[0047] Step 101, obtaining the number of files to be processed and a preset handle upper limit;

[0048] A handle is an identifier used to identify an object or project. It can be used to describe a form, file, etc. It is essentially a special smart pointer used to reference memory blocks or objects managed by other systems (such as databases and operating systems) in programming. The establishment of handles stems from the problem of memory management mechanism, especially the use of virtual addresses. The system uses handles to record changes in data addresses.

[0049] File handle (HFILE): Used to identify files and file streams, such as a file handle opened by the CreateFile function.

[0050] In the embodiment of the present invention, the number of files to be processed may be first obtained, and a handle upper limit may be configured to determine the file processing upper limit of the server and whether the files to be processed exceed the processing capacity of the server.

[0051] Step 102, dividing the files to be processed to each server according to the number of files and the upper limit of the handle;

[0052] In the embodiment of the present invention, when the number of files to be processed exceeds the upper limit of the handle, the files to be processed exceeding the upper limit of the handle may be divided into other processors of the server cluster for processing.

[0053] A server cluster is a group of servers that are managed together and participate in workload management. A server cluster connects two or more servers through a high-speed network, allowing these servers to work together to provide users with services such as applications, system resources, and data. When a client interacts with a server cluster, from the client's perspective, the server cluster is equivalent to having only one server.

[0054] Step 103, calculating the number of remaining handles of each server;

[0055] Step 104, calculating the maximum number of processes of each server according to the number of remaining handles;

[0056] A process is a running activity of a program in a computer on a certain data set. It is the basic unit for resource allocation in the system and the foundation of the operating system structure. In the early computer structure designed for processes, processes are the basic execution entities of programs; in the contemporary computer structure designed for threads, processes are the containers of threads. A program is a description of instructions, data and their organizational forms, and a process is the entity of a program.

[0057] After obtaining the allocated files to be processed, the server can calculate the number of remaining handles of each, so as to determine the maximum number of processes that can be started simultaneously according to the number of remaining handles.

[0058] In the embodiment of the present invention, a process is a program, and the number of handles used each time is calculated in the program. Therefore, the maximum number of processes that can be started can be calculated according to the number of remaining handles of the server.

[0059] Step 105: Start a process according to the maximum number of processes, and use the started process to process the file to be processed.

[0060] After the maximum number of processes on each server is determined, a corresponding number of processes can be started according to the maximum number of processes to concurrently process different files to be processed.

[0061] Before performing data processing, the present invention first determines the number of files, and then determines the system resources. Through configuration and setting, it is clear to start different cluster servers under different numbers; then, through the file handle call of multiple servers, the concurrent processing of multiple files and the full utilization of resources are realized, thereby improving the processing efficiency of multiple files and the resource utilization rate.

[0062] See also Figure 2 , Figure 2A flowchart of a method for processing data of multiple files provided in another embodiment of the present invention. Specifically, the following steps may be included:

[0063] Step 201, obtaining the number of files to be processed and a preset handle upper limit;

[0064] In the embodiment of the present invention, the number of files to be processed may be firstly obtained, and a handle upper limit may be configured to determine the file processing upper limit of the server and whether the files to be processed exceed the processing capacity of the server.

[0065] In one example, the handle limit may be the file processing limit of the server, or may be an upper limit determined according to the object actually performing data processing. For example, when an encryption machine needs to be called to perform a file decryption operation, the handle upper limit may be the file handle upper limit of the encryption machine. The embodiment of the present invention does not impose any specific restrictions on the handle upper limit.

[0066] Step 202, grouping the files to be processed according to the number of files and the upper limit of the handle to obtain a plurality of file groups;

[0067] Step 203, respectively assigning each file group to different servers;

[0068] In the embodiment of the present invention, the handle upper limit may be a preset fixed upper limit data, which may be used as a basis for splitting files to achieve grouping of the files to be processed and distributing them to different servers. Each server processes a group of data.

[0069] Step 204, calculating the number of remaining handles of each server;

[0070] After obtaining the allocated files to be processed, the server can calculate the number of remaining handles for each, so as to determine the maximum number of processes that can be started simultaneously based on the number of remaining handles.

[0071] In one example, the step of calculating the number of remaining handles of each server may include the following sub-steps:

[0072] S41, obtaining the number of used file handles of each server;

[0073] S42, calculating the difference between the handle upper limit and the number of used file handles to obtain the number of remaining handles of each server.

[0074] In the embodiment of the present invention, the number of used file handles of the server can be determined according to the number of API calls to open the file.

[0075] By obtaining the number of used file handles, you can combine the handle upper limit to calculate the number of remaining handles on each server.

[0076] Step 205, calculating the maximum number of processes of each server according to the number of remaining handles;

[0077] In the embodiment of the present invention, step 205 may include the following sub-steps:

[0078] S51, obtaining the number of required handles of each process;

[0079] S52, calculating the maximum number of processes of each server according to the number of remaining handles and the number of required handles.

[0080] In the specific implementation, the number of handles required by different processes is different, and the number of remaining handles on different servers is also different. Based on the number of remaining handles on each server and the number of handles required by the processes on each server, the maximum number of processes that can be executed concurrently on each server can be calculated. By running the maximum number of processes, the server resources can be fully utilized.

[0081] Step 206: Start a process according to the maximum number of processes, and use the started process to process the file to be processed.

[0082] In the embodiment of the present invention, after the maximum number of processes on each server is determined, a corresponding number of processes can be started according to the maximum number of processes to concurrently process different to-be-processed files.

[0083] Before performing data processing, the present invention first determines the number of files, and then determines the system resources. Through configuration and setting, it is clear to start different cluster servers under different numbers; then, through the file handle call of multiple servers, the concurrent processing of multiple files and the full utilization of resources are realized, thereby improving the processing efficiency of multiple files and the resource utilization rate.

[0084] See also Figure 3 , Figure 3 A structural block diagram of a multi-file data processing device provided by an embodiment of the present invention.

[0085] An embodiment of the present invention provides a multi-file data processing device, comprising:

[0086] The file quantity and handle upper limit acquisition module 301 is used to acquire the file quantity of the files to be processed and the preset handle upper limit;

[0087] A partitioning module 302 is used to partition the files to be processed to each server according to the number of files and the upper limit of the handle;

[0088] The remaining handle number calculation module 303 is used to calculate the remaining handle number of each server;

[0089] The maximum process number calculation module 304 is used to calculate the maximum process number of each server according to the number of remaining handles;

[0090] The file processing module 305 is used to start a process according to the maximum number of processes and use the started process to process the file to be processed.

[0091] In the embodiment of the present invention, the division module 302 includes:

[0092] The grouping submodule is used to group the files to be processed according to the number of files and the upper limit of the handle to obtain multiple file groups;

[0093] The allocation submodule is used to allocate each file group to different servers respectively.

[0094] In the embodiment of the present invention, the remaining handle number calculation module 303 includes:

[0095] The submodule for obtaining the number of used handles is used to obtain the number of used file handles of each server;

[0096] The remaining handle number calculation submodule is used to calculate the difference between the handle upper limit and the number of used file handles to obtain the remaining handle number of each server.

[0097] In the embodiment of the present invention, the maximum process number calculation module 304 includes:

[0098] The submodule for obtaining the number of required handles is used to obtain the number of required handles of each process;

[0099] The maximum process number calculation submodule is used to calculate the maximum number of processes of each server based on the number of remaining handles and the number of required handles.

[0100] An embodiment of the present invention further provides an electronic device, the device comprising a processor and a memory:

[0101] The memory is used to store the program code and transmit the program code to the processor;

[0102] The processor is used to execute the multi-file data processing method of the embodiment of the present invention according to the instructions in the program code.

[0103] The embodiment of the present invention further provides a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the multi-file data processing method of the embodiment of the present invention.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0106] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0107] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0111] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0112] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing data of multiple files. It is characterized in that include: Get the number of files to be processed and the preset handle limit; Dividing the to-be-processed files onto each server according to the number of files and the handle upper limit; Calculating the number of remaining handles of each of the servers; Calculate the maximum number of processes of each server according to the number of remaining handles; A process is started according to the maximum number of processes, and the to-be-processed file is processed by using the started process.

2. The method according to claim 1, It is characterized in that The step of dividing the to-be-processed files onto each server according to the number of files and the handle upper limit includes: Grouping the to-be-processed files according to the number of files and the handle upper limit to obtain a plurality of file groups; Distribute each file group to different servers.

3. The method according to claim 1, It is characterized in that The step of calculating the number of remaining handles of each server comprises: Obtain the number of used file handles of each of the servers; The difference between the handle upper limit and the number of used file handles is calculated to obtain the number of remaining handles of each of the servers.

4. The method according to claim 1, It is characterized in that The step of calculating the maximum number of processes of each server according to the number of remaining handles includes: Get the number of required handles for each process; The maximum number of processes of each server is calculated according to the number of remaining handles and the number of required handles.

5. A data processing device for multiple files, It is characterized in that include: A file quantity and handle upper limit acquisition module is used to obtain the file quantity of the files to be processed and the preset handle upper limit; A partitioning module, used for partitioning the to-be-processed files onto each server according to the number of files and the handle upper limit; A remaining handle number calculation module, used to calculate the remaining handle number of each server; A maximum process number calculation module, used to calculate the maximum process number of each server according to the remaining handle number; The file processing module is used to start a process according to the maximum number of processes, and use the started process to process the to-be-processed file.

6. The device according to claim 5, It is characterized in that The division module comprises: A grouping submodule, used for grouping the to-be-processed files according to the number of files and the upper limit of the handle to obtain a plurality of file groups; The allocation submodule is used to allocate each file group to different servers respectively.

7. The device according to claim 5, It is characterized in that The remaining handle number calculation module includes: A submodule for obtaining the number of used handles, used to obtain the number of used file handles of each of the servers; The remaining handle number calculation submodule is used to calculate the difference between the handle upper limit and the number of used file handles to obtain the remaining handle number of each server.

8. The device according to claim 5, It is characterized in that The maximum process number calculation module includes: The submodule for obtaining the number of required handles is used to obtain the number of required handles of each process; The maximum process number calculation submodule is used to calculate the maximum process number of each server according to the remaining handle number and the required handle number.

9. An electronic device, It is characterized in that The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the multi-file data processing method according to any one of claims 1 to 4 according to the instructions in the program code.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium is used to store program codes, and the program codes are used to execute the multi-file data processing method according to any one of claims 1 to 4.

Images