File processing method, apparatus, device, medium, and computer program product

Abstract

The application relates to the technical field of big data resource scheduling, and provides a file processing method and device, computer equipment, a storage medium and a computer program product. The application can improve the resource utilization rate of each subnode in file data processing, thereby improving the processing efficiency of a large number of files in the whole file processing system. The method comprises the following steps: obtaining a to-be-processed file set corresponding to a file processing task through an idle subnode, loading file data in the to-be-processed file set into a cache pool through a data reading operation, and obtaining the file data in the cache pool through a data processing operation to obtain a file processing result; wherein the proportion of the process number corresponding to the data reading operation and the data processing operation of the idle subnode is balanced according to the relative size of the processing speed corresponding to the data reading operation and the data processing operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data resource scheduling, and in particular to a file processing method and device, computer equipment, a storage medium and a computer program product. BACKGROUND

[0002] With the continuous development of information technology, the amount of data generated and processed by business platform systems on a daily basis has increased substantially. From the perspective of storage cost and access efficiency, such files are usually stored in the form of sizes of about 10M or less. Efficient processing of these files is of great significance to data sedimentation and information mining of business platform systems.

[0003] However, in the current technology, big data processing frameworks such as HADOOP and HIVE are inefficient in processing a large number of the above-mentioned files, because the file system HDFS is not designed for efficient processing of a large number of the above-mentioned files. SUMMARY

[0004] Therefore, it is necessary to provide a file processing method, device, computer equipment, storage medium and computer program product to solve the above technical problems.

[0005] In one embodiment, a file processing method is provided, which comprises:

[0006] allocating, by a master node, a file processing task to an idle node in at least one node;

[0007] obtaining, by the idle node, a set of files to be processed corresponding to the file processing task, loading file data in the set of files to be processed to a cache pool by using a data reading operation, and obtaining the file data in the cache pool by using a data processing operation to process and obtain a file processing result;

[0008] The ratio of the number of processes corresponding to the data reading operation and the data processing operation is balanced according to the relative size of the processing speed corresponding to the data reading operation and the data processing operation.

[0009] In one embodiment, the master node and at least one node are mounted with a first shared storage and a second shared storage; the first shared storage is used to store the set of files to be processed; and the second shared storage is used to store the file processing result.

[0010] The file processing task is allocated by the master node to the idle node in the form of a file list.

[0011] The obtaining, by the idle sub-node, of the set of to-be-processed files corresponding to the file processing task comprises: obtaining, by the idle sub-node, the set of to-be-processed files from the first shared storage according to the file list sent by the master node.

[0012] The method further comprises: storing, by the idle sub-node, the file processing result to the second shared storage.

[0013] In one embodiment, the method further comprises: in a case where the task allocation for the idle sub-node is a first-time allocation, configuring, by the master node, the task amount corresponding to the file processing task as a preset amount; and in a case where the task allocation for the idle sub-node is a non-first-time allocation, determining, by the master node, the task amount corresponding to the file processing task according to the processing time consumption of the idle sub-node for a historical file processing task.

[0014] In one embodiment, the process corresponding to the data reading operation comprises a production process, the process corresponding to the data processing operation comprises a consumption process, the cache pool comprises a plurality of cache pool sub-areas, the number of the cache pool sub-areas corresponds to the number of the production processes, and each cache pool sub-area is respectively associated with one production process.

[0015] The loading, by the idle sub-node, of the file data in the set of to-be-processed files to the cache pool comprises: allocating, by the idle sub-node using an allocation process, the files in the set of to-be-processed files to the production processes, and reading and loading, by the production processes, the file data in the files respectively allocated to the production processes to the cache pool sub-areas associated with the production processes in a block manner, and reading and processing, by the consumption process, the file data in the cache pool sub-areas in a row manner.

[0016] In one embodiment, the head of the cache pool sub-area stores a head address pointer and a tail address pointer, the head address pointer is used to indicate the head address of the stored file data in the cache pool sub-area, and the tail address pointer is used to indicate the tail address of the stored file data in the cache pool sub-area.

[0017] The loading, by the idle sub-node, of the file data to the cache pool sub-area associated with the production process comprises: writing, by the idle sub-node using the production process, the file data to the tail of the stored file data in the cache pool sub-area according to the tail address pointer, and updating the tail address pointer.

[0018] The method further comprises: if the available writing space of the cache pool sub-area is insufficient when the idle sub-node using the production process writes the file data to the tail of the stored file data in the cache pool sub-area, suspending, by the idle sub-node, the production process from writing the file data to the cache pool sub-area.

[0019] In one embodiment, the reading file data in each cache pool sub-area includes: reading file data from a head of stored file data in the cache pool sub-area by the free sub-node using the consumer process according to the head address pointer, and updating the head address pointer;

[0020] The method further includes: after the head address pointer is updated, if the head address indicated by the updated head address pointer is the same as the tail address indicated by the tail address pointer, resetting the head address pointer and the tail address pointer by the free sub-node using the consumer process;

[0021] The method further includes: when the free sub-node reads file data from the cache pool sub-area using the consumer process, if the stored file data in the cache pool sub-area is less than a preset number of lines, moving the stored file data to a head of the cache pool sub-area by the free sub-node using the consumer process, and updating the head address pointer and the tail address pointer.

[0022] In one embodiment, the cache pool includes a state announcement area for indicating a state of each cache pool sub-area; the method further includes: obtaining an access request of the production process to the cache pool by the free sub-node using an announcement process for maintaining the state announcement area, and allowing the production process to access an associated cache pool sub-area if a state of the associated cache pool sub-area is in an idle state; obtaining an access request of the consumer process to the cache pool by the free sub-node using the announcement process, and allowing the consumer process to access a cache pool sub-area in an idle state if the cache pool has the cache pool sub-area in the idle state.

[0023] In one embodiment, the method further includes: setting a first access time length for the production process by the free sub-node using the announcement process when the production process is allowed to access the associated cache pool sub-area, and setting a second access time length for the consumer process by the free sub-node using the announcement process when the consumer process is allowed to access the cache pool sub-area in the idle state.

[0024] In one embodiment, a file processing device is further provided, and the device includes:

[0025] a task allocation module, configured to allocate a file processing task to a free sub-node in at least one sub-node by a master node;

[0026] The file processing module is configured to acquire a set of to-be-processed files corresponding to the file processing task through the idle sub-node, load file data in the set of to-be-processed files into a cache pool by using a data reading operation, and obtain the file data in the cache pool by using a data processing operation to process and obtain a file processing result.

[0027] The ratio of the number of processes corresponding to the data reading operation and the data processing operation is balanced according to the relative size of the processing speed corresponding to the data reading operation and the data processing operation.

[0028] In an embodiment, the master node and the at least one sub-node are mounted with a first shared storage and a second shared storage; the first shared storage is configured to store the set of to-be-processed files; the second shared storage is configured to store the file processing result; the task allocation module is further configured to allocate the file processing task to the idle sub-node in the form of a file list by using the master node; the file processing module is further configured to acquire the set of to-be-processed files from the first shared storage according to the file list sent by the master node by using the idle sub-node; and the file processing module is further configured to store the file processing result into the second shared storage by using the idle sub-node.

[0029] In an embodiment, the apparatus further includes a task amount configuration module configured to, in a case where the task allocation to the idle sub-node is a first allocation, configure a task amount corresponding to the file processing task as a preset amount by using the master node; and in a case where the task allocation to the idle sub-node is a non-first allocation, determine the task amount corresponding to the file processing task according to a processing time consumption of the idle sub-node for a historical file processing task by using the master node.

[0030] In an embodiment, the process corresponding to the data reading operation includes a production process; the process corresponding to the data processing operation includes a consumption process; the cache pool includes a plurality of cache pool sub-areas, the number of the cache pool sub-areas corresponds to the number of the production processes, each cache pool sub-area is associated with one production process; and the file processing module is further configured to allocate files in the set of to-be-processed files to each production process by using an allocation process by using the idle sub-node, read file data in each allocated file in a block manner and load the file data into a cache pool sub-area associated with the production process by using each production process, and read file data in each cache pool sub-area in a row manner and process the file data by using the consumption process.

[0031] In one embodiment, the head of the cache pool sub-area stores a head address pointer and a tail address pointer; the head address pointer is used to indicate a head address of the stored file data in the cache pool sub-area, and the tail address pointer is used to indicate a tail address of the stored file data in the cache pool sub-area; the file processing module is further configured to write file data to the tail of the stored file data in the cache pool sub-area by the idle sub-node using the production process according to the tail address pointer, and update the tail address pointer; the apparatus further comprises a write suspension module configured to suspend the production process from writing file data to the cache pool sub-area by the idle sub-node if the available write space of the cache pool sub-area is insufficient when the idle sub-node writes file data to the tail of the stored file data in the cache pool sub-area using the production process.

[0032] In one embodiment, the file processing module is further configured to read file data from the head of the stored file data in the cache pool sub-area by the idle sub-node using the consumption process according to the head address pointer, and update the head address pointer.

[0033] The apparatus further comprises a pointer resetting module configured to reset the head address pointer and the tail address pointer by the idle sub-node using the consumption process if the head address indicated by the updated head address pointer is the same as the tail address indicated by the tail address pointer after the head address pointer is updated.

[0034] The apparatus further comprises a data moving module configured to move the stored file data to the head of the cache pool sub-area and update the head address pointer and the tail address pointer by the idle sub-node using the consumption process if the stored file data in the cache pool sub-area is less than a preset number of lines when the idle sub-node reads file data from the cache pool sub-area using the consumption process.

[0035] In one embodiment, the cache pool comprises a state announcement area used to indicate the state of each cache pool sub-area; the apparatus further comprises a production process access processing module configured to obtain an access request of the production process to the cache pool by the idle sub-node using an announcement process used to maintain the state announcement area, and allow the production process to access the associated cache pool sub-area if the state of the associated cache pool sub-area of the production process is in an idle state; and a consumption process access processing module configured to obtain an access request of the consumption process to the cache pool by the idle sub-node using the announcement process, and allow the consumption process to access the cache pool sub-area in an idle state if the cache pool has a cache pool sub-area in an idle state.

[0036] In one embodiment, the apparatus further comprises an access duration setting module configured to set, by the idle child node, a first access duration for the producer process when the idle child node allows the producer process to access the associated cache pool sub-area using the announcement process, and set, by the idle child node, a second access duration for the consumer process when the idle child node allows the consumer process to access the cache pool sub-area in the idle state using the announcement process.

[0037] In one embodiment, a computer device is also provided. The computer device comprises a memory and a processor. The memory stores a computer program. The processor implements the following steps when executing the computer program: allocating, by a master node, a file processing task to an idle child node in at least one child node; obtaining, by the idle child node, a set of to-be-processed files corresponding to the file processing task, loading file data in the set of to-be-processed files into a cache pool using a data reading operation, and obtaining the file data in the cache pool using a data processing operation to process and obtain a file processing result; wherein a ratio of a number of processes corresponding to the data reading operation and the data processing operation is balanced according to a relative size of processing speeds corresponding to the data reading operation and the data processing operation.

[0038] In one embodiment, a computer readable storage medium is also provided. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the following steps: allocating, by a master node, a file processing task to an idle child node in at least one child node; obtaining, by the idle child node, a set of to-be-processed files corresponding to the file processing task, loading file data in the set of to-be-processed files into a cache pool using a data reading operation, and obtaining the file data in the cache pool using a data processing operation to process and obtain a file processing result; wherein a ratio of a number of processes corresponding to the data reading operation and the data processing operation is balanced according to a relative size of processing speeds corresponding to the data reading operation and the data processing operation.

[0039] In one embodiment, a computer program product is also provided. The computer program product comprises a computer program. The computer program is executed by a processor to implement the following steps: allocating, by a master node, a file processing task to an idle child node in at least one child node; obtaining, by the idle child node, a set of to-be-processed files corresponding to the file processing task, loading file data in the set of to-be-processed files into a cache pool using a data reading operation, and obtaining the file data in the cache pool using a data processing operation to process and obtain a file processing result; wherein a ratio of a number of processes corresponding to the data reading operation and the data processing operation is balanced according to a relative size of processing speeds corresponding to the data reading operation and the data processing operation.

[0040] The file processing method, device, computer device, storage medium and computer program product, by the master node, allocate the file processing task to the idle sub-node in the at least one sub-node, by the idle sub-node, obtain the set of files to be processed corresponding to the file processing task and load the file data in the set of files to be processed to the cache pool by using the data reading operation and obtain the file data in the cache pool by using the data processing operation to obtain the file processing result; wherein the proportion of the process number corresponding to the data reading operation and the data processing operation of the idle sub-node is balanced according to the relative size of the processing speed corresponding to the data reading operation and the data processing operation. The scheme can be responsible for the file processing task of the system in the sub-node by the master node and manage the sub-node, separate the data reading operation and the data processing operation on the sub-node by setting up the cache pool, and appropriately match the process number corresponding to each operation according to the relative size of the processing speed of the two types of operations, solve the problem of processing speed difference existing in parallel execution of the two types of operations, and balance between the data reading operation and the data processing operation, thereby achieving the technical effect of improving the resource utilization rate of each sub-node in file data processing, and further improving the processing efficiency of the entire file processing system for a large number of files. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The application environment diagram of the file processing method in one embodiment;

[0042] Figure 2 The flowchart of the file processing method in one embodiment;

[0043] Figure 3 The file data access schematic diagram based on the cache pool in one embodiment;

[0044] Figure 4 The structure schematic diagram of the cache pool in one embodiment;

[0045] Figure 5 The structure block diagram of the file processing device in one embodiment;

[0046] Figure 6 The internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0048] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0049] The file processing method provided by the present application can be applied to an application environment as shown in Figure 1 The application environment can be referred to as a file processing system, which can be applied to a cloud server to efficiently process a large number of files (small files) stored in the form of about 10M or less. Specifically, as shown in Figure 1 According to the required file processing scale of the actual scene, a virtual machine node (hereinafter referred to as a node) can be dynamically segmented from the resource pool of the cloud server, and a file processing system in the form of a Linux processing cluster is constructed by the nodes. The file processing system includes a master node and at least one sub-node, and solves the speed difference problem between data reading operation (IO intensive operation) and data processing operation (CPU intensive operation) on the sub-node by setting up a cache pool and concurrent control, so as to balance between data reading operation and data processing operation, improve the utilization rate of single node resources (IO, CPU), and further improve the processing capacity of the entire file processing system for a large number of small files.

[0050] The file processing method provided by the present application is described below based on the application environment as shown in Figure 1 and in combination with each embodiment and the corresponding drawings.

[0051] In one embodiment, as shown in Figure 2 a file processing method is provided, which can include the following steps:

[0052] Step S201, distributing a file processing task to an idle sub-node in at least one sub-node by a master node;

[0053] Step S202, obtaining a set of files to be processed corresponding to the file processing task by the idle sub-node, loading file data in the set of files to be processed to a cache pool by a data reading operation, and obtaining the file data in the cache pool by a data processing operation to process and obtain a file processing result; wherein the ratio of the number of processes corresponding to the data reading operation and the data processing operation is balanced according to the relative size of the processing speed corresponding to the data reading operation and the data processing operation.

[0054] Specifically, in the present embodiment, in combination with Figure 1The file processing system can be constructed based on a resource pool of a cloud server. The file processing system includes at least two nodes. The at least two nodes can include a master node that can be used to allocate file processing tasks and manage the nodes in the file processing system, and at least one sub-node that can be used to process files. The number of sub-nodes can be configured according to the actual file processing scale required by the scene. In some embodiments, for the master node and the sub-nodes in the file processing system, two shared storages, including a first shared storage and a second shared storage, can be mounted simultaneously. The first shared storage can be used to store a set of files to be processed, that is, the first shared storage can store a set of files to be processed by the file processing system, and the file processing results obtained after the files are processed by the sub-nodes are stored by the second shared storage. Each node in the file processing system can access the first shared storage and the second shared storage.

[0055] Specifically, for the master node of the file processing system, on the one hand, it can be used to manage all nodes of the entire file processing system. Specifically, a list of sub-nodes can be maintained to complete the management. When the system needs to adjust the sub-nodes by using the elasticity of the cloud server during the actual processing of the files, the involved sub-nodes can actively notify the master node, and the master node can update the list of sub-nodes synchronously. On the other hand, the master node of the file processing system is also responsible for allocating file processing tasks. The file processing tasks can be allocated to an idle sub-node of at least one sub-node of the file processing system through the master node, that is, the file processing tasks are allocated to one or more idle sub-nodes in the system through the master node for processing.

[0056] For the file processing task, in combination with the first shared storage and the second shared storage, in some embodiments, the step S201 of assigning the file processing task to the idle sub-node by the master node can include: assigning the file processing task in the form of a file list to the idle sub-node by the master node, so as to assign the set of files to be processed on the first shared storage to one or more idle sub-nodes in the system for processing. Correspondingly, the step S202 of obtaining the set of files to be processed corresponding to the file processing task by the idle sub-node can include: obtaining the corresponding set of files to be processed from the first shared storage according to the file list after the idle sub-node receives the file list sent by the master node, wherein the file list can be generated and maintained by the master node. In this embodiment, after the idle sub-node processes the set of files to be processed to obtain the corresponding file processing result, the above method can further include the following steps: storing the file processing result to the second shared storage by the idle sub-node. In addition, after processing the set of files to be processed, the idle sub-node can also notify the master node that the file processing is completed, so as to inform the master node that the next set of files to be processed can be assigned to the idle sub-node for processing, thereby improving the file processing efficiency.

[0057] For the file processing task, further, the master node can configure the task amount for the idle sub-node according to the task assignment and processing of the idle sub-node. In this regard, in some embodiments, the above method further includes: in the case of first assignment of the task to the idle sub-node, configuring the task amount corresponding to the file processing task as a preset amount by the master node; in the case of non-first assignment of the task to the idle sub-node, determining the task amount corresponding to the file processing task according to the processing time of the idle sub-node for the historical file processing task by the master node. In actual application, the task amount corresponding to the file processing task can be measured by the number of files corresponding to the file processing task. In the case of first assignment of the file processing task to the idle sub-node by the master node, the idle sub-node can be assigned a set of files to be processed containing a preset amount (such as 100,000) of files by the master node, and in the case of non-first assignment of the file processing task to the idle sub-node by the master node, the task amount corresponding to the file processing task can be determined by the master node according to the processing time of the idle sub-node for the historical file processing task. Specifically, the processing time of each file processing task of each idle sub-node can be registered by the master node, so that when assigning the task amount to the idle sub-node, the task amount of the file processing task can be appropriately increased or decreased by the master node according to the processing time of the idle sub-node for the last file processing task (historical file processing task) based on the task amount corresponding to the last file processing task, so as to reasonably assign the file processing task amount to the idle sub-node.

[0058] For the step S202, specifically, after the file processing task is assigned to the idle sub-node by the master node, the idle sub-node can further obtain the set of to-be-processed files corresponding to the file processing task, load the file data in the set of to-be-processed files into the cache pool by using the data reading operation (IO-intensive operation), and obtain the file data in the cache pool by using the data processing operation (CPU-intensive operation) to process and obtain the file processing result. Specifically, the process of processing the corresponding set of to-be-processed files by the idle sub-node after obtaining the file processing task from the master node mainly includes reading the file data in each file in the set of to-be-processed files by the idle sub-node and processing the file data into the corresponding file processing result by the idle sub-node. The reading of the file and the processing mainly include two parts of the data reading operation (IO-intensive operation) and the data processing operation (CPU-intensive operation). The processing speed of the data reading operation and the data processing operation is different. The data reading operation is a slow operation relative to the data processing operation. If the two operations are executed in series, the data processing operation will be in an idle state most of the time, and the resources of the data processing operation cannot be fully utilized. In order to make the resources of the data processing operation and the data reading operation more fully and evenly utilized, a cache pool is set up in the sub-node to isolate the data reading operation (IO-intensive operation) and the data processing operation (CPU-intensive operation), that is, the data reading operation (IO-intensive operation) is responsible for loading the file data in the set of to-be-processed files into the cache pool, and the data processing operation (CPU-intensive operation) obtains the file data from the cache pool for processing. The two types of operations are executed in parallel in the sub-node. Since the data reading operation is a slow operation relative to the data processing operation, and the data processing operation is a fast operation relative to the data reading operation, the ratio of the number of processes corresponding to the data reading operation and the data processing operation needs to be balanced according to the relative size of the processing speed of the data reading operation and the data processing operation, so that the processing of the file data in the sub-node is more compact and efficient. For example, the ratio of the number of processes corresponding to the data processing operation and the data reading operation can be appropriately set to 1:2 according to the number of CPU cores, so that the whole processing process is more compact and efficient.

[0059] The file processing method can allocate and manage file processing tasks of a system in a sub-node by a master node, separate data reading operations and data processing operations on the sub-node by setting a cache pool, and appropriately match the respective process numbers of the two types of operations according to the relative size of the processing speed of the two types of operations, to solve the problem of the difference in processing speed of parallel execution of the two types of operations, so as to balance the data reading operations and the data processing operations, thereby achieving the technical effect of improving the resource utilization rate of each sub-node in file data processing, and further improving the processing efficiency of the entire file processing system for a large number of files.

[0060] For the cache pool of the sub-node, further, as shown in Figure 3 In one embodiment, the process corresponding to the data reading operation includes a production process, the process corresponding to the data processing operation includes a consumption process, the cache pool includes a plurality of cache pool sub-areas, the number of the cache pool sub-areas corresponds to the number of the production processes, and each cache pool sub-area is associated with a production process. Specifically, through the bridge of the cache pool, the processing processes in the sub-node can be divided into two groups. One group can be responsible for reading file data in a to-be-processed file and writing the file data into the cache pool, which is called a production process and corresponds to a data reading operation. The other group can be responsible for reading file data from the cache pool and processing the file data into corresponding file processing results according to requirements, which is called a consumption process and corresponds to a data processing operation. In actual application, to avoid additional time consumption caused by frequent creation and destruction of processes by the kernel, the production process and the consumption process can exist after being created until all file processing is completed.

[0061] In this embodiment, the cache pool is a public resource and can be accessed by the production process and the consumption process, but the access cannot be performed simultaneously, so the access mode of the cache pool by the production process and the consumption process needs to be configured to enable efficient and orderly file data processing based on the cache pool. As shown in Figure 4 A larger memory area can be created as the cache pool, the size of the cache pool can be set to 1Mx the number of production processes, and then the cache pool can be further divided into a plurality of sub-areas of a fixed size, which are called cache pool sub-areas (such as sub-area 1, sub-area 2, …, sub-area N-2, sub-area N-1, and sub-area N). The size of each sub-area is 1M and is fixedly associated with a production process, that is, each production process only writes file data into one cache pool sub-area in the cache pool, and the consumption process can access any cache pool sub-area in the cache pool, that is, as long as no production process writes file data into the cache pool sub-area, the consumption process can attempt to read file data from the cache pool sub-area.

[0062] In the embodiment, for the file data access of the production process and the consumption process in the cache pool, each idle sub-node faces a set of files to be processed, and the files in the set of files to be processed need to be evenly distributed to each production process of the idle sub-node. The distribution task can be completed by the parent process of the production process, which is called a distribution process. Thus, in some embodiments, the loading of the file data in the set of files to be processed to the cache pool by the data reading operation in step S202 specifically includes: distributing the files in the set of files to be processed to each production process by the idle sub-node using the distribution process, and reading the file data in the files distributed to each production process in a block manner and loading the file data to the cache pool sub-area associated with each production process by each production process, and reading the file data in each cache pool sub-area in a row manner by the consumption process for processing.

[0063] In the embodiment, specifically, the set of files to be processed is regarded as a whole, and the distribution process can distribute a fixed number (e.g., 10,000) of files from the set of files to be processed to one production process at a time. After the production process processes a batch of files, the production process can notify the distribution process, and then obtain the next batch of files to be processed from the distribution process until all the files in the set of files to be processed are processed. The production process can load the file data in the files distributed to the production process to the cache pool sub-area associated with the production process. After the distribution process distributes the files to the production process, the production process generally processes the files one by one according to the steps of opening the file, reading the file, and closing the file. The production process reads the file in a block manner. The file system is constructed on the basis of a disk block, and the disk block is the basic unit of each IO. Thus, when the production process reads the file, one system call reads an integer multiple of the size of the disk block from the file and loads the file data to the cache pool sub-area associated with the production process. The production process reads the file data in a loop until the end of the file. When each production process does not process a file, the production process cannot process the next file to avoid data disorder.

[0064] For the file data access of the production process and the consumption process in the cache pool, further, in some embodiments, as Figure 4As shown, the head of the cache pool sub-area stores a first address pointer and a tail address pointer, the first address pointer is used to indicate the first address of the stored file data in the cache pool sub-area, and the tail address pointer is used to indicate the tail address of the stored file data in the cache pool sub-area. Specifically, the head of each cache pool sub-area in the cache pool can store two pointers, one is the first address of the stored file data in the cache pool sub-area, recorded as s pointer (first address pointer), and the other is the tail address of the stored file data in the cache pool sub-area, recorded as e pointer (tail address pointer). Based on this, in some embodiments, the step of loading the respective associated cache pool sub-area in the above-mentioned embodiment specifically includes: using the production process to write file data to the tail of the stored file data in the cache pool sub-area according to the tail address pointer by the idle sub-node, and updating the tail address pointer, that is, the production process updates the e pointer address stored in the head of the cache pool sub-area after appending and writing file data to the tail of the stored file data and finishing writing the file data; and the step of reading the file data in each cache pool sub-area in the above-mentioned embodiment specifically includes: using the consumption process to read file data from the head of the stored file data in the cache pool sub-area according to the first address pointer by the idle sub-node, and updating the first address pointer, that is, the consumption process reads file data from the head of the stored file data, and also updates the s pointer address stored in the head of the cache pool sub-area after finishing reading the file data, so that the production process and the consumption process can efficiently and orderly access the file data in the cache pool.

[0065] Further, for reading file data from the cache pool sub-area by the consumption process using the idle sub-node, in some embodiments, the above-mentioned method can further include the following steps: after updating the first address pointer, if the first address indicated by the updated first address pointer is the same as the tail address indicated by the tail address pointer, resetting the first address pointer and the tail address pointer by the consumption process using the idle sub-node. Specifically, after updating the s pointer by the consumption process reading file data from the cache pool sub-area, if it is found that the address indicated by the s pointer is equal to the address indicated by the e pointer, the consumption process needs to reset the addresses indicated by the s pointer and the e pointer to the position of "the first address of the sub-area + the size of the head of the sub-area".

[0066] Further, for the idle child node loading file data into the cache pool sub-area by the production process, in some embodiments, the above method can further include the following steps: when the idle child node writes file data into the tail of the stored file data in the cache pool sub-area by the production process, if the available write space of the cache pool sub-area is insufficient, the idle child node suspends the production process to write file data into the cache pool sub-area. Specifically, when the production process loads file data into the tail of the stored file data in the cache pool sub-area, if the size of the file data to be loaded is greater than the available write space of the cache pool sub-area, the loading of the file data is suspended until the available write space of the cache pool sub-area is sufficient.

[0067] Therefore, in some embodiments, the above method can further include the following steps: when the idle child node reads file data from the cache pool sub-area by the consumption process, if the stored file data in the cache pool sub-area is less than a preset number of rows, the idle child node moves the stored file data to the beginning of the cache pool sub-area by the consumption process and updates the head pointer and the tail pointer. Specifically, the consumption process reads file data from the cache pool sub-area in a row manner, when the stored file data in the cache pool sub-area is less than a preset number of rows, such as one row, the idle child node moves the stored file data to the beginning of the cache pool sub-area by the consumption process and updates the s pointer address and the e pointer address of the cache pool sub-area, so that the production process associated with the cache pool sub-area can continue to load the remaining file data and avoid row disorder, and the s pointer address and the e pointer address corresponding to the head of the cache pool sub-area can be updated after the consumption process and the production process access the cache pool sub-area.

[0068] For the cache pool of the idle child node, such as Figure 4In some embodiments, the cache pool can further include a state announcement area for indicating the state of each cache pool sub-area. The state can include an idle state and a used state. Specifically, the state announcement area can store a binary sequence for maintaining the state (idle or used) of each cache pool sub-area, with a size of 1 bit x number of areas, rounded up to an integer multiple of 8, and each 1-bit value is "0" or "1", where "0" indicates that the state of the cache pool sub-area is idle, and "1" indicates that the state of the cache pool sub-area is used. The position of each 1-bit in the area is obtained by calculating the position of the cache pool sub-area it maintains, as follows: reading the binary sequence from left to right, the first bit identifies the starting address of the cache pool sub-area as: (cache pool starting address + state announcement area size) + (1-1) x 1M, the second bit identifies the starting address of the cache pool sub-area as: (cache pool starting address + state announcement area size) + (2-1) x 1M, and so on, and the Nth bit identifies the starting address of the cache pool sub-area as: (cache pool starting address + state announcement area size) + (N-1) x 1M. Further, the state announcement area of the cache pool can be maintained by an independent process in the idle sub-node, referred to as the announcement process, and the production process and the consumption process need to obtain access permission from the announcement process when accessing the cache pool.

[0069] For the access of the production process to the cache pool, the method can further include the following steps: obtaining, by the idle sub-node, the access request of the production process to the cache pool by using the announcement process for maintaining the state announcement area, and allowing the production process to access the cache pool sub-area associated with the production process if the state of the cache pool sub-area associated with the production process is in the idle state. For the access of the consumption process to the cache pool, the method can further include the following steps: obtaining, by the idle sub-node, the access request of the consumption process to the cache pool by using the announcement process, and allowing the consumption process to access the cache pool sub-area in the idle state if the cache pool has the cache pool sub-area in the idle state. Specifically, when the production process and the consumption process need to access the cache pool, the access permission is first obtained from the announcement process. The production process is fixedly associated with a cache pool sub-area. When the production process needs to load file data into the associated cache pool sub-area, the announcement process needs to be inquired whether the associated cache pool sub-area is in the idle state. If yes, the production process obtains the access permission and is allowed to access the cache pool sub-area. The consumption process can access all cache pool sub-areas in the cache pool. When a consumption process needs to read file data from the cache pool, the access permission of the cache pool sub-area in the idle state is also obtained from the announcement process. Before the consumption process and the production process access the cache pool or after the consumption process and the production process finish accessing the cache pool and release the resources, the announcement process is notified and the state of the corresponding cache pool sub-area in the state announcement area is updated by the announcement process. In some embodiments, a request queue can be arranged in the announcement process. According to the principle of first request first allocation of resources, the announcement process obtains the access request from the head of the request queue and allocates resources for the access request, and the new access request is placed at the tail of the request queue.

[0070] Further, for the access permission of the production process and the consumption process to the cache pool, in some embodiments, the method can further include the following steps: setting, by the idle sub-node, a first access time length for the production process when the production process is allowed to access the associated cache pool sub-area by using the announcement process, and setting, by the idle sub-node, a second access time length for the consumption process when the consumption process is allowed to access the cache pool sub-area in the idle state by using the announcement process. Specifically, the cache pool is a public resource. At the same time, only one process (the production process or the consumption process) is allowed to access each cache pool sub-area in the cache pool. In order to avoid that one process occupies one cache pool sub-area for too long, the announcement process can set the access time length of the production process or the consumption process to a time amount, such as 3 seconds for the first access time length and the second access time length. When the access time of the production process or the consumption process to the cache pool sub-area meets the corresponding access time length, the access of the production process or the consumption process is stopped, and the announcement process is informed that the access permission of the cache pool sub-area has been released.

[0071] It should be understood that although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0072] Based on the same inventive concept, the embodiments of the present application also provide a file processing device for implementing the above-mentioned file processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more file processing device embodiments provided below can refer to the limitations of the file processing method described above, which will not be repeated here.

[0073] In one embodiment, as shown in Figure 5 A file processing device is provided, which can include:

[0074] A task allocation module 501 is configured to allocate a file processing task to an idle sub-node in at least one sub-node by a master node;

[0075] A file processing module 502 is configured to obtain a set of files to be processed corresponding to the file processing task by the idle sub-node, load file data in the set of files to be processed to a cache pool by a data reading operation, and obtain file data in the cache pool by a data processing operation to process and obtain a file processing result.

[0076] The ratio of the number of processes corresponding to the data reading operation and the data processing operation is balanced according to the relative size of the processing speed corresponding to the data reading operation and the data processing operation.

[0077] In one embodiment, the master node and the at least one slave node mount a first shared storage and a second shared storage; the first shared storage is used to store the set of files to be processed; the second shared storage is used to store the file processing result; the task allocation module 501 is further configured to allocate, by the master node, the file processing task to the idle slave node in the form of a file list; the file processing module 502 is further configured to acquire, by the idle slave node, the set of files to be processed from the first shared storage according to the file list sent by the master node; and the file processing module 502 is further configured to store, by the idle slave node, the file processing result to the second shared storage.

[0078] In one embodiment, the device 500 further comprises a task quantity configuration module configured to, in a case where the task allocation to the idle slave node is a first-time allocation, configure, by the master node, a task quantity corresponding to the file processing task as a preset quantity; and in a case where the task allocation to the idle slave node is a non-first-time allocation, determine, by the master node, the task quantity corresponding to the file processing task according to a processing time consumption of the idle slave node for a historical file processing task.

[0079] In one embodiment, the process corresponding to the data reading operation comprises a production process; the process corresponding to the data processing operation comprises a consumption process; the cache pool comprises a plurality of cache pool sub-areas, the number of the cache pool sub-areas corresponding to the number of the production processes, each cache pool sub-area being associated with one production process; and the file processing module 502 is further configured to allocate, by the idle slave node, files in the set of files to be processed to the production processes by using the allocation process, read file data in the files allocated to each production process in a block manner by using the production process, load the file data to the cache pool sub-area associated with the production process, and read the file data in each cache pool sub-area in a row manner by using the consumption process to process and handle the file data.

[0080] In one embodiment, the head of each cache pool sub-area stores a head address pointer and a tail address pointer; the head address pointer is used to indicate a head address of the file data stored in the cache pool sub-area, and the tail address pointer is used to indicate a tail address of the file data stored in the cache pool sub-area; the file processing module 502 is further configured to write, by the idle slave node, file data to the tail of the file data stored in the cache pool sub-area according to the tail address pointer by using the production process, and update the tail address pointer; and the device 500 further comprises a write suspension module configured to, when the idle slave node writes file data to the tail of the file data stored in the cache pool sub-area by using the production process, suspend, by the idle slave node, the production process to write file data to the cache pool sub-area if the available write space of the cache pool sub-area is insufficient.

[0081] In one embodiment, the file processing module 502 is further configured to read file data from a head of the file data stored in the cache pool sub-area according to the head address pointer by the consumer process through the idle sub-node, and update the head address pointer;

[0082] The device 500 further comprises a pointer resetting module configured to reset the head address pointer and the tail address pointer by the consumer process through the idle sub-node if the head address indicated by the updated head address pointer is the same as the tail address indicated by the tail address pointer after the head address pointer is updated.

[0083] The device 500 further comprises a data moving module configured to move the stored file data to a head of the cache pool sub-area and update the head address pointer and the tail address pointer by the consumer process through the idle sub-node if the stored file data in the cache pool sub-area is less than a preset number of lines when the file data is read from the cache pool sub-area by the consumer process through the idle sub-node.

[0084] In one embodiment, the cache pool comprises a state announcement area for indicating states of the cache pool sub-areas; the device 500 further comprises a production process access processing module configured to acquire an access request of the production process to the cache pool through the announcement process for maintaining the state announcement area by the idle sub-node, and allow the production process to access an associated cache pool sub-area if a state of the associated cache pool sub-area is in an idle state; and a consumer process access processing module configured to acquire an access request of the consumer process to the cache pool through the announcement process by the idle sub-node, and allow the consumer process to access a cache pool sub-area in the idle state if the cache pool sub-area exists in the idle state.

[0085] In one embodiment, the device 500 further comprises an access duration setting module configured to set a first access duration for the production process when the production process is allowed to access the associated cache pool sub-area by the idle sub-node through the announcement process, and set a second access duration for the consumer process when the consumer process is allowed to access the cache pool sub-area in the idle state by the idle sub-node through the announcement process.

[0086] Each of the above modules can be realized by software, hardware, and a combination thereof in whole or in part. Each of the above modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each of the above modules.

[0087] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 6 The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data such as file processing tasks, a set of files to be processed, file processing results, etc. The network interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a file processing method.

[0088] Those skilled in the art can understand that Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0089] In one embodiment, a computer device is provided, which includes a memory and a processor, and the memory stores a computer program. The processor implements the following steps when executing the computer program:

[0090] allocating, by a master node, a file processing task to an idle node in at least one sub-node; obtaining, by the idle node, a set of files to be processed corresponding to the file processing task, and loading file data in the set of files to be processed to a cache pool by using a data reading operation, and obtaining file data in the cache pool by using a data processing operation to process and obtain a file processing result; wherein the ratio of the number of processes corresponding to the data reading operation and the data processing operation is balanced according to the relative size of the processing speed corresponding to the data reading operation and the data processing operation.

[0091] In one embodiment, the master node and at least one sub-node mount a first shared storage and a second shared storage; the first shared storage is configured to store the set of files to be processed; the second shared storage is configured to store the file processing result; and the processor further implements the following steps when executing the computer program: distributing, by the master node, the file processing task in the form of a file list to the idle node; obtaining, by the idle node, the set of files to be processed from the first shared storage according to the file list sent by the master node; and storing, by the idle node, the file processing result to the second shared storage.

[0092] In one embodiment, the processor, when executing the computer program, further implements the following steps: in a case that the task allocation for the idle sub-node is a first allocation, configuring, by the master node, a task amount corresponding to the file processing task as a preset amount; in a case that the task allocation for the idle sub-node is a non-first allocation, determining, by the master node, the task amount corresponding to the file processing task according to a processing time consumption of the idle sub-node for a historical file processing task.

[0093] In one embodiment, the process corresponding to the data reading operation comprises a production process; the process corresponding to the data processing operation comprises a consumption process; the cache pool comprises a plurality of cache pool sub-areas, the number of the cache pool sub-areas corresponding to the number of the production processes, each cache pool sub-area being associated with one production process; the processor, when executing the computer program, further implements the following steps: allocating, by the idle sub-node using an allocation process, files in the set of files to be processed to the production processes, and reading, by the production processes, file data in the files allocated to the production processes respectively in a block manner and loading the file data to the cache pool sub-areas associated with the production processes respectively, and reading, by the consumption process, file data in the cache pool sub-areas in a row manner for processing.

[0094] In one embodiment, the head of the cache pool sub-area stores a head address pointer and a tail address pointer; the head address pointer is used to indicate a head address of the stored file data in the cache pool sub-area, and the tail address pointer is used to indicate a tail address of the stored file data in the cache pool sub-area; the processor, when executing the computer program, further implements the following steps: writing, by the idle sub-node using the production process, file data to the tail of the stored file data in the cache pool sub-area according to the tail address pointer, and updating the tail address pointer; when the idle sub-node using the production process writes file data to the tail of the stored file data in the cache pool sub-area, if the available writing space of the cache pool sub-area is insufficient, suspending, by the idle sub-node, the production process from writing file data to the cache pool sub-area.

[0095] In one embodiment, the processor, when executing the computer program, further implements the following steps: reading, by the free sub-node, file data from the head of the stored file data in the cache pool sub-area according to the head address pointer by the consumption process, and updating the head address pointer; after the head address pointer is updated, if the head address indicated by the updated head address pointer is the same as the tail address indicated by the tail address pointer, resetting, by the free sub-node, the head address pointer and the tail address pointer by the consumption process; when reading, by the free sub-node, file data from the cache pool sub-area by the consumption process, if the stored file data in the cache pool sub-area is less than a preset number of lines, moving, by the free sub-node, the stored file data to the head of the cache pool sub-area by the consumption process, and updating the head address pointer and the tail address pointer.

[0096] In one embodiment, the cache pool comprises a state announcement area for indicating the state of each cache pool sub-area; the processor, when executing the computer program, further implements the following steps: obtaining, by the free sub-node, an access request of the production process to the cache pool by an announcement process for maintaining the state announcement area, and allowing the production process to access the associated cache pool sub-area if the state of the associated cache pool sub-area of the production process is in an idle state; obtaining, by the free sub-node, an access request of the consumption process to the cache pool by the announcement process, and allowing the consumption process to access the cache pool sub-area in the idle state if there is a cache pool sub-area in the idle state in the cache pool.

[0097] In one embodiment, the processor, when executing the computer program, further implements the following steps: setting, by the free sub-node, a first access time length for the production process by the announcement process when allowing the production process to access the associated cache pool sub-area, and setting, by the free sub-node, a second access time length for the consumption process by the announcement process when allowing the consumption process to access the cache pool sub-area in the idle state.

[0098] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program, when executed by a processor, implements the following steps:

[0099] allocating, by the master node, a file processing task to a free node among the at least one slave node; obtaining, by the free node, a set of files to be processed corresponding to the file processing task, and loading file data in the set of files to be processed into a cache pool by using a data reading operation, and obtaining the file data in the cache pool by using a data processing operation to process and obtain a file processing result; wherein a ratio of a number of processes corresponding to the data reading operation and the data processing operation is balanced according to a relative size of processing speeds corresponding to the data reading operation and the data processing operation.

[0100] In one embodiment, the master node and the at least one slave node mount a first shared storage and a second shared storage; the first shared storage is used to store the set of files to be processed; the second shared storage is used to store the file processing result; the computer program, when executed by the processor, further implements the following steps: distributing, by the master node, the file processing task in the form of a file list to the free node; obtaining, by the free node, the set of files to be processed from the first shared storage according to the file list sent by the master node; and storing, by the free node, the file processing result to the second shared storage.

[0101] In one embodiment, the computer program, when executed by the processor, further implements the following steps: in a case that the task allocation for the free node is a first allocation, configuring, by the master node, a task amount corresponding to the file processing task as a preset amount; in a case that the task allocation for the free node is a non-first allocation, determining, by the master node, the task amount corresponding to the file processing task according to a processing time consumption of the free node for a historical file processing task.

[0102] In one embodiment, the process corresponding to the data reading operation includes a production process; the process corresponding to the data processing operation includes a consumption process; the cache pool includes a plurality of cache pool sub-areas, a number of the cache pool sub-areas corresponding to a number of the production processes, each cache pool sub-area being associated with one production process; the computer program, when executed by the processor, further implements the following steps: distributing, by the free node, files in the set of files to be processed to each production process by using an allocation process, and reading file data in each distributed file in a block manner and loading the file data into a cache pool sub-area associated with each production process by using each production process, and reading file data in each cache pool sub-area in a row manner and processing the file data by using the consumption process.

[0103] In one embodiment, the head of the cache pool sub-area stores a head address pointer and a tail address pointer; the head address pointer is used to indicate the head address of the stored file data in the cache pool sub-area, and the tail address pointer is used to indicate the tail address of the stored file data in the cache pool sub-area; when the computer program is executed by the processor, the following steps are further implemented: the idle sub-node uses the production process to write file data to the tail of the stored file data in the cache pool sub-area according to the tail address pointer, and updates the tail address pointer; when the idle sub-node uses the production process to write file data to the tail of the stored file data in the cache pool sub-area, if the available writing space of the cache pool sub-area is insufficient, the idle sub-node suspends the production process to write file data to the cache pool sub-area.

[0104] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: the idle sub-node uses the consumption process to read file data from the head of the stored file data in the cache pool sub-area according to the head address pointer, and updates the head address pointer; after the head address pointer is updated, if the head address indicated by the updated head address pointer is the same as the tail address indicated by the tail address pointer, the idle sub-node uses the consumption process to reset the head address pointer and the tail address pointer; when the idle sub-node uses the consumption process to read file data from the cache pool sub-area, if the stored file data in the cache pool sub-area is less than a preset number of lines, the idle sub-node uses the consumption process to move the stored file data to the head of the cache pool sub-area, and updates the head address pointer and the tail address pointer.

[0105] In one embodiment, the cache pool includes a state announcement area used to indicate the state of each cache pool sub-area; when the computer program is executed by the processor, the following steps are further implemented: the idle sub-node uses an announcement process used to maintain the state announcement area to obtain an access request of the production process to the cache pool, and allows the production process to access the associated cache pool sub-area if the state of the associated cache pool sub-area is in an idle state; the idle sub-node uses the announcement process to obtain an access request of the consumption process to the cache pool, and allows the consumption process to access the cache pool sub-area in an idle state if the cache pool has a cache pool sub-area in an idle state.

[0106] In one embodiment, the computer program, when executed by the processor, further implements the following steps: setting, by the idle child node, a first access duration for the producer process when allowing the producer process to access the associated cache pool sub-area by using the announcement process, and setting, by the idle child node, a second access duration for the consumer process when allowing the consumer process to access the cache pool sub-area in the idle state by using the announcement process.

[0107] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the following steps:

[0108] allocating, by the master node, a file processing task to an idle child node in the at least one child node; obtaining, by the idle child node, a set of to-be-processed files corresponding to the file processing task, and loading file data in the set of to-be-processed files into a cache pool by using a data reading operation, and obtaining file data in the cache pool by using a data processing operation to process and obtain a file processing result; wherein the ratio of the number of processes corresponding to the data reading operation and the data processing operation is balanced according to the relative size of the processing speed corresponding to the data reading operation and the data processing operation.

[0109] In one embodiment, the master node and the at least one child node mount a first shared storage and a second shared storage; the first shared storage is used to store the set of to-be-processed files; the second shared storage is used to store the file processing result; the computer program, when executed by the processor, further implements the following steps: distributing, by the master node, the file processing task to the idle child node in the form of a file list; obtaining, by the idle child node, the set of to-be-processed files from the first shared storage according to the file list sent by the master node; storing, by the idle child node, the file processing result to the second shared storage.

[0110] In one embodiment, the computer program, when executed by the processor, further implements the following steps: in the case of first allocation of the task to the idle child node, configuring, by the master node, the task amount corresponding to the file processing task as a preset amount; in the case of non-first allocation of the task to the idle child node, determining, by the master node, the task amount corresponding to the file processing task according to the processing time consumption of the idle child node for the historical file processing task.

[0111] In one embodiment, the data reading operation corresponds to a production process; the data processing operation corresponds to a consumption process; the cache pool comprises a plurality of cache pool sub-areas, the number of the cache pool sub-areas corresponding to the number of the production processes, each cache pool sub-area being associated with a production process; the computer program, when executed by the processor, further implements the following steps: the idle sub-node allocates files in the set of files to be processed to the production processes by using the allocation process, and reads file data in the allocated files in a block manner by using the production processes and loads the file data into the associated cache pool sub-areas, and reads file data in the cache pool sub-areas in a row manner by using the consumption processes for processing.

[0112] In one embodiment, the head of the cache pool sub-area stores a head address pointer and a tail address pointer; the head address pointer is used to indicate a head address of the stored file data in the cache pool sub-area, and the tail address pointer is used to indicate a tail address of the stored file data in the cache pool sub-area; the computer program, when executed by the processor, further implements the following steps: the idle sub-node writes file data to the tail of the stored file data in the cache pool sub-area according to the tail address pointer by using the production process, and updates the tail address pointer; when the idle sub-node writes file data to the tail of the stored file data in the cache pool sub-area by using the production process, if the available writing space of the cache pool sub-area is insufficient, the idle sub-node suspends the production process from writing file data to the cache pool sub-area.

[0113] In one embodiment, the computer program, when executed by the processor, further implements the following steps: the idle sub-node reads file data from the head of the stored file data in the cache pool sub-area according to the head address pointer by using the consumption process, and updates the head address pointer; after the head address pointer is updated, if the head address indicated by the updated head address pointer is the same as the tail address indicated by the tail address pointer, the idle sub-node resets the head address pointer and the tail address pointer by using the consumption process; when the idle sub-node reads file data from the cache pool sub-area by using the consumption process, if the stored file data in the cache pool sub-area is less than a preset number of rows, the idle sub-node moves the stored file data to the head of the cache pool sub-area by using the consumption process, and updates the head address pointer and the tail address pointer.

[0114] In one embodiment, the cache pool comprises a state announcement area for indicating the state of each cache pool sub-area; the computer program, when executed by the processor, further implements the following steps: the idle sub-node acquires the access request of the production process to the cache pool by using an announcement process for maintaining the state announcement area, and allows the production process to access the associated cache pool sub-area if the state of the cache pool sub-area associated with the production process is in the idle state; the idle sub-node acquires the access request of the consumption process to the cache pool by using the announcement process, and allows the consumption process to access the cache pool sub-area whose state is in the idle state if there is a cache pool sub-area whose state is in the idle state in the cache pool.

[0115] In one embodiment, the computer program, when executed by the processor, further implements the following steps: the idle sub-node sets a first access time length for the production process by using the announcement process when allowing the production process to access the associated cache pool sub-area, and sets a second access time length for the consumption process by using the announcement process when allowing the consumption process to access the cache pool sub-area whose state is in the idle state.

[0116] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0117] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0118] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A file processing method characterized by, The method comprises: allocating, by a master node, a file processing task to an idle node in at least one sub-node; obtaining, by the idle node, a set of to-be-processed files corresponding to the file processing task, and allocating, by an allocation process, a certain number of files in the set of to-be-processed files to each production process corresponding to a data reading operation, and reading, by each production process, file data in the file allocated thereto in a block manner and loading the file data into a respective associated cache pool sub-area in a cache pool, and reading, by a consumption process corresponding to a data processing operation, file data in each cache pool sub-area in a row manner for processing to obtain a file processing result; wherein the cache pool comprises a plurality of cache pool sub-areas, the number of cache pool sub-areas corresponds to the number of production processes, and each cache pool sub-area is associated with a production process; and the production process processes one file and then processes the next file; wherein the head of the cache pool sub-area stores a first address pointer and a tail address pointer; the respective associated cache pool sub-area loaded into the cache pool comprises writing file data to the tail of the stored file data in the cache pool sub-area according to the tail address pointer and updating the tail address pointer; and the reading of file data in each cache pool sub-area comprises reading file data from the head of the stored file data in the cache pool sub-area according to the first address pointer and updating the first address pointer; if the first address indicated by the updated first address pointer is the same as the tail address indicated by the tail address pointer, the first address pointer and the tail address pointer are reset; and if the stored file data in the cache pool sub-area is less than a preset number of rows, the stored file data is moved to the head of the cache pool sub-area, and the first address pointer and the tail address pointer are updated; wherein the ratio of the number of processes corresponding to the data reading operation and the data processing operation is balanced according to the relative size of the processing speed corresponding to the data reading operation and the data processing operation.

2. The method of claim 1, wherein, The method further comprises: in a case where the allocation of the task to the idle node is the first allocation, configuring, by the master node, the task amount corresponding to the file processing task as a preset amount; in a case where the allocation of the task to the idle node is not the first allocation, determining, by the master node, the task amount corresponding to the file processing task according to the processing time consumption of the idle node for a historical file processing task.

3. The method of claim 1, wherein, The respective associated cache pool sub-area loaded into the cache pool further comprises: when the idle node writes file data to the tail of the stored file data in the cache pool sub-area using the production process, if the available write space of the cache pool sub-area is insufficient, the idle node suspends the production process from writing file data to the cache pool sub-area.

4. The method of claim 1, wherein, The preset number of rows is one row.

5. The method of claim 1, wherein, The cache pool comprises a state announcement area for indicating the state of each cache pool sub-area; and the method further comprises: The idle sub-node acquires, by using the announcement process, an access request of the production process to the cache pool, and allows the production process to access the associated cache pool sub-area when the state of the cache pool sub-area associated with the production process is idle; The idle sub-node acquires, by using the announcement process, an access request of the consumption process to the cache pool, and allows the consumption process to access the cache pool sub-area whose state is idle when the cache pool has a cache pool sub-area whose state is idle.

6. The method of claim 5, wherein, The method further comprises: The idle sub-node sets, by using the announcement process, a first access time length for the production process when the production process is allowed to access the associated cache pool sub-area, and sets, by using the announcement process, a second access time length for the consumption process when the consumption process is allowed to access the cache pool sub-area whose state is idle.

7. A file processing apparatus characterized by comprising: The device comprises: A task allocation module configured to allocate, by a master node, a file processing task to an idle sub-node in at least one sub-node; A file processing module configured to acquire, by the idle sub-node, a set of to-be-processed files corresponding to the file processing task, and allocate, by using an allocation process, a certain number of files in the set of to-be-processed files to each production process corresponding to a data reading operation, and read, by using each production process, file data in the file allocated to the production process in a block manner and load the file data to a cache pool sub-area associated with the production process in a cache pool, and read, by using a consumption process corresponding to a data processing operation, file data in each cache pool sub-area in a row manner for processing to obtain a file processing result; wherein the cache pool comprises a plurality of cache pool sub-areas, the number of the cache pool sub-areas corresponds to the number of the production processes, and each cache pool sub-area is associated with one production process; the production process processes one file and then processes the next file; The head of each cache pool sub-area stores a first address pointer and a tail address pointer; The file processing module is further configured to write file data to the tail of the stored file data in the cache pool sub-area according to the tail address pointer and update the tail address pointer, and read file data from the head of the stored file data in the cache pool sub-area according to the first address pointer and update the first address pointer; A pointer resetting module configured to reset the first address pointer and the tail address pointer if the first address indicated by the updated first address pointer is the same as the tail address indicated by the tail address pointer; A data moving module configured to move the stored file data in the cache pool sub-area to the head of the cache pool sub-area if the stored file data in the cache pool sub-area is less than a preset number of rows, and update the first address pointer and the tail address pointer; The ratio of the number of processes corresponding to the data reading operation and the data processing operation is set according to the relative size of the processing speed of the data reading operation and the data processing operation.

8. The apparatus of claim 7, wherein, The device further comprises: The task amount configuration module is configured to, when the task allocation to the idle child node is the first allocation, configure, by the master node, the task amount corresponding to the file processing task as a preset amount; and when the task allocation to the idle child node is not the first allocation, determine, by the master node, the task amount corresponding to the file processing task according to the processing time of the idle child node for a historical file processing task.

9. The apparatus of claim 7, wherein, The device further comprises: The write suspension module is configured to, when the idle child node writes file data to the tail of the stored file data in the cache pool sub-area by using the production process, suspend, by the idle child node, the production process from writing file data to the cache pool sub-area if the available write space of the cache pool sub-area is insufficient.

10. The apparatus of claim 7, wherein, The preset number of rows is one row.

11. The apparatus of claim 7, wherein, The cache pool comprises a state announcement area for indicating the state of each cache pool sub-area; and the device further comprises: The production process access processing module is configured to acquire, by the idle child node, an access request of the production process to the cache pool by using an announcement process for maintaining the state announcement area, and allow the production process to access the associated cache pool sub-area if the state of the cache pool sub-area associated with the production process is in an idle state. The consumption process access processing module is configured to acquire, by the idle child node, an access request of the consumption process to the cache pool by using the announcement process, and allow the consumption process to access the cache pool sub-area in an idle state if the cache pool has a cache pool sub-area in an idle state.

12. The apparatus of claim 11, wherein, The device further comprises: The access time length setting module is configured to set, by the idle child node, a first access time length for the production process when the production process is allowed to access the associated cache pool sub-area by using the announcement process, and set, by the idle child node, a second access time length for the consumption process when the consumption process is allowed to access the cache pool sub-area in an idle state by using the announcement process.

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

14. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

15. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images