Batch data processing method, apparatus, device, and medium

Abstract

The application provides a batch data processing method and device, equipment and medium, which can be applied to the technical field of big data and cloud computing, and can be used in the financial field and other fields. The method comprises the following steps: scanning a file storage path of a file storage system in a target time period, obtaining a target data file meeting a storage condition, and storing the target data file in a local memory; in response to a task receiving request, receiving a batch calculation task, the batch calculation task comprising a plurality of calculation tasks; determining target data related to each calculation task in the batch calculation task based on batch data in the target data file stored in the local memory; and executing each calculation task based on the target data of each calculation task, to obtain a task processing result of each calculation task. The method improves data read-write performance and task processing efficiency.

Description

Technical Field

[0001] This application relates to the fields of big data and cloud computing technology, and in particular to a batch data processing method, apparatus, device and medium. Background Technology

[0002] In the fintech field, intelligent computing has become an important application area in the trading market. Examples include monitoring and confirming pending events, batch processing of financial product trading events, and batch calculation of financial product net asset value. These various intelligent computing needs in the fintech field can be considered as data processing needs. Currently, due to the large number of trading events and financial calculations, batch processing of large amounts of data is required.

[0003] Currently, data processing or computation tasks are generally generated by applications. These applications can be various client-facing, interactive programs, and each application corresponds to an application server, which in turn retrieves the computation tasks. The data required for the actual processing of these computation tasks needs to be read from a database.

[0004] However, due to limitations in read and write performance between the database and application server, when dealing with a large amount of data to be processed, data read and write failures or slower speeds may occur, resulting in low data processing efficiency. Summary of the Invention

[0005] This application provides a batch data processing method, apparatus, device, and medium to solve the technical problem of low processing efficiency due to significant differences between different protocol files.

[0006] Firstly, this application provides a batch data processing method applied to distributed computing nodes, the method comprising:

[0007] Scan the file storage path of the file storage system during the target time period to obtain target data files that meet the storage conditions, and store the target data files in local memory. The file storage system is used to store at least one data file that the application server has pre-exported from the database. The data file includes batch data to be processed.

[0008] In response to a task receiving request, a batch computing task is received, wherein the batch computing task includes multiple computing tasks;

[0009] Based on the batch data in the target data file stored in the local memory, determine the target data involved in each computing task in the batch computing task;

[0010] Based on the target data of each computing task, each computing task is executed to obtain the task processing results of each computing task.

[0011] Secondly, this application provides a batch data processing device located on a distributed computing node, the device comprising:

[0012] The file acquisition unit is used to scan the file storage path of the file storage system within a target time period, obtain the target data file that meets the storage conditions, and store the target data file in local memory. The file storage system is used to store at least one data file pre-exported from the database by the application server. The data file includes batch data to be processed.

[0013] The task receiving unit is used to receive batch computing tasks in response to a task receiving request, wherein the batch computing tasks include multiple computing tasks.

[0014] A data determination unit is used to determine the target data involved in each computing task in the batch computing task based on the batch data in the target data file stored in the local memory.

[0015] The task execution unit is used to execute each computing task according to the target data of each computing task, and obtain the task processing results of each computing task.

[0016] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0017] The memory stores computer-executed instructions;

[0018] The processor executes computer execution instructions stored in the memory to implement the batch data processing method as described in the first aspect.

[0019] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the batch data processing method as described in the first aspect.

[0020] The batch data processing method provided in this application can scan the file storage path of a file storage system within a target time period to obtain target data files that meet the storage conditions. The file storage system stores at least one data file pre-exported from a database by the application server, and the data file includes batch data to be processed. By constraining the target time period, data files can be read at a specific time, improving data reading efficiency. The target data file can be a target data file that meets the storage conditions, enabling partial file retrieval and distributed storage of data files, avoiding the concentration of a large number of data files on a single node. Subsequently, in response to a task reception request, batch computation tasks can be received, and the target data involved in each computation task can be determined based on the batch data in the target data files stored in local memory. The computation tasks are executed according to the target data of each computation task, obtaining the task processing results of each computation task. Since both the computation tasks and the target data involved in the computation tasks can originate locally, placing the data and tasks on a single node allows for direct retrieval of relevant target data from the local machine during task execution, eliminating the need to temporarily read data from the database. Direct reading of local data is faster, improving data read / write performance and thus task processing efficiency. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] Figure 1 This is an application system architecture diagram of a batch data processing method provided in an embodiment of this application;

[0023] Figure 2 This is a flowchart of one embodiment of a batch data processing method provided in this application;

[0024] Figure 3 This is an example diagram of task distribution provided in an embodiment of this application;

[0025] Figure 4 This is a flowchart of yet another embodiment of a batch data processing method provided in this application;

[0026] Figure 5 This is an example diagram of task execution provided in an embodiment of this application;

[0027] Figure 6 This is an example diagram of data file distribution provided in an embodiment of this application;

[0028] Figure 7 This is a structural example diagram of one embodiment of a batch data processing device provided in this application;

[0029] Figure 8 This is a block diagram of an electronic device used to implement the batch data processing method provided in this application.

[0030] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] The batch data processing method, apparatus, equipment, and medium provided in this application can be applied to the fields of big data and cloud computing technology, as well as financial technology or other related fields. By placing data and tasks in a node, it is possible to directly obtain relevant target data from the local machine during task execution. Direct reading of local data is faster, and task processing efficiency can be improved by enhancing data read and write performance.

[0033] In related technologies, with the rapid development of fintech, banks and other financial institutions can provide various application services to users through applications. Currently, the processing device corresponding to these applications can be an application server. Application servers can handle various financial data processing needs. For processing various types of financial data, it is generally necessary to read relevant data from a database and perform corresponding processing tasks on the data read from the database. Due to the limitations of the read / write performance of the application server and the database, when the data processing demand of the application server is high, i.e., when processing data in batches, it needs to read and write a large amount of data to the database, resulting in high data read / write pressure on the application server and the database, low data read / write speed, and consequently low data processing efficiency.

[0034] To address the aforementioned technical issues, this technical solution considers that the limitation in data read / write operations stems from the fact that when application servers process large amounts of data, they need to simultaneously read and write large amounts of data from the database, resulting in significant data read / write pressure within a short period. Therefore, to solve this problem, data preloading can be employed to store the data, allowing it to be retrieved locally when a computational task is available to execute the corresponding task. Furthermore, to address the issues of large-scale data storage and processing, a distributed cluster can be used to implement data pre-caching and computational task execution, thereby improving data processing efficiency.

[0035] Accordingly, the technical solution of this application scans the file storage path of the file storage system within a target time period to obtain target data files that meet the storage conditions. The file storage system stores at least one data file pre-exported from the database by the application server, and the data file includes batch data to be processed. By constraining the target time period, data files can be read at a specific time, improving data reading efficiency. The target data file can be a target data file that meets the storage conditions, enabling partial file retrieval and distributed storage of data files, avoiding the concentration of a large number of data files on a single node. Subsequently, in response to task reception requests, batch computation tasks can be received, and the target data involved in each computation task can be determined based on the batch data in the target data files stored in local memory. Computation tasks are executed according to the target data of each computation task, obtaining the task processing results of each computation task. Since both the computation tasks and the target data involved in the computation tasks can originate locally, placing the data and tasks on a single node allows for direct retrieval of relevant target data from the local machine during task execution, eliminating the need to temporarily read data from the database. Direct reading of local data is faster, improving data read / write performance and thus task processing efficiency.

[0036] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0037] Figure 1 This application provides an application system architecture diagram for a batch data processing method according to an embodiment of the present application. The system may include a distributed cluster 11 and a file storage system 12, which may have wired or wireless communication connections. The file storage system 12 may also establish a wired or wireless communication connection with a database 13.

[0038] Database 13 can store a large amount of financial data, such as the value of individual stocks and account balances. This embodiment does not impose excessive limitations on the specific type of data or its application area. File storage system 12 and database 13 can also be connected to external applications, which can be installed on electronic device 14. These external applications can send data export commands to database 13, controlling it to export data related to the computing task in advance as data files from database 13 to file storage system 12. File storage system 12 can include the pre-exported data files.

[0039] The distributed cluster 11 may include at least one computing node 110. For any distributed computing node 110 in the distributed cluster 11, the batch data processing method of this application can be configured. Figure 1 The diagram shows one such distributed computing node, and other distributed computing nodes have the same functionality. Based on the batch data processing method of this application, the nodes scan the file storage path of the file storage system during the target time period to obtain the target data file that meets the storage conditions. Through subsequent steps such as file storage, task reception, and task execution, the nodes obtain the task processing results and realize the task execution of the data file. Local data acquisition can reduce data read and write requirements, thereby improving data read and write speed and data processing efficiency.

[0040] In addition, in one possible design, at least one computing node 110 in the distributed cluster 11 can connect to the result database. After the data processing is completed and the task processing result is obtained, the task processing node can be pushed to the local thread pool first, and multiple task processing nodes in the result thread pool can be stored in the result database at a certain time to achieve unified storage of results. This eliminates the need to consume too much memory for data push during the data processing process, thereby reducing the processing power consumption of the nodes and further improving the data processing efficiency.

[0041] like Figure 2 The diagram shown is a flowchart of one embodiment of a batch data processing method provided in this application. This method can be configured as a batch data processing device, which can be located in a distributed computing node. The batch data processing method may include the following steps:

[0042] 201: Scan the file storage path of the file storage system during the target time period, obtain the target data file that meets the storage conditions, and store the target data file in local memory. The file storage system is used to store at least one data file that the application server has pre-exported from the database. The data file includes batch data to be processed.

[0043] Optionally, the target time period can be an idle time period. A time period with low database read / write volume can be used as the target time period. For example, the period from midnight to 5:00 AM can be used as the target time period. Alternatively, real-time monitoring of read / write volume between the server and the database can be performed, and when the read / write volume is determined to be less than the read / write threshold, that time point can be used as the start time, and the time point added to the data export duration can be used as the end time. The start and end times can then be used as the target time period. When scanning the file storage path of the file storage system during the target time period, the file storage path can be scanned according to a preset scanning frequency.

[0044] The file storage system can be an operating system configured on the storage device for storing files, such as an FSS (File System Store) system, FTP (File Transfer Protocol), etc. In this embodiment, the file storage system can pre-export at least one data file from the database and store it.

[0045] At least one data file can include sharded data files and unsharded data files. Sharded data files can be split into at least one sub-data file. Unsharded data files can be distributed directly to the distributed computing nodes without being split. The target data file can include sub-data files of sharded data files and unsharded data files. Of course, if a sharded data file is not split, the target data file can also include the complete sharded data file.

[0046] Optionally, meeting the storage conditions can mean that sub-data files of a sharded data file are determined to be stored on a distributed computing node, and / or that non-sharded data files are determined to be stored on a distributed computing node. Specifically, this means that a sub-data file or a data file is allocated to a distributed computing node, such as when the file identifier of the data file, obtained through hash calculation, matches the node identifier of the distributed computing node. For example, if the file identifier's hash value is 1, and the distributed computing node's node identifier is 1, it can be determined that the data file or sub-data file with the file identifier 1 meets the storage conditions of the distributed computing node with node identifier 1.

[0047] The batch data processing method of this application can be executed by a distributed computing node, which can be a computing node in a distributed cluster. The distributed cluster can have an access connection with a file storage system, and the file storage system can have an access connection with a database. The file storage system can pre-export and store at least one data file.

[0048] The application server can be the server corresponding to the client. The client can initiate computing tasks to the application server, and the application server can retrieve data from the database based on the computing tasks. In practical applications, when there are a large number of computing tasks, it may lead to a large processing pressure on the database. For computing tasks with lower real-time processing requirements, the technical solution of this application can be adopted.

[0049] Local memory can be the storage of distributed computing nodes, and can be used to store received target data files. Target data files can be data files distributed to distributed computing nodes.

[0050] Optionally, the file storage path can be the storage address of at least one data file on a distributed computing node. The file storage path can be pre-set, enabling accurate writing and reading of data files. For example, the file storage path can include information such as the node name of the distributed node, the application name, the data file name, the data type, and the timestamp. The file storage path can be represented as follows:

[0051] ~ / ${nodeName (node ​​name)} / ${app (application name)} / ${dataName (data file name)}_${dataType (data type)}_${timestamp (timestamp)}

[0052] In practical storage, sharded and non-sharded data files can be stored separately. For example, sharded data files can be stored in a first file path, and non-sharded data files can be stored in a second file path. The first and second file paths can be different. For instance, the first file path may include information such as the node name of the distributed node, the application name, the sharding attribute identifier, the data file name, the data type, and the timestamp. The second file path may also include the same information. Furthermore, a non-sharding attribute identifier can be added between the application name and the data file name. By using sharding and non-sharding attribute identifiers, the sharding attributes of data files stored in different storage paths can be distinguished, enabling efficient storage management of data files based on storage paths.

[0053] 202: In response to a task receiving request, receive batch computation tasks, which include multiple computation tasks.

[0054] Task reception requests can be sent by the task coordination node. Batch computing tasks refer to data processing tasks initiated by application servers or other terminal devices, which rely on data in data files to complete. Computation tasks can be sent by application servers or other terminal devices to the task coordination node, which then distributes the computing tasks.

[0055] Optionally, task reception can be performed at any time period to ensure timely task processing. Alternatively, task reception can be performed within a target time period, which can be an idle period. By constraining task reception by time period, task execution can be achieved within that specific time frame. Specifically, receiving computational tasks can involve receiving batch computational tasks.

[0056] 203: Based on the batch data in the target data file stored in local memory, determine the target data involved in each computing task in the batch computing task.

[0057] Optionally, step 203 may include: determining the target data involved in each computation task of the batch computation task based on the batch data in the target data file stored in local memory.

[0058] The target data file can store batch data, and batch computation tasks can include batch computation tasks targeting the batch data. A batch computation task can include multiple computation tasks. That is, computations targeting the batch data in the actual target file can be initiated through batch computation tasks, and the target data corresponding to each computation task can be obtained. The batch data in the target data file may partially involve computation tasks. Therefore, it is necessary to determine the target data matching each computation task from the batch data in the target data file, based on the task requirements of each computation task. The target data matching each computation task can refer to data matching the task information of the computation task. The computation task can include task information, such as at least one of the following: task name, task batch number, task parameters (JSON), and task-related data. The task information can define the data requirements of the task, and the target data related to the computation task can be obtained through the task information. The target data involved in a computation task can include at least one piece of data.

[0059] 204: Based on the target data of each computing task, execute each computing task separately and obtain the task processing results of each computing task.

[0060] Optionally, step 204 may include: inputting the target data of each batch computing task into the corresponding computing task to perform data processing, and obtaining the task processing result of the computing task. When there are multiple target data, the target data of each computing task can be input sequentially to perform data processing, and the task processing result of each computing task can be obtained. The batch processing result of the batch computing task may include the task processing results corresponding to multiple computing tasks respectively.

[0061] In this embodiment, the file storage path of the file storage system is scanned during a target time period to obtain target data files that meet the storage conditions. The file storage system stores at least one data file pre-exported from the database by the application server, and the data file includes batch data to be processed. By constraining the target time period, data files can be read at a specific time, improving data reading efficiency. The target data file can be a target data file that meets the storage conditions, allowing for partial file retrieval and distributed storage of data files, avoiding the concentration of a large number of data files on a single node. Subsequently, in response to a task reception request, batch computation tasks can be received, and the target data involved in each computation task can be determined based on the batch data in the target data file stored in local memory. The computation tasks are executed according to the target data of each computation task, obtaining the task processing results of each computation task. Since both the computation tasks and the target data involved in the computation tasks can originate locally, placing the data and tasks on a single node allows for direct retrieval of relevant target data from the local machine during task execution, eliminating the need to temporarily read data from the database. Direct reading of local data is faster, improving data read / write performance and thus task processing efficiency.

[0062] Furthermore, based on any of the above embodiments, in response to a task sending request, receiving a computing task includes:

[0063] In response to a task receive request sent by the task coordinating node, receive a task execution notification message sent by the task coordinating node;

[0064] Based on the task execution notification message, determine the batch calculation tasks.

[0065] like Figure 3 The diagram shown is an example of task distribution provided in an embodiment of this application. (Reference) Figure 3 The coordinating node 31 can receive task execution notification messages from the online interface and distribute the computing tasks to each distributed computing node 32 in response to these messages. The coordinating node 32 can also receive batch computing task execution notification messages and distribute the computing tasks to each distributed computing node in response to these messages.

[0066] Optionally, the task coordination node can connect to the application server's interface, other terminal device interfaces, or online interfaces. It receives batch computation tasks via the online interface and sends these tasks to the distributed computing nodes using PRC (Remote Procedure Call) or SOCKET protocols. The distributed computing nodes can receive computation tasks sent via PRC or SOCKET.

[0067] The task execution notification message can include either a task execution notification message sent directly through the online interface or a task execution notification message determined by a batch file. Specifically, determining the batch computation tasks based on the task execution notification message can include: receiving the task execution notification message sent through the online interface to obtain the batch computation tasks in the task execution notification message. It can also include: obtaining the task execution notification message corresponding to the batch file, parsing the batch file, and obtaining the batch computation tasks corresponding to the batch file.

[0068] For example, during task distribution, the data types required by the distributed computing tasks can be matched with the data types of the data actually stored on each computing node. This ensures that the data types required by the distributed computing tasks are the same as the data types stored on the computing nodes, enabling normal data storage and use. Furthermore, in practical applications, each computing node receives the same computing tasks, allowing for concurrency of similar computing tasks and improving task processing efficiency. Additionally, various computing tasks can be distributed to the corresponding computing nodes by pre-associating the task type with the node identifier. Of course, in this method, the node identifier can also be adapted to the actual data type, achieving corresponding management and distribution of data and tasks, improving the effectiveness and relevance of task distribution, and increasing data processing efficiency.

[0069] In this embodiment, computational tasks can be coordinated through a task coordination node, enabling the simultaneous distribution of batch computational tasks and improving task distribution efficiency. Furthermore, the task coordination node allows for the effective distribution of computational tasks, achieving efficient processing of these tasks.

[0070] like Figure 4 As shown, with Figure 2 The difference in the steps of the illustrated embodiment is that, after receiving the computing task, step 203 further includes:

[0071] 401: Push batch computation tasks to the task thread pool.

[0072] Step 204: Based on the target data of each computation task, execute each computation task separately and obtain the task processing results of each computation task, including:

[0073] 402: By using the target threads in the task thread pool, the data processing of the target data corresponding to each computing task is executed to obtain the task processing results of each computing task.

[0074] Optionally, the task thread pool can include multiple target threads, each of which can be used to execute computation tasks. Concurrent execution of multiple target threads allows for parallel execution of computation tasks, improving execution efficiency and speed. When there are a large number of batch computation tasks, these tasks can be executed in batches using one or more target threads in the task thread pool. After obtaining the processing results of each computation task, these results are pushed to the result thread pool for pre-storage of the computation task results.

[0075] In this embodiment, after determining the batch computation tasks, they can be pushed to a task thread pool. The task thread pool can store these tasks, achieving effective storage. Through the target threads in the task thread pool, data processing of the target data corresponding to each computation task can be executed separately, enabling the independent execution of batch computation tasks, avoiding task omissions, and ensuring task execution integrity. Furthermore, the target threads in the task thread pool can achieve efficient execution of computation tasks, improving their execution efficiency and accuracy.

[0076] Furthermore, based on any of the above embodiments, a computational task is performed according to the target data to obtain the task processing result, and then the process further includes:

[0077] The task processing results of each computing task are pushed to the result thread pool for pre-storage of the results;

[0078] In response to the result push task, the task processing results corresponding to the multiple computing tasks stored in the result thread pool are stored in the result database.

[0079] For ease of understanding, such as Figure 5 The diagram shown is an example of task execution provided in an embodiment of this application. (Reference) Figure 5Distributed computing nodes can receive distributed computing tasks and push them to task thread pool 51. Then, computing task 52 can be executed through the target thread in the task thread pool. During execution, computing task 52 can call the task implementation method 521 and access the target data stored in local memory 522 during the task implementation process, executing the computing task corresponding to the target data to obtain the task processing result. The task processing result is pushed to the result thread pool 53. The result thread pool 53 can pre-store the results. And when a result push task 54 exists, executing the result push task 54 will store the multiple task processing results stored in the result thread pool 53 together in the result database 55.

[0080] Optionally, before responding to the result push task, the process may further include: determining that the result thread pool meets the result push conditions. Specifically, meeting the result push conditions can involve detecting the number of task processing results in the result thread pool; if the number of results is greater than or equal to a threshold, then the result thread pool is determined to meet the push conditions. Pushing task processing results only after reaching the threshold allows for the simultaneous push of multiple task processing results, reducing the efficiency of task processing result pushes and thus reducing the read / write frequency of the result database, thereby reducing the read / write pressure on the result database. Alternatively, meeting the result push conditions can also involve detecting that the current time has reached the target push time, then determining that the result thread pool meets the push conditions. By setting a target push time, task processing results can be pushed at a specific time, reducing the read / write pressure on the result database and achieving efficient result pushes. The target push time can be the idle time of the result database.

[0081] In this embodiment, the task processing results of each computing task can be pushed to a result thread pool, which pre-stores the results. When a result push task is determined to exist, the task processing results corresponding to the multiple computing tasks stored in the result thread pool are stored in the result database. By pre-storing the results in the result thread pool, unified pushing and saving of the task processing results of computing tasks can be achieved, reducing the frequency of result saving and thus improving data storage security and accuracy.

[0082] Furthermore, based on any of the above embodiments, the target data involved in each computing task in the batch computing task is determined based on the batch data in the target data file stored in local memory, including:

[0083] Read the local data involved in each computing task in the batch computing task from the batch data of the target data file stored in local memory;

[0084] The corresponding target data is determined based on the local data for each computing task.

[0085] Optionally, the target data for each computation task can be determined using its local data. The target data for a computation task may include the target data for that computation task itself.

[0086] In this embodiment, local data related to each computing task in a batch computing task can be read from the batch data of the target data file stored in local memory, so as to determine the corresponding target data based on the local data corresponding to each computing task. By reading local data, local acquisition of computing task-related data can be achieved. Acquiring data locally can greatly improve the data read and write speed, thereby improving the acquisition efficiency of target data for each computing task and effectively enhancing the data read and write speed.

[0087] Furthermore, based on any of the above embodiments, after reading the local data involved in the computing task from the target data file stored in local memory, the method further includes:

[0088] Read other data related to each computing task in the batch computing task from the database;

[0089] The target data is determined based on the local data corresponding to each computing task, including:

[0090] The target data is obtained by combining the local data corresponding to each computing task in the batch computing task with other data.

[0091] Alternatively, the other data can be data from the database related to the computation task but not stored in the target data file. Reading this other data reduces read / write restrictions on the local target data file, ensuring access to all data related to the computation task and improving the security of the task execution.

[0092] In this embodiment, for each computing task, other data involved in each computing task in the batch computing task can be read from the database, realizing the database reading of partial data of each computing task. Combined with local data reading, the amount of data read and write performed from the database can be effectively reduced, thereby reducing the data read and write pressure between the database and the database and improving data read and write efficiency.

[0093] Furthermore, based on any of the above embodiments, scanning the file storage path of the file storage system within a target time period to obtain target data files that meet the storage conditions includes:

[0094] Scan the file storage paths of the file storage system within the target time period to obtain at least one data file;

[0095] Identify at least one computing node corresponding to the distributed cluster, wherein the at least one computing node includes a distributed computing node.

[0096] Based on the file information corresponding to at least one data file, determine the target computing node corresponding to at least one data file from at least one computing node;

[0097] Based on the target computing nodes corresponding to at least one data file, the target data file of the target computing node is determined to be a distributed computing node, so as to read the target data file from the file storage path.

[0098] In this embodiment, at least one data file can be obtained by scanning the file storage path of the file storage system within a target time period. Based on the file information corresponding to each of the at least one data file, the target computing node corresponding to each data file can be determined, thus enabling the distribution of data to computing nodes based on the file information of each data file. By using the target computing nodes corresponding to each data file, the target data files belonging to the distributed computing nodes can be obtained, enabling the reading of target data files from the distributed computing nodes and improving the efficiency and accuracy of data file distribution.

[0099] Furthermore, based on any of the above embodiments, the distributed computing node is a computing node in a distributed cluster, and the distributed cluster includes at least one computing node. Determining the target computing node corresponding to at least one data file from the at least one computing node based on the file information corresponding to at least one data file includes:

[0100] At least one data file is fragmented and classified to obtain fragmented data files and non-fragmented data files;

[0101] The fragmented data file is split into fragments to obtain at least one sub-data file.

[0102] For a sharded data file, determine the target computing node corresponding to each sub-data file of the sharded data file from at least one computing node;

[0103] For non-sharded data files, determine the target computing node corresponding to the non-sharded data file from at least one computing node, or determine a specific node from at least one computing node, and determine the specific node as the target computing node for the non-sharded data file.

[0104] Optionally, for a sharded data file, determining the target computing node corresponding to each sub-data file of the sharded data file from at least one computing node may include: for the sharded data file, using a hash algorithm to determine the target computing node corresponding to the sharded data file from at least one computing node.

[0105] Each data file can be associated with a shard identifier, which can be either a shardable identifier or a non-shardable identifier. The shard identifier of each data file determines its sharding category. For example, if a data file's shard identifier is shardable, the data file can be a sharded data file. If a data file's shard identifier is non-shardable, the data file can be a non-sharded data file.

[0106] Optionally, for a non-sharded data file, determining the target computing node corresponding to the non-sharded data file from at least one computing node may include: for the non-sharded data file, using a hash algorithm to determine the target computing node corresponding to the non-sharded data file from at least one computing node.

[0107] Hash algorithms can be used to achieve distributed storage of at least one sub-file of a fragmented data file. They can also be used to achieve distributed storage of non-fragmented data files, improving data file security and orderliness, and enabling the regular storage and use of data files.

[0108] The specific node can be a node in the distributed cluster used to store non-sharded data files, and can be obtained through node specification or node filtering. Node specification can specifically designate any computing node in the distributed cluster as the specific node. Node filtering can specifically perform storage space checks on each computing node in the distributed cluster, and select computing nodes whose storage space meets the usage requirements as the specific nodes. Storage space usage requirements can include, for example, storage space exceeding a storage threshold, or storage space occupancy being less than a occupancy threshold.

[0109] For ease of understanding, such as Figure 6 The diagram shown is an example of data file distribution provided in an embodiment of this application. (Reference) Figure 6 Data in database 61 can be distributed to file storage system 62. File storage system 62 can identify fragmented and non-fragmented data files among the received data files. For example, fragmented data files can be named "pFile-1, pFile-2", and non-fragmented data files can be named "npFile-1, npFile-2". Fragmented data files can be divided into several sub-data files.

[0110] For example, "pFile-1" can be divided into n sub-data files such as "pFile-1-1, pFile-1-2, ..., pFile-1-n". Each sub-data file can have its target computing node determined by a hash algorithm. Assume at least one computing node 631-63n in the distributed cluster 63, and a specific node 64. After being fragmented, the data files can be distributed across the distributed cluster for separate storage. For example, pFile-1-1 is distributed to distributed computing node 631, which is the target computing node for sub-data file pFile-1-1; pFile-1-2 is distributed to distributed computing node 632, which is the target computing node for sub-data file pFile-1-2; and pFile-1-n is distributed to distributed computing node 63n, which is the target computing node for sub-data file pFile-1-n.

[0111] For non-fragmented files, distributed storage or storage on specific nodes can be used. For example, npFile-1 can be stored in a distributed manner on distributed computing node 63n, or npFile-1 can be stored on a specific node 64.

[0112] In this embodiment, at least one data file can be segmented and classified to obtain segmented data files and non-segmented data files. By classifying segmented and non-segmented data files, at least one data file can be distributed and stored according to its segmentation category. For segmented data files, at least one sub-data file can be identified and distributed through at least one computing node, achieving distributed storage of at least one sub-data file. For non-segmented data files, they can be directly distributed or stored on specific nodes, achieving distributed or targeted storage. Distributed storage enables the regular storage of data files, improving storage efficiency and regularity. Furthermore, the regularly stored sub-data files and non-segmented data files allow for fast retrieval and use, improving file utilization efficiency.

[0113] like Figure 7 The diagram shown is a structural schematic of one embodiment of a batch data processing device provided in this application. This batch data processing device can be configured with the batch data processing method provided in the foregoing embodiments. The batch data processing device can be located in a distributed computing node. The batch data processing device 700 may include the following units:

[0114] File acquisition unit 701: used to scan the file storage path of the file storage system during the target time period, obtain the target data file that meets the storage conditions, and store the target data file in local memory. The file storage system is used to store at least one data file pre-exported from the database by the application server. The data file includes batch data to be processed.

[0115] Task receiving unit 702: Used to receive batch computing tasks in response to a task receiving request, wherein the batch computing tasks include multiple computing tasks;

[0116] Data determination unit 703: used to determine the target data involved in each computing task in the batch computing task based on the batch data in the target data file stored in local memory;

[0117] Task execution unit 704: is used to execute each computing task according to the target data of each computing task, and obtain the task processing results of each computing task.

[0118] As one embodiment, the task receiving unit 702 includes:

[0119] The message receiving module is used to respond to the task receiving request sent by the task coordination node and receive the task execution notification message sent by the task coordination node.

[0120] The task determination module is used to determine batch calculation tasks based on task execution notification messages.

[0121] As yet another embodiment, it also includes:

[0122] The task push unit is used to push batch computing tasks to the task thread pool;

[0123] Task execution unit 704 includes:

[0124] The task execution module is used to perform data processing on the target data corresponding to each computing task through the target threads in the task thread pool, and obtain the task processing results of each computing task.

[0125] As yet another embodiment, it also includes:

[0126] The result push unit is used to push the task processing results of each computing task to the result thread pool for result pre-storage.

[0127] The result storage unit is used to respond to the result push task by storing the task processing results corresponding to multiple computing tasks stored in the result thread pool into the result database.

[0128] As another embodiment, the data determination unit 703:

[0129] The local read module is used to read local data involved in each computing task in the batch computing task from the batch data of the target data file stored in local memory;

[0130] The target determination module is used to determine the corresponding target data based on the local data corresponding to each computing task.

[0131] As yet another embodiment, it also includes:

[0132] Other reading units are used to read other data involved in each computing task in the batch computing task from the database;

[0133] The target determination module includes:

[0134] The data synthesis submodule is used to combine the local data and other data corresponding to each computing task in the batch computing task to obtain the corresponding target data.

[0135] As another embodiment, the file acquisition unit 701 includes:

[0136] The first determining module is used to scan the file storage path of the file storage system within the target time period to obtain at least one data file;

[0137] The node determination module is used to determine at least one computing node corresponding to the distributed cluster, wherein the at least one computing node includes a distributed computing node.

[0138] The file distribution module is used to determine the target computing node corresponding to at least one data file from at least one computing node based on the file information corresponding to at least one data file.

[0139] The second determining module is used to determine the target data file of the target computing node as a distributed computing node based on the target computing node corresponding to at least one data file, so as to read the target data file from the file storage path.

[0140] As another embodiment, the target data file includes sub-data files of the fragmented data file and / or non-fragmented files, and the file distribution module includes:

[0141] The fragmentation and classification submodule is used to fragment and classify at least one data file to obtain fragmented data files and non-fragmented data files.

[0142] The file fragmentation submodule is used to process fragmented data files into fragments to obtain at least one sub-data file.

[0143] The first distribution submodule is used to determine the target computing node corresponding to each sub-data file of the sharded data file from at least one computing node for the sharded data file;

[0144] The second distribution submodule is used to determine the target computing node corresponding to the non-sharded data file from at least one computing node, or to determine a specific node from at least one computing node as the target computing node for the non-sharded data file.

[0145] The batch data processing apparatus of this application can be used to execute batch data processing methods. For details on the specific execution methods and technical effects of each step, please refer to the embodiments related to batch data processing methods, which will not be repeated here.

[0146] The aforementioned electronic devices can be either wireless or wired terminals. A wireless terminal can be a device that provides voice and / or other service data connectivity to a user, a handheld device with wireless connectivity, or other processing devices connected to a wireless modem. The wireless terminal can communicate with one or more core network devices via a Radio Access Network (RAN). The wireless terminal can be a mobile terminal, such as a mobile phone (or "cellular" phone) or a computer with a mobile terminal, for example, a portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile device, which exchanges voice and / or data with the RAN. Furthermore, a wireless terminal can also be a Personal Communication Service (PCS) phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA), or other similar devices. Wireless terminals can also be referred to as systems, subscriber units, subscriber stations, mobile stations, mobile stations, remote stations, remote terminals, access terminals, user terminals, user agents, user devices, or user equipment, without limitation. Optionally, the aforementioned electronic devices can also be smartwatches, tablets, etc.

[0147] Figure 8This is a block diagram illustrating an electronic device according to an exemplary embodiment. The device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc. The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0148] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the batch data processing method described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0149] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0150] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0151] Multimedia component 808 includes a screen that provides an output interface between electronic device 800 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When electronic device 800 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0152] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0153] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0154] Sensor assembly 814 includes one or more sensors for providing status confirmation of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0155] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0156] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described batch data processing method.

[0157] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0158] A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the aforementioned batch data processing method of the electronic device.

[0159] In addition, this application also provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement any of the batch data processing methods provided in any of the above embodiments.

[0160] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the batch data processing methods provided in any of the above embodiments.

[0161] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0162] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A batch data processing method, characterized in that, Applied to distributed computing nodes, the method includes: Scan the file storage path of the file storage system during the target time period to obtain target data files that meet the storage conditions, and store the target data files in local memory. The file storage system is used to store at least one data file that the application server has pre-exported from the database. The data file includes batch data to be processed. In response to a task receiving request, a batch computing task is received, wherein the batch computing task includes multiple computing tasks; Based on the batch data in the target data file stored in the local memory, determine the target data involved in each computing task in the batch computing task; Based on the target data of each computing task, each computing task is executed to obtain the task processing results of each computing task. The target data file includes sub-data files of a fragmented data file and / or non-fragmented files; the step of scanning the file storage path of the file storage system within a target time period to obtain the target data file that meets the storage conditions includes: Scan the file storage path of the file storage system during the target time period to obtain at least one data file; determine at least one computing node corresponding to the distributed cluster, wherein at least one computing node includes the distributed computing node; At least one data file is fragmented and classified to obtain fragmented data files and non-fragmented data files; The fragmented data file is fragmented to obtain at least one sub-data file; for each fragmented data file, a target computing node corresponding to each sub-data file of the fragmented data file is determined from at least one computing node; for each non-fragmented data file, a target computing node corresponding to the non-fragmented data file is determined from at least one computing node, or a specific node is determined from at least one computing node, and the specific node is determined as the target computing node of the non-fragmented data file. Based on the target computing node corresponding to at least one of the data files, the target computing node is determined as the target data file of the distributed computing node, so as to read the target data file from the file storage path.

2. The method according to claim 1, characterized in that, The step of receiving batch computation tasks in response to a task sending request includes: In response to a task receiving request sent by a task coordination node, receive a task execution notification message sent by the task coordination node; Based on the task execution notification message, determine the batch calculation tasks.

3. The method according to claim 1, characterized in that, After receiving the computing task, the process further includes: Push the batch computing tasks to the task thread pool; The step of executing each computation task according to the target data of each computation task and obtaining the task processing results of each computation task includes: The target threads in the task thread pool are used to perform data processing on the target data corresponding to each computing task, thereby obtaining the task processing results of each computing task.

4. The method according to claim 1, characterized in that, After executing each computational task based on the target data of each computational task and obtaining the task processing results of each computational task, the method further includes: The task processing results of each computing task are pushed to the result thread pool for pre-storage of the results; In response to the result push task, the task processing results corresponding to the multiple computing tasks stored in the result thread pool are stored in the result database.

5. The method according to claim 1, characterized in that, The determination of the target data involved in each computing task in the batch computing task based on the batch data in the target data file stored in the local memory includes: Read the local data involved in each computing task in the batch computing task from the batch data of the target data file stored in the local memory; The corresponding target data is determined based on the local data corresponding to each computing task.

6. The method according to claim 5, characterized in that, After reading the local data involved in each computing task in the batch computing task from the batch data of the target data file stored in the local memory, the method further includes: Read other data related to each computing task in the batch computing task from the database; The step of determining the corresponding target data based on the local data corresponding to each computing task includes: The local data corresponding to each computing task in the batch computing task is combined with the other data to obtain the corresponding target data.

7. A batch data processing device, characterized in that, Located on a distributed computing node, the device includes: The file acquisition unit is used to scan the file storage path of the file storage system within a target time period, obtain the target data file that meets the storage conditions, and store the target data file in local memory. The file storage system is used to store at least one data file pre-exported from the database by the application server. The data file includes batch data to be processed. The task receiving unit is used to receive batch computing tasks in response to a task receiving request, wherein the batch computing tasks include multiple computing tasks. A data determination unit is used to determine the target data involved in each computing task in the batch computing task based on the batch data in the target data file stored in the local memory. The task execution unit is used to execute each computing task according to the target data of each computing task, and obtain the task processing results of each computing task. The target data file includes sub-data files of a fragmented data file and / or non-fragmented files, and the file acquisition unit includes: The first determining module is used to scan the file storage path of the file storage system within a target time period to obtain at least one data file; A node determination module is used to determine at least one computing node corresponding to a distributed cluster, wherein the at least one computing node includes the distributed computing node. The file distribution module includes: The fragmentation and classification submodule is used to fragment and classify at least one data file to obtain fragmented data files and non-fragmented data files. The file fragmentation submodule is used to fragment the fragmented data file to obtain at least one sub-data file. The first distribution submodule is used to determine, for the sharded data file, the target computing node corresponding to each sub-data file of the sharded data file from at least one computing node; The second distribution submodule is used to determine, for the non-sharded data file, the target computing node corresponding to the non-sharded data file from at least one of the computing nodes, or to determine a specific node from at least one of the computing nodes, and determine the specific node as the target computing node of the non-sharded data file; The second determining module is used to determine the target computing node as the target data file of the distributed computing node based on the target computing node corresponding to at least one of the data files, so as to read the target data file from the file storage path.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement any one of the batch data processing methods as described in claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement any one of the batch data processing methods as described in claims 1-6.

Images