A data processing system and model training method

Abstract

The embodiment of the present invention provides a data processing system and a model training method, which relate to the field of big data processing technology. The system includes at least one task node, and each task node includes: at least one data computing subnode, at least one model training subnode and a shared memory; each data computing subnode is a subnode in a real-time data computing system, and each model training subnode is a subnode in a model training system; each data computing subnode is used to perform a specified processing operation on the acquired first real-time data to be processed, obtain a first processing result, and store the first processing result in the shared memory; each model training subnode is used to read the first processing result from the shared memory, use the first processing result to perform model training, and obtain a trained target model. Compared with the prior art, the solution provided by the embodiment of the present invention can improve the timeliness of the processing results of real-time data.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a data processing system and a model training method. Background Art

[0002] At present, with the increasing requirements of various industries for the timeliness of data processing, many real-time data computing systems have emerged to process real-time data and obtain processing results, such as real-time data features; and with the increasing timeliness requirements of various model training algorithms for the above processing results, it is required that the processing results obtained by the real-time data computing system can be efficiently pushed to the model training system for training.

[0003] However, in the related art, due to the lack of integration between the real-time data computing system and the model training system, the processing results obtained by the real-time data computing system in real time need to be written to the disk before they can be read and used by the model training system, which makes the timeliness of the real-time data processing results poor.

[0004] Based on this, how to integrate the real-time data computing system with the model training system and improve the timeliness of the real-time data processing results has become a problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the embodiment of the present invention is to provide a data processing system and a model training method to realize the integration of a real-time data computing system and a model training system, and improve the timeliness of the processing results of real-time data. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a data processing system, the system comprising at least one task node, each task node comprising: at least one data computing subnode, at least one model training subnode and a shared memory; wherein each data computing subnode is a subnode in a real-time data computing system, and each model training subnode is a subnode in a model training system;

[0007] Each data computing sub-node is used to perform a specified processing operation on the acquired first real-time data to be processed, obtain a first processing result, and store the first processing result in the shared memory;

[0008] Each model training sub-node is used to read the first processing result from the shared memory, and use the first processing result to perform model training to obtain a trained target model.

[0009] Optionally, in a specific implementation, each model training sub-node is also used to store the target model in the shared memory; each data calculation sub-node is also used to obtain the target model from the shared memory, and use the target model to process the obtained second real-time data to be processed, obtain a second processing result, and store the second processing result in the shared memory.

[0010] Optionally, in a specific implementation, each task node is provided with a shared memory management service; each data computing sub-node is further used for:

[0011] Registering the data information of the first processing result into the shared memory management service;

[0012] Obtaining a first reference address of the data information in the shared memory management service, and sending the first reference address to each model training sub-node;

[0013] The data information includes: a node identifier of the data calculation subnode, a storage address of the first processing result in the shared memory, and a variable value corresponding to the first processing result;

[0014] Each model training sub-node reads the first processing result from the shared memory, including:

[0015] Each model training sub-node reads the data information from the shared memory management service according to the received first reference address, and reads the first processing result from the shared memory according to the data information;

[0016] Each data computing sub-node stores the first processing result in the shared memory, including:

[0017] Each data computing sub-node stores the first processing result in the shared memory according to the first memory protocol; wherein the data information includes: a protocol identifier of the first memory protocol.

[0018] Optionally, in a specific implementation, each task node is provided with a shared memory management service; each model training sub-node is further used for:

[0019] Registering the model information of the target model into the shared memory management service;

[0020] Acquire a second reference address of the model information in the shared memory management service, and send the second reference address to each data computing sub-node;

[0021] The model information includes: a node identifier of the model training subnode, a storage address of the target model in the shared memory, and a variable value corresponding to the target model;

[0022] Each data computing sub-node obtains the target model from the shared memory, including:

[0023] Each data computing sub-node reads the model information from the shared memory management service according to the received second reference address, and reads the target model from the shared memory according to the model information;

[0024] Each model training sub-node stores the target model in the shared memory, including:

[0025] Each model training sub-node stores the target model in the shared memory according to the second memory protocol; wherein the model information includes: a protocol identifier of the second memory protocol.

[0026] Optionally, in a specific implementation, the real-time data computing system is a distributed real-time data computing system; the model training system is a distributed model training system; each task node is provided with a state management service, and each data processing sub-node is also used for:

[0027] Acquire the first real-time data to be processed from a message queue of at least one data source;

[0028] Inserting a designated identifier into each message queue according to a preset cycle; wherein the designated identifier includes: partition information of the message queue and an offset address of the currently read data in the message queue;

[0029] In the state management service, the data state corresponding to each message queue determined based on each designated identifier is recorded; wherein the data state corresponding to each message queue is used to characterize: the offset address in the message queue of the data obtained by each data computing node from the message queue.

[0030] Optionally, in a specific implementation, the number of the data sources is multiple; before each data processing sub-node performs the specified processing operation on the acquired first real-time data to be processed, each data computing sub-node is further used to:

[0031] Determine whether the specified identifiers inserted in the message queue of each data source are aligned;

[0032] If aligned, a specified processing operation is performed on the first acquired real-time data to be processed.

[0033] Optionally, in a specific implementation, after each data computing sub-node crashes and restarts, each data computing sub-node is further used to:

[0034] Obtaining the data status corresponding to each message queue recorded in the status management service;

[0035] Each data computing sub-node obtains the first to-be-processed real-time data from a message queue of at least one data source, including:

[0036] Each data calculation sub-node starts from the offset address represented by each acquired data state and acquires the first real-time data to be processed from each message queue.

[0037] In a second aspect, an embodiment of the present invention provides a model training method, which is applied to any target model training subnode in any target task node of a data processing system; wherein the data processing system includes: at least one task node, each task node includes: at least one data computing subnode, at least one model training subnode and shared memory, each data computing subnode is a subnode in a real-time data computing system, and each model training subnode is a subnode in a model training system; the method includes:

[0038] Reading a first processing result from a target shared memory included in a target task node to which the target model training subnode belongs; wherein the first processing result is obtained by each first data calculation subnode included in the target task node performing a specified processing operation on the acquired first real-time data to be processed, and is stored in the target shared memory;

[0039] The model is trained using the read first processing result to obtain a trained target model.

[0040] Optionally, in a specific implementation, the real-time data computing system is a distributed real-time data computing system; the model training system is a distributed model training system; and the method further includes:

[0041] The target model is stored in the target shared memory, so that each first data computing sub-node included in the target task node obtains the target model from the target shared memory, and uses the target model to process the obtained second real-time data to be processed, obtains a second processing result, and stores the second processing result in the target shared memory.

[0042] Optionally, in a specific implementation, each task node is provided with a shared memory management server, and the method further includes:

[0043] Registering the model information of the target model into the target shared memory management service set by the target task node;

[0044] Obtaining a second reference address of the model information in the target shared memory management service, and sending the second reference address to each data computing sub-node included in the target task node; wherein the model information includes: the node identifier of the above-mentioned target model training sub-node, the storage address of the target model in the target shared memory and the variable value corresponding to the target model;

[0045] The storing the target model to the target shared memory so that each first data computing sub-node included in the target task node obtains the target model from the target shared memory includes:

[0046] According to the second memory protocol, the target model is stored in the target shared memory, so that each data computing sub-node included in the target task node reads the model information from the target shared memory management service according to the received second reference address, and reads the target model from the target shared memory according to the model information; wherein the model information includes: the protocol identifier of the second memory protocol.

[0047] Beneficial effects of the embodiments of the present invention:

[0048] By applying the solution provided by the embodiment of the present invention, a data processing system can be established based on the real-time data computing system and the model training system. The data processing system includes at least one task node, and each task node includes: at least one data computing sub-node in the real-time data computing system, and at least one model training sub-node in the model training system, and each task node may also include a shared memory.

[0049] Furthermore, during the operation of the data processing system, for each task node, each data computing sub-node in the task node can obtain the real-time data to be processed, and perform a specified processing operation on the obtained real-time data to obtain a processing result. Afterwards, the data computing sub-node can store the processing result in the shared memory of the task node. In this way, each model training sub-node in the task node can directly read the above processing result from the shared memory of the task node, and use the read processing result to perform model training to obtain a trained target model.

[0050] Based on this, in an embodiment of the present invention, the data computing subnode in the real-time data computing system and the model training subnode in the model training system can be deployed collaboratively on the same task node, that is, the integration of the real-time data computing system and the model training system can be realized. Furthermore, the processing results obtained by the data computing subnode processing the real-time data can be directly placed in the memory of the task node to which it belongs, and the model training subnode can directly read the above processing results from the memory of the task node to which it belongs for model training. In this way, the process of dropping the above processing results to the disk when transmitting between the data computing subnode and the model training subnode can be reduced, and the efficient transmission of the above processing results between the data computing subnode and the model training subnode can be achieved, thereby improving the timeliness of the processing results of real-time data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0052] Figure 1 A schematic diagram of the structure of a data processing system provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of the structure of another data processing system provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a specific example of a specific implementation method provided by an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a specific example of a specific implementation method provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0057] In the related technologies, due to the lack of integration between the real-time data computing system and the model training system, the processing results obtained by the real-time data computing system in real time need to be written to the disk before they can be read and used by the model training system, which makes the timeliness of the real-time data processing results poor. Based on this, how to realize the integration of the real-time data computing system and the model training system and improve the timeliness of the real-time data processing results is a problem that needs to be solved urgently.

[0058] In order to solve the above technical problems, an embodiment of the present invention provides a data processing system.

[0059] The data processing system can be applied to various application scenarios that require data processing timeliness and need to use the processing results of real-time data for model training, such as real-time recommendation application scenarios and various risk control scenarios.

[0060] Furthermore, in the above application scenarios, it is necessary to use a real-time data computing system to process real-time data to obtain processing results, and it is necessary to use a model training system to use the above processing results to perform model training to obtain a trained target model. Therefore, the data processing system is established based on various real-time data computing systems and various model training systems.

[0061] For example, the real-time data computing system can be: Flume (real-time log collection system), Spark Streaming (streaming data processing system), Storm (distributed real-time big data processing system), Flink (distributed streaming data stream processing system), etc., and the model training system can be: TensorFlow, PyTorch, Caffe (Convolutional Architecture for Fast Feature Embedding, convolutional neural network architecture), Ray, etc. Exemplarily, the above-mentioned data processing system can be established by combining the above-mentioned Flink and Ray.

[0062] It should be emphasized that the embodiments of the present invention do not limit the application scenarios of the data processing system, and the real-time data computing system and model training system used to establish the data processing system. Therefore, according to the requirements of the application scenarios in actual applications, a matching real-time data computing system and model training system can be selected to establish a data processing system provided by the embodiments of the present invention.

[0063] A data processing system provided by an embodiment of the present invention includes at least one task node, each task node includes: at least one data computing sub-node, at least one model training sub-node and a shared memory; wherein each data computing sub-node is a sub-node in a real-time data computing system, and each model training sub-node is a sub-node in a model training system;

[0064] Each data computing sub-node is used to perform a specified processing operation on the acquired first real-time data to be processed, obtain a first processing result, and store the first processing result in the shared memory;

[0065] Each model training sub-node is used to read the first processing result from the shared memory, and use the first processing result to perform model training to obtain a trained target model.

[0066] As can be seen above, by applying the solution provided by the embodiment of the present invention, a data processing system can be established based on the real-time data computing system and the model training system. The data processing system includes at least one task node, and each task node includes: at least one data computing sub-node in the real-time data computing system, and at least one model training sub-node in the model training system, and each task node may also include a shared memory.

[0067] Furthermore, during the operation of the data processing system, for each task node, each data computing sub-node in the task node can obtain the real-time data to be processed, and perform a specified processing operation on the obtained real-time data to obtain a processing result. Afterwards, the data computing sub-node can store the processing result in the shared memory of the task node. In this way, each model training sub-node in the task node can directly read the above processing result from the shared memory of the task node, and use the read processing result to perform model training to obtain a trained target model.

[0068] Based on this, in an embodiment of the present invention, the data computing subnode in the real-time data computing system and the model training subnode in the model training system can be deployed collaboratively on the same task node, that is, the integration of the real-time data computing system and the model training system can be realized. Furthermore, the processing results obtained by the data computing subnode processing the real-time data can be directly placed in the memory of the task node to which it belongs, and the model training subnode can directly read the above processing results from the memory of the task node to which it belongs for model training. In this way, the process of dropping the above processing results to the disk when transmitting between the data computing subnode and the model training subnode can be reduced, and the efficient transmission of the above processing results between the data computing subnode and the model training subnode can be achieved, thereby improving the timeliness of the processing results of real-time data.

[0069] A data processing system provided by an embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0070] Figure 1 A schematic diagram of the structure of a data processing system provided by an embodiment of the present invention. Figure 1 As shown, the data processing system includes at least one task node 110 , each task node includes at least one data computing sub-node 101 , at least one model training node 102 and a shared memory 103 .

[0071] Each data computing sub-node 101 is a data computing sub-node in the real-time data computing system, and each model training sub-node 102 is a model training sub-node in the model training system.

[0072] Optional, such as Figure 2 As shown, the real-time data computing system to which each of the above-mentioned data computing sub-nodes 101 belongs is a distributed real-time data computing system; correspondingly, the model training system to which each of the above-mentioned model training sub-nodes 102 belongs is a distributed model training system.

[0073] Among them, when both the distributed real-time data computing system and the distributed model training system have master nodes for system management, the master node in the distributed real-time data computing system and the master node in the distributed model training system can be set on the same task node in a data processing system provided by an embodiment of the present invention.

[0074] Furthermore, in an embodiment of the present invention, each task node can be understood as a data processing server, and the data computing sub-nodes 101 and model training nodes 102 included on each task node are data computing programs for real-time data processing and model training programs for model training, and the shared memory is the memory space determined in the data processing server for storing the processing results obtained by each data computing sub-node 101 for processing real-time data, so as to be read and used by each model training node 102.

[0075] That is to say, in the embodiment of the present invention, the data computing subnode 101 and the model training subnode 102 belonging to the real-time data computing system and the model training system are integrated into the same data processing server, that is, the two programs belonging to the real-time data computing system and the model training system, which are used for real-time data processing and model training respectively, are integrated into the same hardware device, and further in the memory of the hardware device, a memory space is determined for storing the processing results obtained by the program for real-time data processing on the real-time data, so that the program for model training can read the processing results from the memory space for model training. In this way, since the memory space determined above can be used by the data computing subnode 101 and the model training subnode 102 in common, the memory space determined above can be called shared memory.

[0076] Among them, for each task node 110, each data calculation sub-node 101 in the task node 110 can be used to: obtain the first real-time data to be processed, thereby performing a specified processing operation on the obtained first real-time data to be processed to obtain a first processing result.

[0077] For example, the first processing operation may be at least one of the following operations: missing value filling and cleaning, dimension table associated attribute value processing, data one-hot encoding, bucket processing, etc.; of course, the above operations are merely examples of the first processing operation, not limitations.

[0078] After obtaining the first processing result, each data computing sub-node 101 may store the first processing result in the shared memory 103 included in the task node 110 .

[0079] Furthermore, each model training sub-node 102 in the task node 110 can read the first processing results stored in the above-mentioned data calculation sub-nodes 101 from the shared memory 103 included in the task node 110, and use the read first processing results to perform model training to obtain a trained target model.

[0080] Among them, each model training sub-node 102 may pre-store a specified algorithm, so that after reading the above-mentioned first processing result, the above-mentioned first processing result can be used to train the above-mentioned specified algorithm, thereby obtaining a trained model.

[0081] Optionally, each model training sub-node 102 in the task node 110 can communicate with a business system used to perform various data processing tasks. Thus, after obtaining the trained model, each model training sub-node 102 can directly send each model to the business system, so that the business system can use the received models to complete the corresponding data processing tasks.

[0082] It can be understood that in many cases, for each task node, each data computing sub-node 101 needs to utilize the model trained by each model training sub-node 102 to implement when processing the acquired real-time data.

[0083] Therefore, in order to ensure that each data computing sub-node 101 can efficiently obtain the model trained by each model training sub-node 102, after each model training sub-node 102 obtains the target model through training, it can also store the target model in the shared memory of the task node for acquisition and use by each data computing sub-node 101 in the task node.

[0084] Based on this, optionally, in a specific implementation,

[0085] Each model training sub-node 102 is also used to store the target model in the shared memory 103;

[0086] Each data computing sub-node 101 is further used to obtain a target model from the shared memory 103 , and use the target model to process the obtained second real-time data to be processed to obtain a second processing result, and store the second processing result in the shared memory 103 .

[0087] In this specific implementation, for each task node 110, each model training subnode 102 in the task node 110 can also store the target model in the shared memory 103 of the task node 110 after training. In this way, when each data calculation subnode 101 obtains the second real-time data to be processed, it can obtain the above target model from the shared memory 103 of the task node 110, so that the obtained second real-time data to be processed can be processed using the obtained target model to obtain the second processing result.

[0088] Furthermore, considering that each model training sub-node 102 in the task node 110 may need to use the above-mentioned second processing result to perform model training, each data calculation sub-node 101 can continue to store the above-mentioned second processing result in the shared memory 103 of the task node 110.

[0089] That is to say, in this specific implementation, for each task node 110, each data calculation sub-node 101 and each model training sub-node 102 in the task node 110 can realize information interaction.

[0090] Among them, for each task node 110, each data calculation sub-node 101 in the task node 110 can obtain the target model trained by each model training sub-node 102 in the task node 110 from the shared memory 103 of the task node 110, and use the target model to process the acquired real-time data, and then store the processing result in the shared memory 103 of the task node 110 again; and each model training sub-node 102 in the task node 110 can obtain the processing result processed by each data calculation sub-node 101 in the task node 110 from the shared memory 103 of the task node 110, and use the processing result to perform model training, and then store the trained target model in the shared memory 103 of the task node 110 again.

[0091] By repeating this cycle, efficient information output between each data calculation sub-node 101 and each model training sub-node 102 in the task node 110 can be achieved, thereby improving the efficiency of real-time data processing and model training.

[0092] Moreover, from a system perspective, since the above-mentioned processing results and target models are only stored in the shared memory 103, there is no need to occupy each data calculation sub-node 101 to store the above-mentioned target model, and there is no need to occupy the memory of each model training sub-node 102 to store the above-mentioned processing results. Therefore, the memory waste caused by one piece of data occupying two pieces of memory can be reduced.

[0093] Based on this, in an embodiment of the present invention, the data computing subnode in the real-time data computing system and the model training subnode in the model training system can be deployed collaboratively on the same task node, that is, the integration of the real-time data computing system and the model training system can be realized. Furthermore, the processing results obtained by the data computing subnode processing the real-time data can be directly placed in the memory of the task node to which it belongs, and the model training subnode can directly read the above processing results from the memory of the task node to which it belongs for model training. In this way, the process of dropping the above processing results to the disk when transmitting between the data computing subnode and the model training subnode can be reduced, and the efficient transmission of the above processing results between the data computing subnode and the model training subnode can be achieved, thereby improving the timeliness of the processing results of real-time data.

[0094] Optionally, in a specific implementation, a shared memory management service may be provided in each task node 110. The so-called shared memory management service may be understood as a class description, for example, it may be a section of Java or C++ code executed in a process, or it may be a section of code attached to each data computing sub-node 101 or each model training sub-node 102. The shared memory management service may provide a storage area for storing information of various types of data in the shared memory 103, for example, data information of the first processing result, model information of the target model, etc. Exemplarily, the data information of the first processing result may include: variable value, memory physical address, occupied size, etc. In addition, the shared memory service needs to provide an external access port so that other processes or internal threads can access it.

[0095] Furthermore, in this specific implementation, for each task node 110, each data computing sub-node 101 in the task node 110 can also be used to perform the following steps 11-12:

[0096] Step 11: registering the data information of the first processing result into the shared memory management service;

[0097] Step 12: Obtain the first reference address of the data information in the shared memory management service, and send the first reference address to each model training sub-node;

[0098] The data information includes: a node identifier of the data calculation subnode, a storage address of the first processing result in the shared memory, and a variable value corresponding to the first processing result.

[0099] In this specific implementation, after each data computing sub-node 101 in the task node 110 stores the obtained first processing result in the shared memory 103 in the task node 110 , the data information of the first processing result can be determined.

[0100] Among them, the data information of the first processing result can be used to uniquely determine the above-mentioned first processing result from the shared memory 103 in the task node 110, that is, the data information uniquely corresponds to the first processing result, and thus, the above-mentioned first processing result can be read from the shared memory 103 in the task node 110 according to the data information.

[0101] Based on this, the data information may include: the node identifier of the data calculation subnode 101, the storage address of the first processing result in the shared memory 103 of the task node 110, and the variable value corresponding to the first processing result. The variable value corresponding to the first processing result refers to: the reference variable of the first processing result. Optionally, the structure of the above variable value can be: the node identifier of the data calculation subnode that generates the first processing result & the storage address of the first processing result in the shared memory of the task node & the data size of the first processing result. Exemplary: Node 2 & 0xffffffff & 20kb (Kilobyte).

[0102] After determining the data information of the first processing result, each data computing sub-node 101 in the task node 110 may register the data information in the shared memory management service.

[0103] For example, if the shared memory management service is a piece of Java code executed in the process, the data information of the above-mentioned first processing result can be packaged into a data packet, and the variable value corresponding to the data packet can be set, so that the variable value is written into the JAVA code, and the data packet is stored in the designated storage location in the task node 110.

[0104] Furthermore, after completing the registration of the data information of the above-mentioned first processing result, each data calculation sub-node 101 in the task node 110 can obtain the first reference address of the data information of the above-mentioned first processing result in the shared memory management service, and send the first reference address to each model training sub-node 102 in the task node 110.

[0105] The first reference address is used to determine the data information of the first processing result in the shared memory management service, that is, the first reference address can be understood as: the storage address of the data information of the first processing result in the shared memory management service. In addition, the first reference address can uniquely determine the data information of the first processing result in the shared memory management service, that is, the first reference address uniquely corresponds to the data information of the first processing result, so that the data information of the first processing result can be determined from the shared memory management service according to the first reference address.

[0106] The first reference address may be a variable. The variable includes: the node identifier of the data computing subnode that generates the first processing result & the storage address of the data information of the first processing result in the shared memory management service & the data size of the data information of the first processing result. For example: node 1 & 0xffffffff & 5kb.

[0107] Optionally, each data computing sub-node 101 in the task node 110 may use a custom function to send the first reference address to each model training sub-node 102 in the task node 110 .

[0108] Based on this, in this specific implementation, each of the above-mentioned model training sub-nodes reads the first processing result from the shared memory 103, which may include the following step 13:

[0109] Step 13: Each model training sub-node 120 reads data information from the shared memory management service according to the received first reference address, and reads the first processing result from the shared memory 103 according to the data information.

[0110] Since the above-mentioned first reference address uniquely corresponds to the data information of the above-mentioned first processing result, each model training sub-node 102 in the task node 110 can read the data information of the above-mentioned first processing result from the shared memory management service of the task node 110 according to the received first reference address; further, since the data information of the above-mentioned first processing result uniquely corresponds to the first processing result, each model training sub-node 102 in the task node 110 can read the above-mentioned first processing result from the shared memory 103 of the task node 110 according to the data information of the above-mentioned first processing result.

[0111] For example, Figure 3 The figure shows a schematic diagram of a specific example of this specific implementation method.

[0112] Furthermore, when storing the first processing result in the shared memory 103 , each first data computing sub-node 101 may use different memory protocols to convert the first processing result into different formats for storage.

[0113] For example, the Arrow memory protocol can be used to convert the first processing result into the Arrow listing data format for storage. The Arrow memory protocol can be regarded as a universal memory protocol, thereby providing a standard data exchange format, realizing seamless linking of data between different systems, and saving the time required for data format conversion during data transmission between different systems.

[0114] Based on this, optionally, in a specific implementation, each data computing sub-node 101 stores the first processing result in the shared memory 103, which may include the following step 14:

[0115] Step 14: Each data computing sub-node stores the first processing result in the shared memory according to the first memory protocol;

[0116] The data information includes: a protocol identifier of the first memory protocol.

[0117] In this specific implementation, each data computing sub-node 101 can store the first processing result in the shared memory 103 according to the preset first memory protocol. In addition, in order to ensure that each model training sub-node 102 can smoothly read the first processing result, each data computing sub-node 101 needs to inform each model training sub-node 102 of the first memory protocol adopted.

[0118] Based on this, the protocol identifier of the first memory protocol adopted above can be added to the data information of the above-mentioned first processing result, so that each model training sub-node 102 can obtain the first memory protocol adopted above from the data information after obtaining the data information of the above-mentioned first processing result.

[0119] Of course, optionally, each data computing sub-node 101 can also directly send a notification about the first memory protocol to each model training sub-node 102 to inform each model training sub-node 102 that the above-mentioned first processing result is stored in the shared memory 103 in the format corresponding to the first memory protocol.

[0120] Similar to the above specific implementation, when each model training sub-node 102 stores the trained target model in the shared memory 103, each model training byte 102 needs to inform each data calculation sub-node 101 of the relevant information about the stored target model.

[0121] Based on this, optionally, in a specific implementation, each task node 110 is provided with a shared memory management service;

[0122] Furthermore, in this specific implementation, for each task node 110, each model training sub-node 102 in the task node 110 can also be used to perform the following steps 21-22:

[0123] Step 21: Register the model information of the target model into the shared memory management service;

[0124] Step 22: Obtain a second reference address of the model information in the shared memory management service, and send the second reference address to each data computing sub-node;

[0125] The model information includes: the node identifier of the model training subnode, the storage address of the target model in the shared memory, and the variable value corresponding to the target model;

[0126] Accordingly, in this specific implementation, each data computing sub-node 101 obtains the target model from the shared memory 103, which may include the following step 23:

[0127] Step 23: Each data computing sub-node 101 reads the model information from the shared memory management service according to the received second reference address, and reads the target model from the shared memory 103 according to the model information.

[0128] Optionally, in a specific implementation, each model training sub-node 102 stores the target model in the shared memory 103, which may include the following step 24:

[0129] Step 24: Each model training sub-node stores the target model in the shared memory according to the second memory protocol;

[0130] The model information includes: a protocol identifier of the second memory protocol.

[0131] It should be noted that the specific implementation of the above steps 21-24 is similar to the specific implementation of the above steps 11-14, and will not be repeated here.

[0132] In many cases, the task node 110 may fail, resulting in a crash. To avoid data reuse after the task node 110 is restarted, the data used by each data computing sub-node 101 and model training sub-node 102 in the task node 110 may be state managed to achieve exactly-once semantics of the data.

[0133] Based on this, optionally, in a specific implementation, each task node 110 is provided with a state management service, and each data processing sub-node 101 can also be used to perform the following steps 31-33:

[0134] Step 31: Obtain first real-time data to be processed from a message queue of at least one data source;

[0135] Step 32: inserting a specified identifier into each message queue according to a preset period;

[0136] The designated identifier includes: the partition information of the message queue and the offset address of the currently read data in the message queue;

[0137] Step 33: In the state management service, record the data state corresponding to each message queue determined based on each designated identifier;

[0138] The data state corresponding to each message queue is used to represent: the offset address in the message queue of the data obtained by each data computing node from the message queue.

[0139] In this specific implementation, each data processing sub-node 101 can obtain the first real-time data to be processed from the message queue of at least one data source; thus, a specified identifier can be inserted into each message queue used to read the first real-time data to be processed according to a preset period and the duration corresponding to each preset period.

[0140] Among them, the designated identifier inserted in each message queue includes: the partition information of the message queue and the offset address of the currently read data in the message queue. Among them, since a message queue can have multiple partitions, it is necessary to determine the message queue partition where the currently read data is located, and the offset address of the currently read data in the message queue partition. In other words, the partition information of the message queue included in the above-mentioned designated identifier is: the information of the message queue partition to which the currently read data belongs, and the offset address of the currently read data in the message queue included in the above-mentioned designated identifier is: the offset address of the currently read data in the message queue partition to which it belongs. For example, the designated identifier can be: Barrier ID.

[0141] In this way, according to the designated identifier inserted in each message queue, each data processing sub-node 101 can determine which data has been read by itself and which data has not been read by itself in the message queue in which it is reading data. Furthermore, the data status corresponding to each message queue can be determined by each designated identifier, and the data status corresponding to each message queue determined based on each designated identifier is recorded in the set state management service, and the data status is used to characterize: which data has been read and which data has not been read in the message queue. Among them, the data status may include: the node identifier of the data processing sub-node, the partition information of the message queue, the offset address of the currently read data in the message queue, the Barrier ID, the statistical information of the data processing sub-node and other related status information; for example, the statistical information of the data processing sub-node may include sum information or count information, and other related status information may include bucket information, etc. This is all reasonable.

[0142] The so-called recording of the data status corresponding to each message queue is: persisting the data status corresponding to each message queue determined above, and marking the designated identifier corresponding to this persistence.

[0143] Based on this, optionally, in a specific implementation, on the basis of the above steps 31-33, after each data computing sub-node 101 crashes and restarts, each data computing sub-node 101 can also be used to perform the following step 34:

[0144] Step 34: Obtain the data status corresponding to each message queue recorded in the status management service;

[0145] Accordingly, in this specific implementation, each data computing sub-node obtains the first real-time data to be processed from the message queue of at least one data source, which may include the following step 311:

[0146] Step 311: Each data calculation sub-node starts from the offset address represented by each acquired data state and acquires the first real-time data to be processed from each message queue.

[0147] In this specific implementation, after each data computing sub-node 101 crashes and restarts, in order to implement the exactly-once semantics of the data, each data computing sub-node 101 can first obtain the data status corresponding to each message queue recorded in the status management service, thereby determining the offset address of each partition of each message queue in which the data computing node 101 obtains the data from each message queue, that is, determining which data in each message queue has been read and which data has not been read.

[0148] In this way, for each message queue, in order to avoid repeatedly reading the data in the message queue, each data calculation sub-node 101 can obtain the first real-time data to be processed from the message queue in the message queue partition represented by the data state corresponding to the message queue, starting from the offset address represented by the data state corresponding to the message queue.

[0149] That is to say, each data computing sub-node 101 can first determine the message queue partition where the data last read before the crash is located, and the offset address of the data last read before the crash in the message queue partition according to the data status corresponding to each message queue. Therefore, after the crash is restarted, each data computing sub-node 101 can obtain the first real-time data to be processed from the message queue partition determined above, starting from the offset address determined above.

[0150] In this way, for each message queue, each data computing sub-node 101 can start from the first data in the unread data in the message queue, and sequentially obtain each data that has not been read before the shutdown and restart, so as to obtain the first real-time data to be processed.

[0151] Furthermore, when there are multiple data sources, each data computing sub-node 101 may obtain the first real-time data to be processed from the message queues of the multiple data sources.

[0152] In this way, each message queue can be stream-aligned, that is, each message queue has temporal correlation, and thus each message queue needs to be aligned based on the inserted designated identifiers. The designated identifiers inserted in each message queue all reach the offset position of the data corresponding to the same time point in the message queue.

[0153] Based on this, optionally, in a specific implementation, before each data processing sub-node 101 performs a specified processing operation on the acquired first real-time data to be processed, it can also be used to perform the following step 35:

[0154] Step 35: Determine whether the designated identifiers inserted in the message queue of each data source are aligned; if aligned, perform a designated processing operation on the acquired first real-time data to be processed.

[0155] In this specific implementation, after acquiring the first real-time data to be processed, each data processing sub-node 101 may first determine whether the designated identifiers inserted in the message queue of each data source are aligned.

[0156] When the result is determined to be aligned, it means that each message queue is aligned based on the above-mentioned designated identifiers, so that each data processing sub-node 101 can perform a designated processing operation on the acquired first real-time data to be processed.

[0157] When the result is determined to be misaligned, it means that the message queues are not aligned based on the specified identifier inserted above, and thus, each data processing sub-node 101 does not perform the specified processing operation on the acquired first real-time data to be processed, but waits for the alignment of the specified identifier inserted in the message queue of each data source before performing the specified processing operation on the acquired first real-time data to be processed.

[0158] Among them, the so-called waiting for the alignment of the specified identifier inserted in the message queue of each data source means: for the message queue corresponding to the later time of the inserted specified identifier, you can stop reading data from the message queue, and continue to read data from the message queue corresponding to the earlier time of the inserted specified identifier, until the time corresponding to the specified identifier inserted in the message queue is the same as the time corresponding to the later specified identifier mentioned above, and then perform the specified processing operation on the first real-time data to be processed obtained.

[0159] For example, Figure 4As shown, the computing worker is a data computing sub-node, the training worker is a model training sub-node, and the two consumption data queues are message queues corresponding to the two data sources used by the computing worker to obtain the first real-time data to be processed; the state management is a state management service set for a task node, and the feature engineering is the first processing operation performed by the computing worker on the obtained first real-time data to be processed.

[0160] In this way, when the computing worker obtains the first real-time data to be processed from the two consumer data queues and performs the first processing operation on the obtained first real-time data to be processed, it is necessary to wait for the two Barrier IDs in the two consumer data queues to arrive at the offset position of the data corresponding to the same time point in the message queue, and persist the data status corresponding to the two consumer data queues. The so-called persistence means: storing the data status corresponding to the two consumer data queues into a file, and the file can be stored in a distributed storage system.

[0161] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0162] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0163] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data processing system, characterized in that: The system includes at least one task node, each task node includes: at least one data computing subnode, at least one model training subnode and a shared memory; wherein each data computing subnode is a subnode in a real-time data computing system, and each model training subnode is a subnode in a model training system; Each data computing sub-node is used to perform a specified processing operation on the acquired first real-time data to be processed, obtain a first processing result, and store the first processing result in the shared memory; Each model training sub-node is used to read the first processing result from the shared memory, and use the first processing result to perform model training to obtain a trained target model; Each task node is equipped with a shared memory management service; each data computing sub-node is also used for: Registering the data information of the first processing result into the shared memory management service; Obtaining a first reference address of the data information in the shared memory management service, and sending the first reference address to each model training sub-node; The data information includes: a node identifier of the data calculation subnode, a storage address of the first processing result in the shared memory, and a variable value corresponding to the first processing result; Each model training sub-node reads the first processing result from the shared memory, including: Each model training sub-node reads the data information from the shared memory management service according to the received first reference address, and reads the first processing result from the shared memory according to the data information.

2. The system according to claim 1, characterized in that Each model training sub-node is also used to store the target model in the shared memory; Each data computing sub-node is also used to obtain the target model from the shared memory, and use the target model to process the obtained second real-time data to be processed to obtain a second processing result, and store the second processing result in the shared memory.

3. The system according to claim 1, characterized in that Each data computing sub-node stores the first processing result in the shared memory, including: Each data computing sub-node stores the first processing result in the shared memory according to the first memory protocol; wherein the data information includes: a protocol identifier of the first memory protocol.

4. The system according to claim 2, characterized in that Each task node is equipped with a shared memory management service; each model training sub-node is also used for: Registering the model information of the target model into the shared memory management service; Acquire a second reference address of the model information in the shared memory management service, and send the second reference address to each data computing sub-node; The model information includes: a node identifier of the model training subnode, a storage address of the target model in the shared memory, and a variable value corresponding to the target model; Each data computing sub-node obtains the target model from the shared memory, including: Each data computing sub-node reads the model information from the shared memory management service according to the received second reference address, and reads the target model from the shared memory according to the model information; Each model training sub-node stores the target model in the shared memory, including: Each model training sub-node stores the target model in the shared memory according to the second memory protocol; wherein the model information includes: a protocol identifier of the second memory protocol.

5. The system according to claim 1 or 2, characterized in that: The real-time data computing system is a distributed real-time data computing system; the model training system is a distributed model training system; each task node is provided with a state management service, and each data processing sub-node is also used for: Acquire the first real-time data to be processed from a message queue of at least one data source; Inserting a designated identifier into each message queue according to a preset cycle; wherein the designated identifier includes: partition information of the message queue and an offset address of the currently read data in the message queue; In the state management service, the data state corresponding to each message queue determined based on each designated identifier is recorded; wherein the data state corresponding to each message queue is used to characterize: the offset address in the message queue of the data obtained by each data computing node from the message queue.

6. The system according to claim 5, characterized in that The number of the data sources is multiple; before each data processing sub-node performs the specified processing operation on the acquired first real-time data to be processed, each data calculation sub-node is also used to: Determine whether the specified identifiers inserted in the message queue of each data source are aligned; If aligned, a specified processing operation is performed on the first acquired real-time data to be processed.

7. The system according to claim 5, characterized in that After each data computing sub-node crashes and restarts, each data computing sub-node is also used to: Obtaining the data status corresponding to each message queue recorded in the status management service; Each data computing sub-node obtains the first to-be-processed real-time data from a message queue of at least one data source, including: Each data calculation sub-node starts from the offset address represented by each acquired data state and acquires the first real-time data to be processed from each message queue.

8. A model training method, characterized in that: The method is applied to any target model training subnode in any target task node of a data processing system; wherein the data processing system comprises: at least one task node, each task node comprises: at least one data computing subnode, at least one model training subnode and a shared memory, each data computing subnode is a subnode in a real-time data computing system, and each model training subnode is a subnode in a model training system; the method comprises: Reading a first processing result from a target shared memory included in a target task node to which the target model training subnode belongs; wherein the first processing result is obtained by each first data calculation subnode included in the target task node performing a specified processing operation on the acquired first real-time data to be processed, and is stored in the target shared memory; Performing model training using the read first processing result to obtain a trained target model; Each task node is provided with a shared memory management service; each data computing sub-node included in the target task node registers the data information of the first processing result into the target shared memory management service set by the target task node; obtains the first reference address of the data information in the target shared memory management service, and sends the first reference address to each model training sub-node; wherein the data information includes: the node identifier of the data computing sub-node, the storage address of the first processing result in the target shared memory and the variable value corresponding to the first processing result; Reading a first processing result from a target shared memory included in a target task node to which the target model training subnode belongs includes: The data information is read from the target shared memory management service according to the received first reference address, and the first processing result is read from the target shared memory according to the data information.

9. The method according to claim 8, characterized in that The real-time data computing system is a distributed real-time data computing system; the model training system is a distributed model training system; the method further includes: The target model is stored in the target shared memory, so that each first data computing sub-node included in the target task node obtains the target model from the target shared memory, and uses the target model to process the obtained second real-time data to be processed, obtains a second processing result, and stores the second processing result in the target shared memory.

10. The method according to claim 9, characterized in that Each task node is provided with a shared memory management server, and the method further comprises: Registering the model information of the target model into the target shared memory management service set by the target task node; Obtaining a second reference address of the model information in the target shared memory management service, and sending the second reference address to each data computing sub-node included in the target task node; wherein the model information includes: the node identifier of the above-mentioned target model training sub-node, the storage address of the target model in the target shared memory and the variable value corresponding to the target model; The storing the target model to the target shared memory so that each first data computing sub-node included in the target task node obtains the target model from the target shared memory includes: According to the second memory protocol, the target model is stored in the target shared memory, so that each data computing sub-node included in the target task node reads the model information from the target shared memory management service according to the received second reference address, and reads the target model from the target shared memory according to the model information; wherein the model information includes: the protocol identifier of the second memory protocol.

Images