A model training system

Abstract

An embodiment of the present invention discloses a model training system, which includes a central service node and multiple device nodes. The central service node is used to obtain device information of each device node, determine each training subtask, the device node corresponding to each training subtask, and the model training data to be trained based on the device information and pre-determined model information; send each training subtask and the model training data to the device node corresponding to each training subtask; each device node is used to receive other device training data sent by other device nodes other than the node, execute the training subtask based on the pre-acquired local training data, device training data, model training data to be trained, and model offline package, obtain its own device training data, and send its own device training data to the central service node and each device node other than the node at preset time intervals. The present invention achieves efficient resource utilization and improves model training efficiency.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a model training system. Background Art

[0002] With the rapid development of deep learning technology, various deep learning models requiring training are widely used in fields such as natural language processing and computer vision. However, the scale and complexity of these models are also increasing. As a result, traditional single-machine training methods are no longer able to meet these training requirements. To address this issue, distributed training methods have emerged. By distributing models across multiple devices for parallel processing, they effectively improve the efficiency of model training and inference.

[0003] Existing distributed model training methods typically involve a central server partitioning the training data and model. Each device processes a batch of data and uses this data to train the partitioned model components. Ultimately, the final model is derived from the trained component models. While these methods address the training of large models to some extent, distributing model slices across different devices requires complex coordination and model synchronization mechanisms, increasing system complexity and implementation difficulty. Furthermore, the models trained on each device in this approach rely solely on the limited dataset provided by the central server, resulting in low generalization and data processing accuracy. Summary of the Invention

[0004] The present invention provides a model training system that eliminates the need for model slicing, reducing system complexity. The model can capture richer data features during training, improving the model's generalization capabilities and data processing accuracy.

[0005] In a first aspect, an embodiment of the present invention provides a model training system, comprising a central service node and multiple device nodes, wherein the central service node is connected to each device node, and each device node has a model offline package deployed thereon; the model offline package is predetermined based on the device resources of each device node;

[0006] The central service node is used to obtain device information of each device node, determine each training subtask, the device node corresponding to each training subtask, and the model data to be trained based on the device information and the predetermined model information; and send each training subtask and the model data to be trained to the device node corresponding to each training subtask;

[0007] Each device node is configured to receive other device training data sent by each device node other than the current node, perform the training subtask based on the pre-acquired local training data, the other device training data, the model data to be trained, and the model offline package, obtain its own device training data, and send the own device training data to the central service node and each device node other than the current node at a preset time interval;

[0008] The central service node is further configured to generate target model parameters based on the training data of each of the devices itself, and send the target model parameters to each of the device nodes, so that each of the device nodes obtains a target model based on the target model parameters.

[0009] Furthermore, each device node is specifically used to obtain the device training resources at the current moment, determine the required resources of the training subtask, determine the training strategy of the training subtask based on the device training resources and the required resources, and execute the training subtask based on the training strategy.

[0010] Furthermore, the multiple device nodes also include at least one candidate node, and each candidate node is specifically used to send a join training instruction to the central service node, and receive other device training data of other device nodes except the candidate node sent by the central service node based on the join training instruction, determine the training strategy and training subtask of the candidate node according to the other device training data, and execute the training subtask based on the training strategy.

[0011] Furthermore, each device node is specifically used to determine the model training progress corresponding to each time interval based on the other device training data and the own device training data, and update the training strategy based on the model training progress, the other device training data and the own device training data.

[0012] Furthermore, each device node is specifically used to determine whether there is a data conflict between its own device training data and the training data of other devices. If there is a data conflict between its own device training data and the training data of other devices, the training data of its own device and the training data of other devices are sent to the central service node, and the conflict-free training data after conflict processing sent by the service node is received.

[0013] Furthermore, the self-device training data and the other-device training data include data labels, update parameters and output results, and each device node is specifically used to perform consistency detection on the data labels, the output results and the update parameters of the self-device training data and the other-device training data, respectively, to obtain label consistency detection results, parameter consistency detection results and output consistency detection results; if the parameter consistency detection result is inconsistent or the label consistency detection result is inconsistent or the output consistency detection result is inconsistent, it is determined that there is a data conflict between the self-device training data and the other-device training data.

[0014] Furthermore, the central service node includes a pre-established distributed database, which includes multiple storage nodes; the central service node is specifically used to obtain identification information of the own device training data sent by each device node, perform consistent hash calculation on the own device training data based on the identification information, and determine the target storage node corresponding to the own device training data according to the calculation result, and store the own device training data to the target storage node.

[0015] Furthermore, each device node is also used to determine the data index of the requested data based on the training subtask, and search for the requested data in the local cache based on the data index. If the requested data is not found in the local cache, a data request instruction is generated based on the data index, the data request instruction is sent to the central service node, and the requested data returned by the central service node is received, and the requested data is stored in the local cache.

[0016] Furthermore, the central service node is also used to receive the data request instruction sent by the device node, perform the consistent hash calculation on the data index of the data request instruction, and obtain the requested storage node; determine the requested data through the requested storage node, and return the requested data to the device node.

[0017] Furthermore, the central service node is also used to update the training subtask and the device node corresponding to the training subtask based on the self-device training data of each device node and the device information of each device node, and send the updated training subtask to the device node corresponding to the training subtask.

[0018] Furthermore, each device node is also used to receive a model usage instruction sent by a user, obtain model output information based on the model usage instruction and the target model, and display the model output information to the user.

[0019] In an embodiment of the present invention, the system includes a central service node and multiple device nodes, the central service node is connected to each device node, and a model offline package is deployed on each device node; the model offline package is predetermined based on the device resources of each device node; the central service node is used to obtain device information of each device node, determine each training subtask, the device node corresponding to each training subtask, and the model data to be trained based on each device information and predetermined model information; send each training subtask and the model data to be trained to the device node corresponding to each training subtask; each device node is used to receive other device training data sent by each device node other than the node, execute the training subtask based on the pre-acquired local training data, device training data, model data to be trained, and model offline package, obtain its own device training data, and send its own device training data to the central service node and each device node other than the node at preset time intervals; the central service node is also used to generate target model parameters based on each device training data, and send the target model parameters to each device node, so that each device node obtains the target model based on the target model parameters. In the system of the embodiment of the present invention, the model offline package is determined based on the device resources, ensuring that each device node can effectively train the model within its capabilities. The training subtasks are distributed through the central service node, so that the training subtasks are evenly distributed among the device nodes, avoiding the situation where some nodes are overloaded while other nodes are idle. Multiple device nodes can process different training subtasks in parallel and quickly send training data to the central service node, which accelerates the update iteration process of the model parameters and significantly improves the speed of model training. The device node gives priority to using local training data, reducing dependence on the central service node or other device nodes. At the same time, the target model obtained will be more personalized and can better adapt to different users and scenarios. By integrating the training results from different device nodes, the model can learn a wider range of data features, thereby improving generalization capabilities. That is, the system of the embodiment of the present invention achieves efficient utilization of resources, while improving model training efficiency and enhancing model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A first structural diagram of a model training system provided by an embodiment of the present invention;

[0022] Figure 2A schematic diagram of merging non-conflicting device training data provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram illustrating that there is no conflict in device training data provided by an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of sharing conflict-free device training data provided by an embodiment of the present invention;

[0025] Figure 5 A first schematic diagram showing conflicting device training data provided by an embodiment of the present invention;

[0026] Figure 6 A first schematic diagram of merging conflicting device training data provided by an embodiment of the present invention;

[0027] Figure 7 A second schematic diagram showing a conflict in device training data provided by an embodiment of the present invention;

[0028] Figure 8 A second schematic diagram of merging conflicting device training data provided by an embodiment of the present invention;

[0029] Figure 9 A second structural diagram of a model training system provided by an embodiment of the present invention;

[0030] Figure 10 This is a third structural diagram of a model training system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0032] Figure 1 This is a first structural diagram of a model training system provided by an embodiment of the present invention. The embodiment of the present invention can reduce the complexity of the system, improve the generalization ability of the model and the accuracy of data processing. The information collected in the system of the embodiment of the present invention is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse. Reference Figure 1 , the model training system provided in the embodiment of the present invention includes:

[0033] A central service node 110 and multiple device nodes 120; the central service node 110 is connected to each device node 120, and a model offline package is deployed on each device node 120; the model offline package is predetermined based on the device resources of each device node 120; the central service node 110 is used to obtain device information of each device node 120, and determine each training subtask, the device node 120 corresponding to each training subtask and the model data to be trained based on each device information and predetermined model information; each training subtask and the model data to be trained are sent to the device node 120 corresponding to each training subtask; each device node 120 is used to receive other device training data sent by each device node other than the current node, execute training subtasks based on the pre-acquired local training data, other device training data, model data to be trained and model offline package, obtain its own device training data, and send its own device training data to the central service node 110 and each device node other than the current node at preset time intervals;

[0034] The central service node 110 is further configured to generate target model parameters based on the training data of each device, and send the target model parameters to each device node 120 , so that each device node 120 obtains a target model based on the target model parameters.

[0035] The central service node 110 is connected to each device node 120. Each device node 120 is a mobile device used to perform model training tasks, such as a mobile phone, tablet computer, or laptop computer. In this solution, the central service node 110 can be located on the same device as a device node, or it can be independent of each device node 120 and located on a separate device. A model offline package is deployed on the central service node 110 and each device node 120, and each model offline package is determined based on the device resources of each device node 120. Device resources refer to the hardware and software resources possessed by a device, such as the processor, central processing unit (CPU), graphics processing unit (GPU), memory, running memory, storage space for long-term data storage, operating system, software environment, battery, and heat dissipation capacity. Different devices have different device resource conditions, and therefore, different model offline packages are also used. For example, lightweight model offline packages can be deployed on resource-constrained device nodes (such as mobile phones and tablet computers). A lightweight model offline package is a model offline package that is lightweighted by processing the original model, removing unnecessary redundant functions and data structures. The original and more complex model offline packages can be deployed on device nodes with more abundant resources (such as laptops).

[0036] After each device node 120 and the central service node 110 have deployed the model offline package, they can connect to each other via device networking. Device networking is a network system that connects multiple devices via a wireless or wired network. Device networking enables communication and collaboration between each device node 120 and the central service node 110. For example, after each device node 120 and the central service node 110 have deployed the model offline package, if a user is using a mobile device such as a mobile phone, they can open the application responsible for deploying the model offline package on the device node 120 and join the device network by entering a group password or scanning a group QR code. The application can automatically search for nearby devices in the same group that have the same application running and are in a connectable state (device nodes connected to the same device network). The application automatically searches for nearby devices in the same group that have the same application running and are in a connectable state. If other devices in the same group are found, the application can prioritize establishing a point-to-point connection with the other device in the same group, for example, via Bluetooth or a wireless network, to directly transmit networking information and data between the devices. If a point-to-point connection fails, the application can connect through the central service node 110, which allocates a connection path and coordinates data transmission. If the user is using a device such as a laptop, they can choose to connect via a wireless or wired network. Once connected, the application on the laptop performs a handshake communication with other devices in the same group. In this solution, network security protocols can be used to encrypt network communication data to prevent data theft and tampering during transmission. Furthermore, the application can monitor the network connection status in real time. If a network interruption occurs, the application can automatically attempt to reconnect, ensuring network stability.

[0037] After the central service node 110 successfully connects to each device node 120, the central service node 110 can obtain the device information of each device node 120 to determine each training subtask, the device node 120 corresponding to each training subtask, and the model training data based on the device information and model information. Device information includes the device node identifier, device resource information, and device status information. The device node identifier uniquely identifies the device node, and device status information includes the device's operating status and operating mode. Model information includes model structure information, model parameter information, training configuration information, data requirements, and resource requirements. Structure information includes the model type, such as convolutional neural network or recurrent neural network. Model structure information also includes the number of layers, the number of nodes per layer, and the activation function type. Model parameter information includes the model's initial parameter values ​​or weights, as well as parameter update rules and learning rates. Training configuration information includes the optimization algorithm and regularization method used during model training. Data requirements include the training data format and preprocessing requirements, as well as the data batch size required for each training cycle. Resource requirement information includes device resource requirements and network bandwidth requirements during model training. Training subtasks are smaller, manageable subtasks that the central service node 110 breaks down the entire model training task into. Model training data is pre-determined by the central service node 110 and is applicable to all device nodes 120 training models.

[0038] Specifically, after the central service node 110 establishes a connection with each device node 120, the central service node 110 can obtain the device information of each device node 120. Simultaneously, the central service node 110 organizes and analyzes the model information and decomposes the model training task based on the model structure and training algorithm, obtaining multiple training subtasks. For example, for a deep neural network model, different network layers can be used as different training subtasks. For example, the training process of the convolutional layer can be used as a training subtask, and the training process of the fully connected layer can be used as a training subtask. At the same time, the model training data can be divided according to each training subtask to obtain the model training data corresponding to each training subtask. After determining each training subtask, the central service node 110 determines the training subtask suitable for each device node 120 based on the device resources required by each training subtask and the device information of each device node 120. For example, the training subtask suitable for each device node 120 can be determined based on its CPU performance, GPU performance, and memory capacity. For example, computationally intensive convolutional layer training tasks can be assigned to higher-configuration device nodes (such as laptops), while simple data preprocessing and subsequent data optimization tasks can be assigned to lower-configuration device nodes (such as mobile phones). After determining each training subtask and its corresponding device node 120, the training subtask and the model training data are sent to the corresponding device node 120 based on the identifier of each device node.

[0039] After receiving the training subtask, device node 120 can obtain local training data and begin executing the training subtask based on the local training data and the model data to be trained. Local training data refers to data stored by device node 120 itself and used for model training. Local training data can be data collected with user permission when a user uses device node 120 and completes user instructions, such as user behavior data and user habit data. Specifically, while executing a training subtask, each device node 120 can send device training data (its own device training data) generated during the execution of the training subtask to other device nodes at preset time intervals, and send its own device training data to other device nodes. At the same time, device node 120 can also receive device training data sent by other device nodes, namely, other device training data, and then continue executing the training subtask based on the local training data, other device training data, model data to be trained, and the model offline package.

[0040] In this solution, optionally, each device node 120 is specifically used to obtain the device training resources at the current moment, determine the required resources for the training subtask, determine the training strategy for the training subtask based on the device training resources and the required resources, and execute the training subtask based on the training strategy.

[0041] The training strategy instructs device node 120 on how to efficiently execute a training subtask. Device training resources are the computing resources available within device node 120 for executing a training subtask, and the required resources for a training subtask are the resources consumed to execute the training subtask. In an optional embodiment, upon receiving a training subtask, device node 120 may obtain device training resources, such as the current CPU utilization, GPU utilization, memory usage, and remaining battery life. After obtaining the device training resources, the required resources for the training subtask are analyzed to determine the minimum resource thresholds for the required resources, such as the required CPU and GPU resources. Simultaneously, data fusion is performed on local training data, other device training data, and the model's training data to be trained to form a complete training dataset. Furthermore, based on the analysis of the complete training dataset, device training resources, and required resources, the model's learning rate is adjusted, the batch size is changed or determined, and the model architecture is optimized to determine the training strategy for the training subtask. For example, when GPU utilization exceeds 90%, the parallelism of the training task can be reduced, thereby reducing the batch size and training intensity. If the remaining power of the device node 120 is lower than 20%, the training subtask is automatically suspended, and after the power is restored to 50%, the training subtask is automatically resumed.

[0042] In this solution, optionally, each device node 120 is specifically used to determine the model training progress corresponding to each time interval based on other device training data and its own device training data, and update the training strategy based on the model training progress, other device training data and its own device training data.

[0043] Specifically, during the execution of the training subtask, each device node 120 can send its device training data to the central service node 110 and other device nodes at preset time intervals, thereby reporting the model training progress and intermediate results to the central service node 110 and other device nodes. After receiving the training data of other devices, each device node 120 can analyze the training data of other devices and its own device training data at the moment to determine the model training progress at the current moment, and adjust its own training strategy based on the model training progress at the current moment, the training data of other devices, and its own device training data to maintain the consistency of the overall model training progress. For example, each device node 120 sends the current model training progress and intermediate results, including gradient information and parameter update values, to the central service node 110 and other device nodes every 10 minutes. The device node 120 receives the model training progress and intermediate results sent by other device nodes. Based on the received model training progress and intermediate results, it compares its own model training progress with the model training progress of other device nodes. If the model training progress of a device node is found to be too fast or too slow, the training speed can be adjusted appropriately, such as by adjusting the learning rate or batch size in the training strategy, to maintain consistency in the overall model training progress. The system in this solution can flexibly adapt to the training capabilities and progress differences of each device node 120, achieving efficient resource utilization and consistent model training progress, thereby improving overall training efficiency and model performance.

[0044] In this solution, optionally, the multiple device nodes 120 also include at least one candidate node, and each candidate node is specifically used to send a join training instruction to the central service node 110, and receive other device training data of other device nodes other than the candidate node sent by the central service node 110 based on the join training instruction, determine the training strategy and training subtask of the candidate node according to the other device training data, and execute the training subtask based on the training strategy.

[0045] Among them, the candidate node is a new device node added during the model training process. The candidate node is connected to the central service node 110, but when the model training just starts, the candidate node has not yet participated in the model training process. When the candidate node needs to participate in model training, the candidate node can send a join training instruction to the central service node 110. After receiving the join training instruction sent by the candidate node, the central service node 110 can send other device training data of each device node other than the candidate node to the candidate node. The candidate node can analyze the received other device training data, determine the model training progress based on the other device training data, and determine the training strategy and training subtasks of the candidate node based on the model training progress and other device training data. At this time, the candidate node has not yet started to participate in the model training process, so the candidate node's training strategy is determined only based on the other device training data (that is, excluding the local training data of the candidate device). When a candidate node starts to execute the training strategy, it means that the candidate node has started to participate in the model training process. Then, the candidate node, like other device nodes, can receive training data from other devices at preset time intervals, determine the model training progress corresponding to each time interval of other device nodes except this node based on the training data of other devices, and update the training strategy based on the model training progress, training data of other devices and its own device training data.

[0046] Exemplarily, each device node 120 connected to the central service node 110 begins executing its own training task at 13:00. The preset time interval is 10 minutes. At 13:30, if the candidate node needs to participate in model training, the candidate node sends a join training instruction to the central service node 110 and analyzes the other device training data received from the central service node 110 to determine the model training progress. The candidate node determines the training strategy and training subtasks of the candidate node based on the other device training data and the model training progress. The candidate node begins executing the training strategy at 13:30 (the training strategy at this time is determined only based on the training data of other devices, and the candidate node has not yet generated its own device training data). The candidate node receives the training data of other devices at 13:40, at which point the candidate node has also generated its own device training data. Therefore, the candidate node can update its training strategy based on the training data of other devices and its own device training data.

[0047] In this solution, optionally, the central service node 110 is also used to update the training subtask and the device node 120 corresponding to the training subtask based on the device training data of each device node 120 and the device information of each device node 120, and send the updated training subtask to the device node 120 corresponding to the training subtask.

[0048] Specifically, in the process of executing training subtasks, each device node 120 can send its device training data to the central service node 110 and other device nodes at preset time intervals, thereby reporting the model training progress and intermediate results to the central service node 110 and other device nodes. After receiving the device training data of each device node 120, the central service node 110 analyzes the device training data of each device node 120 to evaluate the training effect and resource utilization of each device node 120. Based on the training effect and resource utilization of each device node 120, the central service node 110 can assign computing resource-intensive training subtasks to device nodes with stronger performance, or assign simple training subtasks to device nodes with limited resources. For example, when a device node 120 is idle for a long time, some training subtasks of other device nodes can be assigned to the device node 120 to improve the overall training efficiency.

[0049] In this way, the central service node 110 can dynamically adjust training tasks and resource allocation to adapt to the changes and capability differences of each device node 120, thereby improving the efficiency and effectiveness of the entire distributed training system.

[0050] In this solution, optionally, each device node 120 is specifically used to determine whether there is a data conflict between its own device training data and the training data of other devices. If there is a data conflict between its own device training data and the training data of other devices, the device training data and the training data of other devices are sent to the central service node 110, and the conflict-free training data after the conflict is processed is received from the service node.

[0051] Among them, the self-device training data and other device training data include data labels, update parameters and output results. Data conflict is the inconsistency generated by different device nodes 120 when processing the same or related data in a distributed training environment. For example, model parameter update conflict: different device nodes 120 have different update values ​​for the same model parameter, resulting in inconsistent model state. Gradient calculation conflict: During the backpropagation process, there are differences in the gradients calculated by different device nodes 120. Inconsistent data labels: For the same data sample, different device nodes 120 determine different labels or prediction results. Data conflicts will cause inconsistent states of the model on different device nodes 120. Resolving conflicts can ensure that the model states on all device nodes 120 remain consistent, thereby obtaining stable and reliable training results. Therefore, after receiving training data from other devices, each device node 120 can perform a conflict analysis on its own training data and the training data from other devices, including determining whether there is a data conflict between its own training data and the training data from other devices. If there is a data conflict between its own training data and the training data from other devices, the conflicting data will be sent to the central service node 110. The central service node 110 can then process the conflicting data using pre-defined conflict rules and obtain conflict-free data (i.e., conflict-free training data (no longer conflicting between its own training data and the training data from other devices). Furthermore, the central service node 110 sends the conflict-free training data to the device node 120.

[0052] In this solution, optionally, each device node 120 is specifically used to perform consistency detection on the data labels, output results and update parameters of its own device training data and the training data of other devices, and obtain label consistency detection results, parameter consistency detection results and output consistency detection results; if the parameter consistency detection result is inconsistent or the label consistency detection result is inconsistent or the output consistency detection result is inconsistent, it is determined that there is a data conflict between the own device training data and the training data of other devices.

[0053] Specifically, the own device training data and the other device training data include data labels, update parameters and output results. After receiving the other device training data, the device node 120 performs consistency detection on the data labels, output results and update parameters of the own device training data and the other device training data respectively, and obtains the label consistency detection result, the parameter consistency detection result and the output consistency detection result. If any one of the three consistency detection results is inconsistent, it is determined that there is a data conflict in the other device training data. The device node 120 can send the conflicting training data to the central service node 110 so that the central service node 110 can handle the conflict. If the label consistency detection result, the parameter consistency detection result and the output consistency detection result are all consistent, the device node 120 can directly merge the own device training data and the other device training data to obtain conflict-free training data.

[0054] For example, Figure 2 This is a schematic diagram of merging non-conflicting device training data provided by an embodiment of the present invention. Figure 2 As shown in the figure, the black data represents the initial device training data of the device node, the red data represents the updated portion of the device node's output after a training interval, and the blue data represents data merged from other device nodes. The initial device training data for device node 1 is 1a, 2a, 1b, 2b, 1c, 2c, 3a. The initial device training data for device node 2 is 1a, 2a, 1b, 2b, 1c, 2c, 3a. The initial device training data for device node 3 is 1a, 2a, 1b, 2b, 1c, 2c, 3a. After a training interval, the output of device node 1 is 1a, 2a, 1b, 2b, 1c, 2c, 3a, 5a. The output of device node 2 is 1a, 2a, 1b, 2b, 1c, 2c, 3a, 5b. The output of device node 3 is 1a, 2a, 1b, 2b, 1c, 2c, 3a, 5c. Since device nodes 1, 2, and 3 all have the same key parameter update (i.e., 3a), and the newly added parameters 5a, 5b, and 5c are different on all nodes, there is no conflict and they can be safely merged. The merged device training data (conflict-free training data) is 1a, 2a, 1b, 2b, 1c, 2c, 3a, 5a, 5b, 5c.

[0055] In this solution, the central service node 110 will simultaneously receive the device training data sent by each device node. Figure 3 This is a schematic diagram showing that there is no conflict in the device training data provided by the embodiment of the present invention. Figure 3As shown, the output results of device node 1 are 1a, 2a. The output results of device node 2 are 1b, 2b. The output results of device node 3 are 1c, 2c. After determining that there is no conflict between the output results, the central service node 110 merges them to obtain 1a, 2a, 1b, 2b, 1c, 2c. Figure 4 Schematic diagram of sharing conflict-free device training data provided by an embodiment of the present invention. Figure 4 As shown, device node 1, device node 2, and device node 3 can share the merged output results 1a, 2a, 1b, 2b, 1c, 2c.

[0056] For example, Figure 5 This is a first schematic diagram of a device training data conflict provided by an embodiment of the present invention. Figure 5 As shown in the figure, after device node 1 is retrained based on the data of other devices (1a, 2a, 1b, 2b, 1c, 2c), the output result is: 1a, 2a, 1b, 2b, 1c, 2c, 3a. However, device nodes 2 and 3 are not updated synchronously. Therefore, it is determined that there is a conflict in their output results. The central service node 110 processes them and obtains the merged output result: 1a, 2a, 1b, 2b, 1c, 2c, 3a. Figure 6 This is a first schematic diagram of merging conflicting device training data provided by an embodiment of the present invention. Figure 6 As shown, device node 1, device node 2, and device node 3 can share the merged output results: 1a, 2a, 1b, 2b, 1c, 2c, 3a.

[0057] In this solution, when the central service node 110 cannot clearly determine the conflict between the training results generated by different device nodes based on the conflict rules, it can select a benchmark result set and let all device nodes share it synchronously, thereby ensuring the consistency and stability of model training. Figure 7 A second schematic diagram of a device training data conflict provided by an embodiment of the present invention. Figure 7 As shown, device node 1 is based on Figure 6After continuing to train the model using the shared device training data, the updated output results are: 1a, 2a, 1b, 2b, 1c, 2c, 3a, 4a. Device node 2 also receives the updated output results: 1a, 2a, 1b, 2b, 1c, 2c, 3a, 4A. Device node 3, however, remains unchanged, and its output results are: 1a, 2a, 1b, 2b, 1c, 2c, 3a. a, b, and c represent different elements in the model parameter updates or training results, while A is an updated version of a specific element, representing a different value or state. In other words, 1a, 2a, 1b, 2b, 1c, and 2c represent the consistent set of model parameters or results obtained by each device node during the initial training phase. 3a represents the new parameter update obtained by device node A after retraining. 4a represents another new parameter update obtained by device node A after further training. 4A represents a different parameter update obtained by device node B after retraining, and 4A conflicts with 4a obtained by node A. The conflict between 4A and 4a may be due to different updates to the same model parameters, and therefore cannot be simply merged or superimposed. In this case, central service node 110 can detect inconsistencies between the output results of device nodes 1 and 2 and those of other device nodes or each other, but cannot automatically determine which result is superior or more appropriate for the current model state. Central service node 110 then uses its own dataset (assuming it is 1a, 2a, 1b, 2b, 1c, 2c, 3a) as a benchmark, instructing device nodes 1, 2, and 3 to synchronously share this benchmark result set and discard the portions of their results that are inconsistent with the benchmark (i.e., 4a and 4A). Figure 8 This is a second schematic diagram of merging conflicting device training data provided by an embodiment of the present invention. Figure 8 As shown in the figure, device nodes 1, 2, and 3 can share the merged output results: 1a, 2a, 1b, 2b, 1c, 2c, 3a. In this solution, conflict detection and processing can be performed on the training data of each device to ensure data consistency and the effectiveness of model training during distributed training.

[0058] The target model parameters are the model parameters corresponding to the trained model, ultimately determined by the central service node 110. After each device node 120 completes each training subtask and the central service node 110 collects all device training data from each device node 120, the central service node 110 can calculate the target model parameters based on the collected device training data from each device node 120. For example, this can be done by performing a weighted average of the parameters sent by each device node 120, selecting the optimal parameters, or performing other forms of model aggregation. After obtaining the target model parameters, the target model parameters are sent to each device node 120. Each device node 120 receives the target model parameters and obtains a target model based on the target model parameters.

[0059] The system of this embodiment includes a central service node and multiple device nodes, the central service node is connected to each device node, and a model offline package is deployed on each device node; the model offline package is predetermined based on the device resources of each device node; the central service node is used to obtain device information of each device node, determine each training subtask, the device node corresponding to each training subtask, and the model data to be trained based on each device information and predetermined model information; send each training subtask and the model data to be trained to the device node corresponding to each training subtask; each device node is used to receive other device training data sent by each device node other than the node, execute training subtasks based on the pre-acquired local training data, device training data, model data to be trained, and model offline package, obtain its own device training data, and send its own device training data to the central service node and each device node other than the node at preset time intervals; the central service node is also used to generate target model parameters based on each device training data, and send the target model parameters to each device node, so that each device node obtains the target model based on the target model parameters. In the system of this embodiment, the model offline package is determined based on device resources, ensuring that each device node can effectively train the model within its capabilities. The training subtasks are distributed through the central service node, so that the training subtasks are evenly distributed among the device nodes, avoiding the situation where some nodes are overloaded while other nodes are idle. Multiple device nodes can process different training subtasks in parallel and quickly send training data to the central service node, which accelerates the update and iteration process of the model parameters and significantly improves the speed of model training. The device nodes give priority to using local training data, reducing dependence on the central service node or other device nodes. At the same time, the target model obtained will be more personalized and can better adapt to different users and scenarios. By integrating the training results from different device nodes, the model can learn a wider range of data features, thereby improving generalization capabilities. That is, the system of this embodiment achieves efficient utilization of resources, while improving model training efficiency and enhancing model performance.

[0060] Figure 9 A second structural diagram of a model training system provided by an embodiment of the present invention is shown in FIG. Figure 9As shown, the system includes: a central service node 110 and multiple device nodes 120, the central service node 110 is connected to each device node 120, and each device node 120 is deployed with a model offline package; the central service node 110 is used to obtain device information of each device node 120, and determine each training subtask, the device node 120 corresponding to each training subtask and the model training data according to each device information and predetermined model information; each training subtask and the model training data are sent to the device node 120 corresponding to each training subtask; each device node is used to receive data other than the node itself. The other device training data sent by each other device node is used to perform training subtasks based on the pre-acquired local training data, other device training data, model data to be trained, and model offline package to obtain its own device training data, and send its own device training data to the central service node 110 and other device nodes other than this node at preset time intervals; the central service node 110 is also used to generate target model parameters based on each own device training data, and send the target model parameters to each device node 120, so that each device node 120 obtains the target model based on the target model parameters. The central service node 110 includes a pre-established distributed database 111, which includes multiple storage nodes; the central service node 110 is specifically used to obtain the identification information of the own device training data sent by each device node, perform consistent hash calculation on the own device training data based on the identification information, and determine the target storage node corresponding to the own device training data based on the calculation result, and store the own device training data in the target storage node.

[0061] Among them, the distributed database 111 is a database system, and the distributed database 111 is composed of multiple storage nodes. The storage node is responsible for storing and managing data. Each storage node can independently perform data read and write operations, and work with other storage nodes to achieve global consistency of data and high availability of the system. The various storage nodes are distributed in different geographical locations and server clusters to improve data availability and fault tolerance. The consistent hashing algorithm is used to evenly distribute data to multiple storage nodes to balance the load and minimize the migration overhead between nodes. Specifically, the central service node 110 can build a virtual hash ring, which represents all possible hash values, for example, a value from 0 to 2. 32-1 Each storage node is mapped to one or more positions on the hash ring, and a storage node can have multiple hash values ​​to support load balancing. When the central service node 110 receives the device training data sent by each device node 120, it determines the device identifier from each device training data, performs a consistent hash calculation on the device identifier, and obtains a calculation result. Based on the calculation result, it determines the storage node where the device training data should be stored, i.e., the target storage node.

[0062] For example, assume that distributed database 111 has three storage nodes, and their positions on the hash ring are: storage node 1: position 100; storage node 2: position 200; and storage node 3: position 300. A consistent hashing algorithm is used to hash the device identification number (device node identifier). Assume the hash result is 150. Starting from 150 and searching clockwise, the first storage node found is at position 200, i.e., storage node 2. Therefore, the target storage node can be determined to be storage node 2.

[0063] In this solution, after receiving the self-device training data, the storage node can perform an integrity check on the received data, specifically checking whether the self-device training data was truncated or corrupted during transmission. If the integrity check passes, the node can further verify that the self-device training data conforms to the expected format and standards, such as by checking the data type, size, and structure. After the self-device training data passes all checks performed by the storage node, the storage node can store the self-device training data. After successfully storing the self-device training data, the storage node can mark the stored self-device training data, for example, by updating the version number of the self-device training data (adding a version identifier, such as an incrementing number or a timestamp, to the data storage record).

[0064] Specifically, when a new self-device training data is stored in the distributed database 111, its corresponding version number will be incremented. Correspondingly, when each device node 120 reads other device training data from the central service node 110, it can not only obtain the data itself, but also the version number corresponding to the data. Before using other device training data for training, the device node 120 can check its version number to confirm whether the other device training data is the latest version of the data. If its version number is not the latest, the device node 120 can re-read the latest version of the other device training data from the distributed database 111. In this way, the distributed database 111 can maintain data consistency between different storage nodes and ensure the accuracy and reliability of the data.

[0065] In this solution, each device node 120 is optionally further configured to determine a data index of the requested data based on the training subtask, search the local cache for the requested data based on the data index, and if the requested data is not found in the local cache, generate a data request instruction based on the data index, send the data request instruction to the central service node 110, receive the requested data returned by the central service node 110, and store the requested data in the local cache. The central service node 110 is further configured to receive the data request instruction sent by the device node 120, perform a consistent hash calculation on the data index of the data request instruction, and obtain a requested storage node; determine the requested data through the requested storage node, and return the requested data to the device node 120.

[0066] Data indexes are used to quickly retrieve data records. Each data index is associated with a specific data record and uniquely identifies and locates that data record. Specifically, the distributed database 111 stores device training data for each device. When each device node 120 needs device training data, it can search for that data in the distributed database 111. Specifically, when a device node 120 needs to search for its own device data while performing a training subtask, the device node 120 can generate a data index based on the data being searched (the requested data). Using the data index, the requested data is preferentially searched in the local cache of the device node 120. If the requested data is not found in the local cache, a request instruction is generated based on the data index. The request instruction is sent to the central service node 110, which performs a consistent hash calculation on the data index and, based on the calculation result, determines the storage node where the requested data is located, i.e., the requesting storage node. The requested data is then searched in the requesting storage node and returned to the device node 120. After receiving the request data, the device node 120 stores the request data in a local cache, so that when the request data is needed later, the request data can be quickly determined from the local cache.

[0067] The system of this embodiment includes: a central service node and multiple device nodes 120, the central service node is connected to each device node, and a model offline package is deployed on each device node; the central service node is used to obtain device information of each device node, and determine each training subtask, the device node corresponding to each training subtask, and the model data to be trained based on each device information and predetermined model information; each training subtask and the model data to be trained are sent to the device node corresponding to each training subtask; each device node is used to receive other device training data sent by each device node other than the current node, execute training subtasks based on pre-acquired local training data, other device training data, model data to be trained, and model offline package, obtain its own device training data, and send its own device training data to the central service node and each device node other than the current node at preset time intervals; the central service node is also used to generate target model parameters based on its own device training data, and send the target model parameters to each device node, so that each device node obtains the target model based on the target model parameters. The central service node includes a pre-established distributed database, which includes multiple storage nodes. The central service node is specifically used to obtain the identification information of the device training data sent by each device node, perform a consistent hash calculation on the device training data based on the identification information, and determine the target storage node corresponding to the device training data based on the calculation result, and store the device training data in the target storage node. In the system of this embodiment, through multiple storage nodes, the distributed database can provide higher data storage capacity and redundancy, enhancing the reliability and fault tolerance of the system. The distributed database significantly improves the efficiency, reliability, and scalability of the training model, while reducing network communication overhead and enhancing data security and privacy protection.

[0068] Figure 10 The third structural diagram of a model training system provided by an embodiment of the present invention is as follows: Figure 10As shown, the system includes a central service node 110 and multiple device nodes 120. The central service node 110 is connected to each device node 120. The central service node 110 is used to obtain device information of each device node 120, determine each training subtask, the device node 120 corresponding to each training subtask and the model data to be trained according to each device information and predetermined model information; send each training subtask and the model data to be trained to the device node 120 corresponding to each training subtask; each device node 120 is used to receive other device training data sent by each device node other than the node, execute training subtasks based on pre-acquired local training data, other device training data, model data to be trained and model offline package, obtain its own device training data, and send its own device training data to the central service node 110 and each device node other than the node at preset time intervals; the central service node 110 is also used to generate target model parameters based on each own device training data, and send the target model parameters to each device node 120, so that each device node 120 obtains a target model 121 based on the target model parameters. Each device node 120 includes a target model 121. Each device node 120 is further configured to receive a model usage instruction sent by a user, obtain model output information based on the model usage instruction and the target model 121, and display the model output information to the user.

[0069] Target model 121 is a trained model that can be used directly by users. Specifically, after model training is completed, each device node 120 has a corresponding target model 121. Users can send model usage instructions to target model 121 through the user interface of device node 120. Based on the model usage instructions, device node 120 can use target model 121 to determine the model output information corresponding to the model usage instructions and display the model output information to the user.

[0070] For example, a user opens the application corresponding to the model offline package on a mobile phone or laptop and sends a model usage instruction via voice or text input, such as "Book me a high-speed rail ticket from Beijing to Shanghai tomorrow, second-class." After receiving the model usage instruction, the application preprocesses it, for example, converting the voice instruction into text. The text instruction is then sent to the locally stored target model 121. Target model 121 uses natural language processing techniques to semantically understand and analyze the model usage instruction. Through lexical analysis, syntactic analysis, and semantic role labeling, it identifies key information in the model usage instruction, such as the departure point "Beijing," the destination "Shanghai," the departure time "tomorrow," and the seat type "second-class." Based on this identified key information, target model 121 determines that it needs to interface with an external ticketing system. Target model 121 internally stores interface information for different ticketing systems and selects the appropriate ticketing system interface based on factors such as the user's region and commonly used ticketing platform. A booking request containing the parsed key information is then sent to the ticketing system. After receiving a booking request, the ticketing system determines eligible trains and available tickets based on the request and returns this information to the target model 121. After receiving the information returned by the ticketing system, the target model 121 organizes and filters it, removing train information that does not meet the user's requirements, such as trains that do not offer two seats or trains with inconsistent departure times. The organized train information is formatted and displayed in a list format, with each list item containing the train number, departure time, arrival time, and fare information. Furthermore, the target model 121 returns this organized train information to the application, which displays it on the user interface for easy viewing and selection. After viewing the train information on the user interface, the user selects the train they wish to book and confirms the booking. The application sends the user's confirmation to the target model 121. The target model 121 interacts again with the ticketing system and, in accordance with the ticketing system's interface specifications, sends a booking confirmation request, including the train selection and passenger information (which the user can set in advance in the application). The ticketing system processes the booking request, locking seats, and performing payment operations. After the operation is completed, the booking result is returned to target model 121. Target model 121 feeds the booking result from the ticketing system back to the application, which then displays the final booking result to the user on the user interface. If the booking is successful, the train number, seat number, and order number are displayed. If the booking fails, the specific reason for the failure is displayed, and corresponding solutions are provided.

[0071] In this embodiment, the system includes a central service node and multiple device nodes, the central service node being connected to each device node. The central service node is configured to obtain device information from each device node, determine each training subtask, the device node corresponding to each training subtask, and the model data to be trained based on each device information and predetermined model information, and send each training subtask and the model data to be trained to the device node corresponding to each training subtask. Each device node is configured to receive other device training data sent by other device nodes other than the node, execute training subtasks based on pre-acquired local training data, other device training data, model data to be trained, and model offline packages, obtain its own device training data, and send its own device training data to the central service node and each device node other than the node at preset time intervals. The central service node is further configured to generate target model parameters based on its own device training data and send the target model parameters to each device node so that each device node obtains a target model based on the target model parameters. Each device node includes a target model. Each device node is further configured to receive model usage instructions sent by a user, obtain model output information based on the model usage instructions and the target model, and display the model output information to the user. In this embodiment of the system, each device node not only participates in the model training process but also directly interacts with users, leveraging the trained target model to provide an end-to-end solution. Furthermore, this solution completes the entire process from model training to model usage on the same device, improving model usability and responsiveness while also enhancing the user experience.

[0072] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A model training system, characterized in that: The system includes a central service node and multiple device nodes, wherein the central service node is connected to each device node, and each device node is deployed with a model offline package; the model offline package is predetermined based on the device resources of each device node; The central service node is used to obtain device information of each device node, and determine each training subtask, the device node corresponding to each training subtask, and the model training data according to the device information and the predetermined model information; Sending each training subtask and the model data to be trained to the device node corresponding to each training subtask; Each device node is configured to receive other device training data sent by each device node other than the current node, perform the training subtask based on the pre-acquired local training data, the other device training data, the model data to be trained, and the model offline package, obtain its own device training data, and send the own device training data to the central service node and each device node other than the current node at a preset time interval; The central service node is further configured to generate target model parameters based on the training data of each of the devices itself, and send the target model parameters to each of the device nodes, so that each of the device nodes obtains a target model based on the target model parameters.

2. The system according to claim 1, wherein: Each device node is specifically used to obtain the device training resources at the current moment, determine the required resources of the training subtask, determine the training strategy of the training subtask based on the device training resources and the required resources, and execute the training subtask based on the training strategy.

3. The system according to claim 1, wherein: The multiple device nodes also include at least one candidate node, and each candidate node is specifically used to send a join training instruction to the central service node, and receive other device training data of other device nodes except the candidate node sent by the central service node based on the join training instruction, determine the training strategy and training subtask of the candidate node according to the other device training data, and execute the training subtask based on the training strategy.

4. The system according to any one of claims 2 to 3, characterized in that Each device node is specifically used to determine the model training progress corresponding to each time interval according to the other device training data and the own device training data, and update the training strategy based on the model training progress, the other device training data and the own device training data.

5. The system according to claim 1, wherein: Each device node is specifically used to determine whether there is a data conflict between its own device training data and the training data of other devices. If there is a data conflict between its own device training data and the training data of other devices, the training data of other devices and the own device training data are sent to the central service node, and the conflict-free training data after conflict processing sent by the service node is received.

6. The system according to claim 5, characterized in that The self-device training data and the other-device training data include data labels, update parameters, and output results. Each device node is specifically configured to perform consistency detection on the data labels, the output results, and the update parameters of the self-device training data and the other-device training data, respectively, to obtain a label consistency detection result, a parameter consistency detection result, and an output consistency detection result; If the parameter consistency detection result is inconsistent, the label consistency detection result is inconsistent, or the output consistency detection result is inconsistent, it is determined that there is a data conflict between the own device training data and the other device training data.

7. The system according to claim 1, wherein: The central service node includes a pre-established distributed database, which includes multiple storage nodes; the central service node is specifically used to obtain identification information of the own device training data sent by each device node, perform consistent hash calculation on the own device training data based on the identification information, and determine the target storage node corresponding to the own device training data according to the calculation result, and store the own device training data in the target storage node.

8. The system according to claim 1, wherein: Each device node is further configured to determine a data index of the requested data based on the training subtask, search for the requested data in the local cache based on the data index, and if the requested data is not found in the local cache, generate a data request instruction based on the data index, send the data request instruction to the central service node, receive the requested data returned by the central service node, and store the requested data in the local cache.

9. The system according to claim 8, characterized in that The central service node is also used to receive the data request instruction sent by the device node, perform consistent hash calculation on the data index of the data request instruction, and obtain a request storage node; determine the request data through the request storage node, and return the request data to the device node.

10. The system according to claim 1, wherein: The central service node is also used to update the training subtask and the device node corresponding to the training subtask based on the self-device training data of each device node and the device information of each device node, and send the updated training subtask to the device node corresponding to the training subtask.

Images