Data training optimization method and system

Through the data management layer, the data management layer uniformly manages the tasks to be trained and loads data in batches according to the processing volume of nodes, the problem of data repetition and occupation in large-scale data training is solved, and a more efficient training process is achieved.

CN119514720APending Publication Date: 2025-02-25FUJIAN FUNO MOBILE COMM TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411421038.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2025-02-25

AI Technical Summary

Technical Problem

During large-scale data training, the training process takes a long time due to the need to use massive data, and data repetition and occupation may occur during distributed tasks, which affects the training efficiency.

Method used

The data management layer uniformly manages the tasks to be trained, calculates the total amount of data to be loaded based on the node processing volume of each idle node, and loads the data in batches to avoid data duplication and improve loading efficiency. The training results generated by the idle node during training are not only stored locally, but are also sent to the data management layer to determine whether there are required idle nodes in advance, and send the training results to the corresponding node for data training.

Benefits of technology

Reduce training time, improve training efficiency, avoid repeated data loading, and reduce transmission time and training time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119514720A_ABST
    Figure CN119514720A_ABST
Patent Text Reader

Abstract

The invention relates to a data training optimization method and system, and the method comprises the steps: obtaining the to-be-trained data size of a to-be-trained task and the node processing amount of each idle node through a data management layer, calculating the total amount of to-be-loaded data at the time according to the node processing amount, and carrying out the calculation of the total amount of to-be-loaded data; the data to be trained is loaded and distributed in batches according to the total amount of the current data to be loaded, idle nodes store training results generated by data training in a data management layer, and when the data management layer distributes the total amount of the current data to be loaded next time, if it is judged that idle nodes needing training results exist, the data to be loaded is distributed to the data management layer. If yes, the training results are distributed together when the total amount of the data to be loaded is distributed. Therefore, the to-be-trained tasks are uniformly managed through the data management layer, the to-be-trained data volume is recorded and distributed in batches, repeated data loading and distribution are avoided, the loading and distribution efficiency is improved, training results needed by idle nodes are distributed in advance, the transmission time is shortened, and the training efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an optimization method and system for data training. Background Art

[0002] At present, most artificial intelligence (AI) scenarios require a general or specialized large model for data analysis and result output. The large model is usually programmed in Python and trained with a large amount of training data to obtain the required large model. During the training process, due to the need to use a large amount of training data, the training process is time-consuming. In order to reduce the training time, a distributed task training method is adopted to slice the relevant training tasks and assign them to different machine nodes for task training. Since each machine node needs to perform task training, there may be data duplication and data occupation, which affects the training efficiency. Summary of the invention

[0003] The technical problem to be solved by the present invention is: the present invention provides an optimization method and system for data training, which can reduce the training time and improve the training efficiency.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] In a first aspect, the present invention provides a method for optimizing data training, comprising:

[0006] The data management layer obtains the amount of data to be trained for the task to be trained, and at the same time obtains the node processing capacity of each idle node, and calculates the total amount of data to be loaded at that time according to the node processing capacity;

[0007] The data management layer loads the amount of data to be trained in batches according to the total amount of data to be loaded at that time, and distributes the total amount of data to be loaded at that time to corresponding idle nodes for data training according to the node processing capacity after each loading of the total amount of data to be loaded at that time;

[0008] When performing data training, the idle node stores the generated training results locally, and sends the training results to the data management layer for storage. When the data management layer distributes the total amount of data to be loaded next time, it determines whether there are idle nodes that need the training results. If so, the training results are sent to the corresponding idle nodes for data training when distributing the total amount of data to be loaded next time.

[0009] The beneficial effects of the present invention are as follows: the tasks to be trained are uniformly managed through the data management layer, the total amount of data to be loaded at that time is calculated according to the node processing capacity of each idle node, and the amount of data to be trained for the training task is loaded in batches according to the total amount of data to be loaded at that time, and each time the total amount of data to be loaded is loaded, it is distributed according to the node processing capacity to avoid repeated loading of data and improve the loading efficiency at the same time. The training results generated by the idle nodes during data training are not only stored locally but also stored in the data management layer. When the data management layer distributes the total amount of data to be loaded next time, it will be pre-determined whether there are idle nodes that need training results. If so, the training results will be sent to the corresponding idle nodes for data training when the total amount of data to be loaded is distributed. There is no need to apply for the transmission of training results later, which reduces the transmission time and training duration and improves the training efficiency.

[0010] Optionally, the simultaneously acquiring the node processing capacity of each idle node and calculating the total amount of data to be loaded at that time according to the node processing capacity includes:

[0011] The node processing capacity of each idle node is aggregated to calculate the total amount of data to be loaded at that time;

[0012] or

[0013] The smallest node processing capacity is selected from all node processing capacities, and the total amount of data to be loaded is calculated based on the smallest node processing capacity.

[0014] According to the above description, it can be known that various situations are taken into consideration and various methods are adopted to calculate the total amount of data to be loaded at that time, so as to ensure the flexibility of the calculated total amount of data to be loaded at that time, and to calculate the total amount of data to be loaded at that time by summarizing the node processing capacity of each idle node, so as to ensure the full utilization of each node and improve the utilization rate, and to calculate the total amount of data to be loaded at that time with the minimum node processing capacity, so as to improve the loading efficiency of the total amount of data to be loaded at that time.

[0015] Optionally, when the data management layer distributes the total amount of data to be loaded next time, the method includes:

[0016] It is determined whether there is a new idle node. If so, the total amount of data to be loaded is updated according to the node processing capacity of the new idle node.

[0017] According to the above description, the data management layer will update the idle nodes when distributing the total amount of data to be loaded next time, so as to ensure the real-time performance of the idle nodes and improve the utilization rate of the nodes.

[0018] Optionally, the training result includes a current node ID.

[0019] According to the above description, the training result includes the current node ID, so that the data management layer knows the source of the training result, which is convenient for subsequent tracing.

[0020] In a second aspect, the present invention provides an optimization system for data training, comprising:

[0021] The management module is used for the data management layer to obtain the amount of data to be trained for the task to be trained, and at the same time obtain the node processing capacity of each idle node, and calculate the total amount of data to be loaded at that time according to the node processing capacity;

[0022] A distribution module is used for the data management layer to load the amount of data to be trained in batches according to the total amount of data to be loaded at that time, and distribute the total amount of data to be loaded at that time to the corresponding idle nodes for data training according to the node processing capacity after each loading of the total amount of data to be loaded at that time;

[0023] The training module is used for the idle nodes to store the training results generated locally during data training, and send the training results to the data management layer for storage. When the data management layer distributes the total amount of data to be loaded next time, it is determined whether there are idle nodes that need the training results. If so, the training results are sent to the corresponding idle nodes for data training when the total amount of data to be loaded is distributed next time.

[0024] The beneficial effects of the present invention are as follows: the tasks to be trained are uniformly managed through the data management layer, the total amount of data to be loaded at that time is calculated according to the node processing capacity of each idle node, and the amount of data to be trained for the training task is loaded in batches according to the total amount of data to be loaded at that time, and each time the total amount of data to be loaded is loaded, it is distributed according to the node processing capacity to avoid repeated loading of data and improve the loading efficiency at the same time. The training results generated by the idle nodes during data training are not only stored locally but also stored in the data management layer. When the data management layer distributes the total amount of data to be loaded next time, it will be pre-determined whether there are idle nodes that need training results. If so, the training results will be sent to the corresponding idle nodes for data training when the total amount of data to be loaded is distributed. There is no need to apply for the transmission of training results later, which reduces the transmission time and training duration and improves the training efficiency.

[0025] Optionally, the management module is specifically:

[0026] The node processing capacity of each idle node is aggregated to calculate the total amount of data to be loaded at that time;

[0027] or

[0028] The smallest node processing capacity is selected from all node processing capacities, and the total amount of data to be loaded is calculated based on the smallest node processing capacity.

[0029] According to the above description, it can be known that various situations are taken into consideration and various methods are adopted to calculate the total amount of data to be loaded at that time, so as to ensure the flexibility of the calculated total amount of data to be loaded at that time, and to calculate the total amount of data to be loaded at that time by summarizing the node processing capacity of each idle node, so as to ensure the full utilization of each node and improve the utilization rate, and to calculate the total amount of data to be loaded at that time with the minimum node processing capacity, so as to improve the loading efficiency of the total amount of data to be loaded at that time.

[0030] Optionally, the training module includes:

[0031] The updating module is used to determine whether there is a new idle node. If so, the total amount of data to be loaded at that time is updated according to the node processing capacity of the new idle node.

[0032] According to the above description, the data management layer will update the idle nodes when distributing the total amount of data to be loaded next time, so as to ensure the real-time performance of the idle nodes and improve the utilization rate of the nodes.

[0033] Optionally, the training result includes a current node ID.

[0034] According to the above description, the training result includes the current node ID, so that the data management layer knows the source of the training result, which is convenient for subsequent tracing. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a data training optimization method provided in this embodiment;

[0036] Figure 2 A schematic diagram of the overall process of a data training optimization method provided in this embodiment;

[0037] Figure 3 A schematic diagram of the structure of a data training optimization system provided in this embodiment.

[0038] [Description of Reference Numerals]

[0039] 1. An optimization system for data training;

[0040] 2. Management module;

[0041] 3. Distribution module;

[0042] 4. Training module; 41. Update module. DETAILED DESCRIPTION

[0043] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0044] Embodiment 1

[0045] Please refer to Figure 1 to Figure 2 The present invention provides a data training optimization method, comprising the steps of:

[0046] S1. The data management layer obtains the amount of data to be trained for the task to be trained, and at the same time obtains the node processing capacity of each idle node, and calculates the total amount of data to be loaded at that time according to the node processing capacity;

[0047] In this embodiment, if Figure 2 As shown, a data management layer is pre-constructed, and the data management layer uniformly manages the training tasks, that is, the amount of data to be trained of the tasks to be trained is uniformly distributed to the nodes by the management layer. Specifically, the data management layer obtains the amount of data to be trained of the tasks to be trained, and simultaneously obtains the node processing capacity of each idle node, and calculates the total amount of data to be loaded at that time according to the node processing capacity of each idle node.

[0048] At this time, the node processing capacity of each idle node is obtained in step S1, and the total amount of data to be loaded at that time is calculated according to the node processing capacity, including:

[0049] S11, summarizing the node processing capacity of each idle node to calculate the total amount of data to be loaded at that time;

[0050] or

[0051] The smallest node processing capacity is selected from all node processing capacities, and the total amount of data to be loaded is calculated based on the smallest node processing capacity.

[0052] In this embodiment, if Figure 2As shown, there are two ways for the data management layer to calculate the total amount of data to be loaded at the time according to the node processing capacity of each idle node, namely: Method 1. The node processing capacity of each idle node is aggregated to calculate the total amount of data to be loaded at the time. For example, there are 5 idle nodes, and the node processing capacity of each idle node is: idle node 1 is 5W, idle node 2 is 7W, idle node 3 is 10W, idle node 4 is 10W, and idle node 5 is: 8W, that is, according to method 1, the total amount of data to be loaded at the time = 5W + 7W + 10W + 10W+8W=40W; Method 2. Filter out the smallest node processing capacity from all node processing capacities, and count the total amount of data to be loaded at that time based on the smallest node processing capacity. For example, there are 5 idle nodes, and the node processing capacity of each idle node is: idle node 1 is 5W, idle node 2 is 7W, idle node 3 is 10W, idle node 4 is 10W, and idle node 5 is 8W. That is, the smallest node processing capacity selected according to Method 2 is 5W, that is, the total amount of data to be loaded at that time = 5W+5W+5W+5W+5W=25W.

[0053] S2, the data management layer loads the amount of data to be trained in batches according to the total amount of data to be loaded at that time, and distributes the total amount of data to be loaded at that time to the corresponding idle nodes for data training according to the node processing capacity after each loading of the total amount of data to be loaded at that time;

[0054] In this embodiment, if Figure 2 As shown, the data management layer records the amount of data to be trained in batches according to the total amount of data to be loaded calculated in step S1. For example, the amount of data to be trained is 100W, and the total amount of data to be loaded is 25W, then 100W / 25W=4, that is, 4 is the number of batches, and the amount of data to be trained is loaded in 4 times. If the calculated number of batches is a decimal, it is rounded up. For example, if the total amount of data to be loaded is 40W, then 100W / 40W=2.5, then the number of batches is 3, and the amount of data to be trained is loaded in 3 times. After each loading of the total amount of data to be loaded, the total amount of data to be loaded is distributed to the corresponding idle nodes for training according to the node processing capacity. After the data management layer has completed the distribution, the next total amount of data to be loaded can be loaded, that is, loading in advance can reduce the loading time of subsequent training data.

[0055] S3. When performing data training, the idle node stores the generated training results locally, and sends the training results to the data management layer for storage. When the data management layer distributes the total amount of data to be loaded next time, it determines whether there are idle nodes that need the training results. If so, the training results are sent to the corresponding idle nodes for data training when distributing the total amount of data to be loaded next time.

[0056] In this embodiment, if Figure 2 As shown, the idle node will generate training results when performing data training. The training results will be stored locally, that is, in the idle node itself, and the training results will also be sent to the data management layer for storage, wherein the training results include but are not limited to the current node ID and data range. When the data management layer distributes the total amount of data to be loaded for the next time, it will first determine whether there are idle nodes that require training results. If so, when distributing the total amount of data to be loaded, the required training results will be sent to the corresponding idle nodes for data training. During this process, if the data management layer does not have the corresponding training results, it will be forwarded from the idle nodes that have the corresponding training results.

[0057] At this time, when the data management layer distributes the total amount of data to be loaded next time in step S3, it includes:

[0058] S31. Determine whether there is a new idle node. If so, update the total amount of data to be loaded at that time according to the node processing capacity of the new idle node.

[0059] In this embodiment, if Figure 2 As shown, when the data management layer distributes the total amount of data to be loaded next time, it will determine whether there is a new idle node, that is, update the idle node. If so, the data management layer can first distribute the total amount of data to be loaded this time according to the node processing capacity of the original idle nodes, and then recalculate according to the node processing capacity of the new idle nodes when loading the total amount of data to be loaded next time. It can also first calculate the latest total amount of data to be loaded according to the node processing capacity of the new idle nodes, load according to the latest total amount of data to be loaded, and after loading, distribute the latest total amount of data to be loaded to the idle nodes including the new idle nodes.

[0060] Embodiment 2

[0061] Please refer to Figure 3 The present invention provides a data training optimization system 1, including: a management module 2, a distribution module 3, a training module 4 and an update module 41.

[0062] The management module 2 is used for the data management layer to obtain the amount of data to be trained for the task to be trained, and at the same time obtain the node processing capacity of each idle node, and calculate the total amount of data to be loaded at that time according to the node processing capacity;

[0063] Distribution module 3, used for the data management layer to load the amount of data to be trained in batches according to the total amount of data to be loaded at that time, and distribute the total amount of data to be loaded at that time to the corresponding idle nodes for data training according to the node processing capacity after each loading of the total amount of data to be loaded at that time;

[0064] The training module 4 is used for the idle nodes to store the training results generated locally during data training, and send the training results to the data management layer for storage. When the data management layer distributes the total amount of data to be loaded next time, it is determined whether there are idle nodes that need the training results. If so, the training results are sent to the corresponding idle nodes for data training when the total amount of data to be loaded is distributed next time.

[0065] Specifically, the management module 2 is specifically:

[0066] The node processing capacity of each idle node is aggregated to calculate the total amount of data to be loaded at that time;

[0067] or

[0068] The smallest node processing capacity is selected from all node processing capacities, and the total amount of data to be loaded is calculated based on the smallest node processing capacity.

[0069] Specifically, the training module 4 includes:

[0070] The updating module 41 is used to determine whether there is a new idle node. If so, the total amount of data to be loaded is updated according to the node processing capacity of the new idle node.

[0071] Specifically, the training result includes the current node ID.

[0072] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0073] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0075] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0076] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0077] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0078] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A data training optimization method, characterized in that: include: The data management layer obtains the amount of data to be trained for the task to be trained, and at the same time obtains the node processing capacity of each idle node, and calculates the total amount of data to be loaded at that time according to the node processing capacity; The data management layer loads the amount of data to be trained in batches according to the total amount of data to be loaded at that time, and distributes the total amount of data to be loaded at that time to corresponding idle nodes for data training according to the node processing capacity after each loading of the total amount of data to be loaded at that time; When performing data training, the idle node stores the generated training results locally, and sends the training results to the data management layer for storage. When the data management layer distributes the total amount of data to be loaded next time, it determines whether there are idle nodes that need the training results. If so, the training results are sent to the corresponding idle nodes for data training when distributing the total amount of data to be loaded next time.

2. A data training optimization method as claimed in claim 1, characterized in that: The step of simultaneously acquiring the node processing capacity of each idle node and calculating the total amount of data to be loaded at that time according to the node processing capacity includes: The node processing capacity of each idle node is aggregated to calculate the total amount of data to be loaded at that time; or The smallest node processing capacity is selected from all node processing capacity, and the total amount of data to be loaded is calculated based on the smallest node processing capacity.

3. The data training optimization method according to claim 1, characterized in that: When the data management layer distributes the total amount of data to be loaded next time, it includes: It is determined whether there is a new idle node. If so, the total amount of data to be loaded is updated according to the node processing capacity of the new idle node.

4. The data training optimization method according to claim 1, characterized in that: The training result includes the current node ID.

5. A data training optimization system, characterized in that: include: The management module is used for the data management layer to obtain the amount of data to be trained for the task to be trained, and at the same time obtain the node processing capacity of each idle node, and calculate the total amount of data to be loaded at that time according to the node processing capacity; A distribution module is used for the data management layer to load the amount of data to be trained in batches according to the total amount of data to be loaded at that time, and distribute the total amount of data to be loaded at that time to the corresponding idle nodes for data training according to the node processing capacity after each loading of the total amount of data to be loaded at that time; The training module is used for the idle nodes to store the training results generated locally during data training, and send the training results to the data management layer for storage. When the data management layer distributes the total amount of data to be loaded next time, it is determined whether there are idle nodes that need the training results. If so, the training results are sent to the corresponding idle nodes for data training when the total amount of data to be loaded is distributed next time.

6. A data training optimization system as claimed in claim 5, characterized in that: The management module is specifically: The node processing capacity of each idle node is aggregated to calculate the total amount of data to be loaded at that time; or The smallest node processing capacity is selected from all node processing capacity, and the total amount of data to be loaded is calculated based on the smallest node processing capacity.

7. A data training optimization system as claimed in claim 5, characterized in that: The training module includes: The updating module is used to determine whether there is a new idle node. If so, the total amount of data to be loaded at that time is updated according to the node processing capacity of the new idle node.

8. A data training optimization system as claimed in claim 5, characterized in that: The training result includes the current node ID.