A model training system, method, device, equipment and medium

By determining the order of submodules by training scheduling and control servers, resource sharing is realized in the model training system, the problem of independent resource occupancy between different models is solved, and resource utilization and model design efficiency are improved.

CN114219029BActive Publication Date: 2024-12-03CHINA CONSTRUCTION BANK
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202111543800.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-16
Publication Date
2024-12-03
Estimated Expiration
2041-12-16

AI Technical Summary

Technical Problem

Different models occupy resources independently during the training process, resulting in low resource utilization.

Method used

Design a model training system, including training scheduling server and training control server, determine the submodule order by receiving model identification and sample set, and let each submodule adjust its parameters based on the target sample set and target annotation set to realize resource sharing.

Benefits of technology

This improves resource utilization, avoids the repeated use of resources by different models, and improves the efficiency of model design.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114219029B_ABST
    Figure CN114219029B_ABST
Patent Text Reader

Abstract

The present application relates to the field of data processing technology, and in particular to a model training system, method, device, equipment and medium, which are used to solve the problem that different models in the prior art all use their own submodules and independently occupy various resources. In the application embodiment, when training the model, the second identifier of each submodule contained in the model of the first identifier is obtained, and the model is trained using the submodule corresponding to the second identifier, and different models may use the same submodule, so that some models with the same steps can use the same submodule, avoiding different models using their own submodules and independently occupying various resources, thereby improving resource utilization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a model training system, method, device, equipment and medium. Background Art

[0002] In the construction of intelligent applications, models are widely used in natural language processing, speech recognition, face recognition and other scenarios. Simply put, model training refers to the process in which the model performs abstract calculations on a certain amount of basic data and compares it with the ideal result set, thereby adjusting its own parameters.

[0003] Different models in the prior art may contain the same steps. For example, the model corresponding to natural language processing and the model corresponding to speech recognition both contain the step corresponding to "word segmentation". However, when designing the model, different models use their own sub-modules and independently occupy various resources. Summary of the invention

[0004] The embodiments of the present application provide a model training system, method, apparatus, device and medium to solve the problem in the prior art that different models all use their own sub-modules and independently occupy various resources.

[0005] The embodiment of the present application provides a model training system, the system comprising: a training scheduling server, a training control server, and submodules that perform different functions;

[0006] The training scheduling server is used to receive a first instruction, obtain a first identifier of the model carried in the first instruction, obtain a target sample set and a target annotation set corresponding to the first identifier according to a pre-saved correspondence between the model identifier and the sample set and the annotation set; and send a second instruction carrying the target sample set and the target annotation set to the training control server;

[0007] The training control server is used to determine the second identifier of each submodule included in the model of the first identifier in sequence according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules;

[0008] Each second identified submodule is used to adjust its own parameters based on the target sample set and the target annotation set.

[0009] The present application embodiment provides a model training method, which is applied to a training scheduling server, and the method includes:

[0010] Receiving a first instruction, and obtaining a first identifier of the model carried in the first instruction;

[0011] According to the pre-saved correspondence between the model identifier and the sample set and the annotation set, the target sample set and the target annotation set corresponding to the first identifier are obtained; and the second instruction carrying the target sample set and the target annotation set is sent to the training control server, so that the training control server determines the second identifier of each sub-module contained in the model of the first identifier in turn according to the pre-saved correspondence between the model identifier and the identifier of the sub-module and the correspondence between the order of the sub-modules.

[0012] The present application embodiment provides a model training method, which is applied to a training control server, and the method includes:

[0013] Receive a second instruction carrying a target sample set and a target annotation set; wherein the second instruction also carries a first identifier of the model;

[0014] According to the correspondence between the pre-saved model identifier and the sub-module identifier and the correspondence between the order of the sub-modules, the second identifier of each sub-module contained in the model with the first identifier is determined in turn, so that each sub-module with the second identifier can adjust its own parameters based on the target sample set and the target annotation set.

[0015] The present application embodiment provides a model training method, which is applied to any submodule that performs different functions, and the method includes:

[0016] Obtain target sample set and target annotation set;

[0017] Based on the target sample set and the target annotation set, the parameters thereof are adjusted.

[0018] The present application embodiment provides a model training method, the method comprising:

[0019] Receiving a first instruction, obtaining a first identifier of a model carried in the first instruction, and obtaining a target sample set and a target annotation set corresponding to the first identifier according to a pre-saved correspondence between the model identifier and the sample set and the annotation set;

[0020] Determine in sequence the second identifier of each submodule included in the model of the first identifier according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules;

[0021] Based on the target sample set and the target annotation set, the parameters of each submodule of the second identification are adjusted.

[0022] The present application provides a model training device, the device comprising:

[0023] A first receiving and obtaining module, configured to receive a first instruction and obtain a first identifier of a model carried in the first instruction;

[0024] The first processing module is used to obtain the target sample set and the target annotation set corresponding to the first identifier according to the pre-saved correspondence between the model identifier and the sample set and the annotation set; and send the second instruction carrying the target sample set and the target annotation set to the training control server.

[0025] The present application embodiment provides a model training device, which is applied to a training control server, and the device includes:

[0026] A receiving module, configured to receive a second instruction carrying a target sample set and a target annotation set; wherein the second instruction also carries a first identifier of the model;

[0027] The second processing module is used to determine the second identifier of each submodule included in the model of the first identifier in sequence according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules.

[0028] The embodiment of the present application provides a model training device, which is applied to any submodule that performs different functions, and the device includes:

[0029] An acquisition module, used to acquire a target sample set and a target annotation set;

[0030] The fourth processing module is used to adjust its own parameters based on the target sample set and the target annotation set.

[0031] The present application provides a model training device, the device comprising:

[0032] A second receiving and obtaining module, configured to receive a first instruction, obtain a first identifier of a model carried in the first instruction, and obtain a target sample set and a target annotation set corresponding to the first identifier according to a pre-saved correspondence between the model identifier and the sample set and the annotation set;

[0033] The third processing module is used to determine the second identifier of each submodule included in the model of the first identifier in turn according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules; based on the target sample set and the target annotation set, adjust the parameters of each submodule of the second identifier.

[0034] An embodiment of the present application provides an electronic device, which includes at least a processor and a memory, and the processor is used to perform the steps of any of the above-mentioned model training methods when executing a computer program stored in the memory.

[0035] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of any of the above-mentioned model training methods.

[0036] An embodiment of the present application provides a computer program product, which includes: a computer program code, and when the computer program code is run on a computer, the computer executes the steps of any of the above-mentioned model training methods.

[0037] In an embodiment of the present application, when training a model, a training scheduling server is used to receive a first instruction. After receiving the first instruction, the training scheduling server obtains a first identifier of the model carried in the first instruction, and obtains a target sample set and a target annotation set corresponding to the first identifier based on a pre-saved correspondence between the model identifier and the sample set and the annotation. The model training system trains the model of the first identifier through the target sample set and the target annotation. The training scheduling server sends a second instruction carrying the target sample set and the target annotation set to the training control server. After receiving the second instruction, the training control server determines the second identifier of each submodule contained in the model of the first identifier in turn based on a pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules. Each submodule of the second identifier adjusts its own parameters based on the target sample set and the target annotation set, thereby realizing the training of the model of the first identifier. In the application embodiment, when the model is trained, the second identifier of each sub-module contained in the model with the first identifier is obtained, and the model is trained using the sub-module corresponding to the second identifier, and different models may use the same sub-module, so that some models with the same steps can use the same sub-module, avoiding different models using each of their own sub-modules and independently occupying various resources, thereby improving resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of the structure of a model training system provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of the execution order of submodules of a model provided in an embodiment of the present application combined with a directed acyclic graph;

[0041] Figure 3A schematic diagram of a processing process of a training control server provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the execution process of a training scheduling server provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of a training process provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of a process of a model training method provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a process of a model training method provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of a process of a model training method provided in an embodiment of the present application;

[0047] Fig. 9 A schematic diagram of a process of a model training method provided in an embodiment of the present application;

[0048] Fig.10 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0049] Fig.11 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0050] Fig.12 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0051] Fig.13 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0052] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0054] In an embodiment of the present application, when training a model, a training scheduling server is used to receive a first instruction. After receiving the first instruction, the training scheduling server obtains a first identifier of the model carried in the first instruction, and obtains a target sample set and a target annotation set corresponding to the first identifier based on a pre-saved correspondence between the model identifier and the sample set and the annotation. The model training system trains the model of the first identifier through the target sample set and the target annotation. The training scheduling server sends a second instruction carrying the target sample set and the target annotation set to the training control server. After receiving the second instruction, the training control server determines the second identifier of each submodule contained in the model of the first identifier in turn based on a pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules. Each submodule of the second identifier adjusts its own parameters based on the target sample set and the target annotation set, thereby realizing the training of the model of the first identifier.

[0055] In order to improve resource utilization, the embodiments of the present application provide a model training system, method, apparatus, device and medium.

[0056] Embodiment 1:

[0057] Figure 1 A schematic diagram of the structure of a model training system provided in an embodiment of the present application, wherein the system includes a training scheduling server 101, a training control server 102, and submodules 103 that perform different functions;

[0058] The training scheduling server 101 is used to receive a first instruction, obtain a first identifier of the model carried in the first instruction, and obtain a target sample set and a target annotation set corresponding to the first identifier according to a pre-saved correspondence between the model identifier and the sample set and the annotation set; and send a second instruction carrying the target sample set and the target annotation set to the training control server 102;

[0059] The training control server 102 is used to sequentially determine the second identifier of each submodule included in the model of the first identifier according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules;

[0060] Each second identified submodule 103 is used to adjust its own parameters based on the target sample set and the target annotation set.

[0061] In an embodiment of the present application, in order to implement training of the model, the training scheduling server may receive a first instruction, and after receiving the first instruction, train the model of the first identifier carried in the first instruction. The first instruction may be sent by a staff member through a device such as a client.

[0062] In an embodiment of the present application, if a certain model needs to be trained, a first instruction for model training can be sent to a training scheduling server, and the first instruction carries a first identifier of the model to be trained. In order to train the model, the training scheduling server obtains the first identifier carried in the instruction after receiving the first instruction. The training scheduling server pre-stores the correspondence between the model identifier and the sample set and the annotation set. After obtaining the first identifier, the training scheduling server can determine the target sample set and target annotation set corresponding to the first identifier based on the first identifier and the pre-stored correspondence between the model identifier and the sample set and the annotation set, so that the model of the first identifier can be trained based on the target sample set and the target annotation set.

[0063] After obtaining the target sample set and the target annotation set, the training scheduling server carries the target sample set and the target annotation set in a second instruction and sends it to the training control server. In order to let the training control server know which model to train, the second instruction also carries the first identifier of the model to be trained. After receiving the second instruction, the training control server obtains the first identifier carried in the second instruction.

[0064] The training control server pre-stores the correspondence between the model identifier and the identifier of the sub-module and the correspondence between the order of the sub-modules. After the training control server obtains the first identifier carried in the second instruction, it determines the second identifier of each sub-module corresponding to the model of the first identifier and the order of each sub-module according to the pre-stored correspondence between the model identifier and the identifier of the sub-module and the correspondence between the order of the sub-modules. The second identifier of each sub-module is the second identifier of each sub-module contained in the model of the first identifier.

[0065] For example, each submodule of the model determined by the first identification is submodule a, submodule b and submodule c, and the order of the determined submodules is submodule b, submodule c and submodule a.

[0066] Each second-identified submodule adjusts its own parameters based on the target sample set and the target annotation set. Specifically, when the parameters of each second-identified submodule are adjusted, when the preset conditions are met, the adjusted parameters are obtained. In an embodiment of the present application, submodules with various identifiers are saved, and models with different identifiers may contain the same submodule. Therefore, there is no need to independently design corresponding submodules for each model, thereby saving resources and improving the efficiency of model design.

[0067] In addition, in an embodiment of the present application, the staff can also set the time for the next training of the model of each identifier and the corresponding identifier of the model on the training scheduling server. The training scheduling server determines whether the time for the next training of the model of any identifier has arrived according to a preset time interval. If the time for the next training of the model of any identifier has arrived, the training scheduling server is equivalent to receiving a first instruction. The identifier carried in the first instruction is the first identifier described above. The training scheduling server obtains the target sample set and target annotation set corresponding to the first identifier, and sends a second instruction carrying the target sample set and target annotation set to the training control server. The training control server determines the second identifier of each sub-module contained in the model of the first identifier in turn, and each sub-module of the second identifier is used to adjust its own parameters based on the target sample set and target annotation set.

[0068] In the application embodiment, when the model is trained, the second identifier of each sub-module contained in the model with the first identifier is obtained, and the model is trained using the sub-module corresponding to the second identifier, and different models may use the same sub-module, so that some models with the same steps can use the same sub-module, avoiding different models using each of their own sub-modules and independently occupying various resources, thereby improving resource utilization.

[0069] Embodiment 2:

[0070] In order to accurately adjust the parameters of the model, based on the above embodiment, in the embodiment of the present application, the system further includes at least one storage module 104;

[0071] The training control server 102 is further used to send the target sample set and the target annotation set to any storage module, and send the address of the storage module;

[0072] The storage module 104 is used to receive and store the target sample set and the target annotation set;

[0073] Each second identified submodule 103 is specifically configured to obtain the target sample set and the target annotation set stored in the storage module at the address, and adjust its own parameters based on the target sample set and the target annotation set.

[0074] In the embodiment of the present application, when training the model of the first identification, the training control server saves the acquired target sample set and target annotation set to the storage module, and after receiving the target sample set and target annotation set sent by the training control server, the storage module saves the received target sample set and target annotation set. Each submodule of the second identification adjusts its own parameters based on the target sample set and target annotation set saved in the storage module.

[0075] The training control server can randomly send the target sample set and the target annotation set to any storage module, or send the target sample set and the target annotation set to pre-set corresponding storage modules respectively, wherein the storage module receiving the target sample set and the storage module receiving the target annotation set can be the same or different, and in order to enable the second identified submodule to adjust its own parameters based on the target sample set and the target annotation set, the address of the storage module storing the target sample set and the target annotation set is stored.

[0076] In addition, the task of model training is a graphics processing unit (GPU) resource-intensive task, and resource monitoring can be easily completed through Kubernetes. Therefore, in an embodiment of the present application, the operating environment of the storage module and each submodule of the second identifier can be Kubernetes.

[0077] Embodiment 3:

[0078] In order to accurately adjust the parameters of the model, based on the above embodiments, in the embodiment of the present application, the training control server 102 is specifically used to save the target sample set to a first storage module, and send the first address of the first storage module to the first submodule ranked first in the submodules of the model of the first identifier;

[0079] The first submodule 103 is used to receive the first address, obtain the target sample set stored in the first storage module at the first address, process the target sample set, save the first processing result to the second storage module, and send the second address of the second storage module to the training control server;

[0080] The training control server 102 is further used to receive the address sent by any submodule, and determine whether there are other submodules that are sorted after the submodule sent, and if so, send the address to the other submodule;

[0081] Any other submodule 103 is used to receive the address, obtain each processing result stored in the storage module of the address, process each processing result, save the second processing result to the third storage module, and send the third address of the third storage module to the training control server.

[0082] In an embodiment of the present application, the training control server is used to send the target sample set to the first storage module, and the first storage module stores the received target sample set. In addition, in order to make the submodule of the model of the first identification adjust the parameters of the model based on the samples in the target sample set, the training control server sends the first address of the first storage module to the first submodule ranked first among the submodules of the model of the first identification. For example, each submodule of the model of the first identification is determined to be submodule a, submodule b and submodule c, and the order of the submodules determined is submodule b, submodule c and submodule a, then the training control server sends the first address of the first storage module storing the target sample set to submodule b.

[0083] After receiving the first address, the first submodule ranked first in the submodules of the model of the first identification obtains the target sample set stored in the first storage module of the first address, and processes each sample in the target sample set to determine the first processing result. After obtaining the first processing result, the first submodule sends the first processing result to the second storage module, and in order to enable the submodules ranked after the first submodule in the submodules of the model of the first identification to be processed based on the first processing result, the first submodule sends the second address of the second storage module to the training control server.

[0084] After receiving the second address sent by the first submodule, the training control server sends the second address of the received second storage module to other submodules sorted after the first submodule in order to facilitate the submodules sorted after the first submodule to continue data processing. After receiving the second address sent by the training control server, the other submodules obtain each first processing result stored in the second storage module of the second address, process each first processing result obtained, obtain the processed second processing result, and send the second processing result to the third storage module. The third storage module receives the second processing result sent by the other submodule and saves the received second processing result. The other submodules send the third address of the third storage module to the training control server. After receiving the third address of the third storage module that saves the second processing result, the training controller determines whether there are any submodules after the other submodules. If there are any submodules after the other submodules, the above process continues.

[0085] In order to accurately adjust the parameters of the model, based on the above embodiments, in the embodiment of the present application, the training control server 102 is further used to receive an address sent by any submodule, and if it is determined that there are at least two other submodules sorted after the submodule, then the address is sent to the at least two other submodules; or, if the addresses sent by at least two submodules are received and it is determined that there are other submodules sorted after the two submodules, then the at least two addresses are sent to the other submodules;

[0086] The other submodule 103 is used to obtain each processing result stored in the storage module of the at least two addresses if the at least two addresses are received, process the processing results of the corresponding order in the storage module of the at least two addresses, save the third processing result obtained after processing to a fourth storage module, and send the fourth address of the fourth storage module to the training control server.

[0087] In an embodiment of the present application, a model training usually includes multiple processes. For example, in natural language processing, the processing of a sample set usually includes loading preprocessing, word segmentation, training and other processes, wherein each process may correspond to one or several sub-modules, and some sub-modules may run simultaneously. For example, a sub-module is used to capture local semantic features, and another sub-module is used to capture temporal semantic features. The sub-modules that capture these two features may run simultaneously, and any sub-module that captures these two features does not need to use the processed data processed by another sub-module. Therefore, in order to improve the processing efficiency of the model, the execution order of the sub-modules may be combined with a directed acyclic graph, and the execution order of the sub-modules may be intuitively reflected through the directed acyclic graph and applied in the model training system.

[0088] Specifically, when some submodules can run simultaneously during this training process, after receiving the address sent by any submodule, the training control server determines whether there are at least two other submodules sorted after the submodule. If there are at least two other submodules sorted after the submodule, the address is sent to the at least two other submodules, and the at least two submodules respectively obtain each processing result stored in the address and process each processing result. The at least two submodules respectively obtain each processed processing result and send the corresponding obtained processing results to different storage modules respectively. In addition, the at least two submodules respectively send the addresses of the corresponding storage modules to the training control server.

[0089] If the training control server receives the addresses sent by at least two submodules, it determines whether there are other submodules ranked after the at least two submodules. If there are other submodules ranked after the at least two submodules, that is, the other submodules will continue to perform subsequent processing based on the processing results of the at least two submodules. Therefore, after determining that there are other submodules ranked after the at least two submodules, the training control server sends the at least two addresses to the other submodules. The other submodule obtains each correspondingly stored processing result in the at least two storage modules of the at least two addresses, and obtains the corresponding processing results in the at least two storage modules. The other submodule combines the corresponding processing results in the at least two storage modules, processes each combined result, obtains the processed third processing result, saves the obtained third processing result to the fourth storage module, and sends the fourth address of the fourth storage module to the training control server.

[0090] Since each second-identified submodule processes each sample in the target sample set in sequence, or processes each processing result after the corresponding processing of the target sample set in sequence, each processing result in the at least two storage modules is a processing result after the samples in the target sample set are processed in sequence, that is, the processing results of the same order in the at least two storage modules are the results after the same sample in the target sample set is processed. For example, the fifth processing result in the at least two storage modules is the processing result after the fifth sample in the target sample set is processed. Therefore, in the embodiment of the present application, the other submodule can obtain the processing results of the same order in the at least two storage modules and process the obtained processing results.

[0091] Each processing result may be a matrix or a vector. Taking the at least two storage modules as two storage modules, and the processing results in the two storage modules are both vectors as an example, the processing process of the other submodules may be to combine the vectors of corresponding order to obtain a new vector. Specifically, the vectors obtained from one of the storage modules may be sorted in front, and the vectors of corresponding order obtained from the other storage module may be sorted in the back, and combined into a new vector. Taking the at least two storage modules as two storage modules, and the processing results in the at least two storage modules are both matrices as an example, the processing process of the other submodules may be to perform dot multiplication of matrices of corresponding order to obtain a new matrix. Specifically, the matrix obtained from one of the storage modules may be dot multiplied with the matrix of corresponding order obtained from the other storage module to obtain a new matrix.

[0092] Figure 2 A schematic diagram of the execution order of sub-modules of a model provided in an embodiment of the present application combined with a directed acyclic graph.

[0093] Depend on Figure 2 It can be seen that after submodule 1 is executed, submodule 2 and submodule 4 process the processing result of submodule 1, submodule 3 processes the processing result of submodule 2, and submodule 5 processes the processing results of submodule 4 and submodule 3.

[0094] When recording the order of the submodules, the execution order can be stored in the code through List. Each List contains two attributes, "from" and "to". The value of the attribute is the identifier of the submodule, and all from-to pairs can form a directed acyclic graph.

[0095] The corresponding directed acyclic graph is Figure 2 For example, after receiving the address sent by submodule 1, the training control server determines that there are two other submodules sorted after submodule 1, and sends the address sent by submodule 1 to submodule 2 and submodule 4. Submodule 2 and submodule 4 respectively obtain each processing result stored in the storage module of the address, and process each obtained processing result respectively. In addition, after receiving the two addresses sent by submodule 3 and submodule 4, the training control server sends the two addresses to submodule 5, and submodule 5 obtains the processing results of the corresponding order in the storage modules of the two addresses for processing.

[0096] In order to accurately adjust the parameters of the model, based on the above embodiments, in the embodiment of the present application, the training control server 102 is further used to save the target annotation set and the parameters of each second-identified submodule to the fourth storage module, and send the fifth address of the fifth storage module to the second submodule ranked last in the submodules of the model of the first identification;

[0097] The second submodule 103 is used to obtain the target annotation set and the parameters of each second-identified submodule stored in the fifth storage module of the fifth address, and adjust the parameters of each second-identified submodule according to the target annotation set, the parameters of each second-identified submodule and the fourth processing result obtained by the second submodule.

[0098] In an embodiment of the present application, in order to realize the training of the model, the training control server saves the target annotation set and the parameters of each second-identified sub-module to the fifth storage module. In order to facilitate the training of the model of the first identification, the training control server also sends the fifth address of the fifth storage module to the second sub-module ranked last in the sub-modules of the model of the first identification.

[0099] The second submodule receives the fifth address, and obtains the target annotation set and the parameters of each second-identified submodule stored in the fifth storage module of the fifth address. In addition, the second submodule also receives the address of the storage module that stores the processing results of the previous submodule. Each processing result stored in the storage module is each processing result completed by the submodule that is sorted before the second submodule. The second submodule obtains each processing result stored at the address, processes each processing result, and obtains the fourth processing result after processing. The second submodule determines the loss value based on the fourth processing result and the target annotation set, and adjusts the parameters of each second-identified submodule based on the loss value to obtain the adjusted parameters of each second-identified submodule, thereby realizing the training of the model.

[0100] In order to implement the training of the model, on the basis of the above embodiments, in the embodiment of the present application, the second submodule 103 is further used to send the adjusted parameters of each second identified submodule to the training control server;

[0101] The training control server 102 is further configured to send the adjusted parameters of each second identified submodule to the corresponding identified submodule;

[0102] The submodule 103 is further configured to adjust its own corresponding parameters according to the received adjusted parameters.

[0103] Since in the embodiment of the present application, the submodule ranked last in the submodules with the second identifier is the second submodule, after the parameter adjustment of each submodule with the second identifier is completed, the adjusted parameters of each submodule with the second identifier are sent to the training control server.

[0104] In order to realize the training of the model of the first identification, after receiving the adjusted parameters of the sub-module of each second identification, the training controller sends the adjusted parameters of the sub-module of each second identification to the sub-module of the corresponding identification. After receiving the adjusted parameters, the sub-module of the corresponding identification uses the adjusted parameters to adjust its own corresponding parameters. Specifically, it adjusts its own corresponding parameters to the received corresponding adjusted parameters.

[0105] In addition, in an embodiment of the present application, after adjusting the parameters, each second-identified submodule can re-process the samples in the target sample set or each obtained processing result based on the adjusted parameters, and determine whether the loss function converges according to the processed results and the target annotation set. If the loss function has not converged, continue to adjust the parameters of each second-identified submodule until the loss function converges or other convergence conditions are met.

[0106] Embodiment 4:

[0107] In order to implement the training of the model, on the basis of the above embodiments, in the embodiment of the present application, any second identified submodule 103 is used to send an execution failure instruction to the training control server when the execution fails;

[0108] The training control server 102 is further configured to receive the execution failure instruction sent by the submodule, determine whether the total number of failures of the submodule during this training process exceeds a preset number threshold, and if not, send a re-execution instruction to the submodule;

[0109] The submodule 103 is used to receive the re-execution instruction, and obtain each fifth processing result saved in the storage module of the corresponding address, process each fifth processing result, save the sixth processing result to the sixth storage module, and send the sixth address of the sixth storage module to the training control server.

[0110] In an embodiment of the present application, there may be a situation where a submodule with a second identifier fails to execute. In order to successfully implement the training of the model, if any submodule with a second identifier fails to execute, an execution failure instruction is sent to the training control server. After receiving the execution failure instruction of any submodule with a second identifier, the training control server determines whether the submodule is faulty based on the total number of execution failures of the submodule during this training process. If the total number exceeds the preset number threshold, it means that the submodule may have failed, and there is no need to re-execute the submodule. If the total number does not exceed the number threshold, the training control server sends a re-execution instruction to the submodule.

[0111] After receiving the re-execution instruction sent by the training control server, the sub-module obtains each fifth processing result stored in the storage module of the corresponding address, processes each fifth processing result, and sends the sixth processing result obtained after processing the fifth processing result to the sixth storage module. The sixth storage module saves the received fifth processing result. In addition, the sub-module also sends the sixth address of the sixth storage module storing the sixth processing result to the training control server.

[0112] Embodiment 5:

[0113] In order to realize the training of the model, on the basis of the above embodiments, in the embodiment of the present application, the training control server 102 is also used to determine the execution progress of the training according to the sub-module currently performing data processing and the order of the sub-modules at a preset time interval, and send the execution progress to the training scheduling server.

[0114] In an embodiment of the present application, the training control server is also used to obtain the execution progress of this training and send the obtained execution progress to the training scheduling server. Specifically, the training control server can determine the execution progress of this training according to a preset time interval, wherein the execution progress is determined by obtaining the submodule to which the address of the storage module was sent for the last time before the current time, and determining the execution progress according to the order of the submodule in the submodule sequence. For example, there are 5 submodules in this training process, and the current operation is to the 3rd submodule, then the execution progress can be 3 / 5.

[0115] In order to realize the training of the model, on the basis of the above embodiments, in the embodiment of the present application, the training control server 102 is also used to dynamically register the training scheduling server 101 and maintain heartbeat with the training scheduling server.

[0116] In the embodiment of the present application, in order to realize the training of the model, the training control server is also used to dynamically register the training scheduling server, and inform the training scheduling server that it is online through dynamic registration. In addition, the training control server also maintains a heartbeat with the training scheduling server, and informs the training scheduling server that it is available, so that the training scheduling server can accurately send the second instruction to the training control server, so as to realize the training of the model. Among them, how to perform dynamic registration and how to perform heartbeat maintenance are existing technologies and will not be repeated here.

[0117] In order to improve resource utilization, based on the above embodiments, in the embodiment of the present application, the training control server 102 is also used to delete the data received by the training control server when the training task is executed after a preset time interval after the training task is completed.

[0118] In the embodiment of the present application, in order to avoid the training control server from storing too much useless data and causing waste of resources, the training control server may delete the data received during the execution of this training task after a preset time interval after the execution of this training task is completed. Specifically, the data received by the training control server may be deleted after a preset time interval after receiving the address sent by the last submodule, wherein the data includes the target sample set, the target annotation set, and the address sent by each submodule used in this training process.

[0119] Figure 3 A schematic diagram of a processing procedure of a training control server provided in an embodiment of the present application.

[0120] Depend on Figure 3It can be seen that the loading submodule in the training control server can first receive the second instruction sent by the training scheduling server, wherein the training control server will not directly obtain the data required for the training task, that is, the target sample set and the target annotation set. The data obtained by the training control server is sent by the training scheduling server.

[0121] The model parsing submodule in the training control server obtains each submodule of the model of the first identifier and the order of the submodules, and parses the order of the submodules into a directed acyclic graph.

[0122] The execution thread in the training control server is used to determine each second identified submodule and the order of the submodules, etc. The log search thread is responsible for communicating with each submodule, summarizing the detailed logs of the training execution process, and sending them to the training scheduling server through the communication thread.

[0123] In addition, after a preset time interval after each training task is completed, the post-cleaner in the training control server can delete the data received by the training control server.

[0124] In an embodiment of the present application, all communications between the training scheduling server, the training control server, each second identification submodule and the storage module can be completed using the Hypertext Transfer Protocol (HTTP) protocol and the Representational State Transfer (REST) ​​interface, that is, the training scheduling server can use the HTTP protocol and the REST interface to send the second instruction to the training control server; the training controller can also use the HTTP protocol and the REST interface to send the address of the storage module to each second identification submodule; the training control server can also use the HTTP protocol and the REST interface to send the target sample set and the target annotation set to the storage module; any second identification submodule can also use the HTTP protocol and the REST interface to send the corresponding processing result to the storage module. Specifically, how to use the HTTP protocol and the REST interface for communication is a prior art and will not be repeated here.

[0125] In the embodiment of the present application, after the model training is completed, the training scheduling server can also send the callback content to the device that sent the first instruction through the "callback address", so as to achieve the synchronization of the progress and results of the training task, which is efficient and reliable. In order to ensure the idempotence and integrity of the callback, the data sent to the device that sent the first instruction to the training scheduling server in the embodiment of the present application can include:

[0126]

[0127] Table 1

[0128] As can be seen from Table 1, the callback content includes a field named "callback", whose Chinese name is "business ID", which is data used to distinguish the identifier of each training process; it can also include a field named "TrainId", whose Chinese name is "scheduling ID", which is the ID number of the target sample set and the target annotation set obtained by the training scheduling server each time; of course, it can also include other customizable content.

[0129] Embodiment 6:

[0130] Figure 4 A schematic diagram of the execution process of a training scheduling server provided in an embodiment of the present application.

[0131] Depend on Figure 4 It can be seen that the task triggering thread of the training scheduling server can determine whether the preset time has been reached according to the preset time interval, and if the preset time has been reached, the identification of the corresponding model is obtained. The management thread of the training scheduling server is responsible for sending the second instruction to the training control server, the training progress process of the training scheduling server is used to receive the execution progress sent by the training control server, and the result processing thread of the training scheduling server is used to send the detailed process of the model training to the device that sent the first instruction to the training scheduling server.

[0132] In addition, in an embodiment of the present application, the staff can make the required training execution logic into a corresponding image according to the sub-process of each model and store it in the image warehouse. The training control server can run the corresponding image. After running the corresponding image, each image after running adjusts its own parameters based on the target sample set and the target annotation set.

[0133] Figure 5 A schematic diagram of a training process provided in an embodiment of the present application.

[0134] The staff will make the execution logic of the sub-process of each model into a corresponding image and store it in the image warehouse, and configure the running order when running the image. When training any model, the server where the sub-module is located obtains the corresponding image in the image warehouse in turn according to the running order, and runs the obtained image, which is equivalent to the sub-module of each second identification described in the embodiment of the present application adjusting its own parameters based on the target sample set and the target annotation set.

[0135] In the embodiment of the present application, the preset number threshold may be obtained by the training scheduling server and sent to the training control server. When the training scheduling server obtains the preset number threshold, it may be a training task table corresponding to the model of the first identification obtained, and the training task table contains the preset number threshold. In addition, the training task table also contains fields such as "training task ID", "first identification", "task description", and "callback address". Among them, the ID corresponding to the "training task ID" is different each time the model is trained, the "first identification" is used to identify which model is trained, and the "callback address" can actively transmit the details of the training process back to the side that initiates the training.

[0136] In addition, in an embodiment of the present application, the training scheduling server is also used to create a training instance table during each model training, and the training instance table records each scheduling of the training task, that is, each time information of obtaining data such as the target sample set and the target annotation set, etc., in addition to some basic task information, it also includes "training results" and "training log path". The training log is transmitted back to the training scheduling server in text form and saved to the NAS disk in the form of a "date / training instance ID / log file" hierarchy. The training manager table can also be recorded, which mainly records the heartbeat time of the current training control server.

[0137] Field Name Chinese name Id Identifier Name name Namespace Namespace Image Mirror Command Startup Command Env Environment variables Limits Resource Configuration TtlSecondsAfterFinished Survival time NodeSelector Node Selector

[0138] Table 2

[0139] Table 2 shows the field names and corresponding Chinese names required by the administrator when configuring the submodules in the embodiment of the present application, wherein the field name includes "Id", the corresponding Chinese name is "identifier", which is unique among multiple submodules, and the field is referenced in the submodule sequence to indicate which submodule it is; the field name includes "Name", the corresponding Chinese name is "name", which is specifically the submodule name, which is convenient for the staff to view in the application system; the field name includes "Namespace", the corresponding Chinese name is "namespace", which is specifically the submodule started in the specified namespace in the container environment, which is conducive to GPU resource planning and management; the field name includes "Image", the corresponding Chinese name is "image", which is specifically the specific image required to implement the function of the submodule; the field name It includes "Command", whose corresponding Chinese name is "startup command", which specifically refers to the execution command and parameters after the sub-module is started; the field name includes "Env", whose corresponding Chinese name is "environmental variable", which specifically refers to the parameters passed into the container when the container used by the sub-module is started; the field name includes "Limits", whose corresponding Chinese name is "resource configuration", which specifically refers to the maximum resources that the container can allocate to the sub-module; the field name includes "TtlSecondsAfterFinished", whose corresponding Chinese name is "survival time", which specifically refers to how long the container is retained and destroyed after the sub-module is processed; the field name includes "NodeSelector", whose corresponding Chinese name is "node selector", which specifically refers to the container running on the specified node.

[0140] Embodiment 7:

[0141] Figure 6 A schematic diagram of a process of a model training method provided in an embodiment of the present application, the process comprising:

[0142] S601: Receive a first instruction, and obtain a first identifier of a model carried in the first instruction;

[0143] S602: According to the pre-saved correspondence between the model identifier and the sample set and the annotation set, obtain the target sample set and the target annotation set corresponding to the first identifier; and send a second instruction carrying the target sample set and the target annotation set to the training control server.

[0144] The method is applied to a training scheduling server. The specific process of the training scheduling server executing the model training method can be found in the other embodiments above, and the specific content will not be repeated here.

[0145] Figure 7 A schematic diagram of a process of a model training method provided in an embodiment of the present application, the process comprising:

[0146] S701: Receive a second instruction carrying a target sample set and a target annotation set; wherein the second instruction also carries a first identifier of a model;

[0147] S702: According to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules, sequentially determine the second identifier of each submodule included in the model of the first identifier.

[0148] In a possible implementation, the method further includes:

[0149] The target sample set and the target annotation set are sent to any storage module, and the address of the storage module is sent.

[0150] In a possible implementation manner, sending the target sample set and the target annotation set to any storage module, and sending the address of the storage module includes:

[0151] The target sample set is saved in a first storage module, and a first address of the first storage module is sent to a first submodule ranked first among the submodules of the model of the first identifier;

[0152] Receive the address sent by any submodule, and determine whether there are other submodules that are ranked after the submodule that sends the address. If so, send the address to the other submodule.

[0153] In a possible implementation, the method further includes:

[0154] If an address sent by any submodule is received and it is determined that there are at least two other submodules sorted after the submodule, the address is sent to the at least two other submodules; or, if addresses sent by at least two submodules are received and it is determined that there are other submodules sorted after the two submodules, the at least two addresses are sent to the other submodules.

[0155] In a possible implementation, the method further includes:

[0156] The target annotation set and the parameters of each second-identified submodule are saved in a fifth storage module, and the fifth address of the fifth storage module is sent to the second submodule ranked last in the submodules of the model of the first identification.

[0157] In a possible implementation, the method further includes:

[0158] The adjusted parameters of each submodule with the second identifier are sent to the submodule with the corresponding identifier.

[0159] In a possible implementation, the method further includes:

[0160] Receive the execution failure instruction sent by any sub-module, determine whether the total number of failures of the sub-module during this training process exceeds a preset number threshold, and if not, send a re-execution instruction to the sub-module.

[0161] In a possible implementation, the method further includes:

[0162] According to the preset time interval, the execution progress of this training is determined according to the sub-module currently performing data processing and the order of the sub-modules, and the execution progress is sent to the training scheduling server.

[0163] In a possible implementation, the method further includes:

[0164] Dynamically register the training scheduling server and maintain heartbeat with the training scheduling server.

[0165] In a possible implementation, the method further includes:

[0166] After a preset time interval after the task is completed, the data received by the training control server when the training task is executed is deleted.

[0167] The method is applied to a training control server. The specific process of the training control server executing the model training method can be found in the other embodiments above, and the specific content will not be repeated here.

[0168] Figure 8 A schematic diagram of a process of a model training method provided in an embodiment of the present application, the process comprising:

[0169] S801: Obtain a target sample set and a target annotation set;

[0170] S802: Adjusting its own parameters based on the target sample set and the target annotation set.

[0171] In a possible implementation manner, obtaining a target sample set and a target annotation set includes:

[0172] Receive the address of the storage module sent by the training control server, and obtain the target sample set and the target annotation set stored in the storage module at the address.

[0173] In a possible implementation, the method further includes:

[0174] Receive the address sent by the training control server, obtain each processing result stored in the storage module of the address, process each processing result, save the second processing result to the third storage module, and send the third address of the third storage module to the training control server.

[0175] In a possible implementation, the method further includes:

[0176] If at least two addresses are received, each processing result stored in the storage module of the at least two addresses is obtained, each processing result is processed respectively, the third processing result obtained after processing is saved in the fourth storage module, and the fourth address of the fourth storage module is sent to the training control server.

[0177] In a possible implementation, the method further includes:

[0178] If the submodule is the second submodule ranked last in the submodules of the model with the first identifier, receiving the fifth address of the fifth storage module for storing the target annotation set and the parameters of each submodule with the second identifier sent by the training control server;

[0179] Obtain the target annotation set and the parameters of each second-identified sub-module stored in the fifth storage module of the fifth address, and adjust the parameters of each second-identified sub-module according to the target annotation set, the parameters of each second-identified sub-module and the fourth processing result obtained by processing the second sub-module.

[0180] In a possible implementation, the method further includes:

[0181] If the submodule is the second submodule ranked last among the submodules of the model with the first identifier, the adjusted parameters of each submodule with the second identifier are sent to the training control server.

[0182] In a possible implementation, the method further includes:

[0183] According to the received adjusted parameters, the corresponding parameters of the self are adjusted.

[0184] In a possible implementation, the method further includes:

[0185] When the execution fails, the execution failure instruction is sent to the training control server;

[0186] Receive the re-execution instruction, and obtain each fifth processing result saved in the storage module of the corresponding address, process each fifth processing result, save the sixth processing result to the sixth storage module, and send the sixth address of the sixth storage module to the training control server.

[0187] This method is applied to sub-modules that execute different functions. The specific process of sub-modules that execute different functions executing the model training method can be found in the other embodiments mentioned above, and the specific content will not be repeated here.

[0188] Fig. 9 A schematic diagram of a process of a model training method provided in an embodiment of the present application, the process comprising:

[0189] S901: Receive a first instruction, obtain a first identifier of a model carried in the first instruction, and obtain a target sample set and a target annotation set corresponding to the first identifier according to a pre-saved correspondence between the model identifier and the sample set and the annotation set;

[0190] S902: determining the second identifier of each submodule included in the model with the first identifier in sequence according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules;

[0191] S903: Adjust the parameters of each submodule of the second identifier based on the target sample set and the target annotation set.

[0192] In a possible implementation, adjusting the parameters of each submodule of the second identifier based on the target sample set and the target annotation set includes:

[0193] If it is determined that any submodule is sequenced after at least two parallel submodules, then the processing results of the at least two parallel submodules are obtained, and each processing result is processed respectively; or

[0194] If it is determined that there are at least two other sub-modules that are sequenced after any sub-module, the at least two other sub-modules are controlled to perform processing based on the processing result of the sub-module.

[0195] In a possible implementation, the method further includes:

[0196] The parameters of each second-identified submodule are adjusted according to the target annotation set, the parameters of each second-identified submodule and the fourth processing result obtained by the second submodule.

[0197] In a possible implementation, the method further includes:

[0198] The execution progress of this training is determined according to the submodule currently processing data and the order of the submodules at preset time intervals.

[0199] Embodiment 8:

[0200] Fig.10A schematic diagram of the structure of a model training device provided in an embodiment of the present application, the device comprising:

[0201] A first receiving and obtaining module 1001 is used to receive a first instruction and obtain a first identifier of a model carried in the first instruction;

[0202] The first processing module 1002 is used to obtain the target sample set and target annotation set corresponding to the first identifier according to the pre-saved correspondence between the model identifier and the sample set and the annotation set; and send the second instruction carrying the target sample set and the target annotation set to the training control server.

[0203] Fig.11 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, the device comprising:

[0204] A receiving module 1101 is used to receive a second instruction carrying a target sample set and a target annotation set; wherein the second instruction also carries a first identifier of a model;

[0205] The second processing module 1102 is used to determine the second identifier of each submodule included in the model of the first identifier in sequence according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules.

[0206] In a possible implementation, the second processing module 1102 is further configured to send the target sample set and the target annotation set to any storage module, and send the address of the storage module.

[0207] In a possible implementation, the second processing module 1102 is specifically configured to save the target sample set to a first storage module, and send the first address of the first storage module to the first submodule ranked first among the submodules of the model of the first identifier; receive the address sent by any submodule, and determine whether there are other submodules ranked after the sent submodule, and if so, send the address to the other submodules.

[0208] In a possible implementation, the second processing module 1102 is further used to receive an address sent by any submodule, and if it is determined that there are at least two other submodules sorted after the submodule, then the address is sent to the at least two other submodules; or, if addresses sent by at least two submodules are received, and it is determined that there are other submodules sorted after the two submodules, then the at least two addresses are sent to the other submodules.

[0209] In a possible implementation, the second processing module 1102 is also used to save the target annotation set and the parameters of each second-identified sub-module to a fifth storage module, and send the fifth address of the fifth storage module to the second sub-module ranked last in the sub-modules of the model of the first identification.

[0210] In a possible implementation manner, the second processing module 1102 is further configured to send the adjusted parameters of each submodule with a second identifier to the submodule with a corresponding identifier.

[0211] In a possible implementation, the second processing module 1102 is further used to receive an execution failure instruction sent by any submodule, and determine whether the total number of failures of the submodule during this training process exceeds a preset number threshold. If not, a re-execution instruction is sent to the submodule.

[0212] In a possible implementation, the second processing module 1102 is further configured to determine the execution progress of the training at a preset time interval according to the submodule currently performing data processing and the order of the submodules, and send the execution progress to the training scheduling server.

[0213] In a possible implementation manner, the second processing module 1102 is further configured to dynamically register the training scheduling server and perform heartbeat maintenance with the training scheduling server.

[0214] In a possible implementation manner, the second processing module 1102 is further configured to delete the data received by the training control server when the training task is executed after a preset time interval after the task is completed.

[0215] Fig.12 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, the device comprising:

[0216] An acquisition module 1201 is used to acquire a target sample set and a target annotation set;

[0217] The fourth processing module 1202 is used to adjust its own parameters based on the target sample set and the target annotation set.

[0218] In a possible implementation, the acquisition module 1201 is specifically configured to receive an address of a storage module sent by a training control server, and acquire the target sample set and the target annotation set stored in the storage module at the address.

[0219] In a possible implementation, the acquisition module 1201 is specifically used to receive an address sent by a training control server, acquire each processing result stored in a storage module of the address, process each processing result, save the second processing result to a third storage module, and send the third address of the third storage module to the training control server.

[0220] In a possible implementation, the acquisition module 1201 is also used to, if at least two addresses are received, obtain each processing result stored in the storage modules of the at least two addresses, process each processing result respectively, save the third processing result obtained after processing to a fourth storage module, and send the fourth address of the fourth storage module to the training control server.

[0221] In one possible implementation, the acquisition module 1201 is also used to receive the fifth address of the fifth storage module that stores the target annotation set and the parameters of each second-identified submodule sent by the training control server if the submodule is the second submodule ranked last among the submodules of the model identified by the first identifier; obtain the target annotation set and the parameters of each second-identified submodule stored in the fifth storage module of the fifth address, and adjust the parameters of each second-identified submodule according to the target annotation set, the parameters of each second-identified submodule and the fourth processing result obtained by processing the second submodule.

[0222] In a possible implementation, the acquisition module 1201 is also used to send the adjusted parameters of each second-identified submodule to the training control server if the submodule is the second submodule ranked last in the submodules of the model with the first identifier.

[0223] In a possible implementation, the acquisition module 1201 is further configured to adjust its own corresponding parameters according to the received adjusted parameters.

[0224] In a possible implementation, the fourth processing module 1202 is also used to send an execution failure instruction to the training control server when the execution fails; receive the re-execution instruction, and obtain each fifth processing result stored in the storage module of the corresponding address, process each fifth processing result, save the sixth processing result to the sixth storage module, and send the sixth address of the sixth storage module to the training control server.

[0225] Fig.13 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, the device comprising:

[0226] The second receiving and obtaining module 1301 is used to receive the first instruction, obtain the first identifier of the model carried in the first instruction, and obtain the target sample set and target annotation set corresponding to the first identifier according to the pre-stored correspondence between the model identifier and the sample set and the annotation set;

[0227] The third processing module 1302 is used to determine the second identifier of each submodule included in the model of the first identifier in turn according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules; and adjust the parameters of the submodule of the first identifier based on the target sample set and the target annotation set.

[0228] In one possible implementation, the third processing module 1302 is specifically used to obtain the processing results of the at least two parallel sub-modules if it is determined that any sub-module is sorted after at least two parallel sub-modules, and process each processing result separately; or if it is determined that there are at least two other sub-modules sorted after any sub-module, control the at least two other sub-modules to perform processing based on the processing result of the sub-module.

[0229] In a possible implementation, the third processing module 1302 is further used to adjust the parameters of each second-identified submodule according to the target annotation set, the parameters of each second-identified submodule and the fourth processing result obtained by processing the second submodule.

[0230] In a possible implementation, the third processing module 1302 is further configured to determine the execution progress of the training according to the submodule currently performing data processing and the order of the submodules at a preset time interval.

[0231] Embodiment 9:

[0232] Fig.14 This is a schematic diagram of the structure of an electronic device provided by the present application. Based on the above embodiments, the present application embodiment further provides an electronic device, such as Fig.14 As shown, it includes: a processor 1401, a communication interface 1402, a memory 1403 and a communication bus 1404, wherein the processor 1401, the communication interface 1402, and the memory 1403 communicate with each other through the communication bus 1404;

[0233] The memory 1403 stores a computer program. When the program is executed by the processor 1401, the processor 1401 performs the following steps:

[0234] Receiving a first instruction, and obtaining a first identifier of the model carried in the first instruction;

[0235] According to the pre-saved correspondence between the model identifier and the sample set and the annotation set, the target sample set and the target annotation set corresponding to the first identifier are obtained; and the second instruction carrying the target sample set and the target annotation set is sent to the training control server, so that the training control server determines the second identifier of each sub-module contained in the model of the first identifier in turn according to the pre-saved correspondence between the model identifier and the identifier of the sub-module and the correspondence between the order of the sub-modules.

[0236] Based on the above embodiments, an embodiment of the present invention further provides an electronic device, including: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus.

[0237] The memory stores a computer program, and when the program is executed by the processor 1401, the processor performs the following steps:

[0238] Receive a second instruction carrying a target sample set and a target annotation set; wherein the second instruction also carries a first identifier of the model;

[0239] According to the correspondence between the pre-saved model identifier and the sub-module identifier and the correspondence between the order of the sub-modules, the second identifier of each sub-module contained in the model with the first identifier is determined in turn, so that each sub-module with the second identifier can adjust its own parameters based on the target sample set and the target annotation set.

[0240] In a possible implementation, the method further includes:

[0241] The target sample set and the target annotation set are sent to any storage module, and the address of the storage module is sent.

[0242] In a possible implementation manner, sending the target sample set and the target annotation set to any storage module, and sending the address of the storage module includes:

[0243] The target sample set is saved in a first storage module, and a first address of the first storage module is sent to a first submodule ranked first among the submodules of the model of the first identifier;

[0244] Receive the address sent by any submodule, and determine whether there are other submodules that are ranked after the submodule that sends the address. If so, send the address to the other submodule.

[0245] In a possible implementation, the method further includes:

[0246] If an address sent by any submodule is received and it is determined that there are at least two other submodules sorted after the submodule, the address is sent to the at least two other submodules; or, if addresses sent by at least two submodules are received and it is determined that there are other submodules sorted after the two submodules, the at least two addresses are sent to the other submodules.

[0247] In a possible implementation, the method further includes:

[0248] The target annotation set and the parameters of each second-identified submodule are saved in a fifth storage module, and the fifth address of the fifth storage module is sent to the second submodule ranked last in the submodules of the model of the first identification.

[0249] In a possible implementation, the method further includes:

[0250] The adjusted parameters of each submodule with the second identifier are sent to the submodule with the corresponding identifier.

[0251] In a possible implementation, the method further includes:

[0252] Receive the execution failure instruction sent by any sub-module, determine whether the total number of failures of the sub-module during this training process exceeds a preset number threshold, and if not, send a re-execution instruction to the sub-module.

[0253] In a possible implementation, the method further includes:

[0254] According to the preset time interval, the execution progress of this training is determined according to the sub-module currently performing data processing and the order of the sub-modules, and the execution progress is sent to the training scheduling server.

[0255] In a possible implementation, the method further includes:

[0256] Dynamically register the training scheduling server and maintain heartbeat with the training scheduling server.

[0257] In a possible implementation, the method further includes:

[0258] After a preset time interval after the task is completed, the data received by the training control server when the training task is executed is deleted.

[0259] Based on the above embodiments, an embodiment of the present invention further provides an electronic device, including: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus.

[0260] The memory stores a computer program, and when the program is executed by the processor 1401, the processor performs the following steps:

[0261] Obtain target sample set and target annotation set;

[0262] Based on the target sample set and the target annotation set, the parameters thereof are adjusted.

[0263] In a possible implementation manner, obtaining a target sample set and a target annotation set includes:

[0264] Receive the address of the storage module sent by the training control server, and obtain the target sample set and the target annotation set stored in the storage module at the address.

[0265] In a possible implementation, the method further includes:

[0266] Receive the address sent by the training control server, obtain each processing result stored in the storage module of the address, process each processing result, save the second processing result to the third storage module, and send the third address of the third storage module to the training control server.

[0267] In a possible implementation, the method further includes:

[0268] If at least two addresses are received, each processing result stored in the storage module of the at least two addresses is obtained, each processing result is processed respectively, the third processing result obtained after processing is saved in the fourth storage module, and the fourth address of the fourth storage module is sent to the training control server.

[0269] In a possible implementation, the method further includes:

[0270] If the submodule is the second submodule ranked last in the submodules of the model with the first identifier, receiving the fifth address of the fifth storage module for storing the target annotation set and the parameters of each submodule with the second identifier sent by the training control server;

[0271] Obtain the target annotation set and the parameters of each second-identified sub-module stored in the fifth storage module of the fifth address, and adjust the parameters of each second-identified sub-module according to the target annotation set, the parameters of each second-identified sub-module and the fourth processing result obtained by processing the second sub-module.

[0272] In a possible implementation, the method further includes:

[0273] If the submodule is the second submodule ranked last among the submodules of the model with the first identifier, the adjusted parameters of each submodule with the second identifier are sent to the training control server.

[0274] In a possible implementation, the method further includes:

[0275] According to the received adjusted parameters, the corresponding parameters of the self are adjusted.

[0276] In a possible implementation, the method further includes:

[0277] When the execution fails, the execution failure instruction is sent to the training control server;

[0278] Receive the re-execution instruction, and obtain each fifth processing result saved in the storage module of the corresponding address, process each fifth processing result, save the sixth processing result to the sixth storage module, and send the sixth address of the sixth storage module to the training control server.

[0279] Based on the above embodiments, an embodiment of the present invention further provides an electronic device, including: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus.

[0280] The memory stores a computer program, and when the program is executed by the processor 1401, the processor performs the following steps:

[0281] Receiving a first instruction, obtaining a first identifier of a model carried in the first instruction, and obtaining a target sample set and a target annotation set corresponding to the first identifier according to a pre-saved correspondence between the model identifier and the sample set and the annotation set;

[0282] Determine in sequence the second identifier of each submodule included in the model of the first identifier according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules;

[0283] Based on the target sample set and the target annotation set, the parameters of each submodule of the second identification are adjusted.

[0284] In a possible implementation, adjusting the parameters of each submodule of the second identifier based on the target sample set and the target annotation set includes:

[0285] If it is determined that any submodule is sequenced after at least two parallel submodules, then the processing results of the at least two parallel submodules are obtained, and each processing result is processed respectively; or

[0286] If it is determined that there are at least two other sub-modules that are sequenced after any sub-module, the at least two other sub-modules are controlled to perform processing based on the processing result of the sub-module.

[0287] In a possible implementation, the method further includes:

[0288] The parameters of each second-identified submodule are adjusted according to the target annotation set, the parameters of each second-identified submodule and the fourth processing result obtained by the second submodule.

[0289] In a possible implementation, the method further includes:

[0290] The execution progress of this training is determined according to the submodule currently processing data and the order of the submodules at preset time intervals.

[0291] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0292] The communication interface is used for communication between the above electronic device and other devices.

[0293] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0294] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0295] Embodiment 10:

[0296] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device, and when the program is run on the electronic device, the electronic device implements the following steps when executing:

[0297] The memory stores a computer program, and when the program is executed by the processor, the processor performs the following steps:

[0298] Receiving a first instruction, and obtaining a first identifier of the model carried in the first instruction;

[0299] According to the pre-saved correspondence between the model identifier and the sample set and the annotation set, the target sample set and the target annotation set corresponding to the first identifier are obtained; and the second instruction carrying the target sample set and the target annotation set is sent to the training control server, so that the training control server determines the second identifier of each sub-module contained in the model of the first identifier in turn according to the pre-saved correspondence between the model identifier and the identifier of the sub-module and the correspondence between the order of the sub-modules.

[0300] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by a processor, and when the program runs on the processor, the processor implements the following steps when executing:

[0301] Receive a second instruction carrying a target sample set and a target annotation set; wherein the second instruction also carries a first identifier of the model;

[0302] According to the correspondence between the pre-saved model identifier and the sub-module identifier and the correspondence between the order of the sub-modules, the second identifier of each sub-module contained in the model with the first identifier is determined in turn, so that each sub-module with the second identifier can adjust its own parameters based on the target sample set and the target annotation set.

[0303] In a possible implementation, the method further includes:

[0304] The target sample set and the target annotation set are sent to any storage module, and the address of the storage module is sent.

[0305] In a possible implementation manner, sending the target sample set and the target annotation set to any storage module, and sending the address of the storage module includes:

[0306] The target sample set is saved in a first storage module, and a first address of the first storage module is sent to a first submodule ranked first among the submodules of the model of the first identifier;

[0307] Receive the address sent by any submodule, and determine whether there are other submodules that are ranked after the submodule that sends the address. If so, send the address to the other submodule.

[0308] In a possible implementation, the method further includes:

[0309] If an address sent by any submodule is received and it is determined that there are at least two other submodules sorted after the submodule, the address is sent to the at least two other submodules; or, if addresses sent by at least two submodules are received and it is determined that there are other submodules sorted after the two submodules, the at least two addresses are sent to the other submodules.

[0310] In a possible implementation, the method further includes:

[0311] The target annotation set and the parameters of each second-identified submodule are saved in a fifth storage module, and the fifth address of the fifth storage module is sent to the second submodule ranked last in the submodules of the model of the first identification.

[0312] In a possible implementation, the method further includes:

[0313] The adjusted parameters of each submodule with the second identifier are sent to the submodule with the corresponding identifier.

[0314] In a possible implementation, the method further includes:

[0315] Receive the execution failure instruction sent by any sub-module, determine whether the total number of failures of the sub-module during this training process exceeds a preset number threshold, and if not, send a re-execution instruction to the sub-module.

[0316] In a possible implementation, the method further includes:

[0317] According to the preset time interval, the execution progress of this training is determined according to the sub-module currently performing data processing and the order of the sub-modules, and the execution progress is sent to the training scheduling server.

[0318] In a possible implementation, the method further includes:

[0319] Dynamically register the training scheduling server and maintain heartbeat with the training scheduling server.

[0320] In a possible implementation, the method further includes:

[0321] After a preset time interval after the task is completed, the data received by the training control server when the training task is executed is deleted.

[0322] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by a processor, and when the program runs on the processor, the processor implements the following steps when executing:

[0323] Obtain target sample set and target annotation set;

[0324] Based on the target sample set and the target annotation set, the parameters thereof are adjusted.

[0325] In a possible implementation manner, obtaining a target sample set and a target annotation set includes:

[0326] Receive the address of the storage module sent by the training control server, and obtain the target sample set and the target annotation set stored in the storage module at the address.

[0327] In a possible implementation, the method further includes:

[0328] Receive the address sent by the training control server, obtain each processing result stored in the storage module of the address, process each processing result, save the second processing result to the third storage module, and send the third address of the third storage module to the training control server.

[0329] In a possible implementation, the method further includes:

[0330] If at least two addresses are received, each processing result stored in the storage module of the at least two addresses is obtained, each processing result is processed respectively, the third processing result obtained after processing is saved in the fourth storage module, and the fourth address of the fourth storage module is sent to the training control server.

[0331] In a possible implementation, the method further includes:

[0332] If the submodule is the second submodule ranked last in the submodules of the model with the first identifier, receiving the fifth address of the fifth storage module for storing the target annotation set and the parameters of each submodule with the second identifier sent by the training control server;

[0333] Obtain the target annotation set and the parameters of each second-identified sub-module stored in the fifth storage module of the fifth address, and adjust the parameters of each second-identified sub-module according to the target annotation set, the parameters of each second-identified sub-module and the fourth processing result obtained by processing the second sub-module.

[0334] In a possible implementation, the method further includes:

[0335] If the submodule is the second submodule ranked last among the submodules of the model with the first identifier, the adjusted parameters of each submodule with the second identifier are sent to the training control server.

[0336] In a possible implementation, the method further includes:

[0337] According to the received adjusted parameters, the corresponding parameters of the self are adjusted.

[0338] In a possible implementation, the method further includes:

[0339] When the execution fails, the execution failure instruction is sent to the training control server;

[0340] Receive the re-execution instruction, and obtain each fifth processing result saved in the storage module of the corresponding address, process each fifth processing result, save the sixth processing result to the sixth storage module, and send the sixth address of the sixth storage module to the training control server.

[0341] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by a processor, and when the program runs on the processor, the processor implements the following steps when executing:

[0342] Receiving a first instruction, obtaining a first identifier of a model carried in the first instruction, and obtaining a target sample set and a target annotation set corresponding to the first identifier according to a pre-saved correspondence between the model identifier and the sample set and the annotation set;

[0343] Determine in sequence the second identifier of each submodule included in the model of the first identifier according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules;

[0344] Based on the target sample set and the target annotation set, the parameters of each submodule of the second identification are adjusted.

[0345] In a possible implementation, adjusting the parameters of each submodule of the second identifier based on the target sample set and the target annotation set includes:

[0346] If it is determined that any submodule is sequenced after at least two parallel submodules, then the processing results of the at least two parallel submodules are obtained, and each processing result is processed respectively; or

[0347] If it is determined that there are at least two other sub-modules that are sequenced after any sub-module, the at least two other sub-modules are controlled to perform processing based on the processing result of the sub-module.

[0348] In a possible implementation, the method further includes:

[0349] The parameters of each second-identified submodule are adjusted according to the target annotation set, the parameters of each second-identified submodule and the fourth processing result obtained by the second submodule.

[0350] In a possible implementation, the method further includes:

[0351] The execution progress of this training is determined according to the submodule currently processing data and the order of the submodules at preset time intervals.

[0352] Embodiment 11:

[0353] An embodiment of the present application also provides a computer program product, which, when executed by a computer, implements the model training method described in any of the above method embodiments applied to an electronic device.

[0354] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof, and all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions, and when the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated.

[0355] In the application embodiment, when the model is trained, the second identifier of each sub-module contained in the model with the first identifier is obtained, and the model is trained using the sub-module corresponding to the second identifier, and different models may use the same sub-module, so that some models with the same steps can use the same sub-module, avoiding different models using each of their own sub-modules and independently occupying various resources, thereby improving resource utilization.

[0356] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0357] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0358] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0359] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0360] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A model training system, characterized in that: The system includes: a training scheduling server, a training control server, and submodules that perform different functions; The training scheduling server is used to receive a first instruction, obtain a first identifier of a model carried in the first instruction, and obtain a target sample set and a target annotation set corresponding to the first identifier according to a pre-saved correspondence between the model identifier and the sample set and the annotation set; send a second instruction carrying the target sample set and the target annotation set to the training control server; the second instruction also carries the first identifier of the model to be trained; The training control server is used to determine the second identifier of each submodule included in the model of the first identifier in sequence according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules; Each second identified submodule is used to adjust its own parameters based on the target sample set and the target annotation set; Wherein, the system further comprises: at least one storage module; The training control server is specifically configured to save the target sample set to a first storage module, and send a first address of the first storage module to a first submodule ranked first among the submodules of the model of the first identifier; The first submodule is configured to receive the first address, obtain the target sample set stored in the first storage module at the first address, process the target sample set, save the first processing result to the second storage module, and send the second address of the second storage module to the training control server; The training control server is further used to receive the address sent by any submodule, and determine whether there are other submodules that are sorted after the submodule sent, and if so, send the address to the other submodule; Any other sub-module is used to receive the address, obtain each processing result stored in the storage module of the address, process each processing result, save the second processing result to the third storage module, and send the third address of the third storage module to the training control server.

2. The system according to claim 1, characterized in that The training control server is further configured to receive an address sent by any submodule, and if it is determined that there are at least two other submodules arranged after the submodule, send the address to the at least two other submodules; Or, if addresses sent by at least two submodules are received and it is determined that there are other submodules that are ranked after the two submodules, the at least two addresses are sent to the other submodules; The other sub-modules are used to obtain each processing result stored in the storage modules of the at least two addresses if the at least two addresses are received, process the processing results of the corresponding order in the storage modules of the at least two addresses, save the third processing result obtained after processing to the fourth storage module, and send the fourth address of the fourth storage module to the training control server.

3. The system according to claim 1, characterized in that The training control server is further used to save the target annotation set and the parameters of each second-identified submodule to a fifth storage module, and send the fifth address of the fifth storage module to the second submodule ranked last in the submodules of the model of the first identification; The second submodule is used to obtain the target annotation set and the parameters of each second-identified submodule stored in the fifth storage module of the fifth address, and adjust the parameters of each second-identified submodule according to the target annotation set, the parameters of each second-identified submodule and the fourth processing result obtained by the second submodule.

4. The system according to claim 3, characterized in that The second submodule is further used to send the adjusted parameters of each second identified submodule to the training control server; The training control server is further used to send the adjusted parameters of each submodule of the second identification to the submodule of the corresponding identification; The submodule is further configured to adjust its own corresponding parameters according to the received adjusted parameters.

5. The system according to claim 1, characterized in that Any second identified submodule is used to send an execution failure instruction to the training control server when execution fails; The training control server is further configured to receive the execution failure instruction sent by the submodule, determine whether the total number of failures of the submodule during this training process exceeds a preset number threshold, and if not, send a re-execution instruction to the submodule; The submodule is used to receive the re-execution instruction, and obtain each fifth processing result saved in the storage module of the corresponding address, process each fifth processing result, save the sixth processing result to the sixth storage module, and send the sixth address of the sixth storage module to the training control server.

6. The system according to claim 1, characterized in that The training control server is also used to determine the execution progress of this training according to the sub-module currently performing data processing and the order of the sub-modules at a preset time interval, and send the execution progress to the training scheduling server.

7. The system according to claim 1, characterized in that The training control server is also used to dynamically register the training scheduling server and maintain heartbeat with the training scheduling server.

8. The system according to any one of claims 1 to 7, characterized in that: The training control server is also used to delete the data received by the training control server when the training task is executed after a preset time interval after the task is completed.

9. A model training method, applied to a training scheduling server, characterized in that: The method comprises: Receiving a first instruction, and obtaining a first identifier of the model carried in the first instruction; According to the correspondence between the pre-saved model identifier and the sample set and the annotation set, a target sample set and a target annotation set corresponding to the first identifier are obtained; a second instruction carrying the target sample set and the target annotation set is sent to the training control server, so that the training control server, according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules, sequentially determines the second identifier of each submodule included in the model of the first identifier; the second instruction also carries the first identifier of the model to be trained; so that the training control server saves the target sample set to the first storage module, and sends the first address of the first storage module to the first submodule ranked first among the submodules of the model of the first identifier; so that the first submodule The module receives the first address, obtains the target sample set stored in the first storage module of the first address, processes the target sample set, saves the first processing result to the second storage module, and sends the second address of the second storage module to the training control server; so that the training control server receives the address sent by any sub-module, and determines whether there are other sub-modules sorted after the sent sub-module, and if so, sends the address to the other sub-modules; so that any other sub-module receives the address, obtains each processing result stored in the storage module of the address, processes each processing result, saves the second processing result to the third storage module, and sends the third address of the third storage module to the training control server.

10. A model training method, applied to a training control server, characterized in that: The method comprises: Receive a second instruction carrying a target sample set and a target annotation set; wherein the second instruction also carries a first identifier of the model; According to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules, the second identifier of each submodule included in the model of the first identifier is sequentially determined, so that each submodule of the second identifier adjusts its own parameters based on the target sample set and the target annotation set; Wherein, the method further comprises: The target sample set is saved to a first storage module, and the first address of the first storage module is sent to a first submodule ranked first among the submodules of the model of the first identifier; so that the first submodule receives the first address, obtains the target sample set saved in the first storage module of the first address, processes the target sample set, and saves the first processing result to a second storage module; and sends the second address of the second storage module; Receive the address sent by any sub-module, and determine whether there are other sub-modules that are sorted after the sub-module sent. If so, send the address to the other sub-module; so that any other sub-module receives the address, obtains each processing result stored in the storage module of the address, processes each processing result, saves the second processing result to the third storage module, and sends the third address of the third storage module.

11. A model training method, characterized in that: The method comprises: Receiving a first instruction, obtaining a first identifier of a model carried in the first instruction, and obtaining a target sample set and a target annotation set corresponding to the first identifier according to a pre-saved correspondence between the model identifier and the sample set and the annotation set; Determine in sequence the second identifier of each submodule included in the model of the first identifier according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules; Based on the target sample set and the target annotation set, adjusting the parameters of each submodule of the second identification; Wherein, the method further comprises: The target sample set is saved to a first storage module, and the first address of the first storage module is sent to a first submodule ranked first among the submodules of the model of the first identifier; so that the first submodule receives the first address, obtains the target sample set saved in the first storage module of the first address, processes the target sample set, and saves the first processing result to a second storage module; and sends the second address of the second storage module; Receive the address sent by any sub-module, and determine whether there are other sub-modules that are sorted after the sub-module sent. If so, send the address to the other sub-module; so that any other sub-module receives the address, obtains each processing result stored in the storage module of the address, processes each processing result, saves the second processing result to the third storage module, and sends the third address of the third storage module.

12. The method according to claim 11, characterized in that The adjusting the parameters of each submodule of the second identifier based on the target sample set and the target annotation set includes: If it is determined that any submodule is sequenced after at least two parallel submodules, then the processing results of the at least two parallel submodules are obtained, and each processing result is processed respectively; or If it is determined that there are at least two other sub-modules that are sequenced after any sub-module, the at least two other sub-modules are controlled to perform processing based on the processing result of the sub-module.

13. The method according to claim 11, characterized in that The method further comprises: According to the target annotation set, the parameters of each second-identified submodule and the fourth processing result obtained by processing the second submodule ranked last in the submodules of the first-identified model, the parameters of each second-identified submodule are adjusted.

14. The method according to claim 11, characterized in that The method further comprises: The execution progress of this training is determined according to the preset time interval, the sub-module currently performing data processing, and the order of the sub-modules.

15. A model training device, applied to a training scheduling server, characterized in that: The device comprises: A first receiving and obtaining module, configured to receive a first instruction and obtain a first identifier of a model carried in the first instruction; The first processing module is used to obtain the target sample set and the target annotation set corresponding to the pre-saved model identifier according to the correspondence between the first identifier and the sample set and the annotation set; send the second instruction carrying the target sample set and the target annotation set to the training control server; the second instruction also carries the first identifier of the model to be trained; so that the training control server saves the target sample set to the first storage module, and sends the first address of the first storage module to the first submodule ranked first in the submodules of the model of the first identifier; so that the first submodule receives the first address and obtains all the submodules stored in the first storage module at the first address. The target sample set is processed, the target sample set is processed, the first processing result is saved to the second storage module, and the second address of the second storage module is sent to the training control server; so that the training control server receives the address sent by any sub-module, and determines whether there are other sub-modules sorted after the sent sub-module, and if so, sends the address to the other sub-module; so that any other sub-module receives the address, obtains each processing result saved in the storage module of the address, processes each processing result, saves the second processing result to the third storage module, and sends the third address of the third storage module to the training control server.

16. A model training device, applied to a training control server, characterized in that: The device comprises: A receiving module, configured to receive a second instruction carrying a target sample set and a target annotation set; wherein the second instruction also carries a first identifier of the model; A second processing module, configured to sequentially determine a second identifier of each submodule included in the model of the first identifier according to a pre-saved correspondence between the model identifier and the identifier of the submodule and a correspondence between the order of the submodules; Among them, the second processing module is also used to save the target sample set to the first storage module, and send the first address of the first storage module to the first submodule ranked first in the submodule of the model of the first identifier; so that the first submodule receives the first address, obtains the target sample set stored in the first storage module of the first address, processes the target sample set, and saves the first processing result to the second storage module; sends the second address of the second storage module; receives the address sent by any submodule, and determines whether there are other submodules ranked after the sent submodule, and if so, sends the address to the other submodules; so that any other submodule receives the address, obtains each processing result stored in the storage module of the address, processes each processing result, saves the second processing result to the third storage module, and sends the third address of the third storage module.

17. A model training device, characterized in that: The device comprises: A second receiving and obtaining module, configured to receive a first instruction, obtain a first identifier of a model carried in the first instruction, and obtain a target sample set and a target annotation set corresponding to the first identifier according to a pre-saved correspondence between the model identifier and the sample set and the annotation set; A third processing module is used to determine the second identifier of each submodule included in the model of the first identifier in sequence according to the pre-saved correspondence between the model identifier and the identifier of the submodule and the correspondence between the order of the submodules; based on the target sample set and the target annotation set, adjust the parameters of the submodules included in the model of the first identifier; Among them, the third processing module is also used to save the target sample set to the first storage module, and send the first address of the first storage module to the first submodule ranked first in the submodule of the model of the first identifier; so that the first submodule receives the first address, obtains the target sample set stored in the first storage module of the first address, processes the target sample set, and saves the first processing result to the second storage module; sends the second address of the second storage module; receives the address sent by any submodule, and determines whether there are other submodules ranked after the sent submodule, and if so, sends the address to the other submodules; so that any other submodule receives the address, obtains each processing result stored in the storage module of the address, processes each processing result, saves the second processing result to the third storage module, and sends the third address of the third storage module.

18. An electronic device, characterized in that: The electronic device includes at least a processor and a memory, and the processor is used to execute the steps of the model training method described in any one of claims 9 to 14 when executing a computer program stored in the memory.

19. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, performs the steps of the model training method described in any one of claims 9 to 14.

20. A computer program product, characterized in that Its computer program product includes: computer program code, when the computer program code is run on a computer, it enables the computer to execute the steps of the model training method as described in any one of claims 9 to 14.

Citation Information

Patent Citations

  • Model acquisition method, device, electronic device, and computer-readable storage medium

    CN108197664A

  • Model training method, object detection method and related device

    CN109492698A

  • Dynamic fusion method and device for a machine learning model

    CN109657727A

  • Model training method and device, text processing method and device and electronic equipment

    CN111324732A

  • Classification model training method and device, electronic equipment and storage medium

    CN112560912A