Model training method and device, and storage medium

Abstract

The embodiment of the application discloses a model training method and device and a storage medium, and belongs to the field of artificial intelligence. The method comprises the following steps: after a target sub-node receives global model parameters from a center node, the target sub-node fuses the received global model parameters and local model parameters of a target model of the target sub-node to obtain fused model parameters. In this way, the obtained fused model parameters simultaneously contain local information and global information of the target sub-node. Then, the target model is updated by using the fused model parameters containing the local information and the global information, and the target model is trained by using a local training set. The precision of the target model can be effectively improved. In this way, even if sample data in the local training set used by each sub-node participating in the training is different in distribution or tasks to be completed by the target model on each sub-node are different, the target model trained by using the method of the embodiment of the application can still meet the requirements of each sub-node.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a model training method and device and storage medium. BACKGROUND

[0002] At present, AI models are widely used in various industries. The training of the model requires a large amount of data, but as people's awareness of data protection increases, it is becoming more and more difficult to obtain data, and cross-node model training is becoming more and more important.

[0003] In related technologies, the center node can issue initialization global model parameters of a target model to each sub-node participating in learning, each sub-node can take the initialization global model parameters as the model parameters of the target model above itself, and train the target model through a local training set, and then upload the model parameters of the target model obtained after training to the center node. The center node can jointly average the model parameters reported by each sub-node to obtain updated global model parameters, and then the center node can issue the updated global model parameters to each sub-node again, and each sub-node repeats the above process until the model converges, and the updated global model parameters issued by the center node last time are taken as the model parameters of the target model above itself.

[0004] However, since the sample data distribution in the training set used by each sub-node to train the target model can be different, and the tasks of the target model on each sub-node can also be different, in the case where all sub-nodes use the same global model parameters as the model parameters of their own target model to train the target model, the target models with the same model parameters obtained by each sub-node cannot meet their own needs. SUMMARY

[0005] The embodiments of the present application provide a model training method, device and storage medium, which can make the model trained by each sub-node more suitable for its own needs and improve the accuracy of the model on each sub-node. The technical solution is as follows:

[0006] In a first aspect, a model training method is provided, which includes: receiving global model parameters of a target model from a center node; fusing the global model parameters of the target model and local model parameters to obtain fused model parameters, the local model parameters being model parameters obtained by training the target model based on a local training set; updating the local model parameters of the target model to the fused model parameters, and training the updated target model according to the local training set of the target sub-node.

[0007] wherein the global model parameter is obtained by the center node weighting and averaging the local model parameters of each sub-node according to the number of sample data of the local training set of each sub-node.

[0008] In the embodiments of the present application, since the fusion model parameter is obtained by the target sub-node fusing the local model parameter of the target sub-node and the global model parameter received from the center node, the local information and the global information of the target sub-node are both contained in the fusion model parameter. Therefore, the precision of the target model can be effectively improved by updating the target model with the fusion model parameter and training the target model with the local training set.

[0009] Optionally, the implementation process of fusing the global model parameter and the local model parameter of the target model to obtain the fusion model parameter can be: fusing the global model parameter and the local model parameter of the target model according to the parameter fusion rule of the target sub-node to obtain the fusion model parameter.

[0010] Optionally, the parameter fusion rule of the target sub-node includes a replacement probability of each local parameter group in the local model parameter of the target model, which is determined based on the attribute information of the target sub-node. In this case, the implementation process of fusing the global model parameter and the local model parameter of the target model according to the parameter fusion rule of the target sub-node to obtain the fusion model parameter can be: replacing the local parameter group in the local model parameter with a replacement probability greater than a first threshold with the corresponding parameter group in the global model parameter to obtain the fusion model parameter.

[0011] wherein the attribute information of the target sub-node can include at least one of the data amount and distribution characteristics of the sample data in the local training set, the computing capability of the target sub-node, and the scale information of the target model. The local parameter group is obtained by dividing a plurality of local parameter values in the local model parameter.

[0012] In the embodiments of the present application, since the attribute information of different sub-nodes can be different, each sub-node determines the replacement probability of each local parameter group in the local model parameter of the target model according to the attribute information of the sub-node, and replaces the local parameter group in the local model parameter with a replacement probability greater than a first threshold with the corresponding parameter group in the global model parameter to obtain the fusion model parameter. Therefore, the fusion model parameter obtained by each sub-node can be more suitable for the needs of each sub-node, can effectively reduce the influence of the different distributions of sample data in the training set for training the target model on the precision of the trained model, and can improve the precision of the target model finally trained by each sub-node.

[0013] Optionally, before the global model parameter and the local model parameter of the target model are fused, the parameter search granularity can also be determined according to the attribute information of the target subnode; the plurality of local parameter values included in the local model parameter are grouped according to the parameter search granularity, to obtain a plurality of local parameter groups; the to-be-optimized replacement probability of each local parameter group in the plurality of local parameter groups is obtained; and the to-be-optimized replacement probability of each local parameter group is iteratively optimized according to the local training set, to obtain the replacement probability of each local parameter group.

[0014] The parameter search granularity is used to indicate the number of parameter values included in each parameter group when the plurality of parameter values included in the model parameter are divided into parameter groups. The more the number of parameter values included in the parameter group, the coarser the parameter search granularity. The less the number of parameter values included in the parameter group, the finer the parameter search granularity.

[0015] In the embodiments of the present application, since the data amount of the sample data of the local training set of each subnode, the complexity of the sample data in the local training set, and the computing capability of each subnode and the scale of the target model deployed by each subnode can be different, the parameter search granularity determined by each subnode according to its own attribute information is more suitable for its own needs.

[0016] Optionally, the implementation process of the step of iteratively optimizing the to-be-optimized replacement probability of each local parameter group according to the local training set, to obtain the replacement probability of each local parameter group can be: determining a verification model parameter according to the to-be-optimized replacement probability of each local parameter group, the plurality of local parameter groups and the global model parameter; replacing the local model parameter of the target model with the verification model parameter to obtain a verification model; training the verification model according to the local training set to obtain an updated verification model; testing the updated verification model according to a verification set; if the test result does not satisfy a reference condition, updating the to-be-optimized replacement probability of each local parameter group, taking the updated probability as the to-be-optimized replacement probability, and returning to execute the step of determining a verification model parameter according to the to-be-optimized replacement probability of each local parameter group, the plurality of local parameter groups and the global model parameter, until the test result satisfies the reference condition, and taking the last updated probability as the replacement probability of each local parameter group.

[0017] The reference condition can be a loss function, precision, recall, accuracy, mean intersection over union, etc. of a model.

[0018] In the embodiments of the present application, each sub-node tests the verification model of itself according to the verification set of itself, and continuously optimizes the to-be-optimized replacement probability of each local parameter group in a case where the test result does not satisfy the reference condition, and then replaces part of the parameter values in the local model parameters with the corresponding parameter values in the global model parameters by using the replacement probability of the local parameter group obtained through optimization, to obtain the fusion model parameters, at this time, the fusion model parameters are more suitable for the data distribution and the detection task of the sub-node, and based on the fusion model parameters, the target model is trained, which can make the precision of the model obtained through training higher.

[0019] Optionally, the parameter fusion rule of the target sub-node includes an index of a to-be-replaced local parameter group that is set in advance according to attribute information of the target sub-node, and the local parameter group is obtained by grouping a plurality of local parameter values included in the local model parameters. In this case, the implementation process of fusing the global model parameters and the local model parameters of the target model to obtain the fusion model parameters according to the parameter fusion rule of the target sub-node can be: replacing the to-be-replaced local parameter group in the local model parameters with the corresponding parameter group in the global model parameters according to the index of the to-be-replaced local parameter group, to obtain the fusion model parameters.

[0020] In the embodiments of the present application, the index of the to-be-replaced parameter group is set in advance based on artificial experience, which can reduce the calculation amount of the model training process.

[0021] In a second aspect, a model training apparatus is provided, which has functions to implement the behaviors of the model training method in the first aspect. The model training apparatus includes at least one module for implementing the model training method provided in the first aspect.

[0022] In a third aspect, a model training apparatus is provided, which includes a processor and a memory in its structure. The memory is used to store programs supporting the model training apparatus to execute the model training method provided in the first aspect, and to store data involved in the model training method provided in the first aspect. The processor is configured to execute the programs stored in the memory.

[0023] In a fourth aspect, a computer readable storage medium is provided, which stores instructions, and when the instructions are run on a computer, the computer executes the model training method in the first aspect.

[0024] In a fifth aspect, a computer program product containing instructions is provided, and when the instructions are run on a computer, the computer executes the model training method in the first aspect.

[0025] The technical effects obtained by the second aspect, the third aspect, the fourth aspect and the fifth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be described here.

[0026] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0027] In the embodiments of the present application, after receiving the global model parameters from the center node, the target sub-node fuses the received global model parameters and the local model parameters of the target model of the target sub-node to obtain fusion model parameters. In this way, the obtained fusion model parameters contain both local information and global information of the target sub-node. Then, the target model is updated using the fusion model parameters containing both local information and global information, and the target model is trained using the local training set. This can effectively improve the accuracy of the target model. In this way, even if the sample data in the local training set used by each sub-node participating in the training is different or the tasks to be completed by the target model on each sub-node are different, the target model trained by the method of the embodiments of the present application can still meet the needs of each sub-node. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a system architecture diagram related to a model training method provided by the embodiments of the present application;

[0029] Figure 2 is a structural schematic diagram of a computer device provided by the embodiments of the present application;

[0030] Figure 3 is a flowchart of a model training method provided by the embodiments of the present application;

[0031] Figure 4 is a flowchart of a method for determining the replacement probability of each local parameter group provided by the embodiments of the present application;

[0032] Figure 5 is a flowchart of a method for obtaining fusion model parameters by a target sub-node A provided by the embodiments of the present application;

[0033] Figure 6 is a structural schematic diagram of a model training device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0034] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings.

[0035] Before explaining the embodiments of the present application in detail, the application scenarios of the embodiments of the present application will be described first.

[0036] The model training method provided in the embodiments of the present application can be used in cross-node deep learning scenarios such as federated learning and distributed learning. For example, in an autonomous driving scenario, because the autonomous driving data collected by each company is affected by the self-collection equipment and the weather, building style, traffic identifier, etc. of the region where the company is located, the autonomous driving data owned by different companies is different. In this case, cross-regional data sharing is generally not performed between companies, and the method of the embodiments of the present application can be used to jointly train the autonomous driving models of each company that owns different autonomous driving data, which can reduce the influence of the different autonomous driving data of each company on the precision of the trained model in the federated learning process, and can improve the precision of the model finally trained by each company. For another example, in a commodity information pushing scenario, an e-commerce client includes a commodity information pushing model that pushes commodity information of sold commodities to a customer. The types of commodities included in the commodity information pushed by each client to the customer can be different, that is, the tasks of the commodity information pushing models in each client are different, but the customer groups owned by each client can be the same. In this scenario, the method of the embodiments of the present application is used to jointly train the commodity information pushing models in the multiple clients, and the trained commodity information pushing model can better meet the requirements of each client.

[0037] It should be noted that the above are only some exemplary application scenarios given by the embodiments of the present application, and do not constitute a limitation on the application scenarios of the model training method provided by the embodiments of the present application.

[0038] Figure 1 is a system architecture diagram involved in a model training method provided by the embodiments of the present application. As shown in Figure 1 The system includes a center node 101 and a plurality of sub-nodes 102, wherein the plurality of sub-nodes 102 can be connected to the center node 101 through a wired or wireless network.

[0039] In the embodiments of the present application, the target model can be deployed on the center node 101 and the plurality of sub-nodes 102. The center node 101 issues global model parameters of the target model to the plurality of sub-nodes 102 participating in learning, and correspondingly, each sub-node 102 receives the global model parameters issued by the center node 101 and trains the target model of each sub-node 102 using the method of the embodiments of the present application.

[0040] It should be noted that in a possible implementation manner, the center node 101 can be a server or a server cluster, or a cloud platform capable of coordinating the federated learning process, and each sub-node 102 can be a user terminal such as a smartphone, a tablet computer, a notebook computer, etc.

[0041] Optionally, the center node 101 and each sub-node 102 can be deployed in a cloud environment, for example, the center node 101 and the plurality of sub-nodes 102 are all servers or virtual machines deployed in a cloud data center. Alternatively, the center node 101 and each sub-node can be a computer device deployed in an edge environment, and the embodiments of the present application do not limit this.

[0042] Figure 2 is a structural schematic diagram of a computer device provided by an embodiment of the present application. Figure 1 The center node and / or the target sub-node in the computer device can be implemented by the computer device. Referring to Figure 2 The computer device includes at least one processor 201, a communication bus 202, a memory 203, and at least one communication interface 204.

[0043] The processor 201 can include a general central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a microprocessor, or one or more integrated circuits for implementing the solutions of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0044] The communication bus 202 is used to transmit information between the above-mentioned components. The communication bus 202 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0045] The memory 203 may be a read-only memory (ROM), random access memory (RAM), other types of static storage devices capable of storing static information and instructions, other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by computer equipment, but not limited thereto. The memory 203 may exist independently, be connected to the processor 201 via the communication bus 202, or be integrated with the processor 201.

[0046] Communication interface 204 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 204 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0047] In one embodiment, processor 201 may include one or more CPUs, for example... Figure 2 CPU0 and CPU1 are shown in the diagram.

[0048] In one embodiment, the physical server may include multiple processors, such as Figure 2 The processors 201 and 205 are shown. Each of these processors can be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used for processing data.

[0049] In an embodiment, the computer device can further include an output device 206 and an input device 207. The output device 206 is in communication with the processor 201 and can display information in a variety of ways. For example, the output device 206 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, a projector, or the like. The input device 207 is in communication with the processor 201 and can receive user input in a variety of ways. For example, the input device 207 can be a mouse, a keyboard, a touch screen device, a sensor device, or the like.

[0050] The memory 203 is configured to store program code 208 for implementing the solutions of the present application, and the processor 201 is configured to execute the program code 208 stored in the memory 203. The computer device can implement the following Figure 3 The model training method provided in the embodiments.

[0051] Figure 3 is a flowchart of a model training method provided in the embodiments. The method can be applied to any sub-node in the model training system shown in Figure 1 , and hereinafter, one of the sub-nodes is taken as an example for description, which is referred to as a target sub-node, see Figure 3 , and the method includes the following steps:

[0052] Step 301: receiving global model parameters of a target model from a center node.

[0053] In the embodiments of the present application, the target model is deployed on the center node and each sub-node, each sub-node has its own local training set, and when starting to train the target model, each sub-node trains the target model according to its own local training set to obtain local model parameters of the target model, and then each sub-node uploads the local model parameters to the center node. Correspondingly, the center node receives the local model parameters uploaded by each sub-node, and performs weighted averaging on the received local model parameters of each sub-node according to the sample number of the local training set of each sub-node to obtain global model parameters. Then, the center node distributes the obtained global model parameters to each sub-node, and correspondingly, the target sub-node receives the global model parameters of the target model distributed by the center node.

[0054] It should be noted that the local model parameters uploaded by each sub-node to the center node can include all local parameter values of the target model deployed by itself, or can be partial local parameter values. For example, a certain sub-node can upload local parameter values of all layers of the target model, or upload local parameter values of some layers in the target model. After receiving the local model parameters uploaded by each sub-node, the center node can perform weighted average on the parameter values of the same part of the target model in the local model parameters uploaded by all sub-nodes to obtain global model parameters. For example, one sub-node uploads local parameter values of the first to fifth convolutional layers of the target model to the center node, and another sub-node uploads local parameter values of the first to tenth convolutional layers of the target model to the center node. The center node can perform weighted average on the local parameter values of the first to fifth convolutional layers in the local model parameters uploaded by the two sub-nodes to obtain global parameter values of the first to fifth convolutional layers, and distribute the obtained global parameter values of the first to fifth convolutional layers to each sub-node as global model parameters.

[0055] For example, for the global parameter value ω k in the global model parameters, it can be determined by the following formula:

[0056]

[0057] Where i is the i-th sub-node participating in model training, n i is the number of samples of the local training set of the i-th sub-node, n is the sum of the number of samples of the local training set of all sub-nodes participating in model training, is the corresponding local parameter value in the local model parameters of the i-th sub-node, and k is a positive integer greater than 0 and not less than the number of global model parameters. k

[0058] Step 302: Fuse the global model parameters and the local model parameters of the target model to obtain fused model parameters.

[0059] In the embodiments of the present application, the target sub-node can fuse the global model parameters and the local model parameters of the target model according to its own parameter fusion rule to obtain fused model parameters.

[0060] ​In an implementation, the parameter fusion rule of the target sub-node can be a replacement probability of each local parameter group in the local model parameters of the target model determined based on the attribute information of the target sub-node. The local parameter group is obtained by dividing a plurality of local parameter values in the local model parameters. In this case, the target sub-node can first determine a plurality of local parameter groups based on the attribute information of the target sub-node, and then determine the replacement probability of each local parameter group. Then, the target sub-node replaces the local parameter group in the local model parameters with a replacement probability greater than a first threshold with the corresponding parameter group in the global model parameters to obtain the fusion model parameters.

[0061] As shown in FIG. 30, the target sub-node can determine the local parameter group and determine the replacement probability of each local parameter group in the local model parameters by the following steps 3021-3024. Figure 4

[0062] 3021: Determine a parameter search granularity according to the attribute information of the target sub-node.

[0063] In the embodiments of the present application, the attribute information of the target sub-node can include at least one of the data amount and distribution characteristics of the sample data in the local training set, the computing capability of the target sub-node, and the scale information of the target model.

[0064] The distribution characteristics of the sample data in the local training set are used to indicate the complexity of the sample data in the local training set or the difference between the sample data in the local training set of the target sub-node and the sample data in the local training set of other sub-nodes. The computing capability of the target sub-node can be represented by the number of floating point operations per second of the target sub-node, and of course can also be represented by other parameters. The number of floating point operations per second of the target sub-node is determined by the performance of the hardware device of the target sub-node. For example, when the target sub-node is a notebook computer, the computing capability of the target sub-node is determined by the CPU chip, memory size, bandwidth, etc. of the notebook computer. The scale information of the target model is used to indicate the size of the target model. For example, the scale information of the target model includes the number of layers of the target model or the number of parameters of the target model.

[0065] For different attribute information, the target sub-node can determine the parameter search granularity in different ways. The parameter search granularity is used to indicate the number of parameter values included in each parameter group when the model parameters are divided into a plurality of parameter groups. The more the number of parameter values included in the parameter group, the coarser the parameter search granularity, and the fewer the number of parameter values included in the parameter group, the finer the parameter search granularity.

[0066] ​Exemplarily, when the attribute information of the target sub-node is the data amount of the sample data in the local training set, the target sub-node can determine the parameter search granularity according to how much the data amount of the sample data in the local training set is. If the data amount of the sample data in the local training set is small, the target sub-node can select a coarser parameter search granularity, for example, divide the parameter values of one convolutional layer into a group, that is, as a parameter group, or divide the parameter values of several adjacent convolutional layers into a group. If the data amount of the sample data in the local training set is large, the target sub-node can select a finer parameter search granularity, for example, can take each parameter value as a parameter group.

[0067] Exemplarily, the target sub-node can store a plurality of data amount ranges and the parameter search granularity corresponding to each data amount range pre-set by the center node, based on which the target sub-node can first determine which range of the plurality of pre-set data amount ranges the data amount of the sample data in the local training set belongs to, and then take the parameter search granularity corresponding to the data amount range to which the data amount of the sample data of the target sub-node belongs as the parameter search granularity of the target sub-node.

[0068] For example, the pre-set data amount ranges are three, wherein the data amount of the sample data in the first data amount range is 0 to 100, and the corresponding parameter search granularity is to take the parameter values of two convolutional layers as a parameter group; the data amount of the sample data in the second data amount range is 101 to 300, and the corresponding parameter search granularity is to take the parameter values of one convolutional layer as a parameter group; the data amount of the sample data in the third data amount range is greater than 300, and the corresponding parameter search granularity is to take one parameter value as a parameter group. In this way, when the data amount of the sample data in the local training set of the target sub-node is 180, it corresponds to the second data amount range, and the parameter search granularity of the target sub-node is to take the parameter values of one convolutional layer as a parameter group.

[0069] When the attribute information of the target sub-node is the distribution feature of the sample data in the local training set, and the distribution feature is used to indicate the complexity of the sample data in the local training set, the target sub-node can determine the parameter search granularity according to the complexity of the sample data in the local training set. The more complex the sample data in the local training set is, the finer the parameter search granularity is selected, and the simpler the sample data in the local training set is, the coarser the parameter search granularity is selected.

[0070] Exemplarily, the complexity of the sample data in the local training set can be represented by the amount of annotations of the sample data in the local training set, where the amount of annotations of the sample data in the local training set refers to the total number of annotation information contained in each sample data in the local training set. In this case, the target sub-node can determine the parameter search granularity according to how much the amount of annotations of the sample data in the local training set is, when the amount of annotations of the sample data in the local training set is more, it indicates that the complexity of the sample data in the local training set is higher, and a finer parameter search granularity can be selected, when the amount of annotations of the sample data in the local training set is less, it indicates that the complexity of the sample data in the local training set is lower, and a coarser parameter search granularity can be selected.

[0071] For example, the target sub-node can store a plurality of annotation quantity ranges and the corresponding parameter search granularity of each annotation quantity range set by the center node in advance, and in the model training process, the target sub-node can first determine which range of the plurality of annotation quantity ranges the amount of annotations of the sample data in the local training set belongs to, and then take the parameter search granularity corresponding to the annotation quantity range to which the amount of annotations of the sample data belongs as the parameter search granularity of the target sub-node.

[0072] Alternatively, the complexity of the sample data in the local training set can also be represented by the correlation between the sample data in the local training set. In this case, the target sub-node can first obtain the similarity matrix of the sample data in the local training set, and then obtain the similarity between the sample data in the local training set according to the similarity matrix, which is a value between 0 and 1. The greater the value, the higher the similarity of the sample data in the local training set, and the lower the complexity of the sample data in the local training set, at which time a coarser parameter search granularity can be selected. Correspondingly, the smaller the value, the lower the similarity of the sample data in the local training set, and the higher the complexity of the sample data in the local training set, at which time a finer parameter search granularity can be selected.

[0073] For example, when the similarity of the sample data in the local training set is a value between 0.4 and 0.8, the target sub-node can select a coarser parameter search granularity, for example, taking the parameter value of a convolution layer as a parameter group, when the similarity of the sample data in the local training set is a value between 0 and 0.4, a finer parameter search granularity can be selected, for example, taking each parameter value as a parameter group.

[0074] Optionally, when the attribute information of the target sub-node is a distribution feature of the sample data in the local training set, and the distribution feature is used to indicate the difference between the sample data in the local training set of the target sub-node and the sample data in the local training set of other sub-nodes, the target sub-node can determine the parameter search granularity according to the difference. The greater the difference between the sample data in the local training set of the target sub-node and the sample data in the local training set of other sub-nodes, the finer the selected parameter search granularity. The smaller the difference between the sample data in the local training set of the target sub-node and the sample data in the local training set of other sub-nodes, the coarser the selected parameter search granularity.

[0075] For example, the difference between the sample data in the local training set of the target sub-node and the sample data in the local training set of other sub-nodes can be represented by the cosine similarity of the local model parameters of the target sub-node and the global model parameters received from the center node. In this case, the target sub-node can subtract the corresponding parameter value in the global model parameters from the parameter value in the local model parameters to obtain a first vector, and then update the local model parameters by training the target model on the local training set to obtain reference model parameters. Then, the target sub-node subtracts the corresponding parameter value in the reference model parameters from the parameter value in the local model parameters to obtain a second vector. The cosine similarity of the first vector and the second vector is calculated. The cosine similarity is a value between -1 and 1. The greater the cosine similarity, the smaller the difference between the sample data in the local training set of the target sub-node and the sample data in the local training set of other sub-nodes, and the lower the complexity of the sample data in the local training set of the target sub-node. In this case, a coarser parameter search granularity is selected. The smaller the cosine similarity, the greater the difference between the sample data in the local training set of the target sub-node and the sample data in the local training set of other sub-nodes, and the higher the complexity of the sample data in the local training set of the target sub-node. In this case, a finer parameter search granularity is selected. For example, when the cosine similarity of the target sub-node is less than 0, a finer parameter search granularity can be selected. When the cosine similarity of the target sub-node is greater than or equal to 0, a coarser parameter search granularity can be selected.

[0076] When the attribute information of the target sub-node is its own computing capability, the target sub-node can determine the parameter search granularity according to the strength of its own computing capability. It should be noted that the finer the parameter search granularity selected by the target sub-node, the more local parameter groups the local model parameters divide in the subsequent step, and the greater the amount of calculation required to generate the optimization replacement probability of each local parameter group. Therefore, the target sub-node needs to have stronger computing capability. Based on this, when the computing capability of the target sub-node is strong, a finer parameter search granularity can be selected. When the computing capability of the target sub-node is weak, a coarser parameter search granularity can be selected.

[0077] For example, when the computing capability of the target sub-node is represented by the number of floating point operations per second of the target sub-node, the target sub-node can store a plurality of floating point operation number ranges and the corresponding parameter search granularity of each floating point operation number range preset by the center node. Based on this, the target sub-node can first determine which range of the plurality of preset floating point operation number ranges the number of floating point operations per second of the target sub-node belongs to, and then use the parameter search granularity corresponding to the floating point operation number range to which the number of floating point operations per second of the target sub-node belongs as the parameter search granularity of the target sub-node.

[0078] When the attribute information of the target model is the scale information of the target model, the parameter search granularity can be determined according to the number of layers of the target model or the number of parameters of the target model. When the target model has a large number of layers or a large number of parameters, a coarse parameter search granularity can be selected; when the target model has a small number of layers or a small number of parameters, a fine parameter search granularity can be selected.

[0079] For example, when the number of layers of the target model is 3 to 10, the parameter search granularity determined by the target sub-node can be to use one parameter value as one parameter group; when the number of layers of the target model is 11 to 20, the parameter search granularity determined by the target sub-node can be to use the parameter values of one layer as one parameter group; and when the number of layers of the target model is greater than 20, the target sub-node can use the parameter values of two or three adjacent layers as one parameter group.

[0080] Optionally, in some possible implementations, the target sub-node can also combine multiple types of information in the attribute information of the target sub-node to determine the parameter search granularity, which will not be described herein again.

[0081] 3022: Grouping a plurality of local parameter values in the local model parameters according to the parameter search granularity to obtain a plurality of local parameter groups.

[0082] After determining the parameter search granularity, the target sub-node can group a plurality of local parameter values in the local model parameters according to the number of parameter values included in the parameter group indicated by the parameter search granularity to obtain a plurality of local parameter groups.

[0083] For example, if the parameter search granularity indicates that one parameter group includes the parameter values of one layer, when the target model includes m layers, according to the parameter search granularity, the local parameter values of each layer in the m layers can be used as one local parameter group, thereby obtaining m local parameter groups.

[0084] In addition, the target sub-node can also use the same parameter search granularity to group a plurality of global parameter values included in the global model parameters, thereby obtaining a plurality of global parameter groups of the same number and corresponding to the plurality of local parameter groups.

[0085] 3023: Obtain the to-be-optimized replacement probability of each local parameter group in the plurality of local parameter groups.

[0086] In an implementation manner, the target sub-node can randomly generate the to-be-optimized replacement probability of each local parameter group, or the target sub-node can also set the to-be-optimized replacement probability of each local parameter group as a preset value.

[0087] Continuing the foregoing example, when the target sub-node divides the plurality of local parameter values included in the local model parameters of the target model into m local parameter groups ω1, ω2, ω3, …, ωm, the target sub-node can correspondingly generate m to-be-optimized replacement probabilities p1, p2, p3, …, pm.

[0088] 3024: Iteratively optimize the to-be-optimized replacement probability of each parameter group according to the local training set, to obtain the replacement probability of each parameter group.

[0089] Exemplarily, the target sub-node can iteratively optimize the to-be-optimized replacement probability of each local parameter group according to the local training set, to obtain the replacement probability of each local parameter group, by the following steps A-E.

[0090] A: Determine the verification model parameters according to the to-be-optimized replacement probability of each local parameter group, the plurality of local parameter groups and the global model parameters.

[0091] In the embodiments of the present application, the target sub-node can replace the local parameter group in the local model parameters with the to-be-optimized replacement probability greater than the first threshold value with the corresponding global parameter group in the global model parameters, and take the replaced local model parameters as the verification model parameters.

[0092] For example, the local parameter groups obtained after the plurality of local parameter values in the local model parameters of the target model are divided are ω1, ω2, ω3, …, ωm. The to-be-optimized replacement probability of each local parameter group is p1, p2, p3, …, pm. The global parameter groups obtained after the global parameter values included in the global model parameters are divided are ω1, ω2, ω3, …, ωm. m When the to-be-optimized replacement probability p1, p2, p3, …, pm is greater than the first threshold value, the target sub-node can replace the parameter group ω1, ω2 in the local model parameters with the corresponding global parameter group ω1, ω2 in the global model parameters, to obtain the verification model parameters ω1, ω2.

[0093] B: Replace the local model parameters of the target model with the verification model parameters, to obtain the verification model.

[0094] ​​​​After obtaining the verification model parameters, the target sub-node can directly replace the plurality of local parameter values included in the local model parameters of the target model with the parameter values included in the verification model parameters, so as to obtain the verification model.

[0095] Still taking the above example as an example, the local model parameters in the target model are replaced with the verification model parameters ω1, ω2, to obtain the verification model.

[0096] C: training the verification model according to the local training set to obtain an updated verification model.

[0097] After obtaining the verification model, the target sub-node can train the verification model according to the local training set to update the model parameters of the verification model, so as to obtain an updated verification model. The target sub-node can perform a specified number of iterations of training the verification model according to the local training set to obtain the updated verification model, or the target sub-node can perform multiple rounds of iteration training according to the local training set until the updated verification model converges.

[0098] After obtaining the updated verification model, in an implementation manner, the target sub-node can compare the loss function of the updated verification model with a preset reference loss value. If the loss function of the updated verification model is not greater than the preset reference loss value, the updated verification model meets the requirements of the target sub-node. At this time, the target sub-node can take the to-be-optimized replacement probability of each local parameter group as the final replacement probability. If the loss function of the updated verification model is greater than the preset reference loss value, it indicates that the updated verification model does not meet the requirements of the target sub-node. At this time, the target sub-node can update the to-be-optimized replacement probability of each local parameter group by using a reinforcement learning method, a reparameterization method, a multi-arm bandit method or other methods capable of updating the to-be-optimized replacement probability, take the updated probability as the to-be-optimized replacement probability, and return to perform steps A to C until the loss function of the verification model meets the requirements of the target sub-node, and then take the last updated probability as the replacement probability of each local parameter group.

[0099] Optionally, in another possible implementation manner, after obtaining the updated verification model, the target sub-node can also test the updated verification model by steps D and E, and then update the to-be-optimized replacement probability of the local parameter group according to the test result.

[0100] D: testing the updated verification model according to the verification set.

[0101] ​The target sub-node tests the updated verification model by using sample data in the verification set to obtain a test result. Then, the target sub-node can determine whether the test result satisfies the reference condition.

[0102] The reference condition can be an index capable of evaluating the model, such as a loss function, precision, recall, accuracy, mean intersection over union, etc. Embodiments of the present application do not limit this.

[0103] For example, when the reference condition is precision, the target sub-node can input each sample data in the verification set into the verification model, and the verification model can identify each sample data to output an identification result of each sample data. Then, the target sub-node can calculate the ratio between the number of correct identification results and the total number of identification results in the identification results obtained by the verification model, to obtain the precision of the verification model, that is, the test result of the verification model. Then, the precision of the verification model is compared with the preset reference precision. If the precision of the verification model is greater than the preset reference precision, it indicates that the test result of the verification model satisfies the reference condition. If the precision of the verification model is not greater than the preset reference precision, it indicates that the test result of the verification model does not satisfy the reference condition.

[0104] E: If the test result does not satisfy the reference condition, update the to-be-optimized replacement probability of each local parameter group, take the updated probability as the to-be-optimized replacement probability, and return to step A until the test result satisfies the reference condition. Then, take the last updated probability as the replacement probability of each local parameter group.

[0105] If it is determined through step D that the test result does not satisfy the reference condition, the target sub-node can update the to-be-optimized replacement probability of each local parameter group by using reinforcement learning, reparameterization method, multi-arm tiger machine method or other methods capable of updating the to-be-optimized replacement probability, take the updated probability as the to-be-optimized replacement probability, and return to step A until the test result of the verification model satisfies the reference condition. Then, take the last updated probability as the replacement probability of each local parameter group.

[0106] After the replacement probability of each local parameter group is obtained through the above steps, the replacement probability of each local parameter group in the local model parameter is compared with the first threshold in sequence, if the replacement probability is greater than the first threshold, the local parameter group corresponding to the replacement probability in the local model parameter is replaced by the corresponding global parameter group, if the replacement probability is not greater than the first threshold, the local parameter group corresponding to the replacement probability in the local model parameter is retained, after all the local parameter groups in the local model parameter whose replacement probability is greater than the first threshold are replaced by the corresponding global parameter groups, the fusion model parameter of the target model is obtained. Wherein, the number of local parameter groups whose replacement probability is greater than the first threshold can be one or multiple, and the embodiments of the present application do not limit this.

[0107] In another implementation manner, the parameter fusion rule of the target subnode can be the index of the local parameter group to be replaced in the local model parameter which is set in advance according to the attribute information of the target subnode, based on which, the target subnode can replace the local parameter group to be replaced in the local model parameter with the corresponding parameter group in the global model parameter according to the index of the local parameter group to be replaced which is set in advance, to obtain the fusion model parameter. Wherein, the index of the local parameter group to be replaced can be the index of one or multiple convolutional layer parameter groups.

[0108] It should be noted that in this implementation manner, the index of the local parameter group to be replaced can be manually pre-configured in the target subnode based on the attribute information of the target subnode, or can be automatically determined by the target subnode according to its own attribute information after the parameter values in the local model parameter are divided to obtain the local parameter groups, and the embodiments of the present application do not limit this.

[0109] Wherein, when the target subnode automatically determines the index of the local parameter group to be replaced according to its own attribute information, the target subnode can refer to the implementation principle of determining the parameter search granularity based on its own attribute information to control the number of indexes of the local parameter group to be replaced, that is, to control the number of local parameter groups to be replaced, and then select the indexes of the local parameter groups to be replaced based on the determined number of indexes.

[0110] Exemplarily, when the attribute information of the target sub-node is the data quantity of the sample data in the local training set, if the data quantity of the sample data in the local training set is small, the target sub-node can select a small quantity of indexes as the indexes of the local parameter groups to be replaced. For example, when the parameter value of a layer in the local model parameter is a local parameter group, if the data quantity in the local training set is small, the target sub-node can select the index of one local parameter group, that is, take one local parameter group as the parameter group to be replaced. If the data quantity of the sample data in the local training set is large, the target sub-node can select indexes of multiple local parameter groups as the indexes of the local parameter groups to be replaced, for example, pre-set the indexes of the local parameter groups in the 2nd, 3rd, 6th and 7th layers as the indexes of the parameter groups to be replaced.

[0111] When the attribute information of the target sub-node is the distribution feature of the sample data in the local training set, if the distribution feature indicates that the sample data in the local training set is relatively complex or has a large difference with the sample data of other nodes, the target sub-node can select more indexes of the local parameter groups as the indexes of the parameter groups to be replaced. If the distribution feature indicates that the sample data in the local training set is relatively simple or has a small difference with the sample data of other nodes, the target sub-node can select less indexes of the local parameter groups as the indexes of the parameter groups to be replaced. The way of judging the complexity of the sample data in the local training set or the difference with the sample data of other nodes can refer to the foregoing description, and the embodiments of the present application will not be repeated here.

[0112] When the attribute information of the target sub-node is the computing capability of the target sub-node, if the computing capability of the target sub-node itself is strong, more indexes of the local parameter groups can be selected as the indexes of the parameter groups to be replaced, and if the computing capability of the target sub-node itself is weak, less indexes of the local parameter groups can be selected as the indexes of the parameter groups to be replaced. The way of judging the computing capability of the target sub-node itself can refer to the foregoing description, and the embodiments of the present application will not be repeated here.

[0113] When the attribute information of the target sub-node is the scale information of the target model, if the scale information indicates that the target model has a large number of layers or a large number of parameters, the target sub-node can select more indexes of the local parameter groups as the indexes of the parameter groups to be replaced, and if the scale information indicates that the target model has a small number of layers or a small number of parameters, the target sub-node can select less indexes of the local parameter groups as the indexes of the parameter groups to be replaced. The way of judging the number of layers or the number of parameters of the target model can refer to the foregoing description, and the embodiments of the present application will not be repeated here.

[0114] Step 303: update the local model parameters of the target model to the fusion model parameters, and train the target model according to the local training set of the target sub-node.

[0115] After obtaining the fusion model parameters, the target sub-node replaces all the local model parameters of the target model with the fusion model parameters, then trains the target model with the sample data in the local training set, and obtains the model parameters after training as the updated local model parameters, and uploads the local model parameters to the center node. After receiving the local model parameters uploaded by the target sub-node and the local model parameters uploaded by other sub-nodes, the center node can continue to use the method of step 301 to weight and average the received updated local model parameters to obtain updated global model parameters, and then distribute the updated global model parameters to each sub-node participating in learning again. Correspondingly, each sub-node can continue to train the target model of itself by using the method of steps 301-303. This cycle is repeated for multiple rounds until the target model on the target sub-node converges. The target sub-node obtains the trained target model, and at this time, the target sub-node can stop training.

[0116] For example, in Figure 5 , Figure 5 is the local model parameter of the target sub-node A, Ω Global = [ω1, ω2, ω3, …, ω m is the global model parameter received by the target sub-node from the center node, and the local model parameter is fused with the global model parameter Ω Global = [ω1, ω2, ω3, …, ω m , and the obtained is the fusion model parameter, which is used to replace the local model parameter in the target model. After that, the target model is trained using the local data set of the target sub-node A.

[0117] ​In the embodiment of the present application, after receiving the global model parameters from the center node, the target sub-node fuses the received global model parameters and the local model parameters of the target model of the target sub-node itself to obtain fused model parameters. In this way, the fused model parameters obtained contain both the local information and the global information of the target sub-node. Then, the target model is updated using the fused model parameters containing both the local information and the global information, and the target model is trained using the local training set. This can effectively improve the precision of the target model. In this way, even if the sample data in the local training sets used by the sub-nodes participating in the training are different in distribution or the tasks to be completed by the models on the sub-nodes are different, the model trained by the method of the embodiment of the present application can still meet the needs of the sub-nodes.

[0118] Next, the training effects of the target models of the model training method provided by the embodiment of the present application and other model training methods in two different application scenarios are exemplarily described.

[0119] The first application scenario: the federated learning system includes four sub-nodes and a center node. The target model capable of completing a classification task is deployed on each of the four sub-nodes and the center node. The images in the local training sets on the four sub-nodes are respectively four different style images of product images, artistic images, clip art and real images, and the classes of the objects contained in the images of the local training sets of each sub-node are the same. For example, the local training set in the sub-node 1 contains product images of pencils, and the local training sets in the sub-nodes 2-4 contain artistic images of pencils, clip art of pencils and real images of pencils respectively. In this case, the classification accuracies of the target models on the four sub-nodes trained by the method of the embodiment of the present application and the target models trained by other methods are shown in Table 1:

[0120] Table 1 Comparison of classification accuracies of models on each sub-node trained by different methods

[0121]

[0122] The second application scenario: the federated learning system includes three sub-nodes and a center node. The target model for completing a detection task is deployed on each of the three sub-nodes and the center node. The target models on the three sub-nodes need to complete three different detection tasks of face detection, unmanned driving detection and traffic identifier detection respectively. Since the detection tasks of the three sub-nodes are different, the sample data in the local training sets of the three sub-nodes are also different. In this case, the accuracies of the target models on the three sub-nodes trained by the method of the embodiment of the present application and the target models trained by other methods when completing the detection tasks are shown in Table 2:

[0123] Table 2 comparison table of detection accuracy of models on each sub-node trained by different methods

[0124] Subnode 1 Subnode 2 Subnode 3 Average accuracy Single-node model training method 54.99 59.07 94.71 69.59 Federal average model training method 50.99 59.91 93.39 68.10 Federal learning model training method of a heterogeneous network 48.32 61.98 92.28 67.73 Federal batch normalization model training method 52.58 60.44 94.28 69.10 Model training method of the embodiment of the present application 56.04 63.44 94.25 71.24

[0125] It can be seen from the comparison table of training effects in the above two application scenarios that the training method of the embodiments of the present application can better guarantee the recognition accuracy of the models on the multiple sub-nodes after jointly training the models of the sub-nodes.

[0126] Next, the model training device provided by the embodiments of the present application will be introduced.

[0127] Referring to Figure 6 The embodiments of the present application provide a model training device 600, which comprises:

[0128] The receiving module 601 is configured to perform step 301 in the foregoing embodiments.

[0129] The fusion module 602 is configured to perform step 302 in the foregoing embodiments.

[0130] The training module 603 is configured to perform step 303 in the foregoing embodiments.

[0131] Optionally, the fusion module 602 is configured to:

[0132] According to the parameter fusion rule of the target sub-node, the global model parameters and the local model parameters of the target model are fused to obtain the fused model parameters.

[0133] Optionally, the parameter fusion rule of the target sub-node comprises a replacement probability of each local parameter group in the local model parameters of the target model determined based on the attribute information of the target sub-node, and the fusion module 602 is mainly configured to:

[0134] Replace the local parameter group with a replacement probability greater than a first threshold in the local model parameters with the corresponding parameter group in the global model parameters to obtain the fused model parameters.

[0135] Optionally, the fusion module 602 further comprises:

[0136] A determination unit configured to determine a parameter search granularity according to the attribute information of the target sub-node.

[0137] A grouping unit configured to group a plurality of local parameter values included in the local model parameters according to the parameter search granularity to obtain a plurality of local parameter groups.

[0138] An acquisition unit configured to acquire a to-be-optimized replacement probability of each local parameter group in the plurality of local parameter groups.

[0139] An iteration unit is configured to iteratively optimize the to-be-optimized replacement probability of each local parameter group according to the local training set, to obtain the replacement probability of each local parameter group.

[0140] Optionally, the iteration unit is mainly configured to:

[0141] determine the verification model parameters according to the to-be-optimized replacement probability of each local parameter group, the plurality of local parameter groups and the global model parameters;

[0142] replace the local model parameters of the target model with the verification model parameters to obtain a verification model;

[0143] train the verification model according to the local training set to obtain an updated verification model;

[0144] test the updated verification model according to the verification set;

[0145] If the test result does not satisfy the reference condition, the to-be-optimized replacement probability of each local parameter group is updated, the updated probability is taken as the to-be-optimized replacement probability, and the step of determining the verification model parameters according to the to-be-optimized replacement probability of each local parameter group, the plurality of local parameter groups and the global model parameters is executed again until the test result satisfies the reference condition, and the last updated probability is taken as the replacement probability of each local parameter group.

[0146] Optionally, the parameter fusion rule of the target sub-node includes an index of a to-be-replaced local parameter group that is set in advance according to attribute information of the target sub-node, the local parameter group is obtained by grouping a plurality of local parameter values included in the local model parameters, and the fusion module 602 is further configured to:

[0147] replace the to-be-replaced local parameter group in the local model parameters with a corresponding parameter group in the global model parameters according to the index of the to-be-replaced local parameter group, to obtain the fusion model parameters.

[0148] Optionally, the attribute information of the target sub-node includes at least one of a data amount and a distribution feature of sample data in the local training set, a computing capability of the target sub-node and scale information of the target model.

[0149] To sum up, after receiving the global model parameters from the center node, the target sub-node fuses the received global model parameters and the local model parameters of the target model of the target sub-node to obtain the fused model parameters. In this way, the obtained fused model parameters contain both the local information and the global information of the target sub-node. Then, the target model is updated by using the fused model parameters containing both the local information and the global information, and the target model is trained by using the local training set. In this way, the precision of the target model can be effectively improved. Even in the case that the sample data in the local training set used by each sub-node participating in the training is different in distribution or the tasks to be completed by the model on each sub-node are different, the model trained by using the method of the embodiment of the present application can still meet the needs of each sub-node.

[0150] It should be noted that: the model training device provided in the above embodiment is only used as an example to illustrate the division of the above functional modules. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the model training device and the model training method provided in the above embodiment belong to the same concept, and the specific implementation process is described in detail in the method embodiment, which will not be repeated here.

[0151] In the above embodiments, all or part of the steps can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the steps can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (for example: coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example: infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be magnetic media (for example: floppy disk, hard disk, magnetic tape), optical media (for example: digital versatile disc (DVD)), or semiconductor media (for example: solid state disk (SSD)) and the like.

[0152] A person of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program instructing related hardware, and the program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read only memory, a magnetic disk or an optical disk, etc.

[0153] The above description does not limit the embodiments of the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A model training method, characterized in that, The method is applied to a target sub-node in a plurality of sub-nodes, the plurality of sub-nodes are deployed with a target model, and the method comprises the following steps: receiving global model parameters of the target model from a center node; fusing the global model parameters of the target model and local model parameters to obtain fused model parameters, wherein the local model parameters are model parameters obtained by training the target model based on a local training set; updating the local model parameters of the target model to the fused model parameters, and training the updated target model according to the local training set of the target sub-node; wherein the trained target model is a model for completing a classification task, images in the local training sets of different sub-nodes are different in style, and images in the local training sets of the plurality of sub-nodes contain objects of the same category.

2. The method of claim 1, wherein, The step of fusing the global model parameters of the target model and the local model parameters to obtain the fused model parameters comprises the following steps: fusing the global model parameters of the target model and the local model parameters according to a parameter fusion rule of the target sub-node to obtain the fused model parameters.

3. The method of claim 2, wherein, The parameter fusion rule of the target sub-node comprises replacement probabilities of each local parameter group in the local model parameters of the target model determined based on attribute information of the target sub-node, and the step of fusing the global model parameters of the target model and the local model parameters according to the parameter fusion rule of the target sub-node to obtain the fused model parameters comprises the following steps: replacing a local parameter group in the local model parameters with a replacement probability greater than a first threshold with a corresponding parameter group in the global model parameters to obtain the fused model parameters.

4. The method of claim 3, wherein, The method further comprises the following steps: determining a parameter search granularity according to the attribute information of the target sub-node; grouping a plurality of local parameter values included in the local model parameters according to the parameter search granularity to obtain a plurality of local parameter groups; obtaining a to-be-optimized replacement probability of each local parameter group in the plurality of local parameter groups; iteratively optimizing the to-be-optimized replacement probability of each local parameter group according to the local training set to obtain a replacement probability of each local parameter group.

5. The method of claim 4, wherein, The step of iteratively optimizing the to-be-optimized replacement probability of each local parameter group according to the local training set to obtain a replacement probability of each local parameter group comprises the following steps: determining verification model parameters according to the to-be-optimized replacement probability of each local parameter group, the plurality of local parameter groups, and the global model parameters; replacing the local model parameters of the target model with the verification model parameters to obtain a verification model; training the verification model according to the local training set to obtain an updated verification model; testing the updated verification model according to a verification set; If the test result does not satisfy the reference condition, the to-be-optimized replacement probability of each local parameter group is updated, the updated probability is taken as the to-be-optimized replacement probability, and the step of determining the verification model parameter according to the to-be-optimized replacement probability of each local parameter group, the plurality of local parameter groups and the global model parameter is executed until the test result satisfies the reference condition, and the replacement probability of each local parameter group is taken as the last updated probability.

6. The method of claim 2, wherein, The parameter fusion rule of the target subnode includes an index of a to-be-replaced local parameter group that is set in advance according to attribute information of the target subnode, the local parameter group is obtained by grouping a plurality of local parameter values included in the local model parameter, and the global model parameter and the local model parameter of the target model are fused according to the parameter combination rule of the target subnode to obtain the fusion model parameter. According to the index of the to-be-replaced local parameter group, the to-be-replaced local parameter group in the local model parameter is replaced by a corresponding parameter group in the global model parameter to obtain the fusion model parameter.

7. The method according to any of claims 3-6, characterized by, The attribute information of the target subnode includes at least one of a data volume and distribution characteristics of sample data in the local training set, a computing capability of the target subnode, and scale information of the target model.

8. A model training apparatus characterized by comprising: The device is applied to a target subnode in a plurality of subnodes, and the plurality of subnodes are all deployed with a target model. The receiving module is configured to receive a global model parameter of the target model from a center node. The fusion module is configured to fuse the global model parameter and a local model parameter of the target model to obtain a fusion model parameter, the local model parameter being a model parameter obtained by training the target model based on a local training set. The training module is configured to update the local model parameter of the target model to the fusion model parameter, and train the updated target model according to a local training set of the target subnode. The target model after training is a model used to complete a classification task, images in the local training sets of different subnodes are of different styles, and the images in the local training sets of the plurality of subnodes contain objects of the same category.

9. The apparatus of claim 8, wherein, The fusion module is configured to: fuse the global model parameter and the local model parameter of the target model according to a parameter fusion rule of the target subnode to obtain the fusion model parameter.

10. The apparatus of claim 9, wherein, The parameter fusion rule of the target subnode includes a replacement probability of each local parameter group in the local model parameter of the target model determined based on attribute information of the target subnode, and the fusion module is mainly configured to: replace a local parameter group with a replacement probability greater than a first threshold in the local model parameter to a corresponding parameter group in the global model parameter to obtain the fusion model parameter.

11. The apparatus of claim 10, wherein, The fusion module further includes: A determination unit configured to determine a parameter search granularity according to attribute information of the target subnode. A grouping unit configured to group a plurality of local parameter values included in the local model parameter according to the parameter search granularity to obtain a plurality of local parameter groups. An acquisition unit is configured to acquire a to-be-optimized replacement probability of each local parameter group in the plurality of local parameter groups; An iteration unit is configured to iteratively optimize the to-be-optimized replacement probability of each local parameter group according to the local training set, to obtain a replacement probability of each local parameter group.

12. The apparatus of claim 11, wherein, The iteration unit is mainly configured to: determine a verification model parameter according to the to-be-optimized replacement probability of each local parameter group, the plurality of local parameter groups, and the global model parameter; replace the local model parameter of the target model with the verification model parameter to obtain a verification model; train the verification model according to the local training set to obtain an updated verification model; test the updated verification model according to a verification set; if a test result does not satisfy a reference condition, update the to-be-optimized replacement probability of each local parameter group, take the updated probability as the to-be-optimized replacement probability, and return to execute the step of determining a verification model parameter according to the to-be-optimized replacement probability of each local parameter group, the plurality of local parameter groups, and the global model parameter, until the test result satisfies the reference condition, and take the last updated probability as the replacement probability of each local parameter group.

13. The apparatus of claim 9, wherein, The parameter fusion rule of the target subnode includes an index of a to-be-replaced local parameter group that is set in advance according to attribute information of the target subnode, the local parameter group is obtained by grouping a plurality of local parameter values included in the local model parameter, and the fusion module is mainly configured to: replace, according to the index of the to-be-replaced local parameter group, a to-be-replaced local parameter group in the local model parameter with a corresponding parameter group in the global model parameter to obtain the fusion model parameter.

14. The apparatus of any of claims 10-13, wherein, The attribute information of the target subnode includes at least one of a data amount and a distribution feature of sample data in the local training set, a computing capability of the target subnode, and scale information of the target model.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by a computer to implement the steps of the method in any one of claims 1-7. The storage medium stores a computer program, and the computer program is executed by a computer to implement the steps of the method in any one of claims 1-7.

Images