Model data processing method, communication device, medium, system and controller
By initializing global and local models in federated learning, performing training with input data and performing weight decay and sliding average, the problem of large communication overhead and reduced generalization performance in federated learning is solved, and efficient model training and Improved generalization performance.
Patent Information
- Application Number
- CN202411276625.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2025-01-10
AI Technical Summary
In federated learning, due to the use of large-scale clients and large models, the communication overhead is too large, data heterogeneity reduces the generalization performance of the model, and the client update gradient numerical value is uneven, resulting in a reduction in the global model convergence rate.
By initializing the global model and local model, combining the input data to perform the training process, using the cross entropy loss function for update processing, and performing weight decay and sliding average when the model converges, limiting the update amount range, and updating the model parameters using the arithmetic averaging method to ensure that Communication efficiency and generalization performance.
It effectively reduces communication overhead, improves the generalization performance and convergence rate of the model, ensures the equal contribution of each client during the training process, and reduces the impact of data heterogeneity on the model.
Smart Images
Figure CN119293630A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a model data processing method, a communication device, a medium, a system and a controller. Background Art
[0002] Model data (or parameter information) processing is the core link of artificial intelligence (AI) methods such as machine learning. In order to improve processing efficiency and data security level, many attempts have been made in related technologies. Among them, federated learning (FL) adopts a distributed learning process to decompose model training operations onto multiple devices, and then transmits model parameters (or model update data) to a central server for aggregation, while the original training data is always retained on the local device. Based on this, multiple distributed devices (clients) can realize the joint training operation of the model without sharing data, which is especially suitable for application scenarios such as medical and financial where data privacy (security) needs to be protected.
[0003] However, federated learning FL has the following technical problems: first, the large number of clients and the widespread use of large models LM (Large Model) or foundational models FM (Foundational Model) may cause its communication overhead to exceed the limit; second, the heterogeneity of client data, which is not independent and identically distributed, reduces the generalization performance of the global model; third, the update gradient values of some clients are too large, which may overwhelm the update amount of other clients; fourth, poor network communication and other reasons may cause the device to be unable to participate in the training of the model in every round, which reduces the convergence rate of the global model.
[0004] In related technologies, compression (processing) methods such as quantization and sparsification are used to address the communication overhead problem. Variance reduction, adaptive optimization and other (processing) methods are also proposed to address the data heterogeneity problem. However, it is difficult to take into account the advantages of the above two types of (processing) methods. While improving the generalization of the model, redundant parameters must also be transmitted, which causes the communication volume to exceed the standard. Summary of the invention
[0005] An embodiment of the present invention discloses a model data processing method, whose core processing process includes a first initialization step, a second training step, and a third construction step; the first initialization step obtains first model data and uses the first model data to initialize second model data; the second training step obtains seventh input data and executes a first training process in combination with the second model data to determine a second loss function and perform a first update process on the second model data; and then, the third construction step determines the third model data when the second model data converges.
[0006] The first model data includes description information of the first model, the second model data includes description information of the second model, the third model data includes description information of the third model, and the second loss function includes a cross entropy loss function.
[0007] Specifically, the first model is a global model, the second model is a local model, and the third model is a target model; the global model shares data within a preset first scope, and the local model exclusively uses data on a preset client or a preset device; the target model is the model to be optimized in the first training process, and the first scope includes the server side, the communication device side and / or the client side.
[0008] The seventh input data includes image data, which comes from a preset local device or client. According to the first update process, a first update amount (060) of the local model, namely g(i, t), is also obtained.
[0009] Specifically, the first model data may come from the server side, and the local device or the client side includes at least one hardware unit, which is independently distributed in space with the server side and / or exchanges data through the communication device side.
[0010] Furthermore, the model data processing method may also be provided with a fourth iterative step; if the number of times the first update processing is executed reaches a preset first threshold value K, the fourth iterative step may perform a second update processing on the second model data and obtain a second update amount of the second model or the local model, i.e., u(t, i).
[0011] The fourth iteration step may also obtain an update amount set of the second model data, perform global aggregation processing on the update amount set, and then use a weight decay method to update the first model data.
[0012] Specifically, the second update processing can utilize the third momentum data for sliding average, symbolic operation and / or perform a third update processing on its sliding average data; its symbolic operation can limit the value range of the second update amount to within the same order of magnitude, and its second update amount can come from different local devices or different clients.
[0013] Furthermore, the model data processing method can also obtain a second update amount according to the second update processing; wherein the third momentum is the local model momentum, i.e. m(i, t); the local model momentum and the first update amount can also be weighted averaged or linearly combined by the first adjustment parameter β1 and the second adjustment parameter β2.
[0014] The updating process of the first model data may adopt an arithmetic mean method, and the weight coefficient of the arithmetic mean method is given by the ratio of the number of local samples of model training to the number of all samples on the local training participating clients.
[0015] Correspondingly, an embodiment of the present invention also discloses a communication device, whose core components include a second training unit and a third construction unit; the second training unit obtains the seventh input data and executes a first training process in combination with the second model data, determines the second loss function and performs a first update process on the second model data; the third construction unit determines the third model data when the second model data converges and exchanges communication data sets between a preset server end and at least one client end.
[0016] Specifically, its communication data set includes first model data, second model data and / or third model data; its second loss function includes a cross-entropy loss function; its communication data set also includes a first update amount of the local model obtained by the first update processing, namely g(i, t).
[0017] The first model data may come from a server, and its local device or client includes at least one hardware unit, which is independently distributed in space with the server and / or exchanges data via a communication device.
[0018] Furthermore, the communication device may also be provided with a fourth iteration unit; if the number of times its first update processing is executed reaches a preset first threshold, the fourth iteration unit will perform a second update processing on the second model data and obtain a second update amount of the second model or the local model, that is, u (t, i); and its fourth iteration unit may also obtain an update amount set of the second model data, and perform global aggregation processing on the update amount set, and then use a weight decay method and other processes to update its first model data.
[0019] Among them, the second update processing includes using the third momentum data for sliding average, symbolic operation and / or a third update processing on the sliding average data; the symbolic operation limits the value range of the second update amount to within the same order of magnitude, and the second update amount comes from different local devices or different clients.
[0020] Furthermore, the communication device can also obtain a second update amount according to the second update processing; its third momentum is the local model momentum, that is, m(i, t); the local model momentum and the first update amount can be weighted averaged or linearly combined through the first adjustment parameter β1 and the second adjustment parameter β2; and the update process of the first model data can adopt the arithmetic averaging method, and the weight coefficient of the arithmetic averaging method is given by the ratio of the number of local samples of model training to the number of all samples on the local training participating clients.
[0021] Similarly, an embodiment of the present invention also discloses a computer storage medium, a machine learning system and a controller; the computer storage medium includes a storage medium body for storing a computer program; when the computer program is executed by a microprocessor, any of the above model data processing methods can be implemented; the machine learning system is provided with the above computer storage medium and / or any of the above communication devices; the controller can also be provided with any of the above communication devices, machine learning systems and / or computer storage media; the above products can all be used to solve the same technical problems, while improving the efficiency of model data processing and the communication bottleneck in the distributed machine learning process.
[0022] In summary, the present invention initializes the second model data with the first model data, and then executes the first training process in combination with the seventh input data to determine the second loss function and performs the first update processing, and then determines the third model data when the second model data converges; if the number of executions of the first update processing reaches the preset first threshold K, the second update processing is performed and the second update amount is obtained, so as to update the first model data through global aggregation processing and weight decay process; its product combines symbolic operation to compress data to improve communication efficiency and take into account the model generalization performance at the same time; it is particularly suitable for model data processing in artificial intelligence AI scenarios such as federated learning FL.
[0023] It should be noted that the terms "first", "second" and similar terms used in this article are only for describing the various components in the technical solution, and do not constitute a limitation of the technical solution, nor can they be understood as an indication or suggestion of the importance of the corresponding elements; elements with "first", "second" and similar terms indicate that the corresponding technical solution contains at least one of the element. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solution of the present invention and facilitate further understanding of the technical effects, technical features and purposes of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings. The accompanying drawings constitute an essential part of the specification and are used together with the embodiments of the present invention to illustrate the technical solution of the present invention, but do not constitute a limitation to the present invention.
[0025] The same reference numerals in the accompanying drawings represent the same components, and the details of each drawing are as follows.
[0026] Figure 1 The process of the embodiment of the method of the present invention is shown in FIG. Figure 1 .
[0027] Figure 2 The process of the embodiment of the method of the present invention is shown in FIG. Figure 2 .
[0028] Figure 3 The process of the embodiment of the method of the present invention is shown in FIG. Figure 3 .
[0029] Figure 4 The process diagram of the product embodiment of the present invention is shown in FIG. Figure 1 .
[0030] Figure 5 The machine learning system embodiment of the present invention is shown in FIG. Figure 1 .
[0031] Figure 6 The process diagram of the product embodiment of the present invention is shown in FIG. Figure 2 .
[0032] Figure 7 The machine learning system embodiment of the present invention is shown in FIG. Figure 2 .
[0033] Figure 8 This is the second local model loss function (cross entropy loss) structure of the method and product of the present invention.
[0034] Fig. 9 It is the first update amount gi(t) of the second local model of the method and product of the present invention.
[0035] Fig.10 It is the second update amount ui(t) of the second local model of the method and product of the present invention.
[0036] Fig.11 It is the second local model momentum mi(t) of the method and product of the present invention.
[0037] Fig.12 It is the first global model iteration structure of the method and product of the present invention.
[0038] Fig.13 These are comparative data on the generalized performance of the methods and products of the present invention.
[0039] Fig.14 The process of the embodiment of the method of the present invention is shown in FIG. Figure 4 .
[0040] Fig.15 It is a schematic diagram of the structure of the device and system embodiment of the present invention.
[0041] Fig.16 Schematic diagram of the layout structure of the product embodiment of the present invention Figure 1 .
[0042] Fig.17 Schematic diagram of the layout structure of the product embodiment of the present invention Figure 2 .
[0043] Fig.18 Schematic diagram of the layout structure of the product embodiment of the present invention Figure 3 .
[0044] The meanings of the various reference numerals / numbers are as follows.
[0045] 010-Server side; 020-communication equipment (module / device) end; 030-Client; 050-loss function, L(i); 051-serial number, i; 052-cross entropy loss function, f(w,i); 053-local model, w(i); 055-second loss function; 060-the first update of the local model, i.e. g(i, t); 061- (number of iterations) preset parameter, t, t is a positive integer; 062-(first update processing times) first threshold, i.e. K, where K is a positive integer; 070-the second update of the local model, i.e. u(t, i); 071-Sign operation, sign[]; 072-first adjustment parameter, β1; 073-local model momentum, i.e. m(i, t); 074-second adjustment parameter, β2; 080-the global model after the t+1th round of update (the first model data), t is a positive integer; 081-the global model after the tth round of update (the first model data), t is a positive integer; 082-first weighting parameter, γ1; 083-Number of clients n, n is a positive integer; 084-second weighting parameter, γ2; 088-(arithmetic mean) weight coefficient; 100-M, N or R client (equipment / system / device / vehicle / controller) end, M, N, R are positive integers; 110-client receiving module; 120 client initialization module; 130-client first determination module; 140-Client update module; 150-client second confirmation module; 200-server; 210-(Server side) Initialization module; 220-(communication device) sending module; 230-(communication device) receiving module; 240-(Server side) confirmation module; 310-first device end, such as server end 010; 320 - second device end, such as communication module (end) 020; 330-third device end, such as client 030; 500-communication device; 510-data receiving module, such as a client receiving module; 520-data determination module, such as a first determination module of a client; 530-data update module, such as client update module; 600-Machine Learning System; 610-a first initialization / service (server) unit, such as a client initialization module; 620-second training unit, such as the Mth client, where M is a positive integer; 630-a third construction unit, such as an Nth client, where N is a positive integer; 640-a fourth iteration unit, such as an Rth client, where R is a positive integer or a second confirmation module of the client; 710-first model data, such as global model data; 720-second model data, such as local model data; 730 - third model data, such as target (local) model data; 777-seventh input data, such as local image data; 810-first initialization step; 820-second training step; 821-First training process; 830 - third construction step; 831-(Second model, i.e., local model) first update processing (process); 832-(Second model, i.e., local model) second update processing (process); 833-Second (target model) construction process (process); 840-fourth iteration step; 841-(Second model, i.e., local model) Second update data transmission process (step); 842-(first model, i.e. global model) data updating process (step); 843-(first model, i.e. global model) data transmission process (steps); 900-any client, such as any vehicle or any controller; 901-controller; 903-Computer storage media; 911 - a first client, such as a first vehicle or a first controller; 912 - a second client, such as a second vehicle or a second controller; 91N-Nth client, such as the Nth vehicle or the Nth controller, N is a positive integer; 999-Comparative data of the embodiment; S11-receiving global model process (step); S12-initialize local model process (step); S13-determine the loss function process (step); S14-local model first update process (step); S15-target model determination process (step); S16-local model second update process (step); S17-local model transfer process (step); S21-global model initialization process (step); S22-global model transfer process (step); S23-local model receiving process (step); S24-global model updating process (step); S25-global model transfer process (step); S31-(Server side) global model initialization process (steps); S32-(Server side) global model sending process (steps); S33-(Client) Global Model Receiving Process (Step); S34-(Client) local model initialization process (step); S35-(Client) Loss function determination process (step); S36-(client) local model first update process (step); S37-(client) local model second update process (step); S38-(Client) local model sending process (step); S39-(Server side) local model receiving process (step); S40-(Server-side) global model update process (step); S41-(Server side) global model sending process (steps); S42-(Client) local model determination process (step). DETAILED DESCRIPTION
[0046] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described below are only for explaining the technical solutions of the present invention, rather than for limiting the present invention. In addition, the parts described in the embodiments or the accompanying drawings are only examples of the relevant parts of the present invention, rather than the whole of the present invention.
[0047] like Figure 1 , 2 The model data processing method shown in , 3, 5, and 14 is provided with a first initialization step 810, a second training step 820, and a third construction step 830; the first initialization step 810 obtains the first model data 710 and uses the first model data 710 to initialize the second model data 720; the second training step 820 obtains the seventh input data 777 and performs a first training process 821 in combination with the second model data 720, determines the second loss function 055 and performs a first update process 831 on the second model data 720; and then, the third construction step 830 determines the third model data 730 when the second model data 720 converges.
[0048] Among them, Fig.14 As shown, the first model data 710 includes description information of the first model, the second model data 720 includes description information of the second model, the third model data 730 includes description information of the third model, and the second loss function 055 includes a cross entropy loss function.
[0049] Specifically, Figure 5 , Figure 7 As shown, the first model adopts a global model, the second model adopts a local model, and the third model is a target model; the global model can share data within a preset first range, while the local model exclusively uses data on a preset client 030 or a preset device; the target model is the model to be optimized in the first training process 821, and the first range includes the server side 010, the communication device side 020 and / or the client side 030.
[0050] Among them, Fig.14 As shown, the seventh input data 777 includes image data, which comes from a preset local device or client 030; furthermore, the first update amount 060 of the local model, namely g (i, t), can be obtained according to the first update processing 831.
[0051] Specifically, Figure 7 As shown, the first model data 710 may come from the server end 010 , and the local device or the client end 030 includes at least one hardware unit, which is independently distributed in space with the server end 010 and / or exchanges data through the communication device end 020 .
[0052] Furthermore, the model data processing method may also be provided with a fourth iterative step 840; if the number of executions of the first update processing 831 reaches a preset first threshold 062, i.e., K; then the fourth iterative step 840 will perform a second update processing 832 on the second model data 720 and obtain a second update amount 070 of the second model or local model, i.e., u(t, i).
[0053] The fourth iteration step 840 may also obtain an update amount set of the second model data 720, perform global aggregation processing on the update amount set, and may use a weight decay method to update the first model data 710.
[0054] Specifically, the second update processing 832 includes using the third momentum data to perform sliding average, symbolic operation and / or a third update processing on the sliding average data; the symbolic operation limits the value range of the second update amount 070 to within the same order of magnitude, and the second update amount 070 comes from different local devices or different clients 030.
[0055] Furthermore, the model data processing method can also obtain the second update amount 070 according to the second update processing 832; its third momentum is the local model momentum 073, that is, m(i, t); the local model momentum 073 and the first update amount 060 can be weighted averaged or linearly combined through the first adjustment parameter 072 and the second adjustment parameter 074.
[0056] The updating process of the first model data 710 may adopt an arithmetic mean method, and the weight coefficient of the arithmetic mean method is given by the ratio 088 of the number of local samples of model training to the number of all samples on the local training participating clients.
[0057] Accordingly, if Figure 4 , 6 The communication device 500 shown in , 7, and 15 is provided with a second training unit 520 and a third construction unit 530; the second training unit 520 obtains the seventh input data 777 and performs a first training process 821 in combination with the second model data 720 to determine the second loss function 055 and performs a first update process 831 on the second model data 720; the third construction unit 530 determines the third model data 730 when the second model data 720 converges and exchanges communication data sets between a preset server end 010 and at least one client end 030.
[0058] The communication data set includes the first model data 710 and the second update amount (070) of the local model, namely u(t,i); and the second loss function 055 includes a cross entropy loss function.
[0059] Specifically, the first model data 710 comes from the server end 010 , and the local device or the client end 030 includes at least one hardware unit, and the hardware unit is independently distributed in space with the server end 010 and / or exchanges data through the communication device end 020 .
[0060] Furthermore, if Figure 4 , 5 As shown in , 6, and 15, the communication device 500 may also be provided with a fourth iteration unit 640; if the number of executions of the first update processing 831 reaches a preset first threshold value 062, the fourth iteration unit 640 performs a second update processing 832 on the second model data 720 and obtains a second update amount 070 of the second model or the local model, i.e., u(t, i); further, the fourth iteration unit 640 also obtains an update amount set of the second model data 720, performs a global aggregation process on the update amount set, and then uses a weight decay method to update the first model data 710.
[0061] Similarly, the second update processing 832 can utilize the third momentum data to perform sliding average, symbolic operation and / or perform a third update processing on the sliding average data; its symbolic operation limits the value range of the second update amount 070 to within the same order of magnitude, and its second update amount 070 comes from different local devices or different clients 030.
[0062] Furthermore, the communication device 500 can also obtain a second update amount 070 according to the second update processing 832; its third momentum is the local model momentum 073, that is, m(i, t); it is obtained by weighted averaging or linearly combining the local model momentum 073 and the first update amount 060 through the first adjustment parameter 072 and the second adjustment parameter 074; the update process of the first model data 710 can adopt an arithmetic averaging method, and the weight coefficient of the arithmetic averaging method is given by the ratio 088 of the number of local samples of model training to the number of all samples on the local training participating clients.
[0063] Similarly, if Fig.16 , 17 The computer storage medium 903 shown is provided with a storage medium body for storing a computer program; when the computer program is executed by a microprocessor, any of the above model data processing methods can be implemented; Figure 7 , 15 The machine learning system 600 shown in 16, 17, and 18 is provided with the above-mentioned computer storage medium 903 and / or the communication device 500, and the controller 901 can also solve the same technical problem by providing the above-mentioned communication device 500, the machine learning system 600 and / or the computer storage medium 903, and the implementation process thereof will not be repeated herein.
[0064] In actual applications (scenario), the following can be used: Figures 16 to 18 The layout structure shown and Fig.15 The product structure shown; among them, Fig.14 The method shown can be implemented on the client 030, the server 010 and / or the communication device 020 respectively, and its core processing process includes the following steps: Fig.14 The execution process (steps) shown and / or Fig.15 The hardware (units) shown.
[0065] Specifically, Figure 1 As shown, it can be applied to Figure 7 The client 030 of the layout; the client 030 may be part or all of the machine learning system 600; wherein the model processing process includes the following steps: S11 receives the global model (i.e., the first model data 710) sent by the server 010, and its client 030 includes multiple local training clients; the local training client trains the local model (i.e., the second model data 720) based on the local privacy data, and the local privacy data is the local image data (i.e., the seventh input data 777); due to the poor stability of the communication connection, the local training clients participating in the model training in the first round are random clients; the random local training clients participating in the local model training in this round receive the global model sent by the server 010.
[0066] S12 uses the global model to initialize the local model; and after the random local training client receives the global model sent by the server 010, the local model is initialized using the global model; S13 inputs the local image data into the initialized local model for model training and determines the loss function of the local model.
[0067] Among them, the local image data can also be input into the local model of the random local training client in this round to perform forward model training; or the local image data can be input into the local feature extractor model of the local model, or the local image features can be output, and the local image data can be input into the local classifier model to output the predicted training label; its loss function 050 is as follows Figure 8 shown.
[0068] Specifically, the loss function 050 may be a cross entropy loss function of the local model 053, ie, a cross entropy loss between the training labels and the true labels of the local image data.
[0069] Furthermore, S14 determines the gradient of the local model through back propagation operation according to the loss function 050 of the local model determined above, and can use the stochastic gradient descent algorithm to optimize the parameters of the local model, and perform a first update processing on the local model; wherein, back propagation is used to calculate the gradient of the loss function relative to each model parameter, so as to optimize the model parameters; in addition, other algorithms can also be used to optimize the parameters of the local model, and all of them will fall within the protection scope of the present invention.
[0070] Furthermore, S15 may determine a target local model (ie, the third model data 730 ) when the local model converges; when the local model converges, model training of the local model is stopped, and the local model of the last model training may be determined as the target local model.
[0071] Among them, Figure 2 The figure shows a process of model processing according to another embodiment; the communication efficiency and generalization performance can be improved simultaneously by taking the symbol operation, and the processing process may include a federated learning FL method.
[0072] Specifically, if the above method is applied to the client 030, the machine learning process can be improved through the following two steps S16 and S17.
[0073] On the one hand, S16 can be performed when the number of first update processing times of the local model reaches a preset threshold (i.e., Fig. 9 When the first threshold value 062) K shown is reached, the local model update amount (local model second update amount 070) u(t, i) after the second update processing is determined; and the update amount of the model is processed by sliding average and sign taking using momentum to achieve sliding average update of momentum.
[0074] Specifically, Fig. 9 , 10 , 11, the processing process of the second update processing is given; wherein, K is the preset threshold value of the first update processing times (i.e., the first threshold value 062), g(i, t) is the first update amount of the local model 060, and m(i, t) is the momentum of the local model 073 (i.e., the third momentum); after sliding average and sign operation, u(t, i) is the second update amount of the local model 070, which is the update amount of the local model obtained after the second update processing, and β1 and β2 are adjustable hyperparameters; in addition, S17 can send the local model update amount after the second update processing to the server end 010.
[0075] Furthermore, when the number of times steps S11 to S14 are repeated reaches a preset threshold value K, K is a positive integer; the last local model update amount of each local training client in this round of training after the second update processing can be sent to the server side 010; wherein, the communication module or communication device 500 can be used to connect each client of the client 030 and the server side 010.
[0076] Specifically, by performing the operation of subtracting the signs of the local models, the model update values of all clients 020 are controlled to be at the same order of magnitude; thereby activating the equal contribution of each client 020 in the direction of global model update, reducing the impact of data heterogeneity on local model heterogeneity, thereby alleviating the deviation of the global model and better training the local model.
[0077] Furthermore, if Figure 3 As shown, the above processing can be implemented on the server side 010 through steps S21 to S25; wherein, S21 initializes the global model; S22 sends the initialized global model to the client 020; the server side first initializes the global model, and considering that the communication connection between the server side 010 and the client 030 is unstable, the initialization data sent by the server side 010 to the client 030 adopts the global model; S23 receives the local model update amount after the second update processing sent by the client 030, and S24 globally aggregates the local model update amount, and applies weight decay update to determine the global model.
[0078] The global model can be updated as follows: Fig.12 The structure shown, its parameters include the global model 081 of the tth round; its global aggregation processing can adopt the arithmetic averaging method, and the weight coefficient of the arithmetic averaging method is given by the ratio of the number of local samples of model training to the number of all samples on the local training participating clients 088, and n represents the number of clients selected to participate in the training.
[0079] It should be noted that the essence of weight decay is to add a regularization term to the loss function to reduce the problem of model overfitting; and the above process can be applied to Fig.12 In the global update shown; in addition, the global model can be sent to the client 030 through S25 on the server side 010, or the global model sent by the server side 010 can be received on the client 030; the above-mentioned data exchange process can also be realized through the above-mentioned communication device 500.
[0080] Furthermore, the global model determined by the above process can be sent to the client 030, and steps S12 to S14 are executed on the client 030, while steps S23 to S25 are executed on the server 010, until the local model is updated based on back propagation, so that the local model converges, and then the above training process of the local model is stopped.
[0081] Specifically, a global model with good generalization performance can be obtained through the above training process, and in this process, the client 030 can store the privacy content of the local image data (ie, the seventh input data 777) in a local device or organization to prevent information leakage.
[0082] Furthermore, if Figure 4 , Figure 5 The illustrated embodiment discloses a signaling interaction process between the client 030, the communication device 500 (or the communication module) and the server 010; wherein, S31 implements the process of initializing the global model of the server 010; S32 sends the initialized global model to the client 020 at the server 010; S33 receives the global model (i.e., the first model data 710) sent by the server 010 at the client 020; S34 initializes the local model (i.e., the second model data 720) according to the global model at the client 030; S35 inputs the local image data (i.e., the seventh input data 777) into the initialized local model at the client 030 for model training to determine the loss function of the local model.
[0083] In addition, S36 performs a first update process on the local model based on back propagation according to the loss function of the local model at the client 030; S37 determines the update amount of the local model after the second update process at the client 030 when the number of first update processes of the local model reaches a preset threshold value K; S38 sends the update amount of the local model after the second update process to the server 010 at the client 030.
[0084] Furthermore, S39 receives the local model update amount after the second update processing sent by the client 030 at the server end 010; S40 performs global aggregation processing on the local model update amount at the server end 010, updates and determines the global model; S41 sends the global model to the client 030 at the server end 010; S42 determines the target local model (i.e., the third model data 730) when the local model of the client 030 converges.
[0085] Among them, the communication module or communication device 500 can be used as a medium for signal transmission between the client 030 and the server 010; after S31 to S41 are executed, S33 to S41 are executed in a loop; when it is determined that the local model converges, S42 is executed again.
[0086] Furthermore, a global model with superior generalization performance can be provided for each (not unique) client 020 training, and local data can be stored confidentially, using symbol-taking operations to improve both communication efficiency and generalization performance; this process includes the use of federated learning FL methods to provide prospects for connecting decentralized medical data sources and protecting user data privacy.
[0087] Among them, Fig.13 As shown, based on the CIFRA-10 and CIFAR-100 image datasets, non-iid-1 and non-iid-2 non-independent data partitioning methods can be used to verify the technical effect of the embodiment in simultaneously improving communication efficiency and generalization performance: and the generalization performance of the federated learning FL method and FedAvg, FedLion, SCAFFOLD, FedEF-TopK, FedEF-Sign, and SCALLION methods are respectively shown.
[0088] Specifically, in the non-iid partitioning method, the samples of each local training client are sampled according to the Dirichlet distribution, and the training data set of each local training client includes an unfixed number of image data of each category of image data sets, and the amount of data of each category of image data sets is different; wherein, the partitioning of non-iid-2 is more dispersed than the partitioning of non-iid-1; wherein, Fig.13 As shown, the comparison data 999 of this embodiment shows the best generalization in the CIFAR-100 image dataset.
[0089] Accordingly, Figure 4 The figure discloses the composition structure of the M, N or R client (device / system / apparatus / vehicle / controller) end; it can be provided with a client receiving module 110, a client initialization module 120, a client first determination module 130, a client update module 140, and a client second determination module 150.
[0090] Specifically, the receiving module 110 of the client 030 is used to receive the global model sent by the server 010; the initialization module 120 of the client 030 is used to initialize the local model according to the global model; the first determination module 130 of the client 030 is used to input the local image data into the initialized local model and perform model training to determine the loss function of the local model; the loss function refers to: the cross entropy loss between the training label and the true label of the local image data.
[0091] Further, the updating module 140 of the client 030 performs a first updating process on the local model based on back propagation according to the loss function of the local model; the second determining module 150 of the client 030 is used to determine the target local model when the local model converges; wherein, Fig.10As shown, by performing the operation of taking the sign of the local model difference, the model update values of all clients 030 are leveled to the same order of magnitude.
[0092] Based on this, the equal contribution of each client 030 in the direction of global model update is activated, which reduces the impact of data heterogeneity on local model heterogeneity, thereby alleviating the deviation of the global model and better training the local model.
[0093] Furthermore, the update module 140 of the client 030 can also perform a second update processing on the local model update amount based on momentum and symbolization according to the update amount of the local model; wherein, the device 100 can also be provided with a third determination module of the client 030, which is used to determine the update amount of the local model after the second update processing when the number of first update processing times of the local model reaches a preset threshold K.
[0094] In addition, the sending module or the communication device 500 of the client 030 can be used to send the local model update amount after the second update process to the server 010.
[0095] Among them, Figure 6 As shown, a machine learning system 600 can also be constructed and applied to the server side; and can specifically include an initialization module 210 of the server side 010, a sending module 220 of the server side 010, a receiving module 230 of the server side 010, and a determination module 240 of the server side 010.
[0096] Specifically, its initialization module 210 is used to initialize the global model, the sending module 220 is used to send the initialized global model to the client 030, the receiving module 230 is used to receive the local model update amount after the second update processing sent by the client 030, and the determination module 240 is used to globally aggregate the local model differences, and update and determine the global model using the weight decay method; wherein the above-mentioned sending module 220 is also used to send the global model to the client 030.
[0097] Furthermore, if Figure 7 As shown, the information interaction process of the server side 010, the communication device side 020 and the client side 030 is also disclosed; wherein, the same inventive concept is adopted: on the one hand, the local model update module can be used to perform a first update processing on the local model in the local training client in the back propagation mode, and perform a second update processing on the local model update amount by using the momentum sliding average and taking the sign; on the other hand, the global model update module can be used to perform a global aggregation processing on the local model update amount after the second update processing, and update it by using the weight decay method to determine the global model.
[0098] In addition, the communication module or communication device 500 can be used to transmit the local model update amount of the client 030 after the second update processing to the server side 010, and can transmit the global model of the server side 010 to the client 030; wherein, the client 030 includes an update module that can be provided with a local model, the server side 010 can be provided with an update module for the global model, and the communication module or communication device 500 can be used to connect the above-mentioned client 030 and server side 010.
[0099] It should be noted that the above embodiments are only for more clearly illustrating the technical solutions of the present invention. Those skilled in the art will understand that the implementation methods of the present invention are not limited to the above contents, and obvious changes, replacements or substitutions based on the above contents do not exceed the scope of the technical solutions of the present invention; other implementation methods will also fall within the scope of the present invention without departing from the concept of the present invention.
Claims
1. A model data processing method, characterized in that It includes a first initialization step (810), a second training step (820), and a third construction step (830); wherein: the first initialization step (810) obtains the first model data (710) and uses the first model data (710) to initialize the second model data (720); the second training step (820) obtains the seventh input data (777) and performs a first training process (821) in combination with the second model data (720), determines the second loss function (055) and performs a first update process (831) on the second model data (720); the third construction step (830) determines the third model data (730) when the second model data (720) converges.
2. The model data processing method according to claim 1, wherein: The first model data (710) includes description information of the first model, the second model data (720) includes description information of the second model, the third model data (730) includes description information of the third model, and the second loss function (055) includes a cross entropy loss function.
3. The model data processing method according to claim 2, wherein: The first model is a global model, the second model is a local model, and the third model is a target model; the global model shares data within a preset first scope, and the local model exclusively uses data on a preset client (030) or a preset device; the target model is a model to be optimized in the first training process (821), and the first scope includes a server side (010), a communication device side (020) and / or the client side (030).
4. The model data processing method according to claim 3, wherein: The seventh input data (777) includes image data, and the image data comes from a preset local device or the client (030); according to the first update processing (831), a first update amount (060) of the local model, namely g (i, t) is also obtained.
5. The model data processing method according to claim 4, wherein: The first model data (710) comes from the server side (010), and the local device or the client side (030) includes at least one hardware unit, and the hardware unit and the server side (010) are independently distributed in space and / or exchange data through the communication device side (020).
6. The model data processing method according to any one of claims 1 to 5, further comprising a fourth iteration step (840); wherein: If the number of times the first update process (831) is executed reaches a preset first threshold (062) K, the fourth iteration step (840) performs a second update process (832) on the second model data (720) and obtains a second update amount (070) of the second model or the local model, i.e., u(t, i).
7. The model data processing method according to claim 6, wherein: The fourth iteration step (840) further obtains an update amount set of the second model data (720), performs global aggregation processing on the update amount set, and then uses a weight decay method to update the first model data (710).
8. The model data processing method according to claim 7, wherein: The second update processing (832) includes using the third momentum data to perform sliding average, symbolic operation and / or a third update processing on the sliding average data; the symbolic operation limits the value range of the second update amount (070) to within the same order of magnitude, and the second update amount (070) comes from different local devices or different clients (030).
9. The model data processing method according to claim 8, further comprising obtaining the second update amount (070) according to the second update processing (832); wherein: The third momentum is the local model momentum (073), i.e., m(i, t); it is obtained by weighted averaging or linearly combining the local model momentum (073) and the first update amount (060) using a first adjustment parameter (072) β1 and a second adjustment parameter (074) β2.
10. The model data processing method according to any one of claims 7 to 9, wherein: The updating process of the first model data (710) adopts an arithmetic mean method, and the weight coefficient of the arithmetic mean method is given by the ratio (088) of the number of local samples of model training to the number of all samples on the local training participating clients.
11. A communication device (500), comprising a second training unit (520) and a third construction unit (530); wherein: The second training unit (520) obtains the seventh input data (777) and executes a first training process (821) in combination with the second model data (720), determines a second loss function (055) and performs a first update process (831) on the second model data (720); the third construction unit (530) determines the third model data (730) when the second model data (720) converges and exchanges communication data sets between a preset server end (010) and at least one client end (030).
12. The communication device (500) according to claim 11, wherein: The communication data set includes first model data (710), the second model data (720) and / or third model data (730); the second loss function (055) includes a cross-entropy loss function; the communication data set also includes a first update amount (060) of the local model obtained by the first update processing (831), i.e., g(i, t).
13. The communication device (500) according to claim 12, wherein: The first model data (710) comes from the server side (010), and the local device or the client side (030) includes at least one hardware unit, and the hardware unit and the server side (010) are independently distributed in space and / or exchange data through the communication device side (020).
14. The communication device (500) according to any one of claims 11 to 13, further comprising a fourth iteration unit (640); wherein: If the number of times the first update process (831) is executed reaches a preset first threshold (062) K, the fourth iteration unit (640) performs a second update process (832) on the second model data (720) and obtains a second update amount (070) of the second model or the local model, i.e., u(t, i); the fourth iteration unit (640) also obtains an update amount set of the second model data (720), performs a global aggregation process on the update amount set, and then uses a weight decay method to update the first model data (710).
15. The communication device (500) according to claim 14, wherein: The second update processing (832) includes using the third momentum data to perform sliding average, symbolic operation and / or a third update processing on the sliding average data; the symbolic operation limits the value range of the second update amount (070) to within the same order of magnitude, and the second update amount (070) comes from different local devices or different clients (030).
16. The communication device (500) according to claim 15, further comprising obtaining the second update amount (070) according to the second update process (832); wherein: The third momentum is the local model momentum (073), i.e., m(i, t); the local model momentum (073) and the first update amount (060) are weighted averaged or linearly combined by a first adjustment parameter (072) β1 and a second adjustment parameter (074) β2 to obtain the third momentum; the arithmetic average method is used in the updating process of the first model data (710), and the weight coefficient of the arithmetic average method is given by the ratio (088) of the number of local samples of model training to the number of all samples on the local training participating clients.
17. A computer storage medium (903), comprising a storage medium body for storing a computer program; when the computer program is executed by a microprocessor, the model data processing method according to any one of claims 1 to 10 is implemented.
18. A machine learning system (600), comprising the computer storage medium (903) according to claim 17 and / or the communication device (500) according to any one of claims 11 to 15.
19. A controller (901), comprising the communication device (500) according to any one of claims 11 to 15, the machine learning system (600) according to claim 18 and / or the computer storage medium (903) according to any one of claim 17.