Asynchronous cooperative training method and system based on multi-terminal clustering cooperative interaction mechanism

Abstract

The invention discloses an asynchronous cooperative training method and system based on a multi-terminal clustering cooperative interaction mechanism, and the method comprises the steps: carrying out the knowledge decomposition of a cloud model, building a low-rank parameter bridge between the cloud model and an equipment model through a cloud super network and a multi-terminal attention mechanism, and achieving the multi-terminal clustering cooperative interaction. Direct interaction and knowledge extraction and application of a heterogeneous model parameter space are realized, and the generalization ability and the application effect of an equipment side model are enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of cloud computing and edge computing, and in particular relates to an asynchronous collaborative training method and system based on a multi-terminal clustering synergy enhancement mechanism. Background Art

[0002] With the rapid development of IoT technology and the widespread deployment of edge computing devices, deep neural networks (DNNs) have shown great application potential in resource-constrained environments. However, due to the high cost of data collection or the need for privacy protection, edge devices can often only obtain a limited number of training samples. This leads to overfitting during training, that is, the model performs well on the training set, but the performance drops sharply on the unseen test set.

[0003] Traditional methods to solve model overfitting mainly include data augmentation and regularization techniques. Data augmentation generates more training samples by performing various transformations on the original data (such as rotation, cropping, flipping, scaling, color jittering, etc.), thereby increasing the diversity of the data. This method can effectively increase the amount of data and reduce overfitting. However, data augmentation relies on predefined transformation rules and cannot generate new samples beyond the transformation range, which may still be insufficient to solve the problem of small samples. Regularization techniques such as L1, L2 regularization and Dropout prevent overfitting by suppressing model complexity. L1 regularization reduces unimportant features by sparsely ...

[0004] Transfer learning is another effective strategy. Transfer learning uses a model pre-trained on a large-scale dataset to adapt to small sample tasks by fine-tuning some parameters of the pre-trained model. Specifically, the basic steps of transfer learning include: first, training a deep learning model on a large-scale dataset to obtain a universal feature representation. Then, the weights of the pre-trained model are applied to the small sample task, and some parameters of the model are adjusted to adapt to the specific task by fine-tuning on the small sample data. The advantage of transfer learning is that it can utilize the universal features captured by large-scale datasets, so that it can also show good performance on small sample data. However, transfer learning also has some limitations. Its effect is highly dependent on the similarity between the pre-trained model and the target task. If the two are very different, the effect of transfer learning may be limited.

[0005] Although these studies solve the overfitting problem to some extent, they are still not direct and comprehensive enough for the small sample learning problem of heterogeneous edge devices.

[0006] In summary, the disadvantages of the prior art are:

[0007] (1) Data augmentation and regularization techniques help to reduce model overfitting, but they still have limitations. Data augmentation methods are limited to predefined transformation rules and cannot generate new samples beyond the scope of these rules; while regularization techniques can control the complexity of the model, they may have limited effects when dealing with small sample learning on edge devices and are not sufficient to significantly improve the generalization ability of the model.

[0008] (2) Transfer learning accelerates small sample learning tasks by leveraging the knowledge of pre-trained models, but its effectiveness is highly dependent on the similarity between the pre-trained model and the target task. When the difference between the two is large, the effect of transfer learning will be greatly reduced.

[0009] (3) The existing technical framework does not fully consider the heterogeneity of edge device models and tasks. Different edge devices have differences in computing power, storage space, and energy supply. The uniqueness of the deployed models and the diversity of task requirements all require the technical framework to provide more flexible and customized solutions. This results in the need to prepare a highly matched model for each edge device model in actual deployment, which consumes a lot of storage space. Summary of the invention

[0010] The purpose of the present invention is to solve the problems existing in the prior art and provide an asynchronous collaborative training method and system based on a multi-terminal clustering synergy mechanism. The main function of the method of the present invention is to use the cloud model to assist in realizing asynchronous collaborative training of device-side models in heterogeneous scenarios of edge computing, thereby improving the generalization ability and performance adaptability of device-side models.

[0011] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0012] In a first aspect, the present invention provides an asynchronous collaborative training method based on a multi-terminal clustering synergy enhancement mechanism, comprising the following steps:

[0013] S1: Randomly sample the image data sets of each device-side model on the device side, and send the randomly sampled image data sets to the cloud. After the cloud receives the image data sets sent by the device side, the cloud calculates the weight importance scores by the pre-trained cloud model, and the cloud selects the cloud sub-network according to the importance score sorting;

[0014] S2: After the cloud initializes the parameters of the super network, it continuously iterates and updates the device-side model parameters in a collaborative way between the end and the cloud; in each iteration, the device determines whether the current number of iterations is divisible by the preset knowledge transfer cycle: if not, the self-learning phase is carried out, the parameters are integrated on the device side and the parameter matrix of the device-side model participating in the knowledge transfer is used as the device-side parameters after the knowledge transfer; if it is satisfied, that is, the knowledge transfer phase is carried out, the cloud recalculates the device-side parameters after the knowledge transfer and sends the device-side parameters after the knowledge transfer to the device side; after the device side receives the device-side parameters after the knowledge transfer, it optimizes the device-side model parameters by minimizing the device-side model loss function, and performs the next iteration until the preset maximum number of iterations is reached, and a trained device-side model is obtained on the device side.

[0015] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0016] Preferably, the specific process of step S1 is as follows:

[0017] S11: randomly sampling the image data sets of each device-side model on the device side, and obtaining a specific sample set corresponding to the image data set of each device-side model;

[0018] S12: After the device sends a specific sample set to the cloud, the cloud calculates the pre-trained cloud model Each parameter matrix is ​​for each specific sample set S di The weight importance score of:

[0019] I im =|Wim |·||X im ||2

[0020] Among them, W im Representing cloud models The parameter matrix of the im-th layer; X im Representing cloud models The input activation corresponding to the parameter matrix of the im-th layer; im Representing cloud models The weight importance score of the im-th layer; ||·||2 represents the L2 norm;

[0021] S13: For each device-side model’s image dataset, the cloud-side model is ranked according to the weight importance score. The parameter matrices of each layer are sorted, and the parameter matrix of the cloud model with the largest weight importance score is selected as the cloud sub-network parameters for knowledge transfer with the device-side model.

[0022] Preferably, in the knowledge transfer phase of step S2, the specific process is as follows:

[0023] S21. The device side sets the device side parameters of the tth iteration Send to the cloud, the cloud updates the hypernetwork Parameters;

[0024] S22.Cloud Update Super Network After the parameters are set, the cloud will Parameter matrix W involved in knowledge transfer ci and the parameter matrix W of the device-side model participating in knowledge transfer di Input to the L-layer knowledge transfer network. In each layer of the knowledge transfer network, the cloud sub-network is first recoded using parameter recoding technology. Parameter matrix W involved in knowledge transfer ci and the parameter matrix W of the device-side model participating in knowledge transfer di The low-rank decomposition is performed as follows:

[0025] W ci =B ci A ci

[0026] W di =B di A di

[0027] in, Represents the cloud subnetwork Parameter matrix W involved in knowledge transfer ci The first intermediate matrix of Represents the cloud subnetwork Parameter matrix W involved in knowledge transfer ci The second intermediate matrix of ci Represents the first intermediate matrix B ci The output dimension or the second intermediate matrix A ci The input dimension of r ci <<{N,M} represents the first intermediate matrix B ci The output dimension or the second intermediate matrix A ci The input dimension of is much smaller than the parameter matrix W of the cloud sub-network participating in knowledge transfer ci The input dimension N and output dimension M of Represents the parameter matrix W of the i-th device-side model participating in knowledge transfer di The third intermediate matrix of Represents the parameter matrix W of the i-th device-side model participating in knowledge transfer di The fourth intermediate matrix of di Represents the third intermediate matrix B di The output dimension or the fourth intermediate matrix A di The input dimension of r di <<{n, m} represents the third intermediate matrix B di The output dimension or the fourth intermediate matrix A di The input dimension of is much smaller than the parameter matrix W of the i-th device-side model involved in knowledge transfer di The input dimension is n and the output dimension is m;

[0028] S23. Cloud to Cloud Subnet Parameter matrix W involved in knowledge transfer ci The second intermediate matrix A ci and the parameter matrix W of the i-th device-side model participating in knowledge transfer di The fourth intermediate matrix A di Input into the cloud hypernetwork, first the second intermediate matrix A ci and the fourth intermediate matrix A di Perform tensor flattening to the second intermediate matrix A ci and the fourth intermediate matrix A di After each row expansion operation, the first row expansion vector and the second row expansion vector are obtained respectively. ci and the fourth intermediate matrix A di After the column expansion operation, the first column expansion vector and the second column expansion vector are obtained respectively. After the first row expansion vector, the second row expansion vector, the first column expansion vector and the second column expansion vector are processed by the multi-head attention mechanism, the cloud model knowledge is integrated into the device-side model to obtain the fourth intermediate matrix integrating the cloud model knowledge

[0029] S24. Integrate the device-side parameters after knowledge migration in the cloud in the following way And the device parameters after knowledge migration Send to device:

[0030]

[0031] Preferably, in step S21, the parameters of the hypernetwork are updated in the following manner:

[0032]

[0033]

[0034] Where, ΔW di The parameter matrix W representing the device-side model participating in knowledge transfer di Changes after the gradient descent update step; Represents the device-side parameters for the tth iteration; Indicates the first cycle The device-side parameters for the iteration; represents the cloud hypernetwork parameters before updating; represents the updated cloud hypernetwork parameters; T represents transposition; η represents the learning rate of the hypernetwork; Indicates the first cycle Device-side parameters for iteration Hypernetwork Parameters The gradient of T cycle represents the preset knowledge transfer cycle; t represents the current number of iterations.

[0035] As a preference, in step S23, the fourth intermediate matrix integrating the cloud model knowledge The calculation method is:

[0036]

[0037] MultiAttn1=ω1Attn(φ R (A di ), φ R (A ci ), φ R (A ci ))

[0038] MultiAttn2=ω2Attn(φ L (A di ), φ R (A ci ), φ R (A ci))

[0039] MultiAttn3=ω3Attn(φ R (A di ), φ L (A ci ), φ L (A ci ))

[0040] MultiAttn4=ω4Attn(φ L (A di ), φ L (A ci ), φ L (A ci ))

[0041] Among them, MultiAttn1, MultiAttn2, MultiAttn3, and MultiAttn4 all represent intermediate variables; Attn represents a multi-head attention mechanism; ω1, ω2, ω3, and ω4 are learnable parameters used to balance the proportion of the multi-head attention mechanism; φ R represents the row-wise expansion of the parameter matrix; φ L Represents expanding the parameter matrix column-wise.

[0042] Preferably, in step S2, the corresponding device-side model parameters are updated by stochastic gradient descent on each image data set on the device:

[0043]

[0044] Among them, W′ di Represents the updated device-side model parameters; W di Represents the parameter matrix of the device-side model participating in knowledge transfer; η di is the learning rate of the device-side model; It means to find the gradient; Represents the device-side model loss function.

[0045] In a second aspect, the present invention provides an asynchronous collaborative classification system based on a multi-terminal clustering synergy enhancement mechanism, comprising:

[0046] A data acquisition module, used for acquiring images to be processed;

[0047] A model acquisition module, used to acquire a device-side model trained according to the asynchronous collaborative training method based on the multi-terminal clustering synergy mechanism as described in any one of the solutions of the first aspect above;

[0048] The result acquisition module is used to input the image to be processed into the trained device-side model and output the classification result of the image to be processed.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1) Traditional transfer learning technology is usually limited to specific model structures and task types, while the present invention can meet the knowledge transfer in edge heterogeneous scenarios. Through multi-terminal clustering and asynchronous collaborative training framework, the present invention effectively solves the overfitting problem in terminal small sample learning and achieves collaborative generalization and efficiency improvement.

[0051] 2) Through the innovative parameter adapter technology, the low-rank parameters of the cloud model are queried and learned, which can accurately extract and integrate the key knowledge of the cloud model into the target device model, thereby significantly improving the performance of the device model;

[0052] 3) Through dynamic adjustment and adaptive strategies, dynamic migration and adjustment of knowledge can be performed according to the needs of the current stage during training, which not only improves the efficiency of training, but also gives the device-side model greater flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of the method of the present invention;

[0054] Figure 2 Schematic diagram of the key parts of the method of the present invention; wherein (a) is a schematic diagram of the collaborative training framework; (b) is a schematic diagram of the processing flow of a layer of knowledge transfer network; (c) is a schematic diagram of the processing flow of the cloud super network; (d) is a schematic diagram of the training process of the device-side model;

[0055] Figure 3 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0057] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0058] The main purpose of this invention is to provide an asynchronous collaborative training method based on a multi-terminal clustering synergy mechanism. This method is different from the previous methods that rely on model structure sharing. The research of this invention focuses on developing and improving knowledge transfer and sharing methods between structurally and functionally different models. The core goal of this research is to overcome the challenges of effective multi-terminal knowledge transfer in heterogeneous environments, thereby improving the generalization ability of edge device models in small sample learning scenarios.

[0059] The asynchronous collaborative training method based on multi-terminal clustering synergy mechanism proposed in the present invention promotes direct interaction, extraction and application of knowledge in these parameter spaces by bridging the gap between heterogeneous model parameter spaces. Here, heterogeneous models refer to two models with different model architectures or structures.

[0060] In the method of the present invention, a pre-trained cloud model is obtained and device-side model collections in, Represents the 1st, 2nd, ..., nth device-side models deployed on each device. These device-side models may have different architectures and functions, and each device-side model has its own independent image dataset. in, Respectively represent the first, second, ..., nth device-side model independent image data sets. The method of the present invention is through a hypernetwork Fusion Cloud Model and the parameters of the device-side model to obtain the cloud-side model and device-side model knowledge. Note that the cloud-side model The structure and implemented tasks of the device-side model can be different.

[0061] like Figure 1 As shown, in a preferred implementation of the present invention, the above asynchronous collaborative training method based on the multi-terminal clustering synergy mechanism includes the following steps S1 to S2. The specific implementation process is described below.

[0062] S1: Randomly sample the image data sets of each device-side model on the device side, and send the randomly sampled image data sets to the cloud. After the cloud receives the image data sets sent by the device side, the cloud calculates the weight importance scores by the pre-trained cloud model, and the cloud selects the cloud sub-network according to the importance score sorting.

[0063] It should be noted that in step S1 of the present invention, the present invention designs a knowledge decomposition algorithm of the cloud model, which is used to decompose the cloud model into multiple cloud sub-networks. Considering that each device-side model has different requirements for knowledge, the above-mentioned knowledge decomposition algorithm improves the efficiency of knowledge migration by selecting knowledge that matches the device-side model in the cloud model.

[0064] It should be noted that in step S1 of the embodiment of the present invention, the specific process is as follows:

[0065] S11: randomly sampling the image data sets of each device-side model on the device side, and obtaining a specific sample set corresponding to the image data set of each device-side model;

[0066] It should be noted that in step S11 of the present invention, the image data set of the i-th device-end model in the device-end is Random sampling in order to obtain the corresponding specific sample set

[0067] S12: After the device sends a specific sample set to the cloud, the cloud calculates the pre-trained cloud model Each parameter matrix is ​​for each specific sample set S di The weight importance score of:

[0068] I im =|W im |·||X im ||2

[0069] Among them, W im Representing cloud models The parameter matrix of the im-th layer; X im Representing cloud models The input activation corresponding to the parameter matrix of the im-th layer; im Representing cloud models The weight importance score of the im-th layer; ||·||2 represents the L2 norm;

[0070] S13: For each device-side model’s image dataset, the cloud-side model is ranked according to the weight importance score. The parameter matrices of each layer are sorted, and the parameter matrix of the cloud model with the largest weight importance score is selected as the cloud sub-network parameters for knowledge transfer with the device-side model.

[0071] S2: After the cloud initializes the parameters of the hypernetwork, it continuously iterates and updates the device-side model parameters in a collaborative way. In each iteration, the device determines whether the current number of iterations t meets the preset knowledge transfer cycle T. cycleInteger division: If not satisfied, the self-learning phase is carried out to integrate the parameters on the device side and to use the parameter matrix W of the device side model in the knowledge transfer di As device-side parameters after knowledge migration If satisfied, t%T cycle =0, the knowledge migration phase is carried out, and the cloud recalculates the device-side parameters after knowledge migration. And the device parameters after knowledge migration Sent to the device side; the device side receives the device side parameters after knowledge migration After that, the device optimizes the device model parameters by minimizing the device model loss function and performs the next iteration until the preset maximum iteration round is reached, and a trained device model is obtained on the device. % means remainder.

[0072] It should be noted that the device-side model training process is divided into two parts: Figure 2 (d) shows the knowledge transfer phase and the self-learning phase. Specifically, when the current number of iterations t cannot be satisfied by the preset knowledge transfer cycle T cycle When it is divisible, the self-learning stage is carried out. In the self-learning stage, no processing is performed, that is, When the current number of iterations t can be preset by the knowledge transfer cycle T cycle When the number is divisible, the knowledge transfer phase begins.

[0073] In the knowledge transfer phase of step S2 of this embodiment, Figure 2 As shown, the specific process is as follows:

[0074] S21. The device side sets the device side parameters of the tth iteration Send to the cloud, the cloud updates the hypernetwork Parameters.

[0075] It should be noted that in step S21 of the present invention, the cloud super network Trained simultaneously with the device model, its parameters are defined as The parameters of the hypernetwork are updated as follows:

[0076]

[0077]

[0078] Where, ΔW di The parameter matrix W representing the device-side model participating in knowledge transfer di Changes after the gradient descent update step; Represents the device-side parameters for the tth iteration; Indicates the first cycle The device-side parameters for the iteration; represents the cloud hypernetwork parameters before updating; represents the updated cloud hypernetwork parameters; T represents transposition; η represents the learning rate of the hypernetwork; Indicates the first cycle Device-side parameters for iteration Hypernetwork Parameters gradient.

[0079] S22.Cloud Update Super Network After the parameters, such as Figure 2 As shown in (a), the cloud subnetwork Parameter matrix W involved in knowledge transfer ci and the parameter matrix W of the device-side model participating in knowledge transfer di Input to the L-layer knowledge transfer network. Figure 2 As shown in (b), in each layer of the knowledge transfer network, the parameter recoding technology is first used to recode the cloud sub-network Parameter matrix W involved in knowledge transfer ci and the parameter matrix W of the device-side model participating in knowledge transfer di The low-rank decomposition is performed as follows:

[0080] W ci =B ci A ci

[0081] W di =B di A di

[0082] in, Represents the cloud subnetwork Parameter matrix W involved in knowledge transfer ci The first intermediate matrix of Represents the cloud subnetwork Parameter matrix W involved in knowledge transfer ci The second intermediate matrix of ci Represents the first intermediate matrix B ci The output dimension or the second intermediate matrix A ci The input dimension of r ci <<{N,M} represents the first intermediate matrix B ci The output dimension or the second intermediate matrix A ci The input dimension of is much smaller than the parameter matrix W of the cloud sub-network participating in knowledge transfer ci The input dimension N and output dimension M of Represents the parameter matrix W of the i-th device-side model participating in knowledge transfer di The third intermediate matrix of Represents the parameter matrix W of the i-th device-side model participating in knowledge transfer di The fourth intermediate matrix of di Represents the third intermediate matrix B di The output dimension or the fourth intermediate matrix A di The input dimension of r di <<{n, m} represents the third intermediate matrix B di The output dimension or the fourth intermediate matrix A di The input dimension of is much smaller than the parameter matrix W of the i-th device-side model involved in knowledge transfer di The input dimension is n and the output dimension is m.

[0083] It should be noted that, in step S22 of the present invention, Figure 2 As shown in (a), the cloud sub-network Parameter matrix W involved in knowledge transfer ci and the parameter matrix W of the device-side model participating in knowledge transfer di The dimensions are W ci ∈R N×M , W di ∈R n×m Where N represents the cloud sub-network Parameter matrix W involved in knowledge transfer ci The input dimension of M represents the cloud sub-network Parameter matrix W involved in knowledge transfer ci The output dimension of n represents the i-th device-side model. Parameter matrix W involved in knowledge transfer di The input dimension of m represents the i-th device-side model. Parameter matrix W involved in knowledge transfer di The output dimension of .

[0084] S23. Cloud to Cloud Subnet Parameter matrix W involved in knowledge transfer ci The second intermediate matrix A ci and the parameter matrix W of the i-th device-side model participating in knowledge transfer di The fourth intermediate matrix A di Input into the cloud hypernetwork, first the second intermediate matrix A ci and the fourth intermediate matrix A di Perform tensor flattening to the second intermediate matrix A ci and the fourth intermediate matrix A di After each row expansion operation, the first row expansion vector and the second row expansion vector are obtained respectively. ci and the fourth intermediate matrix A diAfter the column expansion operation, the first column expansion vector and the second column expansion vector are obtained respectively. After the first row expansion vector, the second row expansion vector, the first column expansion vector and the second column expansion vector are processed by the multi-head attention mechanism (Multi-Head Attention, MHA), the cloud model knowledge is integrated into the device-side model to obtain the fourth intermediate matrix integrating the cloud model knowledge

[0085] It should be noted that, in step S23 of the present invention, Figure 2 (c) shows the fourth intermediate matrix integrating cloud model knowledge. The calculation method is:

[0086]

[0087] MultiAttn1=ω1Attn(φ R (A di ), φ R (A ci ), φ R (A ci ))

[0088] MultiAttn2=ω2Attn(φ L (A di ), φ R (A ci ), φ R (A ci ))

[0089] MultiAttn3=ω3Attn(φ R (A di ), φ L (A ci ), φ L (A ci ))

[0090] MultiAttn4=ω4Attn(φ L (A di ), φ L (A ci ), φ L (A ci ))

[0091] Among them, MultiAttn1, MultiAttn2, MultiAttn3, and MultiAttn4 all represent intermediate variables; Attn represents the multi-head attention mechanism; ω1, ω2, ω3, and ω4 are learnable parameters used to balance the proportion of the multi-head attention mechanism; φR represents the parameter matrix expanded by rows; φL represents the parameter matrix expanded by columns.

[0092] S24. Integrate the device-side parameters after knowledge migration in the cloud in the following way And the device parameters after knowledge migration Send to device:

[0093]

[0094] It should be noted that in step S2, the corresponding device-side model parameters are updated by stochastic gradient descent on each image data set on the device:

[0095]

[0096] Among them, W′ di Represents the updated device-side model parameters; W di Represents the parameter matrix of the device-side model participating in knowledge transfer; η di is the learning rate of the device-side model; It means to find the gradient; Represents the device-side model loss function.

[0097] In summary, the advantages of the present invention are:

[0098] (1) The present invention decomposes a pre-trained cloud model into multiple cloud sub-networks, each of which is responsible for interacting with a device-side model, thereby realizing knowledge migration from a single cloud model to multiple edge device-side models.

[0099] (2) In the present invention, the device-side model and the cloud-side model adopt an asynchronous collaborative training framework. The device-side model dynamically receives updates from the cloud-side model based on its own training status and data availability.

[0100] (3) The present invention adopts a parameter adapter-driven knowledge fusion model, which enhances the model's inclusiveness and adaptability to knowledge in different architecture parameter spaces, effectively simplifies the training challenges in the knowledge migration process, and significantly improves the efficiency and performance of migrating knowledge from cloud-side models to device-side models.

[0101] (4) During the model training process, the hypernetwork learns together with the cloud model and the device model, dynamically transferring and adapting knowledge, including the knowledge accumulation of the cloud model throughout the training trajectory.

[0102] It should also be noted that the asynchronous collaborative training method based on the multi-terminal clustering synergy mechanism in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides an asynchronous collaborative classification system based on the multi-terminal clustering synergy mechanism corresponding to the asynchronous collaborative training method based on the multi-terminal clustering synergy mechanism provided in the above embodiment, such as Figure 3 As shown, it includes:

[0103] A data acquisition module, used for acquiring images to be processed;

[0104] A model acquisition module, used to acquire a device-side model trained according to the asynchronous collaborative training method based on the multi-terminal clustering synergy mechanism in the above-mentioned embodiment;

[0105] The result acquisition module is used to input the image to be processed into the trained device-side model and output the classification result of the image to be processed.

[0106] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. An asynchronous collaborative training method based on multi-terminal clustering synergy enhancement mechanism, It is characterized in that The following steps are involved: S1: Randomly sample the image data sets of each device-side model on the device side, and send the randomly sampled image data sets to the cloud. After the cloud receives the image data sets sent by the device side, the cloud calculates the weight importance scores by the pre-trained cloud model, and the cloud selects the cloud sub-network according to the importance score sorting; S2: After the cloud initializes the parameters of the hypernetwork, it continuously iterates and updates the device-side model parameters in a collaborative way between the end and the cloud; In each iteration, the device determines whether the current number of iterations is divisible by the preset knowledge transfer cycle: if not, the self-learning phase is performed, the parameters are integrated on the device side and the parameter matrix of the device-side model participating in the knowledge transfer is used as the device-side parameters after the knowledge transfer; if it is satisfied, the knowledge transfer phase is performed, the cloud recalculates the device-side parameters after the knowledge transfer and sends the device-side parameters after the knowledge transfer to the device side; After the device side receives the device side parameters after knowledge transfer, the device side optimizes the device side model parameters by minimizing the device side model loss function and performs the next iteration until the preset maximum iteration round is reached, and a trained device side model is obtained on the device side.

2. An asynchronous collaborative training method based on a multi-terminal clustering synergy enhancement mechanism as claimed in claim 1, It is characterized in that The specific process of step S1 is as follows: S11: randomly sampling the image data sets of each device-side model on the device side, and obtaining a specific sample set corresponding to the image data set of each device-side model; S12: After the device sends a specific sample set to the cloud, the cloud calculates the pre-trained cloud model Each parameter matrix is ​​for each specific sample set S di The weight importance score of: I im =|W im ·||X im || 2 Among them, W im Representing cloud models The parameter matrix of the im-th layer; X im Representing cloud models The input activation corresponding to the parameter matrix of the im-th layer; im Representing cloud models The weight importance score of the im-th layer; ||·|| 2 represents the L2 norm; S13: For each device-side model’s image dataset, the cloud-side model is ranked according to the weight importance score. The parameter matrices of each layer are sorted, and the parameter matrix of the cloud model with the largest weight importance score is selected as the cloud sub-network parameters for knowledge transfer with the device-side model.

3. The asynchronous collaborative training method based on multi-terminal clustering synergy enhancement mechanism as claimed in claim 2, It is characterized in that In the knowledge transfer phase of step S2, the specific process is as follows: S21. The device side sets the device side parameters of the tth iteration Send to the cloud, the cloud updates the hypernetwork Parameters; S22.Cloud Update Super Network After the parameters are set, the cloud will Parameter matrix W involved in knowledge transfer ci and the parameter matrix W of the device-side model participating in knowledge transfer di Input to the L-layer knowledge transfer network. In each layer of the knowledge transfer network, the cloud sub-network is first recoded using parameter recoding technology. Parameter matrix W involved in knowledge transfer ci and the parameter matrix W of the device-side model participating in knowledge transfer di The low-rank decomposition is performed as follows: W ci =B ci A ci W di =B di A di in, Represents the cloud subnetwork Parameter matrix W involved in knowledge transfer ci The first intermediate matrix of Represents the cloud subnetwork Parameter matrix W involved in knowledge transfer ci The second intermediate matrix of ci Represents the first intermediate matrix B ci The output dimension or the second intermediate matrix A ci The input dimension of r ci <<{N,M} represents the first intermediate matrix B ci The output dimension or the second intermediate matrix A ci The input dimension of is much smaller than the parameter matrix W of the cloud sub-network participating in knowledge transfer ci The input dimension N and output dimension M of Represents the parameter matrix W of the i-th device-side model participating in knowledge transfer di The third intermediate matrix of Represents the parameter matrix W of the i-th device-side model participating in knowledge transfer di The fourth intermediate matrix of di Represents the third intermediate matrix B di The output dimension or the fourth intermediate matrix A di The input dimension of r di <<{n,m} represents the third intermediate matrix B di The output dimension or the fourth intermediate matrix A di The input dimension of is much smaller than the parameter matrix W of the i-th device-side model involved in knowledge transfer di The input dimension is n and the output dimension is m; S23. Cloud to Cloud Subnet Parameter matrix W involved in knowledge transfer ci The second intermediate matrix A ci and the parameter matrix W of the i-th device-side model participating in knowledge transfer di The fourth intermediate matrix A di Input into the cloud hypernetwork, first the second intermediate matrix A ci and the fourth intermediate matrix A di Perform tensor flattening to the second intermediate matrix A ci and the fourth intermediate matrix A di After each row expansion operation, the first row expansion vector and the second row expansion vector are obtained respectively. ci and the fourth intermediate matrix A di After the column expansion operation, the first column expansion vector and the second column expansion vector are obtained respectively. After the first row expansion vector, the second row expansion vector, the first column expansion vector and the second column expansion vector are processed by the multi-head attention mechanism, the cloud model knowledge is integrated into the device-side model to obtain the fourth intermediate matrix integrating the cloud model knowledge S24. Integrate the device-side parameters after knowledge migration in the cloud in the following way And the device parameters after knowledge migration Send to device:

4. The asynchronous collaborative training method based on multi-terminal clustering synergy enhancement mechanism as claimed in claim 3, It is characterized in that In step S21, the parameters of the hypernetwork are updated as follows: Where, ΔW di The parameter matrix W representing the device-side model participating in knowledge transfer di Changes after the gradient descent update step; Represents the device-side parameters for the tth iteration; Indicates the first cycle The device-side parameters for the iteration; represents the cloud hypernetwork parameters before updating; represents the updated cloud hypernetwork parameters; T represents transposition; η represents the learning rate of the hypernetwork; Indicates the first cycle Device-side parameters for iteration Hypernetwork Parameters The gradient of T cycle represents the preset knowledge transfer cycle; t represents the current number of iterations.

5. The asynchronous collaborative training method based on multi-terminal clustering synergy enhancement mechanism as claimed in claim 3, It is characterized in that In step S23, the fourth intermediate matrix integrating the cloud model knowledge The calculation method is: MultiAttn 1 =ω 1 Attn(φ R (A di ),f R (A ci ),f R (A ci )) MultiAttn 2 =ω 2 Attn(φ L (A di ),f R (A ci ),f R (A ci )) MultiAttn 3 =ω 3 Attn(φ R (A di ),f L (A ci ),f L (A ci )) MultiAttn 4 =ω 4 Attn(φ L (A di ),f L (A ci ),f L (A ci )) Among them, MultiAttn 1 ,MultiAttn 2 ,MultiAttn 3 ,MultiAttn 4 All represent intermediate variables; Attn represents the multi-head attention mechanism; ω 1 ,ω 2 ,ω 3 ,ω 4 is a learnable parameter used to balance the ratio of the multi-head attention mechanism; φ R represents the row-wise expansion of the parameter matrix; φ L Represents expanding the parameter matrix column-wise.

6. The asynchronous collaborative training method based on multi-terminal clustering synergy enhancement mechanism as claimed in claim 3, It is characterized in that In step S2, the corresponding device-side model parameters are updated by stochastic gradient descent on each image dataset on the device side: Among them, W d ' i Represents the updated device-side model parameters; W di Represents the parameter matrix of the device-side model participating in knowledge transfer; η di is the learning rate of the device-side model; It means to find the gradient; Represents the device-side model loss function.

7. An asynchronous collaborative classification system based on multi-terminal clustering synergy mechanism, It is characterized in that include: A data acquisition module, used for acquiring images to be processed; A model acquisition module, used to acquire a device-side model trained by the asynchronous collaborative training method based on the multi-terminal clustering synergy enhancement mechanism according to any one of claims 1 to 6; The result acquisition module is used to input the image to be processed into the trained device-side model and output the classification result of the image to be processed.

Images