Learning method and system for wireless human body perception

Abstract

According to the cross-domain multi-source federated learning method for wireless human body perception, the advantages of traditional federated learning and cross-domain transfer learning are combined, and the training process is innovatively optimized. Efficient data model training can be carried out, the accuracy of the model is improved, convergence is accelerated, and the computing overhead of a cloud server is reduced; meanwhile, the number of channels of the client model is adjusted according to the difference of the computing resources of the clients, and efficient model training of the resources is carried out. The problem that the federal learning technology and the transfer learning technology cannot be directly applied to the scene of human behavior recognition by using wireless signals is solved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction, and in particular to a learning framework oriented to wireless human body perception. Background Art

[0002] With the emergence of technologies such as 5G, edge computing, artificial intelligence, and XR, human production and living conditions are changing rapidly. Human action recognition (HAR) technology is emerging, allowing edge devices to perceive human intentions and better interact with humans to provide customized services. The most common human action recognition technology is based on computer vision, but its performance may be significantly affected by lighting conditions and obstacles, and it causes serious privacy leakage issues due to the use of cameras. In addition, human action recognition technology based on inertial sensors is widely used in motion detection, motion monitoring, and health detection. It is invasive and requires users to wear specific equipment, which limits the potential advantages of this technology.

[0003] In recent years, with the advancement of wireless technologies such as 5G, LoRa, millimeter wave radar, and ultrasonic radar, people have become increasingly dependent on wireless technologies to achieve ubiquitous connectivity. At the same time, ubiquitous wireless signals provide a perfect toolbox for connecting the physical and digital worlds with real-time human activity recognition. Compared with computer vision and sensor-based technologies, wireless signal-based methods have two significant advantages. First, wireless signal-based methods can provide all-weather services to large coverage areas. This is because wireless signals are robust to environmental changes, work under different lighting conditions, and have the ability to penetrate obstacles. Second, this method is non-invasive and does not require the purchase of special equipment, so it can be easily deployed to serve a large number of users at a low cost.

[0004] Existing state-of-the-art methods usually require collecting and annotating a large amount of data, and then uploading it to a cloud server with sufficient computing resources to train a deep learning model for wireless signal-based behavior recognition. However, the wireless signal collection and labeling process imposes a huge burden on the participants, making it difficult to collect a sufficiently large dataset.

[0005] Although transfer learning and federated learning have emerged as possible solutions to this problem in computer vision tasks, neither of these two techniques can be directly applied to scenarios where wireless signals are used for human behavior recognition. Transfer learning relies on existing public datasets to obtain prior knowledge and reduce the need for training data. However, there are currently no large wireless signal datasets available. For example, the Chinese patent application with publication number CN115311738A discloses a method for human behavior recognition based on transfer learning. However, the source field on which the transfer learning adopted is video image data, the data acquisition technology is mature, and there are existing large datasets.

[0006] Although federated learning enables multiple users to collaboratively train models by using their personal data in a privacy-preserving manner, training needs to be performed on edge devices with heterogeneous constrained resources. In addition, the non-independent and identically distributed data distribution of wireless signals makes the federated training process more challenging. And the accuracy of the global model of federated learning is affected by the number of participating clients. For example, the Chinese patent application with publication number CN117994863A discloses a human behavior recognition method and recognition system, which is based on human behavior data and trains the model through a federated learning algorithm, thereby reducing the amount of computation required for human behavior recognition and improving the real-time performance of recognition. However, the human behavior data based on it does not have clear data collection methods, data storage and transmission characteristics. The above-mentioned different collection methods, storage and transmission characteristics, such as data collected through wireless signals, will generate non-independent and identically distributed data because of different collection devices and different wireless methods, which makes it difficult for the federated learning model to converge and generalize, making the federated training process challenging and reducing the efficiency of building the human behavior recognition model. Summary of the invention

[0007] The technical problem to be solved by the present invention is to solve the problem that both traditional federated learning and transfer learning technologies cannot be directly applied to scenarios for human behavior recognition using wireless signals. The present invention combines the advantages of federated learning and transfer learning, optimizes the training process, and provides an efficient and accurate model building method for systems that use wireless signals for human behavior recognition.

[0008] The present invention solves the above technical problems through the following technical means:

[0009] The present invention discloses a learning method for wireless human body perception, which comprises the following steps:

[0010] S1. According to the required human behavior classification task, determine the required action set and determine the appropriate data features.

[0011] S2. Establish a task-related physical model based on the physical signals used in the human behavior classification task.

[0012] S3. Based on the modeled physical model and action set, a large amount of simulation data is generated by traversing the action set.

[0013] S4. Use the data set generated by the model to train the deep learning model on the server, and use other related data sets to train the corresponding pre-trained model to obtain a multi-source pre-trained candidate model.

[0014] S5. The client downloads the pre-trained multi-source candidate model and a small amount of simulated data and candidate data from different source domains.

[0015] S6. Use the downloaded model and data to perform client-side multi-source model training.

[0016] S7. By measuring the computing performance on the client and adjusting the number of channels of the client model, the amount of computation required for client training and the amount of communication required for transmitting parameters can be reduced.

[0017] S8. The client uploads the trained model parameters to the server. The server aggregates the parameters of different client models and then sends the aggregated model to the client to start the next round of training until the specified number of training rounds is reached.

[0018] S9. The client receives the global model sent by the server and performs multi-source model integrated reasoning.

[0019] Furthermore, the data feature described in step S1 is: the feature sensed by WiFi is CSI information, the feature sensed by millimeter wave is phase and Doppler frequency shift, and the feature sensed by ultrasound is phase, signal propagation distance and Doppler frequency shift; this feature needs to be common to two different data fields, and data conversion can be performed through simulation generation.

[0020] Furthermore, the physical model described in step S2 can be modeled as a human body mesh model or a moving joint point.

[0021] Furthermore, the simulation data described in step S3 can simulate data of different speeds, different positions, and different user habits by adjusting various parameters in the simulation process.

[0022] Furthermore, the deep learning model described in step S4 is pre-trained through a large-scale simulated data set, so that the deep learning model can have good representation capabilities in application tasks.

[0023] Preferably, the multi-source pre-trained supplementary models described in step S4 are multiple Transformer models, and the models are pre-trained on different source domain datasets by means of supervised pre-training.

[0024] Preferably, the other relevant data sets described in step S4 are IMU-based behavior recognition data sets.

[0025] Further, the step S6 includes the following steps:

[0026] S61. The user collects a small amount of data, inputs the real data and simulated data into the model, uses the MMD distance as one of the loss functions, and uses the cross entropy function as one of the loss functions. The difference in feature distribution between the simulated data field and the real data field is reduced by minimizing the MMD distance, and the classification effect of the real data is optimized by minimizing the cross entropy loss.

[0027] S62. The real data and source domain candidate data are input into the source domain candidate model to obtain the intermediate features after different layers of the model. The intermediate features of the two fields are evaluated by the Wasserstein distance, and the layer with the smallest distance is selected for truncation. Then, the truncated multiple source domain candidate models are integrated with the model pre-trained with simulated data.

[0028] Furthermore, the multi-source model integration reasoning described in step S9 is specifically as follows:

[0029] Based on the size of the model channel parameters, channel pruning is performed on multiple models to further reduce the model inference time on resource-constrained devices. When the model is deployed for inference, the results of multiple models are weighted averaged, with the models participating in federated training having a higher weight.

[0030] The invention also discloses a learning system oriented to wireless human body perception, and the system adopts the above method during execution.

[0031] The advantages of the present invention are: for the wireless signal human behavior recognition system, it combines the advantages of federated learning and transfer learning, optimizes the training process, and realizes data-efficient and resource-efficient federated learning training. The specific embodiment is as follows:

[0032] (1) Optimization process: The present invention proposes that the system can first use the knowledge of multiple source domain pre-trained models to perform knowledge distillation, thereby accelerating the speed of global model training and improving the accuracy of the model.

[0033] (2) Data efficiency: The multi-source domain model migration training and simulated data generation method adopted in the present invention first trains an initial global model on the cloud server through knowledge migration before the start of federated learning, and then conducts federated learning. The process of federated learning convergence also improves the generalization ability of the global model.

[0034] (3) Resource efficiency: The system proposed in the present invention can detect the computing power of the client and then adjust the number of channels of the client model, thereby realizing the adjustment of the client computing tasks, avoiding the problem of resource waste caused by the long execution time of weak computing power devices, and avoiding the problem of excessive computing power by pruning candidate models, thereby realizing resource-efficient federated learning training. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a cross-domain multi-source federated learning method for wireless human body perception according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of a cross-domain multi-source federated learning system module for wireless human body perception according to an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of cross-domain data simulation generation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example 1

[0040] This embodiment provides a learning method for wireless human perception based on handwriting action recognition based on ultrasonic signals. The process is as follows: Figure 1 As shown, the method steps are as follows:

[0041] S1. According to the required human behavior classification task, determine the required action set and determine the appropriate data features.

[0042] Furthermore, the action set includes all human behaviors that need to be classified. For example, for the task of handwriting recognition, the action set is the actions when writing various numbers and letters. For the task of daily behavior recognition, the relevant action sets are walking, lying down and running. The determination of suitable data features is that the commonly used features for WiFi perception are CSI information, the millimeter wave perception is phase and Doppler shift, and the ultrasonic perception is phase, signal propagation distance and Doppler shift and other features. The features need to be common to two different data fields, and data conversion can be performed by simulation generation.

[0043] like Figure 2The classification task in this embodiment is handwriting action recognition based on ultrasound. The action set can be an Arabic numeral or an English letter. The handwritten letter image is skeletonized to facilitate the subsequent trajectory extraction algorithm to obtain the writing trajectory. The change in the distance from the finger to the mobile phone is used as the input feature of the action recognition model to facilitate simulation through image data.

[0044] S2. Establish a task-related physical model based on the physical signals used in the human behavior classification task.

[0045] Furthermore, the physical model is a human body mesh model or a moving joint point.

[0046] like Figure 2 , a physical model for handwriting action recognition based on ultrasonic signals is established. Experiments are conducted using a smartphone. There is a pair of microphones and speakers at the bottom of the phone. The speaker emits an ultrasonic signal, which is reflected by the hand under the phone and received by the microphone. The distance from the finger to the phone can be measured. When the finger is writing, the change in the distance from the finger to the phone is used as the input feature of the action recognition model.

[0047] S3. Based on the modeled physical model and action set, a large amount of simulation data is generated by traversing the action set.

[0048] Furthermore, the simulation data can simulate data of different speeds, different positions, and different user habits by adjusting various parameters in the simulation process.

[0049] like Figure 3 , each writing action is composed of a series of key nodes. The number 5 has four key nodes. The writing habits of different people can be regarded as different arrangements of key nodes. The corresponding arrangement in the standard writing mode is used. The jumping action of the continuous pen is also taken into account. The continuous pen action is not directly displayed in the image, but it will also affect the phase of the ultrasonic signal. The specific simulation process is as follows: first, read the position of each key node, and then search from the first key node along the writing trajectory. If it is not a continuous pen jumping motion, search the following key nodes by depth-first search without passing through other key nodes. If it is a jumping motion, establish a linear equation based on the coordinate information of the starting point and the end point. Then the coordinates of the continuous pen motion trajectory points are obtained. The virtual positions of the microphone and the speaker are set in the image coordinate system, and then the handwriting process is simulated by traversing the writing motion trajectory coordinate set. At the same time, in order to better simulate the real environment, the position between the microphone and the speaker is changed to simulate the different distances and angles between the hand and the mobile phone when writing. The writing trajectory is downsampled and upsampled to simulate the different writing speeds when writing different strokes.

[0050] S4. Use the data set generated by the model to train the deep learning model on the server, and use other related data sets to train the corresponding pre-trained model to obtain a multi-source pre-trained candidate model.

[0051] Furthermore, the deep learning model can have good representation capabilities for this application task.

[0052] Furthermore, after obtaining the simulated data, it needs to be preprocessed and then input into the neural network for training. The obtained distance change sequence is normalized to unify the different writing amplitudes, and then the sequence length is unified through linear interpolation, so that the data at different writing speeds can be input into the deep learning model within the same sequence length, and the data is processed into a 1*256 sequence. After preprocessing, the simulated data set is trained on the cloud server. The skeleton network of the model used can be a common convolutional neural network or a Transformer based on the self-attention mechanism to obtain a pre-trained model on the simulated data set.

[0053] At the same time, multiple other time series data sets are used to train multi-source candidate models. Different models have different representation capabilities. By integrating multiple models, the generalization ability of the model is enhanced.

[0054] Furthermore, the other related data sets are specifically: IMU-based behavior recognition data sets, such as HHAR, UCI, MotionSense, Shoaib, etc., which use a supervised approach to train the corresponding pre-trained models. The pre-trained models are all Transformers, and the loss function is a cross-entropy loss function. A Transformer-based pre-trained model is trained for each source domain data set. Each pre-trained model learns the distribution of features on the corresponding data set, and together constitutes a multi-source pre-trained candidate model set.

[0055] S5. The client downloads the pre-trained multi-source candidate model and a small amount of simulated data and candidate data from different source domains.

[0056] Furthermore, the step S5 is specifically as follows:

[0057] The client joins the collaborative training process by connecting to the server. The client can be a terminal or edge device such as a smartphone or personal computer. Then download the pre-trained model in step S4 and a small amount of simulated data generated in step S3. After the user receives the pre-trained model and simulated data, if the data gap between different categories is large, for example, there are only five categories, the model can obtain good results in real scenarios. When encountering complex tasks, the difference between simulated data and real data will have a negative impact on the classification results. In addition to the main model trained by federated learning, candidate models in other fields and their corresponding small amounts of data are also downloaded.

[0058] S6. Use the downloaded model and data to perform client-side multi-source model training.

[0059] Further, the step S6 includes the following steps:

[0060] S61, the user collects a small amount of data, inputs the real data and simulated data into the model, uses the MMD distance as one of the loss functions, and also uses the cross entropy function as one of the loss functions. By minimizing the MMD distance, the difference in feature distribution between the simulated data domain and the real data domain is reduced, and by minimizing the cross entropy loss, the classification effect of the real data is optimized;

[0061] S62, the real data and the source domain candidate data are input into the source domain candidate model, the intermediate features after different layers of the model are obtained, the intermediate features of the two fields are evaluated by the Wasserstein distance, the layer with the smallest distance is selected for truncation, and then the truncated multiple source domain candidate models are integrated with the model pre-trained by the simulated data.

[0062] In this embodiment, the downloaded simulation data is regarded as the source domain, and the user's own data is regarded as the target domain. When input into a pre-trained deep learning model, the output of the feature extractor is regarded as a feature. The distribution distance between the extracted source domain and target domain features is calculated. The optimization goal is to reduce the distribution distance in the target domain and the difference between the predicted classification and the actual label. The maximum mean deviation (MMD) is used to measure the distance between the two distributions. MMD maps the distribution to the reproducing kernel Hilbert space (RKHS) through a kernel function. At the same time, the cross entropy function is used as the loss of the target domain data in the classification problem. The domain data corresponding to the real data and the candidate model are input into the candidate model, and the intermediate features after each layer are extracted, evaluated by the Wasserstein distance, and the layer with the smallest distance is selected. The subsequent layers are deleted to reduce the amount of calculation, and the compressed candidate model is integrated with the main model participating in federated learning. Models other than the main model are optimized only using cross entropy loss.

[0063] S7. By measuring the computing performance on the client and adjusting the number of channels of the client model, the amount of computation required for client training and the amount of communication required for transmitting parameters can be reduced.

[0064] Furthermore, the step S7 is specifically as follows:

[0065] Run a unified benchmark program on the client to measure the resources of the client device. By running the same computing task, the computing power of the device can be judged based on the completion time of different devices. According to the different computing powers, the channel width of the model can be adjusted accordingly to reduce the amount of calculation in the subsequent steps. At the same time, reducing the channel width can also reduce the bandwidth required for model parameter transmission.

[0066] S8. The client uploads the trained model parameters to the server. The server aggregates the parameters of different client models and then sends the aggregated model to the client to start the next round of training until the specified number of training rounds is reached.

[0067] Furthermore, the step S8 is specifically as follows:

[0068] After the client adjusts the structure of the local model, it performs training in step S6. After completing the local training round, the weight of the main model is uploaded to the server. The server aggregates the weights of the clients. Since the structures of different clients are different, but the model parameters with a small number of channels are subsets of the model parameters with a large number of channels, each subset is averaged during aggregation. After that, the aggregated model is transferred to the client and the next round of training begins until the specified number of training rounds is reached.

[0069] S9. The client receives the global model sent by the server and performs multi-source model integrated reasoning.

[0070] Furthermore, the step S9 is specifically as follows:

[0071] Based on the size of the model channel parameters, channel pruning is performed on multiple models to further reduce the model inference time on resource-constrained devices. When the model is deployed for inference, the results of multiple models are weighted averaged, with the models participating in federated training having a higher weight.

[0072] Example 2

[0073] It should be further explained that, based on the same inventive concept of Example 1, the present invention also provides a learning system for wireless human body perception, and the system includes the following modules:

[0074] 1. The action set establishment module is used to determine the required action set and appropriate data features according to the required human behavior classification task.

[0075] Furthermore, the data features are: the feature sensed by WiFi is CSI information, the feature sensed by millimeter waves is phase and Doppler frequency shift, and the feature sensed by ultrasound is phase, signal propagation distance and Doppler frequency shift; this feature needs to be common to two different data fields, and data conversion can be performed through simulation generation.

[0076] 2. A physical model building module is used to build task-related physical models based on the physical signals used in the human behavior classification task.

[0077] Furthermore, the physical model can be modeled as a human body mesh model or a moving joint point.

[0078] 3. A simulation data generation module is used to simulate and generate a large amount of simulation data by traversing the action set based on the modeled physical model and action set.

[0079] Furthermore, the simulation data can simulate data of different speeds, different positions, and different user habits by adjusting various parameters in the simulation process.

[0080] 4. The server-side training module is used to train the deep learning model on the server using the data set generated by the model, and to train the corresponding pre-trained model using other related data sets to obtain multi-source pre-trained candidate models.

[0081] Furthermore, the deep learning model is pre-trained with a large-scale simulated data set, so that the deep learning model can have good representation capabilities in application tasks. The multi-source pre-trained supplementary model is a plurality of Transformer models, which are pre-trained on different source domain data sets by means of supervised pre-training.

[0082] 5. The server-side model download module is used by the client to download pre-trained multi-source candidate models as well as a small amount of simulated data and candidate data from different source domains.

[0083] 6. Client training module, used to perform client multi-source model training using downloaded models and data.

[0084] Furthermore, the client training module performs the following operations:

[0085] (1) The user collects a small amount of data, inputs the real data and simulated data into the model, uses the MMD distance as one of the loss functions, and also uses the cross entropy function as one of the loss functions. By minimizing the MMD distance, the difference in feature distribution between the simulated data domain and the real data domain is reduced, and by minimizing the cross entropy loss, the classification effect of the real data is optimized;

[0086] (2) The real data and source domain candidate data are input into the source domain candidate model to obtain the intermediate features after different layers of the model. The intermediate features of the two domains are evaluated by the Wasserstein distance, and the layer with the smallest distance is selected for truncation. Then, the truncated multiple source domain candidate models are integrated with the model pre-trained with simulated data.

[0087] 7. Client training optimization module, which is used to reduce the computational workload of client training and the communication workload of transmission parameters by measuring the computing performance on the client and adjusting the number of channels of the client model.

[0088] 8. The client upload module is used to upload the model parameters trained by the client to the server. The server aggregates the parameters of different client models, and then sends the aggregated model to the client to start the next round of training until the specified number of training rounds is reached.

[0089] 9. Multi-source model integration module, used for the client to receive the global model sent by the server and perform multi-source model integration reasoning.

[0090] Furthermore, the multi-source model integration module performs the following operations:

[0091] Based on the size of the model channel parameters, channel pruning is performed on multiple models to further reduce the model inference time on resource-constrained devices. When the model is deployed for inference, the results of multiple models are weighted averaged, with the models participating in federated training having a higher weight.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A learning method for wireless human body perception, characterized in that: The method comprises the following steps: S1. Determine the required action set and appropriate data features according to the required human behavior classification task; S2, establishing a task-related physical model based on the physical signals used in the human behavior classification task; S3, based on the modeled physical model and action set, simulate and generate a large amount of simulation data by traversing the action set; S4. Using the data set generated by the model to train the deep learning model on the server, and using other related data sets to train the corresponding pre-trained model to obtain a multi-source pre-trained candidate model; S5, the client downloads the pre-trained multi-source candidate model and a small amount of simulation data and candidate data from different source domains; S6. Use the downloaded model and data to perform client multi-source model training; S7, by measuring the computing performance on the client and adjusting the number of channels of the client model, the amount of computation for client training and the amount of communication for transmission parameters can be reduced; S8, the client uploads the trained model parameters to the server, the server aggregates the parameters of different client models, and then sends the aggregated model to the client to start the next round of training until the specified number of training rounds is reached; S9. The client receives the global model sent by the server and performs multi-source model integrated reasoning.

2. A learning method for wireless human perception according to claim 1, characterized in that: The data features described in step S1 are: the features sensed by WiFi are CSI information, the features sensed by millimeter waves are phase and Doppler frequency shift, and the features sensed by ultrasound are phase, signal propagation distance and Doppler frequency shift; This feature needs to be common to two different data domains, and data conversion can be performed through simulation generation.

3. A learning method for wireless human perception according to claim 1, characterized in that: The physical model described in step S2 can be modeled as a human body mesh model or a moving joint point.

4. The learning method for wireless human perception according to claim 1, characterized in that: The simulation data described in step S3 can simulate data of different speeds, different positions, and different user habits by adjusting various parameters in the simulation process.

5. The learning method for wireless human perception according to claim 1, characterized in that: The deep learning model described in step S4 is pre-trained through a large-scale simulated data set, so that the deep learning model can have good representation capabilities in application tasks.

6. A learning method for wireless human perception according to claim 1, characterized in that: The multi-source pre-trained supplementary models described in step S4 are multiple Transformer models, and the models are pre-trained on different source domain datasets through supervised pre-training.

7. The learning method for wireless human perception according to claim 1, characterized in that: The other relevant data sets described in step S4 are IMU-based behavior recognition data sets.

8. The learning method for wireless human perception according to claim 1, characterized in that: The step S6 comprises the following steps: S61. The user collects a small amount of data, inputs the real data and simulated data into the model, uses the MMD distance as one of the loss functions, and uses the cross entropy function as one of the loss functions. The difference in feature distribution between the simulated data field and the real data field is reduced by minimizing the MMD distance, and the classification effect of the real data is optimized by minimizing the cross entropy loss. S62. The real data and source domain candidate data are input into the source domain candidate model to obtain the intermediate features after different layers of the model. The intermediate features of the two fields are evaluated by the Wasserstein distance, and the layer with the smallest distance is selected for truncation. Then, the truncated multiple source domain candidate models are integrated with the model pre-trained with simulated data.

9. The learning method for wireless human perception according to claim 1, characterized in that: The multi-source model integration reasoning described in step S9 is specifically as follows: Based on the size of the model channel parameters, channel pruning is performed on multiple models to further reduce the model inference time on resource-constrained devices. When the model is deployed for inference, the results of multiple models are weighted averaged, with the models participating in federated training having a higher weight.

10. A learning system for wireless human body perception, characterized in that: The system includes the following modules: The action set establishment module is used to determine the required action set and appropriate data features according to the required human behavior classification task; The physical model building module is used to build task-related physical models based on the physical signals used in the human behavior classification task. The simulation data generation module is used to simulate and generate a large amount of simulation data by traversing the action set according to the modeled physical model and action set; The server-side training module is used to train the deep learning model on the server using the data set generated by the model, and to train the corresponding pre-trained model using other related data sets to obtain multi-source pre-trained candidate models; The server-side model download module is used by the client to download pre-trained multi-source candidate models and a small amount of simulated data and candidate data from different source domains; The client training module is used to perform client-side multi-source model training using the downloaded model and data; The client training optimization module is used to reduce the amount of computation for client training and the amount of communication for transmission parameters by measuring the computation performance on the client and adjusting the number of channels of the client model; The client upload module is used to upload the model parameters trained by the client to the server. The server aggregates the parameters of different client models and then sends the aggregated model to the client to start the next round of training until the specified number of training rounds is reached; The multi-source model integration module is used for the client to receive the global model sent by the server and perform multi-source model integration reasoning.

Images