Model training method, action recognition method, device, equipment and medium

Through joint training of terminal devices and edge servers, gradient aggregation technology is used to improve the generalization of the action recognition model, solving the problem of the action recognition model performing poorly on unknown samples, achieving higher recognition accuracy and lower cost.

CN118536577BActive Publication Date: 2024-11-26SHENZHEN RES INST OF BIG DATA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410725803.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-06
Publication Date
2024-11-26
Estimated Expiration
2044-06-06

AI Technical Summary

Technical Problem

The performance performance of the action recognition model on unknown samples is poor, resulting in overfitting of the model and reducing the generalization of the action recognition model.

Method used

Through joint training of terminal devices and edge servers, environment-aware data are sampled, local model parameter gradients are calculated, gradient variance is obtained, model parameters are updated according to preset variance thresholds, and gradient aggregation is performed to improve model generalization.

Benefits of technology

It improves the recognition accuracy of the action recognition model on unknown samples, enhances the generalization performance of the model, and reduces the cost of data acquisition and communication.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118536577B_ABST
    Figure CN118536577B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a model training method, an action recognition method, an apparatus, a device and a medium, which belong to the field of artificial intelligence technology, and include: each terminal device calculates the model parameter gradient of the local model on the terminal device according to each subsample in sequence according to the data collection order to obtain a model parameter gradient set, and obtains the gradient variance according to the model parameter gradient set, updates the model parameter gradient set according to the gradient variance and a preset variance threshold, determines the target model parameter gradient according to the updated model parameter gradient set, and sends the target model parameter gradient to an edge server, the edge server performs gradient aggregation on multiple target model parameter gradients to obtain an aggregated gradient, and updates the global model on the edge server according to the aggregated gradient to obtain a target model, and broadcasts the target model to each terminal device, so that each terminal device can perform action recognition through the target model, which can improve the generalization of the action recognition model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, action recognition method, device, equipment and medium. Background Art

[0002] In the related art, the performance of the action recognition model on the training set or known samples is good, and the action recognition model can accurately recognize the action of each sample. However, the performance of the action recognition model on unknown samples is poor. Even if these unknown samples follow the same distribution as the known samples, the action recognition model cannot accurately recognize the action of the unknown samples. This causes the action recognition model to overfit and reduces the generalization of the action recognition model. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a model training method, action recognition method, device, equipment and medium, aiming to improve the generalization of the action recognition model.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a model training method, which is applied to a model training system, wherein the model training system includes multiple terminal devices and an edge server, and the method includes:

[0005] Each of the terminal devices samples from the collected environment perception data distribution to obtain an environment perception sample set, and determines the data collection order of each subsample in the environment perception sample set, and calculates the model parameter gradient of the local model on the terminal device according to each subsample in sequence according to the data collection order to obtain a model parameter gradient set, and obtains the gradient variance according to the model parameter gradient set;

[0006] Each of the terminal devices updates the model parameter gradient set according to the gradient variance and a preset variance threshold, determines a target model parameter gradient according to the updated model parameter gradient set, and sends the target model parameter gradient to the edge server;

[0007] The edge server performs gradient aggregation on multiple target model parameter gradients to obtain an aggregated gradient, updates a global model on the edge server according to the aggregated gradient to obtain a target model, and broadcasts the target model to each of the terminal devices so that each of the terminal devices performs action recognition through the target model.

[0008] In some embodiments, the step of calculating the model parameter gradient of the local model on the terminal device according to each subsample in sequence according to the data collection order to obtain the model parameter gradient set includes:

[0009] According to the data collection order, each of the sub-samples is sequentially input into the local model for action recognition to obtain a predicted action category of each sub-sample; each of the sub-samples has a preset action label;

[0010] Calculate the cross entropy loss according to the predicted action category and the preset action label to obtain the target loss;

[0011] Determining an initial parameter gradient of the local model in the subsample according to the target loss and the model parameters of the local model;

[0012] Obtaining a training round of the local model, and generating noise according to the training round; the noise intensity of the noise is inversely proportional to the square root of the training round;

[0013] Adding the noise to the initial parameter gradient to obtain the model parameter gradient of the local model in the sub-sample;

[0014] The model parameter gradient set is determined according to the model parameter gradient of each sub-sample.

[0015] In some embodiments, updating the model parameter gradient set according to the gradient variance and a preset variance threshold comprises:

[0016] If the gradient variance is less than or equal to the preset variance threshold, and the number of samples in the environment perception sample set is less than the preset number threshold, sampling a new subsample from the environment perception data distribution, and adding the new subsample to the environment perception sample set;

[0017] Calculating a first intermediate parameter gradient of the new sub-sample;

[0018] The first intermediate parameter gradient is added to the model parameter gradient set to update the model parameter gradient set until the gradient variance is greater than the preset variance threshold or the number of samples is greater than or equal to the preset number threshold.

[0019] In some embodiments, the updating of the model parameter gradient set according to the gradient variance and a preset variance threshold further includes:

[0020] If the gradient variance is greater than the preset variance threshold, and the number of samples in the environment perception sample set is less than the preset number threshold, updating the preset variance threshold according to the gradient variance and a preset smoothing parameter;

[0021] Sampling the model parameter gradient set to obtain a second intermediate parameter gradient, and adding a subsample corresponding to the second intermediate parameter gradient to the environment perception sample set;

[0022] adding the second intermediate parameter gradient to the model parameter gradient set to perform an initial update on the model parameter gradient set;

[0023] The model parameter gradient set after the initial update is updated again according to the updated preset variance threshold and the gradient variance of the model parameter gradient set after the initial update until the number of samples is greater than or equal to the preset number threshold.

[0024] In some embodiments, sampling the model parameter gradient set to obtain a second intermediate parameter gradient includes:

[0025] Determining a sampling probability according to the gradient length of each model parameter gradient in the model parameter gradient set;

[0026] The model parameter gradient set is sampled according to a sampling probability to obtain the second intermediate parameter gradient.

[0027] In some embodiments, determining a target model parameter gradient according to the updated model parameter gradient set and sending the target model parameter gradient to the edge server includes:

[0028] Performing mean calculation on the updated model parameter gradient set to obtain the target model parameter gradient;

[0029] Encoding the target model parameter gradients;

[0030] On the same time-frequency block, the encoded target model parameter gradient is sent to the edge server.

[0031] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a motion recognition method, which is applied to a terminal device. The method includes:

[0032] Receiving a target model sent by an edge server; the target model is trained according to the model training method described in the first aspect;

[0033] Acquire target environment perception data;

[0034] The target environment perception data is input into the target model for action recognition.

[0035] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application provides a motion recognition device, which is applied to a terminal device, and the device includes:

[0036] A receiving module, used to receive a target model sent by an edge server; the target model is trained according to the model training method described in the first aspect;

[0037] An acquisition module is used to acquire target environment perception data;

[0038] The action recognition module is used to input the target environment perception data into the target model for action recognition.

[0039] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method described in the first aspect or the action recognition method described in the second aspect when executing the computer program.

[0040] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the model training method described in the first aspect or the action recognition method described in the second aspect.

[0041] The model training method, action recognition method, action recognition device, electronic device and computer-readable storage medium proposed in the present application improve the generalization of the action recognition model through joint training of terminal devices and edge servers. Each terminal device samples from the collected environment perception data distribution to obtain an environment perception sample set. In order to determine the update degree of each sub-sample in the environment perception sample set to the model parameters, the data collection order of each sub-sample in the environment perception sample set is determined. According to the data collection order, the model parameter gradient of the local model on the terminal device is calculated according to each sub-sample in sequence to obtain a model parameter gradient set, so as to update the model parameters using the model parameter gradient set. The computing power resources of the terminal device are limited, and the local model cannot be trained based on a large amount of data. In order to select a suitable number of samples and reduce the data collection cost of the terminal device, the gradient variance is obtained according to the model parameter gradient set, and the model parameter gradient set is updated according to the gradient variance and the preset variance threshold to obtain a suitable number of model parameter gradients for model parameter update. The target model parameter gradient is determined according to the updated model parameter gradient set to determine the accurate gradient for updating the model parameters, and the target model parameter gradient is sent to the edge server so that the edge server can integrate the target model parameter gradients of each terminal device to update the model parameters. In order to improve the generalization performance of the model and speed up the convergence of the model, the edge server performs gradient aggregation on multiple target model parameter gradients to learn the feature representation of data collected by different terminal devices, obtain aggregated gradients, and update the global model on the edge server according to the aggregated gradients to obtain the target model. The target model is broadcast to each terminal device so that each terminal device can perform action recognition through the target model, thereby improving the generalization performance of the action recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the model training method provided in an embodiment of the present application;

[0043] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0044] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.

[0045] Figure 4 yes Figure 1 Another flow chart of step S102 in FIG. 1 ;

[0046] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.

[0047] Figure 6 yes Figure 1 Flow chart of step S103 in FIG.

[0048] Figure 7 is a flow chart of the action recognition method provided by an embodiment of the present application;

[0049] Figure 8 It is a structural diagram of the model training system provided in the embodiment of the present application;

[0050] Fig. 9 is a schematic diagram of the structure of the motion recognition device provided in an embodiment of the present application;

[0051] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] Deep learning technology is a method of characterizing and learning data based on a neural network architecture. This method can not only process high-dimensional data that many other methods cannot efficiently process, such as pictures, audio, video, and natural language, but also complete a series of abstract tasks such as classification, understanding, and generation. In related technologies, deep learning models are used for human action recognition to perform tasks such as human-computer interaction and abnormal behavior detection. However, action recognition models usually have an overfitting effect, which makes the action recognition model perform better on training sets or known samples and can accurately recognize actions for each sample. The model performance on unknown samples is poor. Even if these unknown samples follow the same distribution as known samples, the action recognition model cannot accurately recognize the actions of unknown samples. Therefore, how to improve the generalization of action recognition models has become an urgent problem to be solved.

[0056] Based on this, the embodiments of the present application provide a model training method, action recognition method, device, equipment and medium, aiming to improve the generalization of the action recognition model.

[0057] The model training method, action recognition method, device, equipment and medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the model training method in the embodiments of the present application is described.

[0058] The model training method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The model training method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements the model training method, etc., but is not limited to the above forms.

[0059] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0060] Figure 1 is an optional flow chart of the model training method provided in the embodiment of the present application, which is applied to the model training system, and the model training system includes multiple terminal devices and an edge server. Figure 1 The method may include but is not limited to steps S101 to S103.

[0061] Step S101, each terminal device samples from the collected environment perception data distribution to obtain an environment perception sample set, and determines the data collection order of each sub-sample in the environment perception sample set, and calculates the model parameter gradient of the local model on the terminal device according to each sub-sample in sequence according to the data collection order to obtain a model parameter gradient set, and obtains the gradient variance according to the model parameter gradient set;

[0062] Step S102, each terminal device updates the model parameter gradient set according to the gradient variance and the preset variance threshold, determines the target model parameter gradient according to the updated model parameter gradient set, and sends the target model parameter gradient to the edge server;

[0063] In step S103, the edge server performs gradient aggregation on multiple target model parameter gradients to obtain an aggregated gradient, updates the global model on the edge server according to the aggregated gradient to obtain a target model, and broadcasts the target model to each terminal device so that each terminal device performs action recognition through the target model.

[0064] This application uses federated learning technology to improve the generalization of the action recognition model so that the action recognition model can accurately identify human actions on unknown samples. Traditional centralized machine learning technology uploads all local sample data to a server for model training, while federated learning is based on multiple distributed edge devices or local data samples of the server for model training. By exchanging model parameters instead of data to transfer knowledge, multiple participating nodes can establish a common and powerful deep learning model without sharing data to solve problems such as data privacy, data security, data access rights and heterogeneous data access. Edge intelligence is a potential paradigm for realizing federated learning. Edge intelligence uses base stations with certain computing capabilities instead of marine data centers as federated learning servers to coordinate the federated learning process, and transmits model parameters through wireless communication networks, so that all terminal devices connected to the network have the possibility of collecting data and accessing distributed intelligent model training. At the same time, it also provides terminal devices with convenient and low-latency intelligent services provided by deep learning models. Consider an edge intelligent system, which includes multiple terminal devices and an edge server for parameter aggregation. The terminal device is a hardware device that directly interacts with a preset object, such as a mobile phone, a computer, a wearable device, a robot, etc. The edge server is an edge base station. When conducting federated learning training, terminal devices need to have perception, computing and communication functions. The perception function can obtain specific information in the environment. The computing function enables the terminal device to have a certain computing power to run local deep learning models. The communication function enables the terminal device to send the local deep learning model to the edge server through wireless communication.

[0065] The terminal device obtains information about the surrounding environment through various sensors and perception technologies to obtain the distribution of environmental perception data. The distribution of environmental perception data is a plurality of data samples collected by the terminal device. The data samples can be images obtained by the camera, audio obtained by the recorder, echo signals reflected by objects when the radar waves emitted by the frequency-modulated continuous wave radar propagate in the environment, text, etc. Deep learning is a data-driven algorithm. A well-trained deep learning model usually requires a large number of parameters as support, and the deep learning model usually contains tens of thousands or even millions of parameters. However, terminal devices such as mobile phones, laptops, and a large number of small perception devices deployed are usually energy-constrained devices. Data collection requires energy costs, and frequent model exchanges also consume a lot of communication costs, which increases the burden on the participating training devices. Considering the computing power limitations of the terminal devices, each terminal device uniformly samples a first preset number of sub-samples from the environmental perception data distribution collected by each terminal device to obtain an environmental perception sample set. The first preset number is the minimum sample batch, denoted as bmin, which can be set according to the actual situation. The first preset number can be 4. During uniform sampling, only one subsample is sampled from the environment perception data distribution without overlap each time, and the sampling order of the subsamples is used as the data collection order to determine the data collection order of each subsample in the environment perception sample set. The sampling order can be determined according to the collection time of the subsamples, for example, the sampling order of the subsamples with earlier collection time is earlier, and the sampling order can also be determined randomly. It can be understood that one observation of the environment by the terminal device is regarded as one sampling of the data distribution.

[0066] In the process of deep learning model training, it is necessary to define a loss function as the objective function and minimize the objective function. In order to improve the generalization of the action recognition model, this application aims to directly optimize the population loss rather than the traditional empirical loss, so as to characterize the performance of the deep learning model on all samples that are independent and identically distributed with the training samples, rather than just the samples in the training set, so that this goal pays more attention to the performance of the model on new similar data. Among them, the empirical loss is defined as the mean of the loss function on the training set, and the population loss is defined as the expectation of the loss function when each sample in the training set is replaced by an independent and identically distributed sample. However, this goal cannot be calculated directly, and it is necessary to establish the relationship between the upper bound of the goal and the adjustable hyperparameters of the model so that this goal can be optimized using existing tools. This application characterizes this relationship through the model parameter gradient, which is the derivative of the loss function with respect to the model parameters, and is used to measure the rate at which the loss function changes with the model parameters. Specifically, according to the data collection order, the model parameter gradient of the local model on the terminal device is calculated in sequence according to each sub-sample to obtain a model parameter gradient set. The local model is a model local to the terminal device. The model parameter gradient set includes the gradients generated by each sub-sample on the current local model, which can be expressed as [g1,...,gi,...,gbmin]. i is used to identify the data collection order, indicating the i-th sub-sample, and gi is the model parameter gradient generated by the i-th sub-sample on the local model. The gradient length of each model parameter gradient in the model parameter gradient set is obtained, and the gradient variance is calculated according to each gradient length. The model parameter gradient is a vector, and the gradient length is the modulus of the vector. Each gradient length is expressed as [n1,...,ni,...,nbmin], where ni is the gradient length of gi. The gradient length can reflect the size of the model parameter update, and the gradient variance is used to reflect the stability of the model parameter update. The calculation method is as follows:

[0067] ,

[0068] Among them, Var represents the gradient variance, bmin is the number of subsamples in the environment perception sample set, ni represents the i-th gradient length, is the average value of each length gradient.

[0069] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S206:

[0070] Step S201, according to the data collection order, each sub-sample is sequentially input into the local model for action recognition to obtain the predicted action category of each sub-sample; each sub-sample has a preset action label;

[0071] Step S202, performing cross entropy loss calculation according to the predicted action category and the preset action label to obtain a target loss;

[0072] Step S203, determining the initial parameter gradient of the local model in the subsample according to the target loss and the model parameters of the local model;

[0073] Step S204, obtaining the training round of the local model, and generating noise according to the training round; the noise intensity of the noise is inversely proportional to the square root of the training round;

[0074] Step S205, adding noise to the initial parameter gradient to obtain the model parameter gradient of the local model in the sub-sample;

[0075] Step S206: Determine a model parameter gradient set according to the model parameter gradient of each sub-sample.

[0076] In step S201 of some embodiments, the local model is a model used locally in the terminal device to identify and classify human actions or behaviors. The local model may be a gesture recognition model used to identify hand actions, an activity recognition model used to identify body movements, a behavior recognition model used to identify violent behavior, abnormal behavior, etc. The local model may use a convolutional neural network, a recurrent neural network, a transformer neural network, etc. According to the order of data collection, each sub-sample is sequentially input into the local model for action recognition to obtain a predicted action category of each sub-sample. The predicted action category may be a gesture category, a limb action category, etc. The gesture category may be sliding, finger bending, finger stretching, etc., and the limb action category may be walking, running, jumping, etc. The predicted action category may be determined according to the type of the local model. If the local model is a gesture recognition model, the predicted action category may be a certain specific hand action, through which the terminal device can be non-contact controlled, such as controlling a virtual object such as a game character or a virtual item, controlling the switch of the terminal device, etc. Each sub-sample has a preset action label, which is a pre-marked action category label, that is, a real label.

[0077] In step S202 of some embodiments, for each sub-sample, the predicted action category and the preset action label are input into the cross entropy loss function to perform cross entropy loss calculation to obtain the target loss of the sub-sample. The cross entropy loss function is expressed as:

[0078] ,

[0079] in, represents the target loss of the i-th subsample; represents the probability distribution of the true label of the i-th sub-sample; is the probability distribution of the predicted action category of the i-th sub-sample.

[0080] In step S203 of some embodiments, for each sub-sample, the model parameters of the local model are derived according to the target loss, the gradient of the local model on the sub-sample is determined, and the initial parameter gradient is obtained.

[0081] In step S204 of some embodiments, when training the deep learning model, adding a certain amount of noise is conducive to the deep learning model converging to the correct posterior distribution, thereby effectively alleviating the overfitting effect, and further improving the recognition accuracy of the action recognition model on unknown samples. The noise can be Gaussian white noise, and the noise intensity changes as the training progresses. The change rule is that the noise intensity (noise power) is inversely proportional to the square root of the training rounds, and the training rounds are the number of times the local model has been trained. The initial noise power is a preset multiple of the gradient length of the initial parameter gradient, and the preset multiple is in the range of [0.0001, 0.001]. If the preset multiple is represented by r and the training round is represented by t, then the noise intensity n is represented by It is understandable that as the number of training rounds increases, the local model will gradually learn the patterns and regularities in the sample data, and its sensitivity to noise will decrease, thus reducing the noise power.

[0082] In step S205 of some embodiments, a commonly used solution is to add noise to the loss function or training data. However, due to the highly nonlinear nature of the deep learning model, it is difficult to quantify the impact mechanism and impact range of these operations. In the embodiment of the present application, a certain amount of Gaussian white noise is added to the gradient (initial parameter gradient) to obtain the model parameter gradient of the local model in the subsample.

[0083] In step S206 of some embodiments, the model parameter gradients of the subsamples are arranged in sequence according to the data collection order to determine a model parameter gradient set.

[0084] Through the above steps S201 to S206, a model parameter gradient set can be obtained to quantify the influence of each sub-sample on the model parameter update based on the model parameter gradient set, thereby guiding the model training process and obtaining an action recognition model with better recognition performance.

[0085] During the model optimization process, the data batch size of each training round is kept unchanged, but a preset variance threshold is recorded. At the same time, sampling is not performed according to the traditional uniform sampling, but new single samples are added gradually. Each terminal device collects new sub-samples one by one according to the gradient variance and the preset variance threshold to update the model parameter gradient set until the number of sub-samples in the environmental perception sample set reaches a fixed data batch size to ensure that the data usage does not exceed the traditional solution. The data batch size is the second preset number, denoted as bmax. The second preset number is different from the first preset number. The second preset number is the maximum sample batch, and the second preset number can be 32. The specific update process is as follows.

[0086] See also Figure 3 In some embodiments, step S102 may include but is not limited to steps S301 to S303:

[0087] Step S301: if the gradient variance is less than or equal to a preset variance threshold, and the number of samples in the environment perception sample set is less than a preset number threshold, a new subsample is sampled from the environment perception data distribution, and the new subsample is added to the environment perception sample set;

[0088] Step S302, calculating the first intermediate parameter gradient of the new sub-sample;

[0089] Step S303: adding the first intermediate parameter gradient to the model parameter gradient set to update the model parameter gradient set until the gradient variance is greater than a preset variance threshold or the number of samples is greater than or equal to a preset number threshold.

[0090] In step S301 of some embodiments, the gradient variance threshold of the current training round t is expressed as , the preset variance threshold of the current training round t can be obtained according to the gradient variance threshold, the current sample batch and the second preset number. The preset variance threshold is expressed as , b is the current sample batch, that is, the number of samples in the environmental perception sample set, and the initial value of the current sample batch is bmin. The gradient variance can reflect the gradient difference of the model parameter gradients of each sub-sample. If the gradient variance is less than or equal to the preset variance threshold, it means that the gradient difference is small, and the process of updating the model parameters using the model parameter gradient set is relatively stable. If the gradient variance is greater than the preset variance threshold, it means that the gradient difference is large, which may lead to instability in the training process. If the gradient variance is less than the preset variance threshold, and the number of samples in the environmental perception sample set is less than the preset number threshold (the second preset number), it means that the difference between the sub-samples is small. In order to improve the local model's ability to recognize new samples, a new sub-sample is sampled from the environmental perception data distribution, and the new sub-sample is added to the environmental perception sample set, and the number of samples in the environmental perception sample set is updated, that is, b=b+1.

[0091] In step S302 of some embodiments, referring to steps S201 to S205, the model parameter gradient of the local model on the terminal device is calculated according to the new sub-sample to obtain a first intermediate parameter gradient.

[0092] In step S303 of some embodiments, the first intermediate parameter gradient is added to the model parameter gradient set to update the model parameter gradient set, and the updated model parameter gradient set is represented as [g1, ..., gb+1]. Each time a new sub-sample is added, the gradient length of the model parameter gradient brought by it is calculated. If the gradient variance of the gradient length is less than or equal to the preset variance threshold, and the number of samples is less than the preset number threshold, then a new sub-sample is added, and steps S301 to S303 are repeated until the gradient variance is greater than the preset variance threshold or the number of samples is greater than or equal to the preset number threshold. It should be noted that the gradient variance is also updated with the update of the model parameter gradient set.

[0093] In the above steps S301 to S303, when the influence of the environmental perception sample set on the model training is relatively stable, new sub-samples are added to the environmental perception sample set so that the local model can capture the characteristic details of the new samples, enhance the model's recognition ability for the new samples, and thereby improve the accuracy of the model's action recognition on any sample.

[0094] See also Figure 4 In some embodiments, step S102 may also include but is not limited to steps S401 to S404:

[0095] Step S401: if the gradient variance is greater than or equal to a preset variance threshold, and the number of samples in the environment perception sample set is less than a preset number threshold, the preset variance threshold is updated according to the gradient variance and a preset smoothing parameter;

[0096] Step S402, sampling the model parameter gradient set to obtain a second intermediate parameter gradient, and adding a subsample corresponding to the second intermediate parameter gradient to the environment perception sample set;

[0097] Step S403, adding the second intermediate parameter gradient to the model parameter gradient set to perform an initial update on the model parameter gradient set;

[0098] Step S404, updating the initially updated model parameter gradient set again according to the updated preset variance threshold and the gradient variance of the initially updated model parameter gradient set until the number of samples is greater than or equal to the preset number threshold.

[0099] In steps S401 to S402 of some embodiments, if the gradient variance is greater than or equal to the preset variance threshold, it means that the difference in the model parameter gradients between the samples is large, which may cause instability in the training process, making it difficult for the model to converge or converge to a bad local optimal solution. The number of samples in the environmental perception sample set is less than the preset number threshold, indicating that the number of samples has not reached the fixed data batch size, and data sampling is still needed. In order to ensure the stability of the training process and improve the action recognition ability of the local model, the embodiment of the present application samples a model parameter gradient from the model parameter gradient set as the second intermediate parameter gradient, or samples multiple model parameter gradients from the model parameter gradient set to obtain multiple second intermediate parameter gradients. The subsample corresponding to the second intermediate parameter gradient is added to the environmental perception sample set to update the number of samples. The gradient variance threshold is updated according to the gradient variance and the preset smoothing parameter, and the preset variance threshold is updated according to the updated gradient variance threshold, the current sample batch and the second preset number. The gradient variance threshold is updated as follows:

[0100] ,

[0101] in, is the gradient variance; is the preset smoothing parameter; is the gradient variance threshold.

[0102] In step S403 of some embodiments, a sampled second intermediate parameter gradient is added to the model parameter gradient set to perform an initial update on the model parameter gradient set. Alternatively, multiple sampled second intermediate parameter gradients are added to the model parameter gradient set to obtain a final model parameter gradient set, where the number of samples is equal to the preset number threshold, the model parameter gradient set has bmax model parameter gradients, and the updated preset variance threshold is used for the next round of data sampling. It should be noted that the number of model parameter gradients in the model parameter gradient set is equal to the number of samples.

[0103] In step S404 of some embodiments, if the gradient variance of the model parameter gradient set after the initial update is less than or equal to the preset variance threshold after the update, refer to steps S301 to S303 to update the model parameter gradient set after the initial update again. If the gradient variance of the model parameter gradient set after the initial update is greater than the preset variance threshold after the update, refer to steps S401 to S402 to update the model parameter gradient set after the initial update again. Repeat the above steps until the number of samples is greater than or equal to the preset number threshold, and stop data collection to reduce the data collection cost of the terminal device.

[0104] In the above steps S401 to S404, when the training process is unstable, the current model parameter gradient set is sampled to reduce the gradient variance, ensure the stability of the training process, enable the model to converge to the optimal value, and thus improve the accuracy of action recognition.

[0105] See also Figure 5 In some embodiments, step S402 may also include but is not limited to steps S501 to S502:

[0106] Step S501, determining a sampling probability according to the gradient length of each model parameter gradient in the model parameter gradient set;

[0107] Step S502: Sample the model parameter gradient set according to the sampling probability to obtain a second intermediate parameter gradient.

[0108] In step S501 of some embodiments, for the training of the deep learning model, the contribution of each data sample is not equal, and the model performance can be improved by selectively using some samples. The gradient length of each model parameter gradient in the model parameter gradient set is summed to obtain the length and value. The ratio of the gradient length to the length and value of each model parameter gradient in the model parameter gradient set is used as the sampling probability of the model parameter gradient. The sampling probability is the probability that the model parameter gradient is sampled, which is used to characterize the importance of the model parameter gradient to the parameter update. The larger the sampling probability, the more important the model parameter gradient. The sampling probability is expressed as:

[0109] ,

[0110] Among them, q represents the sampling probability and n represents the gradient length.

[0111] In step S502 of some embodiments, the cumulative probability is calculated according to the sampling probability, a random number is generated, and the model parameter gradient corresponding to the maximum value less than or equal to the random number in the cumulative probability is obtained to obtain the second intermediate parameter gradient. The cumulative probability is the sum of the current sampling probability and all sampling probabilities before the current sampling probability. For example, if the sampling probabilities are 1, 3, 2, and 4, respectively, then the cumulative probabilities are 1, 4, 6, and 10, respectively.

[0112] The above steps S501 to S502 sample the model parameter gradient set according to importance, which allows the edge intelligent system to obtain lower population loss with fewer data samples. At the same time, while keeping the data batch size unchanged, the local model can pay more attention to sub-samples with higher importance, thereby improving the model's action recognition performance.

[0113] See also Figure 6In some embodiments, step 103 may include but is not limited to steps S601 to S603:

[0114] Step S601, performing mean calculation on the updated model parameter gradient set to obtain the target model parameter gradient;

[0115] Step S602, encoding the target model parameter gradient;

[0116] Step S603: sending the encoded target model parameter gradients to the edge server on the same time-frequency block.

[0117] In step S601 of some embodiments, the model parameter gradient generated by one sample is often difficult to represent the entire data distribution, and each time the model parameters are updated, the average value of the model parameter gradient is usually generated by using multiple samples. Specifically, within one training round, after obtaining a model parameter gradient set having a second preset number of model parameter gradients, the average value of each model parameter gradient in the model parameter gradient set is calculated to obtain a target model parameter gradient, and based on the gradient descent algorithm, the target model parameter gradient is used to update the model parameters of the local model.

[0118] In step S602 of some embodiments, when the training round reaches the preset round threshold, the target model parameter gradient at this time is sent to the edge server. Alternatively, when the training round reaches the preset round threshold, the target model parameter gradient at this time is specially encoded. Compared with the traditional orthogonal multiple access, this method can reduce the training delay and the wireless communication time and frequency resource requirements of the terminal device for model training. The process of special encoding is as follows: simulate amplitude modulation of the target model parameter gradient, divide the channel into multiple sub-channels based on orthogonal frequency division multiplexing, and perform serial-to-parallel conversion on the modulated target model parameter gradient based on the multiple sub-channels to obtain multiple signals. Perform truncated channel inversion on the multiple signals, and perform inverse fast Fourier transform on each inverted signal. Add a cyclic prefix to each signal after the inverse fast Fourier transform, and perform parallel-to-serial conversion on the multiple signals after adding the cyclic prefix to obtain the target signal. Multiply the target signal with the preset carrier signal.

[0119] In some embodiments, when the training round reaches a preset round threshold, the model parameters of the local model are sent directly to the edge server, or the model parameters are specially encoded and sent to the edge server, so that the edge server aggregates the model parameters to obtain the target model. The encoding method of the model parameters can refer to the encoding method of the target model parameter gradient, which will not be repeated here.

[0120] In step S603 of some embodiments, orthogonal multiple access technology enables the terminal device to send or receive information in non-overlapping time or frequency domains. Otherwise, due to the aliasing of wireless signals, the information of the terminal device will be interfered with by other devices. Model aggregation can be performed on model parameters or target model parameter gradients that have not been specially encoded based on orthogonal multiple access. Special coding utilizes the waveform superposition characteristics of the multiple access channel, and sends the target model parameter gradient or model parameter to the edge server at the same time or frequency domain to achieve air aggregation. It should be noted that when using orthogonal multiple access, each terminal device needs to use different data accuracy to meet the requirement of adding different intensity noise to the model parameter vector.

[0121] Determine the communication power, and send the model parameters or target model parameter gradients to the edge server according to the communication power. As the training progresses, the communication power sent can be gradually increased so that more noise can be added to the model parameters, reducing the actual power required for sending, especially reducing the energy loss for communication in the early stage of training. In the edge intelligent system, through the joint control of the allocation of communication and perception resources (the number of data samples), it is possible to reduce the demand for communication energy and latency resources without sacrificing or even enhancing the performance of the machine learning model.

[0122] Through the above steps S601 to S603, the target model parameter gradient is sent to the edge server, so that the edge server implements air aggregation based on the target model parameter gradient, thereby improving the generalization of the model.

[0123] The edge server aggregates the gradients of multiple target model parameters from multiple terminal devices to obtain an aggregated gradient. Gradient aggregation can be performed by gradient averaging. Based on the gradient descent algorithm, the global model on the edge server is updated according to the aggregated gradient to obtain the target model. The target model is broadcast to each terminal device so that each terminal device can perform action recognition through the target model. The federated learning algorithm is an iterative algorithm. The algorithm stops after reaching a certain number of iterations or meeting other stopping conditions. Other stopping conditions may be that the communication power reaches a preset power threshold or the training time reaches a preset time threshold.

[0124] When the edge server performs aggregation, Gaussian white noise can also be added to the aggregated gradient. The model parameters of the global model are updated as follows:

[0125] w_new=w_old-a*g+t*n,

[0126] Among them, w_new is the updated model parameter; w_old is the model parameter before the update; g represents the aggregation gradient; a and t both represent weight parameters; n represents the noise intensity of the added Gaussian white noise, and the noise intensity can be determined by referring to the method of the terminal device.

[0127] For specially encoded target model parameter gradients, the edge server needs to decode the target model parameter gradients. Specifically, the preset carrier signal is multiplied with the superimposed waveforms (multiple target model parameter gradients) from multiple terminal devices in sequence. For each multiplication result, after removing the cyclic prefix of the multiplication result, serial-to-parallel conversion is performed to obtain multiple signals. After performing a fast Fourier transform on each signal, parallel-to-serial conversion is performed to obtain the target signal. Multiple target signals are aggregated to obtain an aggregated gradient.

[0128] The distribution of environmental perception data of different terminal devices is often different, and there is a problem of data heterogeneity, which makes the model training quality of each terminal device different, which may lead to the performance degradation of the global model. In order to improve the accuracy of action recognition by the global model, it is necessary to screen the terminal devices involved in the aggregation. In some embodiments, after the edge server receives multiple target model parameter gradients or multiple model parameters from multiple terminal devices, it obtains the remaining storage space capacity of each terminal device. If the remaining storage space capacity is greater than the preset capacity threshold, the same verification data set and loss function are sent to each terminal device. The terminal device calculates the loss of each local model according to the verification data set and the loss function, and sends the loss to the edge server. The edge server screens the target model parameter gradient or model parameter according to the loss, performs gradient aggregation on the screened target model parameter gradient, or performs parameter aggregation on the screened model parameter. When screening, the target model parameter gradient or model parameter sent by the terminal device with smaller loss can be selected. For example, the losses of each terminal device are sorted from small to large, and the target model parameter gradient or model parameter corresponding to the first few losses are selected. The same verification data set and loss function are sent to each terminal device through the edge server to ensure that all terminal devices use the same distributed data and the same evaluation indicators, reducing the impact of data heterogeneity on the global model.

[0129] In other embodiments, in order to improve the generalization and accuracy of the global model, the terminal devices are screened based on the data distribution of different terminal devices, and terminal devices with high data quality and rich data diversity are selected to participate in aggregation, so that the global model can recognize multiple actions in different geographical locations and at different time points. Specifically, each terminal device performs a quality assessment on the sampled environmental perception sample set to obtain a data quality score, and each terminal device performs a data diversity assessment on the sampled environmental perception sample set to obtain a diversity score. If the data quality score is greater than the first preset score threshold, and the diversity score is greater than the second preset score threshold, the target model parameter gradient or model parameter is sent to the edge server, so that the edge server aggregates the target model parameter gradient or model parameter. Both data quality assessment and data diversity assessment can be performed in the form of user feedback or evaluation models.

[0130] In other embodiments, in order to improve the efficiency of model training, terminal devices are screened based on the device performance of the terminal devices. Specifically, each terminal device obtains the processing time of the model parameters of the local model updated a preset number of times according to the environmental perception sample set, and the sending time of sending the target model parameter gradient or model parameter to the edge server. The preset number of times is greater than or equal to 1 and can be set by itself. If the processing time is less than the first preset time threshold, and the sending time is less than the second preset time threshold, the target model parameter gradient or model parameter is sent to the edge server, so that the edge server aggregates the target model parameter gradient or model parameter. The processing time can be used to select terminal devices with faster calculation speeds, and the sending time can be used to select terminal devices with faster communication rates, thereby improving the efficiency of the entire model training.

[0131] The embodiment of the present application provides a model training method, including: each terminal device collects a first preset number of sub-samples, calculates the model parameter gradient of each sub-sample on the local model of the terminal device, obtains a model parameter gradient set, and calculates the gradient variance according to the model parameter gradient set. If the first preset number is less than the second preset number, it is determined whether the gradient variance is greater than the preset variance threshold. If the gradient variance is less than or equal to the preset variance threshold, a new sub-sample is collected to update the model parameter gradient set, and the above collection steps are repeated until the gradient variance is greater than the preset variance threshold. If the gradient variance is greater than the preset variance threshold, the preset variance threshold is updated, and the updated preset variance threshold is used for sample collection in the next training round. And the sampling probability is calculated according to the model parameter gradient set, and the model parameter gradient set is resampled to a gradient batch with a second preset number of model parameter gradients according to the sampling probability. If the first preset number is greater than or equal to the second preset number, the average value of the model parameter gradient set is obtained to obtain the target model parameter gradient, and after the target model parameter gradient is specially encoded, each terminal device sends the target model parameter gradient to the edge server at its own preset communication power in the same time or frequency domain. After decoding each target model parameter gradient, the edge server aggregates each target model parameter gradient in the air to obtain an aggregated model parameter gradient. According to the aggregated model parameter gradient, the model parameters of the global model on the edge server are updated to obtain the target model, and the target model is broadcast to each terminal device.

[0132] It should be noted that the sampling probability is proportional to the gradient length of the model parameter gradient, and the model parameter gradient set can be unbiased upsampled based on the sampling probability.

[0133] Figure 7 is an optional flow chart of the action recognition method provided in the embodiment of the present application, which is applied to a terminal device. Figure 7 The method may include but is not limited to steps S701 to S703:

[0134] Step S701, receiving a target model sent by an edge server;

[0135] Step S702, acquiring target environment perception data;

[0136] Step S703: input the target environment perception data into the target model for action recognition.

[0137] In step S701 of some embodiments, the terminal device receives the target model broadcast by the edge server to perform action recognition using the trained target model.

[0138] In step S702 of some embodiments, the terminal device senses the surrounding environment and obtains target environment perception data. The target environment perception data may be an image, text, audio signal, etc.

[0139] In step S703 of some embodiments, the target environment perception data is input into the target model for action recognition to obtain a predicted action category.

[0140] Through the above steps S701 to S703, the terminal device can perform accurate action recognition.

[0141] See also Figure 8 The embodiment of the present application also provides a model training system, which can implement the above-mentioned model training method, and the model training system includes:

[0142] The terminal device 801 is used to obtain an environment perception sample set by sampling from the collected environment perception data distribution, and determine the data collection order of each subsample in the environment perception sample set, and calculate the model parameter gradient of the local model on the terminal device according to each subsample in sequence according to the data collection order to obtain the model parameter gradient set, and obtain the gradient variance according to the model parameter gradient set; update the model parameter gradient set according to the gradient variance and a preset variance threshold, determine the target model parameter gradient according to the updated model parameter gradient set, and send the target model parameter gradient to the edge server;

[0143] The edge server 802 is used to perform gradient aggregation on multiple target model parameter gradients to obtain aggregated gradients, and update the global model on the edge server according to the aggregated gradient to obtain the target model, and broadcast the target model to each terminal device so that each terminal device can perform action recognition through the target model.

[0144] The terminal device 801 is also used for:

[0145] According to the data collection order, each sub-sample is sequentially input into the local model for action recognition to obtain the predicted action category of each sub-sample; each sub-sample has a preset action label;

[0146] Calculate the cross entropy loss based on the predicted action category and the preset action label to get the target loss;

[0147] Determine the initial parameter gradient of the local model in the subsample according to the target loss and the model parameters of the local model;

[0148] Obtain the training round of the local model, and generate noise according to the training round; the noise intensity of the noise is inversely proportional to the square root of the training round;

[0149] Add noise to the initial parameter gradient to obtain the model parameter gradient of the local model in the subsample;

[0150] According to the model parameter gradients of each sub-sample, a model parameter gradient set is determined.

[0151] The terminal device 801 is also used for:

[0152] If the gradient variance is less than or equal to the preset variance threshold, and the number of samples in the environment perception sample set is less than the preset number threshold, a new sub-sample is sampled from the environment perception data distribution, and the new sub-sample is added to the environment perception sample set;

[0153] Calculate the first intermediate parameter gradient of the new sub-sample;

[0154] The first intermediate parameter gradient is added to the model parameter gradient set to update the model parameter gradient set until the gradient variance is greater than a preset variance threshold or the number of samples is greater than or equal to a preset number threshold.

[0155] The terminal device 801 is also used for:

[0156] If the gradient variance is greater than a preset variance threshold and the number of samples in the environment perception sample set is less than a preset number threshold, the preset variance threshold is updated according to the gradient variance and a preset smoothing parameter;

[0157] Sampling the model parameter gradient set to obtain a second intermediate parameter gradient, and adding a subsample corresponding to the second intermediate parameter gradient to the environment perception sample set;

[0158] adding the second intermediate parameter gradient to the model parameter gradient set to perform an initial update on the model parameter gradient set;

[0159] The model parameter gradient set after the initial update is updated again according to the updated preset variance threshold and the gradient variance of the model parameter gradient set after the initial update until the number of samples is greater than or equal to the preset number threshold.

[0160] The terminal device 801 is also used for:

[0161] Determine the sampling probability according to the gradient length of each model parameter gradient in the model parameter gradient set;

[0162] The model parameter gradient set is sampled according to the sampling probability to obtain the second intermediate parameter gradient.

[0163] The terminal device 801 is also used for:

[0164] Perform mean calculation on the updated model parameter gradient set to obtain the target model parameter gradient;

[0165] Encode the target model parameter gradients;

[0166] On the same time-frequency block, the encoded target model parameter gradients are sent to the edge server.

[0167] See also Fig. 9 The embodiment of the present application also provides a motion recognition device, which is applied to a terminal device and can implement the above-mentioned motion recognition method. The device includes:

[0168] A receiving module 901 is used to receive a target model sent by an edge server;

[0169] An acquisition module 902 is used to acquire target environment perception data;

[0170] The action recognition module 903 is used to input the target environment perception data into the target model for action recognition.

[0171] The specific implementation of the motion recognition device is basically the same as the specific implementation of the above-mentioned motion recognition method, and will not be repeated here.

[0172] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned model training method or action recognition method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0173] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0174] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0175] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the model training method or action recognition method of the embodiments of this application;

[0176] Input / output interface 1003, used to implement information input and output;

[0177] Communication interface 1004, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0178] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0179] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0180] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned model training method or action recognition method.

[0181] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0182] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0183] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0184] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0186] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0187] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0192] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A model training method, characterized in that: Applied to a model training system, the model training system includes multiple terminal devices and an edge server, and the method includes: Each of the terminal devices samples from the collected environment perception data distribution to obtain an environment perception sample set, and determines the data collection order of each subsample in the environment perception sample set, and calculates the model parameter gradient of the local model on the terminal device according to each subsample in sequence according to the data collection order to obtain a model parameter gradient set, and obtains the gradient variance according to the model parameter gradient set; the calculation formula of the gradient variance is: , Among them, Var represents the gradient variance, bmin is the number of subsamples in the environment perception sample set, ni represents the i-th gradient length, and the gradient length is the modulus length of the model parameter gradient. is the average value of each gradient length; Each of the terminal devices updates the model parameter gradient set according to the gradient variance and the preset variance threshold, determines the target model parameter gradient according to the updated model parameter gradient set, and sends the target model parameter gradient to the edge server; the preset variance threshold is expressed as: , Where b is the number of samples in the environment perception sample set; bmax is the maximum number of samples allowed in the environment perception sample set; is the gradient variance threshold; The edge server aggregates the gradients of the plurality of target model parameter gradients to obtain an aggregated gradient, updates the global model on the edge server according to the aggregated gradient to obtain a target model, and broadcasts the target model to each of the terminal devices so that each of the terminal devices performs action recognition through the target model; The step of calculating the model parameter gradient of the local model on the terminal device according to each sub-sample in sequence according to the data collection order to obtain a model parameter gradient set includes: According to the data collection order, each of the sub-samples is sequentially input into the local model for action recognition to obtain a predicted action category of each sub-sample; each of the sub-samples has a preset action label; a cross entropy loss is calculated according to the predicted action category and the preset action label to obtain a target loss; according to the target loss and the model parameters of the local model, an initial parameter gradient of the local model in the sub-sample is determined; the training round of the local model is obtained, and noise is generated according to the training round; the noise intensity of the noise is inversely proportional to the square root of the training round; the noise is added to the initial parameter gradient to obtain the model parameter gradient of the local model in the sub-sample; according to the model parameter gradient of each sub-sample, the model parameter gradient set is determined; The training round is the number of times the local model has been trained, and the initial noise intensity is a preset multiple of the gradient length of the initial parameter gradient, and the preset multiple is in the range of [0.0001, 0.001]; The updating of the model parameter gradient set according to the gradient variance and a preset variance threshold comprises: If the gradient variance is less than or equal to the preset variance threshold, and the number of samples in the environment perception sample set is less than the preset number threshold, a new subsample is sampled from the environment perception data distribution, and the new subsample is added to the environment perception sample set; a first intermediate parameter gradient of the new subsample is calculated; the first intermediate parameter gradient is added to the model parameter gradient set to update the model parameter gradient set until the gradient variance is greater than the preset variance threshold or the number of samples is greater than or equal to the preset number threshold; The updating of the model parameter gradient set according to the gradient variance and the preset variance threshold further includes: If the gradient variance is greater than the preset variance threshold, and the number of samples in the environment perception sample set is less than the preset number threshold, the preset variance threshold is updated according to the gradient variance and the preset smoothing parameter; the model parameter gradient set is sampled to obtain a second intermediate parameter gradient, and a subsample corresponding to the second intermediate parameter gradient is added to the environment perception sample set; the second intermediate parameter gradient is added to the model parameter gradient set to perform an initial update on the model parameter gradient set; the model parameter gradient set after the initial update is updated again according to the updated preset variance threshold and the gradient variance of the model parameter gradient set after the initial update, until the number of samples is greater than or equal to the preset number threshold; The update formula of the gradient variance threshold is expressed as: , in, is the gradient variance; is the preset smoothing parameter; is the gradient variance threshold.

2. The model training method according to claim 1, characterized in that: The step of sampling the model parameter gradient set to obtain a second intermediate parameter gradient includes: Determining a sampling probability according to the gradient length of each model parameter gradient in the model parameter gradient set; The model parameter gradient set is sampled according to a sampling probability to obtain the second intermediate parameter gradient.

3. The model training method according to any one of claims 1 to 2, characterized in that: The step of determining a target model parameter gradient according to the updated model parameter gradient set and sending the target model parameter gradient to the edge server includes: Performing mean calculation on the updated model parameter gradient set to obtain the target model parameter gradient; Encoding the target model parameter gradients; On the same time-frequency block, the encoded target model parameter gradient is sent to the edge server.

4. A method for motion recognition, characterized in that: Applied to a terminal device, the method comprises: Receiving a target model sent by an edge server; the target model is trained by the model training method according to any one of claims 1 to 3; Acquire target environment perception data; The target environment perception data is input into the target model for action recognition.

5. A motion recognition device, characterized in that: Applied to a terminal device, the device comprises: A receiving module, configured to receive a target model sent by an edge server; the target model is trained by the model training method according to any one of claims 1 to 3; An acquisition module is used to acquire target environment perception data; The action recognition module is used to input the target environment perception data into the target model for action recognition.

6. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method described in any one of claims 1 to 3 or the action recognition method described in claim 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the model training method described in any one of claims 1 to 3 or the action recognition method described in claim 4.