Gesture recognition method, device, storage medium and wearable device

Abstract

The application relates to the technical field of biological signal processing, and discloses a gesture recognition method and device, a storage medium and a wearable device. The method comprises the following steps: acquiring historical model parameters of a general gesture recognition model, wherein the general gesture recognition model is obtained through iterative training based on electromyogram signals of historical users; training the general gesture recognition model through calibration electromyogram signals of different calibration users and historical electromyogram signals of a current user to obtain an initial gesture recognition model; training the initial gesture recognition model according to the historical model parameters and calibration electromyogram signals of the current user to obtain a target gesture recognition model; and performing gesture recognition on collected electromyogram signals of the current user through the target gesture recognition model. Compared with the prior art, the application can reduce the influence of training on the next user and improve user experience.

Description

Technical Field

[0001] This application relates to the field of biosignal processing technology, and in particular to a gesture recognition method, apparatus, storage medium and wearable device. Background Technology

[0002] Electromyography (EMG) signals are electrophysiological signals generated by muscle tissue during contraction, which can express the state of muscle movement. Currently, surface electromyography (sEMG) signals can be used to identify subtle hand movements, and are therefore often used to build gesture recognition systems.

[0003] In existing technologies, most gesture recognition systems train a general gesture recognition model using data from a large number of users. However, due to differences in physical condition, muscle strength, and movement habits among individuals, the EMG signals generated when the same action is performed by different people will vary. Therefore, when a new user uses the system, the gesture recognition model can be calibrated using that user's data. To make the calibrated model even more suitable for the new user, it can be fine-tuned using a small amount of labeled data from that new user. However, this can lead to catastrophic forgetting, or overtraining, where the model forgets much of the learned information and becomes more biased towards the new user. Consequently, the training effect is poor when the next user uses the system, impacting the user experience. Summary of the Invention

[0004] The main objective of this application is to provide a gesture recognition method, apparatus, storage medium, and wearable device, aiming to solve the technical problem that after a new user calibrates the gesture recognition model, and then fine-tunes it using a small amount of labeled data from that new user, the resulting gesture recognition model suffers catastrophic forgetting, leading to poor training performance when the next user uses it.

[0005] To achieve the above objectives, embodiments of this application provide a gesture recognition method, the method comprising:

[0006] Obtain historical model parameters of a general gesture recognition model, which is obtained through iterative training based on electromyography signals of historical users;

[0007] The general gesture recognition model is trained using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user to obtain an initial gesture recognition model.

[0008] The initial gesture recognition model is trained based on the historical model parameters and the current user's calibrated electromyography signal to obtain the target gesture recognition model;

[0009] The target gesture recognition model is used to perform gesture recognition on the electromyography signals of the current user.

[0010] In one embodiment, the step of training the general gesture recognition model using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user to obtain an initial gesture recognition model includes:

[0011] Determine the domain similarity between the calibrated electromyography (EMG) signals of different calibrated users and the historical EMG signals of the current user;

[0012] A reference electromyography signal is selected from the calibrated electromyography signals based on the domain similarity.

[0013] The general gesture recognition model is trained using the reference electromyography (EMG) signal and the historical EMG signal to obtain an initial gesture recognition model.

[0014] In one embodiment, the step of training the general gesture recognition model using the reference electromyography signal and the historical electromyography signal to obtain an initial gesture recognition model includes:

[0015] The intermediate layer of the general gesture recognition model obtains the source domain intermediate feature vector corresponding to the reference electromyography signal and the target domain intermediate feature vector corresponding to the historical electromyography signal, respectively.

[0016] Based on the first output layer of the general gesture recognition model, the source domain gesture category probability corresponding to the reference electromyography signal and the main target domain gesture category probability corresponding to the historical electromyography signal are obtained respectively.

[0017] A training loss function is constructed based on the source domain intermediate feature vector, the target domain intermediate feature vector, the source domain gesture category probability, and the main target domain gesture category probability.

[0018] The general gesture recognition model is trained using the first-round training loss function to obtain an initial gesture recognition model.

[0019] In one embodiment, the step of constructing a training loss function based on the source domain intermediate feature vector, the target domain intermediate feature vector, the source domain gesture category probability, and the main target domain gesture category probability includes:

[0020] Construct a joint maximum mean difference loss function based on the intermediate feature vector of the source domain and the intermediate feature vector of the target domain;

[0021] A first cross-entropy loss function is constructed using the source domain gesture category probability, and a minimum class confusion loss function is constructed using the main target domain gesture category probability.

[0022] A training loss function is constructed based on the joint maximum mean difference loss function, the first cross-entropy loss function, and the minimum class confusion loss function.

[0023] In one embodiment, the step of constructing a minimum-class confusion loss function based on the gesture category probabilities of the main target domain includes:

[0024] The probability of the gesture category in the main target domain is adjusted, and the adjusted probability of the gesture category in the main target domain is reweighted.

[0025] The class association reduction operation is performed on the reweighted main target domain gesture category probability, and a minimum class confusion loss function is constructed based on the operation result.

[0026] In one embodiment, the loss function for one round of training is:

[0027] Loss1=L CE (S)+αL JMMD (S,T)+βL MCC (T);

[0028] Where Loss1 is the loss function for the first round of training, L CE (S) is the first cross-entropy loss function, L JMMD (S,T) is the joint maximum mean difference loss function, L MCC (T) is the minimum class confusion loss function, α is the first preset weight parameter, and β is the second preset weight parameter.

[0029] In one embodiment, the step of training the initial gesture recognition model based on the historical model parameters and the calibrated electromyography signal of the current user to obtain the target gesture recognition model includes:

[0030] Obtain the current model parameters of the initial gesture recognition model, and construct a regularization term based on the current model parameters, preset hyperparameters, and the historical model parameters;

[0031] A second cross-entropy loss function is constructed using the regularization term, and the initial gesture recognition model is trained based on the second cross-entropy loss function and the calibrated electromyography signal of the current user to obtain the target gesture recognition model.

[0032] In one embodiment, the step of constructing the second cross-entropy loss function using the regularization term includes:

[0033] The calibration gesture recognition probability corresponding to the calibration electromyography signal of the current user is obtained through the second output layer of the general gesture recognition model.

[0034] The calibration data label corresponding to the calibration electromyography signal is determined, and a second cross-entropy loss function is constructed based on the calibration gesture recognition probability, the calibration data label, and the regularization term.

[0035] In one embodiment, the second cross-entropy loss function is:

[0036]

[0037] Among them, L CE-2 (T′) is the second cross-entropy loss function, C is the number of gesture categories, and y si P is the calibration data label corresponding to the i-th gesture category. i Let λ be the calibration gesture recognition probability corresponding to the i-th gesture category, λ be the preset hyperparameter, and θ be the historical model parameter. 0 These are the current model parameters of the initial gesture recognition model.

[0038] In one embodiment, the step of training the initial gesture recognition model based on the second cross-entropy loss function and the calibrated electromyography signal of the current user to obtain the target gesture recognition model includes:

[0039] A confidence test is performed on the probability of the gesture category in the main target domain to obtain the pseudo-label of the target domain, and the probability of the gesture category in the auxiliary target domain corresponding to the historical electromyography signal is obtained based on the second output layer of the general gesture recognition model.

[0040] A generalized cross-entropy loss function is constructed based on the target domain pseudo-label and the auxiliary target domain gesture category probability, and a second-round training loss function is constructed based on the generalized cross-entropy loss function and the second cross-entropy loss function;

[0041] The initial gesture recognition model is trained using the two-round training loss function to obtain the target gesture recognition model.

[0042] In one embodiment, the generalized cross-entropy loss function is:

[0043]

[0044] Among them, L GCE (T) is the generalized cross-entropy loss function, C is the number of gesture categories, and y ti P is the pseudo-label of the target domain corresponding to the i-th gesture category. iq represents the calibration gesture recognition probability corresponding to the i-th gesture category, and q is a preset adjustment parameter.

[0045] In one embodiment, the second-round training loss function is:

[0046] Loss2=L GCE (T)+γL CE-2 (T′);

[0047] Where Loss2 is the loss function for the second round of training, L GCE (T) is the generalized cross-entropy loss function, L CE-2 (T′) is the second cross-entropy loss function, and γ is the preset third weight.

[0048] In one embodiment, the number of reference electromyography signals is at least two, and the number of target gesture recognition models is the same as the number of reference electromyography signals;

[0049] The step of performing gesture recognition on the collected electromyography signals of the current user using the target gesture recognition model includes:

[0050] The electromyography signals of the current user are collected and the gestures are recognized by each of the target gesture recognition models to obtain the initial recognition results corresponding to each target gesture recognition model.

[0051] The initial recognition results are integrated to obtain the target gesture recognition result.

[0052] In one embodiment, before the step of training the general gesture recognition model using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user, the method further includes:

[0053] Determine if the current user is a first-time user;

[0054] If so, the calibrated electromyography (EMG) signal of the current user is used as the historical EMG signal of the current user, and the step of training the general gesture recognition model using the calibrated EMG signals of different calibrated users and the historical EMG signal of the current user is performed.

[0055] Furthermore, to achieve the above objectives, this application also proposes a gesture recognition device, the device comprising:

[0056] The parameter acquisition module is used to acquire historical model parameters of the general gesture recognition model, which is obtained through iterative training based on electromyography signals of historical users.

[0057] A training module is used to train the general gesture recognition model using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user, in order to obtain an initial gesture recognition model.

[0058] The second-round training module is used to train the initial gesture recognition model based on the historical model parameters and the current user's calibrated electromyography signal to obtain the target gesture recognition model.

[0059] The gesture recognition module is used to perform gesture recognition on the electromyography signals of the current user collected by the target gesture recognition model.

[0060] In addition, to achieve the above objectives, this application also proposes a storage medium storing a gesture recognition program, which, when executed by a processor, implements the gesture recognition method as described above.

[0061] In addition, to achieve the above objectives, this application also proposes a wearable device, which includes: a memory, a processor, and a gesture recognition program stored in the memory and executable on the processor. When the gesture recognition program is executed by the processor, it implements the gesture recognition method described above.

[0062] This application provides a gesture recognition method, apparatus, storage medium, and wearable device. The method includes: acquiring historical model parameters of a general gesture recognition model, wherein the general gesture recognition model is obtained through iterative training based on electromyography (EMG) signals of historical users; training the general gesture recognition model using calibrated EMG signals of different calibrated users and historical EMG signals of the current user to obtain an initial gesture recognition model; training the initial gesture recognition model using the historical model parameters and calibrated EMG signals of the current user to obtain a target gesture recognition model; and performing gesture recognition on the collected EMG signals of the current user using the target gesture recognition model.

[0063] In this embodiment, a general gesture recognition model can first be trained using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user to obtain an initial gesture recognition model. During this process, since the calibrated EMG signals come from different calibrated users, combined with the current user's historical EMG signals, the obtained initial gesture recognition model can be tailored to the current user while preventing excessive deviation from the current user's characteristics. Next, the initial gesture recognition model can be trained again using the historical model parameters of the general gesture recognition model and the current user's calibrated EMG signals to obtain a target gesture recognition model. During this process, since the general gesture recognition model is iteratively trained based on historical users' EMG signals, combined with the current user's calibrated EMG signals, the obtained target gesture recognition model can further ensure that it is tailored to the current user while preventing excessive deviation from the current user's characteristics. Finally, the obtained target gesture recognition model is used to perform gesture recognition on the current user's EMG signals. Compared to existing methods that fine-tune the model using a small amount of labeled data from the user after calibration, this application can first train the model using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user. Then, it can train the model using historical model parameters from a general gesture recognition model and calibrated EMG signals from the current user. This allows the target gesture recognition model to be more suitable for the current user while reducing the occurrence of catastrophic forgetting, thereby reducing the impact on training the next user and improving the user experience. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic diagram of the wearable device structure in the hardware operating environment involved in the embodiments of this application;

[0067] Figure 2 This is a flowchart illustrating the first embodiment of the gesture recognition method of this application;

[0068] Figure 3 This is a schematic diagram of the overall process of the gesture recognition method in this application;

[0069] Figure 4 This is a schematic diagram of the gesture calibration process in the gesture recognition method of this application;

[0070] Figure 5 This document outlines the process for constructing electromyography (EMG) signals in the gesture recognition method of this application.

[0071] Figure 6 This is a flowchart illustrating the second embodiment of the gesture recognition method of this application;

[0072] Figure 7 This is a schematic diagram of model training in the gesture recognition method of this application;

[0073] Figure 8 This is a schematic diagram illustrating the integration of model construction and output results in the gesture recognition method of this application;

[0074] Figure 9 This is a flowchart illustrating the third embodiment of the gesture recognition method of this application;

[0075] Figure 10 This chart compares the accuracy of gesture recognition achieved by this method with that of the traditional method.

[0076] Figure 11 This is a structural block diagram of the first embodiment of the gesture recognition device of this application.

[0077] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0078] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0079] Reference Figure 1 , Figure 1 This is a schematic diagram of the wearable device structure of the hardware operating environment involved in the embodiments of this application.

[0080] like Figure 1As shown, the wearable device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen, and optionally, it may also include a standard wired interface or a wireless interface. In this application, the wired interface of the user interface 1003 may be a USB interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0081] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on wearable devices and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0082] like Figure 1 As shown, the memory 1005, which is identified as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a gesture recognition program.

[0083] exist Figure 1 In the wearable device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the user device; the wearable device calls the gesture recognition program stored in the memory 1005 through the processor 1001 and executes the gesture recognition method provided in the embodiments of this application.

[0084] It should be noted that electromyography (EMG) signals are electrophysiological signals generated by muscle tissue during contraction, and they can express the state of muscle movement. Currently, surface electromyography (sEMG) signals can be used to identify subtle hand movements, and are therefore often used to build gesture recognition systems.

[0085] In existing technologies, most gesture recognition systems train a general gesture recognition model using data from a large number of users. However, due to differences in physical condition, muscle strength, and movement habits among individuals, the EMG signals generated when the same action is performed by different people will vary. Therefore, when a new user uses the system, the gesture recognition model can be calibrated using that user's data. To make the calibrated model even more suitable for the new user, it can be fine-tuned using a small amount of labeled data from that new user. However, this can lead to catastrophic forgetting, or overtraining, where the model forgets much of the learned information and becomes more biased towards the new user. Consequently, the training effect is poor when the next user uses the system, impacting the user experience.

[0086] To address the aforementioned shortcomings, this embodiment provides a gesture recognition method. First, a general gesture recognition model is trained using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user to obtain an initial gesture recognition model. Since the calibrated EMG signals come from different calibrated users, combined with the current user's historical EMG signals, the initial gesture recognition model is tailored to the current user while preventing excessive deviation from the current user's characteristics. Next, the initial gesture recognition model is trained again using historical model parameters of the general gesture recognition model and the current user's calibrated EMG signals to obtain a target gesture recognition model. In this process, since the general gesture recognition model is iteratively trained based on historical users' EMG signals, combined with the current user's calibrated EMG signals, the target gesture recognition model is further tailored to the current user while preventing excessive deviation from the current user's characteristics. Finally, the target gesture recognition model is used to perform gesture recognition on the current user's EMG signals. Compared to existing methods that fine-tune the model using a small amount of labeled data from the user after calibration, this embodiment first trains the model using calibrated electromyography (EMG) signals from different calibrated users and the current user's historical EMG signals. Then, it trains the model using historical model parameters of a general gesture recognition model and the current user's calibrated EMG signals. This allows the target gesture recognition model to be more suitable for the current user while reducing the occurrence of catastrophic forgetting, thereby reducing the impact on training the next user and improving the user experience.

[0087] For ease of understanding, the following is combined with Figures 2 to 11 The gesture recognition method provided in the embodiments of this application will be described in detail.

[0088] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the gesture recognition method of this application. The first embodiment of the gesture recognition method of this application is presented as follows: Figure 2 As shown, in this embodiment, the specific method includes:

[0089] Step S10: Obtain the historical model parameters of the general gesture recognition model. The general gesture recognition model is obtained through iterative training based on the electromyography signals of historical users.

[0090] It is understood that the method of this embodiment can be applied to wearable devices, wherein the wearable device can be an edge device worn on the user's body, such as a smart wristband or smart bracelet, or other wearable devices. This embodiment does not limit the application of such devices. The execution subject of this method embodiment can be the aforementioned wearable device (hereinafter referred to as the device). This embodiment and the following embodiments will be described using the device as an example.

[0091] It should be understood that the aforementioned general gesture recognition model can be the model obtained after the previous training, that is, the model used for gesture recognition in this embodiment before training. The device can be equipped with this general gesture recognition model, meaning that the gesture recognition model can be stored in the device during production or shipment. When a user uses the device, the general gesture recognition model can be trained using the user's own electromyography (EMG) signals, making the model more suitable for that user. Furthermore, when each new user uses the device, the general gesture recognition model obtained in the previous training can be trained using the user's own EMG signals.

[0092] Therefore, the aforementioned historical users refer to users who used the device before the current user. The electromyography (EMG) signals of these historical users are the EMG signals corresponding to each previous user, which can be collected during user use. Consequently, when the current user uses the device, the general gesture recognition model can be obtained by iteratively training using the EMG signals of historical users.

[0093] It should be noted that the aforementioned historical model parameters can be the model parameters corresponding to previous users using the general gesture recognition model. Specifically, they can be the model parameters obtained during the last training, or they can be the model parameters obtained during any previous training. This embodiment does not impose any restrictions on this. The model parameters may include, but are not limited to: weights, biases, connection modes, pooling layer parameters, and hyperparameters.

[0094] The aforementioned device can store historical model parameters of a general gesture recognition model. In specific implementations, when the current user uses the aforementioned device for the first time, or when the current user needs to calibrate the aforementioned device (for example, when the current user finds that the gesture recognition result has a large error after using it for a long time and needs to calibrate it), the calibration function of the aforementioned device can be activated to start training the general gesture recognition model currently stored in the aforementioned device. The aforementioned device can then obtain the historical model parameters of the pre-stored general gesture recognition model.

[0095] Step S20: Train the general gesture recognition model using the calibrated electromyography (EMG) signals of different calibrated users and the historical EMG signals of the current user to obtain the initial gesture recognition model.

[0096] Understandably, before using the aforementioned device, the manufacturer can recruit several (e.g., N) subjects. These N subjects should ideally cover different age groups, genders, and physical differences such as height and weight, and the more subjects, the better, forming the different calibrated users. Then, the device collects the gestures of each subject according to the gesture recognition requirements (e.g., which gestures need to be recognized), obtaining the calibrated electromyography (EMG) signals corresponding to each calibrated user. For ease of understanding, refer to... Figures 3 to 5 To explain, Figure 3 This is a schematic diagram of the overall process of the gesture recognition method in this application. Figure 4 This is a schematic diagram of the gesture calibration process in the gesture recognition method of this application. Figure 5 This document outlines the process for constructing calibration electromyography signals in the gesture recognition method of this application.

[0097] Before use, N subjects can perform corresponding actions (i.e., through motion capture guidance from the aforementioned device, such as video or audio) using the device's motion capture function. Figure 4 The recruited subjects perform corresponding movements under the guidance of the motion acquisition guidance system. The aforementioned device can construct a historical calibration dataset based on the acquired electromyography (EMG) signals. The aforementioned device can acquire the EMG signals corresponding to the N subjects as the aforementioned calibration EMG signals. The calibration EMG signals are labeled motion data. By constructing a corresponding dataset for each subject's calibration EMG signals, N motion datasets can be obtained. These datasets are labeled motion datasets and are stored on a cloud server (i.e., Figure 3 Step a: Construct the historical calibration dataset, that is... Figure 5 Step 1: Collect a large amount of subject movement data (according to the gesture movement requirements, collect labeled datasets of N subjects)).

[0098] After obtaining the calibrated electromyography (EMG) signals from N subjects, considering the potential interference in the signals, the aforementioned device can perform preprocessing operations on each calibrated EMG signal. These preprocessing operations may include, but are not limited to, outlier repair and filtering. Specifically, for example... Figure 5 As shown, the acquired calibrated electromyography signals can undergo preprocessing such as anomaly detection, missing value repair, bandpass filtering, data cleaning, and feature extraction (i.e., Figure 5 Step 2: Data processing (performing anomaly detection, filtering, cleaning, etc. on the dataset).

[0099] For example, since the sensor in the device that collects electromyography signals may have poor contact or external disturbances, there may be abnormally high and abnormally low sampling points in the acquisition results. The device can then determine whether the signal amplitude of each sampling point is higher or lower than the corresponding preset threshold. If so, it can be determined as abnormal and then repaired by the mean difference method.

[0100] After the repair is completed, in order to eliminate noise interference from other frequency bands, the above-mentioned device can use a bandpass filter and set a preset frequency band. Then, the repaired acquisition results can be filtered by the bandpass filter to remove noise interference from other frequency bands besides the preset frequency band, thus completing the filtering operation. The preset frequency band can be selected from 20-500Hz, but it can also be other frequency bands. This embodiment does not limit this.

[0101] After filtering, the device can perform activity segment detection. This is because the device usually collects data in real time during the user's wearing of the device, and most of the signal data is data from non-action states. Therefore, this embodiment can use threshold-based activity segment detection to filter out models with gesture movements. However, since the subject makes the corresponding movements according to the guidance, there is no need to perform activity segment detection here, while the other collected electromyography signals can be detected.

[0102] In order to facilitate subsequent processing, after the above-mentioned device performs active segment detection, it can extract features from the current electromyography signal through mathematical statistics. Specifically, it can include, but is not limited to, at least one of the following: root mean square, mean absolute value, zero-crossing rate, or wavelength of the current electromyography signal. This embodiment does not limit this.

[0103] After the preprocessing operations described above, the calibration dataset is used to transfer gesture knowledge learned from labeled historical data to user data. Therefore, the data quality of the calibration dataset greatly affects the effectiveness of knowledge transfer. Furthermore, to improve subsequent training results, the device can also perform quality assessment on these signals (i.e.,...). Figure 5 Step 3: Data Quality Assessment (Assessing the quality of the datasets collected from each individual and removing low-quality datasets). Data quality assessment includes checking the quality of the preprocessed electromyography (EMG) signals, signal-to-noise ratio, signal fluctuation, and loss rate. In addition, a self-test should be conducted using each subject's data to verify the effectiveness of that subject's data in gesture recognition and classification. If the accuracy of the self-test is below a certain threshold, the source domain data for that subject will be deemed unqualified, removed, and the subject will be re-collected, repeating the above process. Finally, a standardized historical calibration dataset is obtained, which serves as the calibrated EMG signals for the different calibration users and is stored (i.e.,...). Figure 5 Step 4: Historical calibration data storage (storing labeled datasets that conform to standards to form historical calibration datasets)).

[0104] It should be understood that the aforementioned historical electromyography (EMG) signals of the current user can be EMG signals obtained by the device after collecting and detecting the movement segments of the gestures previously made by the current user. Specifically, during use, the device can store the EMG signals corresponding to each gesture made by the current user as the aforementioned historical EMG signals (i.e., Figure 4 (User's historical data). However, considering that if a user uses the device for a long time, a large number of historical electromyography (EMG) signals will be stored. Therefore, the device can store historical EMG signals of the current user within a preset time range from the current moment, that is, store the EMG signals corresponding to the gestures made by the current user recently (i.e., ... Figure 3 Step b: Real-time collection and storage of recent user activity data.

[0105] It should be emphasized that after acquiring the electromyography (EMG) signal by collecting the current user's movements, the above-mentioned preprocessing operations can also be used to process the EMG signal to improve data quality. The specific process is similar to that described above, and this embodiment does not impose any restrictions on it.

[0106] In practice, after obtaining the calibrated electromyography (EMG) signals of different calibrated users and the historical EMG signals of the current user, the general gesture recognition model can be trained using these EMG signals to obtain an initial gesture recognition model. Because the calibrated EMG signals of different calibrated users and the historical EMG signals of the current user are used in this training process, the initial gesture recognition model obtained through training can be made to fit the current user without deviating too much from that user.

[0107] Furthermore, considering that if the current user is using the device for the first time, the device does not store the current user's historical electromyography (EMG) signals, and in order to achieve training, in this embodiment, before the step of training the general gesture recognition model using the calibrated EMG signals of different calibrated users and the current user's historical EMG signals, the following is also included:

[0108] Step S201: Determine whether the current user is a first-time user;

[0109] Step S202: If yes, then use the current user's calibrated electromyography signal as the current user's historical electromyography signal, and perform the step of training the general gesture recognition model using the calibrated electromyography signals of different calibrated users and the current user's historical electromyography signal.

[0110] Understandably, the aforementioned calibration electromyography (EMG) signal can be an EMG signal collected when the user performs a predetermined calibration action. For example, when calibration is required, (the aforementioned device) can guide the current user to perform relevant calibration actions and use the collected EMG signals of these calibration actions as the aforementioned calibration EMG signal.

[0111] It should be noted that the above-mentioned device can determine whether the current user is using the device for the first time by means of the user's logged-in account, etc. Of course, other determination methods can also be used, and this embodiment does not limit them.

[0112] For example, the device can allow users to log in to their accounts before each use. If the current user logs in to an account that is already logged in, it can be determined that the current user is not using the device for the first time. The device has stored the user's historical electromyography (EMG) signals, and the general gesture recognition model can be trained directly using the historical EMG signals and the calibrated EMG signals to obtain the initial gesture recognition model.

[0113] However, when the device detects that the currently logged-in account is a new account, it can determine that the current user is a first-time user. Since the device does not store the user's historical electromyography (EMG) signals, it can guide the current user to perform relevant calibration actions and use the collected EMG signals from these calibration actions as the calibration EMG signals. Figure 4 Users can perform corresponding calibration actions according to the action acquisition guidance system in the device. The device generates user calibration data based on the acquired electromyography signals. The data can also undergo the aforementioned preprocessing operations. The specific process will not be described in detail in this embodiment.

[0114] After obtaining the aforementioned calibrated electromyography (EMG) signal, the device can use the calibrated EMG signal as the current user's historical EMG signal, and train the general gesture recognition model together with the calibrated EMG signals of different calibrated users to obtain the initial gesture recognition model.

[0115] Step S30: Train the initial gesture recognition model based on the historical model parameters and the current user's calibrated electromyography signal to obtain the target gesture recognition model.

[0116] After obtaining the initial gesture recognition model, the device can acquire the calibration electromyography (EMG) signal corresponding to the relevant calibration movements pre-guided by the current user. In other words, to obtain the current user's movement habits, the device can pre-calibrate the current user, obtaining labeled movement data as the aforementioned calibration EMG signal (i.e.,...). Figure 3 Step c: User action habit calibration.

[0117] After obtaining the calibrated electromyography (EMG) signal, the aforementioned device can retrain the initial gesture recognition model using the obtained historical model parameters. Since both training sessions employ a small number of samples, this can be termed "few-shot learning" (i.e., training with a limited number of samples). Figure 3 Step e: Few-shot learning strategy to build action model, that is Figure 4 (Model update calibration). Since the training process used historical model parameters of the general gesture recognition model and the current user's calibration electromyography signal, and the general gesture recognition model is obtained through iterative training based on the historical user's electromyography signal, the historical model parameters will contain some features of the historical user. This also ensures that the trained target gesture recognition model will not deviate too much from the current user while being suitable for the current user.

[0118] Step S40: Perform gesture recognition on the collected electromyography signals of the current user using the target gesture recognition model.

[0119] After obtaining the target gesture recognition model, when the user makes a movement, the device can capture the movement and perform active segment detection to obtain the user's electromyography (EMG) signal. Then, the target gesture recognition model is used to perform gesture recognition on the EMG signal to obtain the recognition result (i.e.,...). Figure 3 Step f: Use of the gesture recognition model.

[0120] In practical implementation, the aforementioned device can pre-construct a corresponding dataset based on the calibrated electromyography signals of N subjects and store it in a cloud server (i.e., Figure 3 Step a: Construct a historical calibration dataset; during use, it can store the historical electromyography signals of the current user's recent gestures in real time (i.e., Figure 3 Step b: Real-time collection and storage of recent user action data; when the current user is performing calibration, the above device can guide the current user to perform corresponding preset calibration actions (i.e. Figure 3 Step c: User movement habit calibration; If it is the first time using the device, the initial gesture recognition model is obtained by training the calibrated electromyography (EMG) signal and the standard EMG signal in the cloud server; If it is not the first time using the device, the initial gesture recognition model is obtained by training the device with historical EMG signals and standard EMG signals; Then, the device combines historical model parameters and calibrated EMG signals to train the initial gesture recognition model to obtain the target gesture recognition model (i.e., ... Figure 3 In step e: constructing an action model using a few-shot learning strategy, the electromyography signal can then be used to recognize the gesture by the target gesture recognition model, thus obtaining the recognition result (i.e., Figure 3 Step f: Use of the gesture recognition model.

[0121] This embodiment can first train the model using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user. Then, it can train the model using historical model parameters of a general gesture recognition model and calibrated EMG signals from the current user. This allows the target gesture recognition model to be more suitable for the current user while reducing the occurrence of catastrophic forgetting, thereby reducing the impact on training the next user and improving the user experience.

[0122] Reference Figure 6 , Figure 6 This is a flowchart illustrating the second embodiment of the gesture recognition method of this application. Based on the first embodiment described above, the second embodiment of the gesture recognition method of this application is proposed.

[0123] like Figure 6 As shown, in order to train a general gesture recognition model by combining calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user, in this embodiment, the steps of training the general gesture recognition model using calibrated EMG signals from different calibrated users and historical EMG signals from the current user to obtain an initial gesture recognition model include:

[0124] Step S21: Determine the domain similarity between the calibrated electromyography (EMG) signals of different calibrated users and the historical EMG signals of the current user.

[0125] It should be noted that, considering that directly using calibrated EMG signals from different calibrated users for training might not be very suitable for the current user, the aforementioned device can first determine the domain similarity between the calibrated EMG signals from different calibrators and the current user's historical EMG signals (when it is the first use, the calibrated EMG signal is used as the historical EMG signal). Domain similarity can be determined by selecting signals from the calibrated EMG signals that are similar to the current user's habits.

[0126] Specifically, this embodiment can use the Maximum Mean Discrepancy (MMD) method to determine the distribution distance between the electromyography (EMG) signal and the current user's historical EMG signal by defining different kernel functions, thus measuring the distribution difference between the two. This can be achieved through the following formula 1:

[0127]

[0128] Wherein, S is the source domain (i.e., the calibrated electromyography signal), T is the target domain (i.e., the historical electromyography signal), and D is the target domain. mmd (S,T) represents the domain similarity between the source and target domains, n s n is the number of source domains. tLet k be the number of target domains, and k(,) be the kernel function, which can be a Gaussian kernel function, etc. Let i be the i-th source domain. Let j be the j-th source domain. For the i-th target domain, Let D be the j-th target domain in the target domain. And when the distributions of the source domain and the target domain are completely identical, D... mmd (S,T)=0.

[0129] Step S22: Select a reference electromyography signal from the calibrated electromyography signals based on domain similarity.

[0130] Understandably, the aforementioned reference electromyography (EMG) signals can be signals from the calibrated EMG signals that are similar to the current user's habits. After obtaining the domain similarity between each calibrated EMG signal and the historical EMG signals using Formula 1, the domain similarities can be sorted from smallest to largest or from largest to smallest. K calibrated EMG signals corresponding to these domain similarities are then selected in ascending order as the aforementioned reference EMG signals. These K reference EMG signals are the EMG signals from the calibrated EMG signals that are similar to the current user's movement habits (i.e.,...). Figure 3 Step d: Filter the reference dataset, that is... Figure 4 The historical calibration dataset and user calibration data are compared for similarity to obtain reference data. These K reference electromyography (EMG) signals are the EMG signals most similar to the current user's habits. The specific value of K can be one or more, and this embodiment does not limit this.

[0131] Step S23: Train the general gesture recognition model by referring to electromyography signals and historical electromyography signals to obtain the initial gesture recognition model.

[0132] After obtaining K reference electromyography (EMG) signals, they can be trained with the current user's historical EMG signals to obtain K initial gesture recognition models (i.e., Figure 3 Step e: Constructing K sets of action models using a few-shot learning strategy. To better illustrate the training process, the following section combines... Figure 7 To explain, Figure 7 This is a schematic diagram of model training in the gesture recognition method of this application.

[0133] During training, the steps described above for training the general gesture recognition model by referencing electromyography (EMG) signals and historical EMG signals to obtain the initial gesture recognition model include:

[0134] Step S231: Obtain the source domain intermediate feature vector corresponding to the reference electromyography signal and the target domain intermediate feature vector corresponding to the historical electromyography signal through the intermediate layer of the general gesture recognition model.

[0135] It should be understood that, as Figure 7 As shown, the above-mentioned general gesture recognition model can include a feature extractor (i.e., Figure 7 (Feature extractor). In obtaining the reference electromyography signal (i.e.,...) Figure 7 (Chinese reference data) and historical electromyography signals (i.e. Figure 7 After initial use (user calibration data, subsequent use (user historical data)), the reference EMG signal and historical EMG signals can be input into the general gesture recognition model. The general gesture recognition model can extract features from the reference EMG signal and historical EMG signal through the feature extractor of the shared parameters, and obtain the source domain intermediate feature vector corresponding to the reference EMG signal (i.e., ... Figure 7 China Z s1 ) and the target domain intermediate feature vector corresponding to the historical electromyography signal (i.e. Figure 7 China Z t1 Based on the above embodiments, when it is the first time using the device, the current user's calibrated electromyography (EMG) signal is used as the historical EMG signal; when it is not the first time using the device, the historical EMG signal is used directly.

[0136] It is important to emphasize that the aforementioned feature extractor can be constructed using a convolutional network architecture, such as a Residual Network (ResNet) or a Visual Geometry Group Network (VGGNet), which can be used to analyze the statistical characteristics of electromyography signals in the time and frequency domains. However, to ensure calibration efficiency, this embodiment can employ a shallower convolutional neural network (CNN) structure.

[0137] In obtaining Z s1 and Z t1 Then, the intermediate layer of the general gesture recognition model (i.e. Figure 7 The FC1 is fully connected to obtain the source domain intermediate feature vector corresponding to the reference electromyography signal (i.e., ...). Figure 7 China Z s2 ) and the target domain intermediate feature vector corresponding to the historical electromyography signal (i.e. Figure 7 China Z t2 The number of intermediate layers is not limited in this embodiment. Figure 7 An intermediate layer is used for explanation.

[0138] Step S232: Based on the first output layer of the general gesture recognition model, obtain the source domain gesture category probability corresponding to the reference electromyography signal and the main target domain gesture category probability corresponding to the historical electromyography signal.

[0139] In obtaining Zs1 Z s2 Z t1 and Z t2 Then, the first output layer of the general gesture recognition model (i.e. Figure 7 The source domain gesture category probability (i.e., FC2) corresponding to the reference electromyography signal is calculated separately. Figure 7 middle ) and the probability of the main target domain gesture category corresponding to the historical electromyography signal (i.e. Figure 7 middle ). Among them, the source domain gesture category probability To reference the probabilities of various gesture categories corresponding to electromyography signals, the probability of gesture categories in the main target domain is calculated. This represents the probability of various gesture categories corresponding to historical electromyography signals.

[0140] Step S233: Construct a training loss function based on the source domain intermediate feature vector, the target domain intermediate feature vector, the source domain gesture category probability, and the main target domain gesture category probability.

[0141] It should be noted that the loss function mentioned above for the first round of training can be the loss function used to train and obtain the initial gesture recognition model. Figure 7 In this embodiment, the reference electromyography (EMG) signal is used as the source domain and the historical EMG signal is used as the target domain. The knowledge learned in the source domain is transferred to the target domain, thus completing the training process of obtaining the initial gesture recognition model (i.e., ...). Figure 7 The first round of training can be called "domain adaptive" training.

[0142] Furthermore, to improve the model's generalization ability and reduce class confusion, this embodiment can combine Joint Maximum Mean Discrepancy (JMMD) and Minimum Class Confusion (MCC) to construct the aforementioned first-round training loss function. The specific process is as follows: Step S233 includes:

[0143] Step S2331: Construct a joint maximum mean difference loss function based on the intermediate feature vectors of the source domain and the intermediate feature vectors of the target domain.

[0144] Continue as Figure 7 As shown, after obtaining the intermediate feature vectors of the source domain and the target domain, a weighted summation operation can be performed on each vector in the intermediate feature vectors of the source domain and the target domain. Specifically, for Z... s1 and Z t1 The first preset weight can be set (i.e. Figure 7 In Perform a weighted operation on Z. s2 and Z t2 A second preset weight can be set (i.e.) Figure 7 In Perform weighted operations.

[0145] Then, a joint maximum mean difference loss function is constructed based on the weighted source domain intermediate feature vector and the weighted target domain intermediate feature vector. The construction process can be implemented according to the following formula 2:

[0146]

[0147] Where S is the source domain, T is the target domain, and L is the destination domain. JMMD (S,T) is the aforementioned joint maximum mean difference loss function (i.e., Figure 7 (jmmmd), n s n is the number of source domains. t Let k be the number of target domains, and k(,) be the kernel function. Multiple Gaussian kernel functions can be used (i.e., Figure 7 By employing multiple sets of kernel functions (Gaussian Kernel 1, Gaussian Kernel 2, ..., Gaussian Kernel n), the problem of biased hyperparameters of a single kernel function across different users can be avoided, thus effectively improving the model's generalization ability. Let be the i-th intermediate feature vector of the source domain. Let j be the j-th intermediate feature vector of the source domain. Let be the i-th intermediate feature vector of the target domain. Let L be the j-th intermediate feature vector in the target domain, L be the number of intermediate layers, i.e., the number of intermediate layers can be increased by stacking, and l be the current intermediate layer.

[0148] Step S2332: Construct the first cross-entropy loss function using the source domain gesture category probability, and construct the minimum class confusion loss function using the main target domain gesture category probability.

[0149] Continue as Figure 7 As shown, the above-mentioned device can also recognize gestures based on the source domain probability. Construct the first cross-entropy loss function (i.e. Figure 7 (CE). Specifically, the above-mentioned device can be based on the source domain gesture recognition probability. And the correct hand gesture type corresponding to the calibrated electromyography signal (i.e. Figure 7 Chinese reference data label y s Together, they are calculated to represent the gesture recognition knowledge learned by the model in the source domain, which can be specifically achieved according to the following formula 3:

[0150]

[0151] Among them, L CE (S) is the first cross-entropy loss function, S is the source domain, C is the number of gesture recognition categories, and y si P is the source domain data label corresponding to the i-th gesture category. i Let be the source domain gesture category probability corresponding to the i-th gesture category.

[0152] It is important to emphasize that, since historical electromyography signals are unlabeled, cross-entropy cannot be directly calculated. Therefore, in this embodiment, the probability of the target domain gesture category can be calculated first. Correction is achieved using a minimal class confusion strategy to reduce confusion between similar classes. The specific process involves the steps outlined above for constructing a minimal class confusion loss function based on the gesture category probabilities of the main target domain, including:

[0153] The probability of the gesture category in the main target domain is adjusted, and the adjusted probability of the gesture category in the main target domain is reweighted. The class association reduction operation is performed on the reweighted probability of the gesture category in the main target domain, and a minimum class confusion loss function is constructed based on the operation results.

[0154] It should be understood that the probability of gesture categories in the main target domain can be determined first. Probability adjustments are made to mitigate the negative impact of overconfident predictions in the target domain. The specific process can be achieved according to the following formula 4:

[0155]

[0156] in, To determine the probability that the adjusted i-th sample (i.e., the historical electromyography signal in the target domain) is predicted as the main target domain gesture category j, P i,j The probability that the i-th sample before adjustment is predicted as the current gesture category j in the main target domain is given by t, which is the readjustment coefficient and can be set according to the actual situation. C is the number of gesture recognition categories, and P is the key factor. i,j′ The probability of predicting the gesture category of the main target domain as the j′-th gesture category for the i-th sample.

[0157] After adjusting the probabilities using Formula 4 above, the adjusted probability of the main target domain gesture category can be reweighted to strengthen the weight of samples with relatively certain predictions in class association. This can be achieved through Formula 5 below:

[0158]

[0159] Among them, W i Let B be the weight of the importance of the i-th sample after reweighting, B be the number of samples in one training iteration, and H() be the entropy formula. To adjust the probability of the gesture category in the main target domain of the current i-th sample, This represents the adjusted probability of the gesture category in the main target domain of the i′ sample.

[0160] After reweighting using Formula 5 above, the class association between all gesture categories can be minimized through a class confusion minimization strategy, thereby reducing the occurrence of class confusion and obtaining the minimum class confusion loss function (i.e. Figure 7 The MCC (Medium-terminal Criterion) can be specifically achieved through the following formula 6:

[0161]

[0162] Among them, L MCC (T) is the minimum class confusion loss function, where T is the target domain and C is the number of gesture recognition categories. Let j be the probability of the main target domain gesture category corresponding to the j-th gesture category. Let be the probability of the main target domain gesture category corresponding to the j′-th gesture category.

[0163] Step S2333: Construct a training loss function based on the joint maximum mean difference loss function, the first cross-entropy loss function, and the minimum class confusion loss function.

[0164] After obtaining the above joint maximum mean difference loss function L JMMD (S,T), First cross-entropy loss function L CE (S) and the minimum class confusion loss function L MCC After (T), a training loss function can be constructed for gradient update. Specifically, it can be obtained using the following formula 7:

[0165] Loss1=L CE (S)+αL JMMD (S,T)+βL MCC (T) Formula 7;

[0166] Where Loss1 is the loss function for the first round of training, α is the first preset weight parameter, and β is the second preset weight parameter, both of which can be set according to the actual situation.

[0167] Step S234: Train the general gesture recognition model using a training loss function to obtain the initial gesture recognition model.

[0168] After obtaining the loss function Loss1 for one round of training, the general gesture recognition model can be trained using this loss function (i.e., ...). Figure 7 (One round of training) is conducted to achieve one round of domain adaptive training and obtain the initial gesture recognition model.

[0169] Furthermore, considering that K sets of reference electromyography (EMG) signals can be selected when choosing reference signals, and when K is multiple, in order to further improve the accuracy of gesture recognition results, in this embodiment, after determining multiple sets of reference EMG signals, the general gesture recognition model can be trained by comparing each set of reference EMG signals with historical EMG signals to obtain K initial gesture recognition models. Then, these K initial gesture recognition models are trained using historical model parameters and calibrated EMG signals to obtain K target gesture recognition models. Finally, when a new gesture is encountered, gesture recognition can be performed using these K target gesture recognition models, and the K recognition results are integrated using a soft voting strategy to obtain the final gesture recognition result. The specific process is as follows: the number of reference EMG signals is at least two, and the number of target gesture recognition models is the same as the number of reference EMG signals.

[0170] Step S40 above includes:

[0171] Step S41: Perform gesture recognition on the collected electromyography signals of the current user using each target gesture recognition model to obtain the initial recognition results corresponding to each target gesture recognition model;

[0172] Step S42: Integrate the initial recognition results to obtain the target gesture recognition result.

[0173] For ease of explanation, refer to Figure 8 , Figure 8 This diagram illustrates the integration of model construction and output results in the gesture recognition method of this application. Figure 8 As shown, after obtaining K reference electromyography signals (i.e. Figure 8 After obtaining reference data 1, reference data 2, ..., reference data K, the general gesture recognition model can be trained according to the above process (i.e., ...). Figure 8 Using user calibration data, user history data, and gesture model rapid calibration training, K target gesture recognition models are obtained (i.e., ... Figure 8 Gesture recognition model 1, gesture recognition model 2, ..., gesture recognition model K);

[0174] When the device detects that the user has made a new action (i.e. Figure 8 (For new user actions), the electromyography signals of the current user can be used to perform gesture recognition through the K target gesture recognition models mentioned above, thereby obtaining K gesture recognition results, which serve as the initial recognition results corresponding to the K target gesture recognition models mentioned above; the initial recognition results are the probability results of each gesture category predicted by each target gesture recognition model for this action.

[0175] Next, the device can average the probabilities of the same gesture category predicted by each target gesture recognition model to obtain the average probability of each gesture category. Then, it selects the gesture category with the highest average probability as the final gesture recognition result, which is the target gesture recognition result (i.e., Figure 3 Step f: The use of the gesture recognition model, that is... Figure 8 The recognition results based on the soft voting strategy are integrated with the gesture recognition results.

[0176] In practical implementation, the aforementioned device can pre-construct a corresponding dataset based on the calibrated electromyography signals of N subjects and store it in a cloud server (i.e., Figure 3 Step a: Constructing a historical calibration dataset); During use, the above device can store in real time the historical electromyography signals of the user's recent gestures (i.e., Figure 3 Step b: Real-time collection and storage of recent user action data; when the current user is performing calibration, the above device can guide the current user to perform corresponding preset calibration actions (i.e. Figure 3 Step c: User movement habit calibration; If it is the first time using the system, K calibrated electromyography (EMG) signals are selected from the calibrated EMG signals as reference EMG signals based on the calibrated EMG signals; If it is not the first time using the system, K calibrated EMG signals are selected from the calibrated EMG signals as reference EMG signals based on historical EMG signals (i.e., ... Figure 3 Step d: Selecting a reference dataset; then, first train the model using calibrated electromyography (EMG) signals and K reference EMG signals to obtain K initial gesture recognition models; then, train each of the K initial gesture recognition models using historical model parameters and calibrated EMG signals to obtain K target gesture recognition models (i.e., ...). Figure 3 In step e: a few-shot learning strategy is used to construct K sets of action models. Finally, these K target gesture recognition models can be used to recognize gestures from the electromyography signal. A soft voting strategy is then used to integrate these K recognition results as the final gesture recognition result (i.e.,...). Figure 3 Step f: Use of the gesture recognition model.

[0177] This embodiment employs joint maximum mean difference alignment to align the joint distribution of specific feature extraction layers between the source and target domains, and uses a minimization of class confusion strategy to correct the prediction results of the target domain data, reducing confusion between similar categories. This further enhances the user experience by minimizing catastrophic forgetting while better catering to the current user. Furthermore, this embodiment can train multiple target gesture recognition models and integrate the recognition results using a soft voting strategy, further improving the accuracy of gesture recognition.

[0178] Reference Figure 9 , Figure 9This is a flowchart illustrating the third embodiment of the gesture recognition method of this application. Based on the above embodiments, the third embodiment of the gesture recognition method of this application is proposed.

[0179] like Figure 9 As shown, in order to train a general gesture recognition model by combining historical model parameters and the current user's calibrated electromyography signal, in this embodiment, step S30 includes:

[0180] Step S31: Obtain the current model parameters of the initial gesture recognition model, and construct regularization terms based on the current model parameters, preset hyperparameters, and historical model parameters.

[0181] It should be noted that, since the above embodiment involves two rounds of training, the model parameters of the initial gesture recognition model after one round of training can be denoted as the current model parameters, and the model parameters of the target gesture recognition model after the previous second round of training can be denoted as the historical model parameters. Therefore, in this embodiment, when performing the second round of training, the device can first obtain the model parameters from the previous second round of training as the historical model parameters, and use the model parameters from the current first round of training as the current model parameters. For ease of explanation later, this embodiment denotes the historical model parameters as θ and the current model parameters as θi. 0 .

[0182] Understandably, the aforementioned regularization term can be a parameter used during training to prevent the model from overfitting. To control the strength of the regularization term, this embodiment can preset a hyperparameter (i.e., the aforementioned preset hyperparameter). The specific hyperparameter can be set according to the actual situation, and this embodiment does not impose any restrictions on it. For ease of subsequent explanation, this embodiment denotes the aforementioned preset hyperparameter as λ. The constructed regularization term can then be denoted as...

[0183] Step S32: Construct a second cross-entropy loss function through a regularization term, and train the initial gesture recognition model based on the second cross-entropy loss function and the current user's calibrated electromyography signal to obtain the target gesture recognition model.

[0184] It should be understood that the aforementioned second cross-entropy loss function can be a loss function used for two rounds of training. To obtain the second cross-entropy loss function, in this embodiment, the step of constructing the second cross-entropy loss function through a regularization term includes:

[0185] Step S321: Obtain the calibration gesture recognition probability corresponding to the calibration electromyography signal of the current user through the second output layer of the general gesture recognition model.

[0186] Continue to refer to Figure 7In this embodiment, the current user's calibrated electromyography (EMG) signal can also be input into the general gesture recognition model. The general gesture recognition model can extract features from the standard EMG signal using the feature extractor with shared parameters to obtain the calibration intermediate feature vector (i.e., ...) corresponding to the calibrated EMG signal. Figure 7 Z′ t1 ).

[0187] In obtaining Z′ t1 Then, the intermediate layer of the general gesture recognition model (i.e. Figure 7 The FC1 is fully connected to obtain the calibration intermediate feature vector (i.e., the calibration intermediate feature vector corresponding to the calibration electromyography signal) from the calibration electromyography signal. Figure 7 Z′ t2 Since these are all consistent with the aforementioned feature extraction and fully connected processes, this embodiment will not elaborate further.

[0188] In obtaining Z′ t1 and Z′ t2 Then, the second output layer of the general gesture recognition model (i.e. Figure 7 The probability of each gesture category corresponding to the calibrated electromyography signal is obtained by calculating the FC3 (FC3) and used as the above-mentioned calibrated gesture recognition probability (i.e., Figure 7 middle The parameters of the first output layer and the second output layer are different.

[0189] Step S322: Determine the calibration data label corresponding to the calibration electromyography signal, and construct the second cross-entropy loss function based on the calibration gesture recognition probability, calibration data label, and regularization term.

[0190] It should be noted that since the user performs actions according to the operation guide during calibration, the aforementioned device can directly obtain the gesture category corresponding to the calibrated electromyography (EMG) as the calibration data label (i.e., ...). Figure 7 (The calibration data label). Then, the calibration data label, the calibration gesture recognition probability, and the above regularization term (i.e., ...) are used. Figure 7 The second cross-entropy loss function (i.e., regularization adjustment) is constructed by adjusting the regularization adjustment. Figure 7 (CE-2).

[0191] Understandably, the second cross-entropy loss function described above is given by the following formula:

[0192]

[0193] Among them, L CE-2 (T′) is the second cross-entropy loss function, C is the number of gesture categories, and y si P is the calibration data label corresponding to the i-th gesture category. iLet λ be the calibrated gesture recognition probability corresponding to the i-th gesture category, λ be a preset hyperparameter, and θ be the historical model parameters. 0 These are the current model parameters for the initial gesture recognition model.

[0194] It should be emphasized that since the amount of data from calibrating electromyography signals is usually small, overtraining may occur during training. This can be controlled by the above-mentioned regularization terms to reduce overtraining.

[0195] Furthermore, after obtaining the second cross-entropy loss function, the device can be trained based on the second cross-entropy loss function. However, to further improve the training effect, in this embodiment, the step of training the initial gesture recognition model based on the second cross-entropy loss function and the current user's calibrated electromyography signal to obtain the target gesture recognition model includes:

[0196] Step S323: Perform a confidence test on the gesture category probability of the main target domain to obtain the target domain pseudo-label, and obtain the auxiliary target domain gesture category probability corresponding to the historical electromyography signal based on the second output layer of the general gesture recognition model.

[0197] Continue as Figure 7 As shown, the aforementioned device can also determine the probability of gesture categories in the main target domain. Perform a confidence test (i.e.) Figure 7 (Medium confidence level detection), which retains samples with high confidence levels and generates pseudo-labels for the target domain (i.e., Figure 7 Chinese counterfeit labels The specific confidence test method can be to calculate the entropy value, that is, to calculate the entropy value of the probability of the gesture category in the main target domain, and select the samples with lower entropy values ​​(i.e., lower uncertainty) as the pseudo-labels of the target domain. This embodiment does not limit the specific number of samples selected. This can be achieved through the following formula 9:

[0198]

[0199] H(e) is the entropy value mentioned above, p i Let be the probability of the main target domain gesture category corresponding to the i-th gesture category, and n be the number of main target domain gesture category probabilities.

[0200] Obtaining pseudo-tags for the target domain Then, the intermediate feature vector Z of the target domain can be simultaneously... t2 After the second output layer (i.e.) Figure 7 (FC3). After passing through the second output layer, the above-mentioned auxiliary target domain gesture category probability (i.e. Figure 7 middle The probability of the gesture category in the auxiliary target domain can be another probability of the various gesture categories corresponding to the historical electromyography signals.

[0201] Step S324: Construct a generalized cross-entropy loss function based on the target domain pseudo-label and the auxiliary target domain gesture category probability, and construct a second-round training loss function based on the generalized cross-entropy loss function and the second cross-entropy loss function.

[0202] In determining the pseudo-label of the target domain and the probability of gesture category in the auxiliary target domain Then, the above generalized cross-entropy loss function can be constructed (i.e. Figure 7 In GCE (Generalized Cross-Entropy), the generalized cross-entropy loss function can mitigate the impact of noise in the labels compared to the traditional cross-entropy loss function. The generalized cross-entropy loss function is as follows:

[0203]

[0204] Among them, L GCE (T) is the generalized cross-entropy loss function, C is the number of gesture categories, and y ti P is the pseudo-label of the target domain corresponding to the i-th gesture category. i Let q be the calibrated gesture recognition probability corresponding to the i-th gesture category, and q be a preset adjustment parameter. When q approaches 0, it becomes the traditional cross-entropy loss function.

[0205] After obtaining the generalized cross-entropy loss function, it can be combined with the second cross-entropy loss function to construct the aforementioned second-round training loss function, which is used to perform a second round of training on the initial gesture recognition model. If the second-round training loss function is denoted as Loss2, then the second-round training loss function is:

[0206] Loss2=L GCE (T)+γL CE-2 (T′) Formula 11;

[0207] Where Loss2 is the loss function for the second round of training, L GCE (T) is the generalized cross-entropy loss function, L CE-2 (T′) is the second cross-entropy loss function, and γ is the preset third weight, which can be set according to the actual situation.

[0208] Step S325: Train the initial gesture recognition model using a two-round training loss function to obtain the target gesture recognition model.

[0209] After obtaining the loss function for the second round of training, the initial gesture recognition model can be trained (i.e., ...). Figure 7Through two rounds of training, a target gesture recognition model can be obtained. Since this embodiment uses the current user's calibrated electromyography signal for supervised training to enhance the model's ability to recognize user signals, the training process for obtaining the target gesture recognition model described above can be called "supervised learning" training.

[0210] This embodiment employs a two-stage few-shot learning strategy of "domain adaptation + supervised learning," which can reduce catastrophic forgetting, improve generalization ability, and increase the utilization rate of historical information. For tasks involving gesture calibration after a new user's first use or after prolonged use, it can better integrate the user's historical unlabeled action data to accurately match user habits.

[0211] Furthermore, referring to Figure 10 , Figure 10 This chart compares the accuracy of gesture recognition achieved using this method with that of traditional methods. Figure 10 As shown, if we denote the gesture recognition method of this scheme as "proposed" and the traditional gesture recognition method as "Baseline," the traditional method involves first calibrating the gesture recognition model using the new user's data, and then fine-tuning the model using a small amount of labeled data from the new user. According to... Figure 10 The data shows that in six test groups (i.e. Figure 10 In tests 1, 2, 3, 4, 5, and 6, the accuracy rates for identifying "proposed" were 95.33%, 96.15%, 93.38%, 95.86%, 97.92%, and 98.7%, respectively, while the accuracy rates for identifying "Baseline" were 88%, 85.71%, 84.11%, 89.35%, 87.5%, and 83.12%, respectively. Clearly, "proposed" consistently outperforms "Baseline" by more than 10%.

[0212] Furthermore, this application also proposes a storage medium storing a gesture recognition program, which, when executed by a processor, implements the gesture recognition method described above.

[0213] In addition, refer to Figure 11 , Figure 11 This is a structural block diagram of the first embodiment of the gesture recognition device of this application; as shown... Figure 11 As shown in the figure, this application also proposes a gesture recognition device, which includes:

[0214] The parameter acquisition module 111 is used to acquire the historical model parameters of the general gesture recognition model, which is obtained through iterative training based on the electromyography signals of historical users.

[0215] The first-round training module 112 is used to train the general gesture recognition model using the calibrated electromyography signals of different calibrated users and the historical electromyography signals of the current user, so as to obtain the initial gesture recognition model.

[0216] The second-round training module 113 is used to train the initial gesture recognition model based on historical model parameters and the current user's calibrated electromyography signal to obtain the target gesture recognition model.

[0217] The gesture recognition module 114 is used to perform gesture recognition on the electromyography signal of the current user collected by the target gesture recognition model.

[0218] This embodiment can first train the model using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user. Then, it can train the model using historical model parameters of a general gesture recognition model and calibrated EMG signals from the current user. This allows the target gesture recognition model to be more suitable for the current user while reducing the occurrence of catastrophic forgetting, thereby reducing the impact on training the next user and improving the user experience.

[0219] As one implementation, the first-round training module 112 is also used to determine whether the current user is a first-time user; if so, the current user's calibrated electromyography signal is used as the current user's historical electromyography signal, and the operation of training the general gesture recognition model is performed using the calibrated electromyography signals of different calibrated users and the current user's historical electromyography signal.

[0220] Based on the first embodiment of the gesture recognition device described in this application, a second embodiment of the gesture recognition device of this application is proposed.

[0221] In this embodiment, the first-round training module 112 is also used to determine the domain similarity between the calibrated electromyography (EMG) signals of different calibrated users and the historical EMG signals of the current user; select a reference EMG signal from the calibrated EMG signals based on the domain similarity; and train the general gesture recognition model using the reference EMG signal and the historical EMG signal to obtain an initial gesture recognition model.

[0222] In one implementation, the first-round training module 112 is further used to obtain the source domain intermediate feature vector corresponding to the reference electromyography signal and the target domain intermediate feature vector corresponding to the historical electromyography signal through the intermediate layer of the general gesture recognition model; to obtain the source domain gesture category probability corresponding to the reference electromyography signal and the main target domain gesture category probability corresponding to the historical electromyography signal based on the first output layer of the general gesture recognition model; to construct a first-round training loss function based on the source domain intermediate feature vector, the target domain intermediate feature vector, the source domain gesture category probability and the main target domain gesture category probability; and to train the general gesture recognition model through the first-round training loss function to obtain the initial gesture recognition model.

[0223] As one implementation, the first-round training module 112 is also used to construct a joint maximum mean difference loss function based on the intermediate feature vector of the source domain and the intermediate feature vector of the target domain; construct a first cross-entropy loss function based on the gesture category probability of the source domain, and construct a minimum class confusion loss function based on the gesture category probability of the main target domain; and construct a first-round training loss function based on the joint maximum mean difference loss function, the first cross-entropy loss function and the minimum class confusion loss function.

[0224] As one implementation, the first-round training module 112 is also used to adjust the probability of the gesture category in the main target domain, and reweight the adjusted probability of the gesture category in the main target domain; perform a class association reduction operation on the reweighted probability of the gesture category in the main target domain, and construct a minimum class confusion loss function based on the operation result.

[0225] As one implementation method, the loss function for one round of training is: Loss1 = L CE (S)+αL JMMD (S,T)+βL MCC (T); where Loss1 is the loss function for one round of training, L CE (S) is the first cross-entropy loss function, L JMMD (S,T) is the joint maximum mean difference loss function, L MCC (T) is the minimum class confusion loss function, α is the first preset weight parameter, and β is the second preset weight parameter.

[0226] In one implementation, the number of reference electromyography (EMG) signals is at least two, and the number of target gesture recognition models is the same as the number of reference EMG signals. The gesture recognition module 114 is also used to perform gesture recognition on the collected EMG signals of the current user through each target gesture recognition model to obtain the initial recognition result corresponding to each target gesture recognition model. The initial recognition results are integrated to obtain the target gesture recognition result.

[0227] Based on the above embodiments of the gesture recognition device of this application, a third embodiment of the gesture recognition device of this application is proposed.

[0228] In this embodiment, the second-round training module 113 is also used to obtain the current model parameters of the initial gesture recognition model, and construct a regularization term based on the current model parameters, preset hyperparameters and historical model parameters; construct a second cross-entropy loss function through the regularization term, and train the initial gesture recognition model according to the second cross-entropy loss function and the current user's calibrated electromyography signal to obtain the target gesture recognition model.

[0229] As one implementation, the second-round training module 113 is also used to obtain the calibration gesture recognition probability corresponding to the calibration electromyography signal of the current user through the second output layer of the general gesture recognition model; determine the calibration data label corresponding to the calibration electromyography signal; and construct a second cross-entropy loss function based on the calibration gesture recognition probability, the calibration data label and the regularization term.

[0230] As one implementation method, the second cross-entropy loss function is: Among them, L CE-2 (T′) is the second cross-entropy loss function, C is the number of gesture categories, and y si P is the calibration data label corresponding to the i-th gesture category. i Let λ be the calibrated gesture recognition probability corresponding to the i-th gesture category, λ be a preset hyperparameter, and θ be the historical model parameters. 0 These are the current model parameters for the initial gesture recognition model.

[0231] As one implementation, the second-round training module 113 is also used to perform a confidence test on the gesture category probability of the main target domain, obtain the target domain pseudo-label, and obtain the auxiliary target domain gesture category probability corresponding to the historical electromyography signal based on the second output layer of the general gesture recognition model; construct a generalized cross-entropy loss function based on the target domain pseudo-label and the auxiliary target domain gesture category probability, and construct a second-round training loss function based on the generalized cross-entropy loss function and the second cross-entropy loss function; train the initial gesture recognition model through the second-round training loss function to obtain the target gesture recognition model.

[0232] As one implementation method, the generalized cross-entropy loss function is: Among them, L GCE (T) is the generalized cross-entropy loss function, C is the number of gesture categories, and y ti P is the pseudo-label of the target domain corresponding to the i-th gesture category. i Let q be the calibration gesture recognition probability corresponding to the i-th gesture category, and q be a preset adjustment parameter.

[0233] As one implementation method, the loss function for the second round of training is: Loss2 = L GCE (T)+γL CE-2 (T′); where Loss2 is the loss function for the second round of training, L GCE (T) is the generalized cross-entropy loss function, L CE-2 (T′) is the second cross-entropy loss function, and γ is the preset third weight.

[0234] Other embodiments or specific implementations of the gesture recognition device described in this application can be found in the above-described method embodiments, and will not be repeated here.

[0235] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0236] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0237] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0238] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A gesture recognition method, characterized in that, The method includes: Obtain historical model parameters of a general gesture recognition model, which is obtained through iterative training based on electromyography signals of historical users; The general gesture recognition model is trained using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user to obtain an initial gesture recognition model. The initial gesture recognition model is trained based on the historical model parameters and the current user's calibrated electromyography signal to obtain the target gesture recognition model; The target gesture recognition model is used to perform gesture recognition on the collected electromyography signals of the current user; The step of training the initial gesture recognition model based on the historical model parameters and the calibrated electromyography signal of the current user to obtain the target gesture recognition model includes: Obtain the current model parameters of the initial gesture recognition model, and construct a regularization term based on the current model parameters, preset hyperparameters, and the historical model parameters; A second cross-entropy loss function is constructed using the regularization term, and the initial gesture recognition model is trained based on the second cross-entropy loss function and the calibrated electromyography signal of the current user to obtain the target gesture recognition model.

2. The method as described in claim 1, characterized in that, The step of training the general gesture recognition model using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user to obtain an initial gesture recognition model includes: Determine the domain similarity between the calibrated electromyography (EMG) signals of different calibrated users and the historical EMG signals of the current user; A reference electromyography signal is selected from the calibrated electromyography signals based on the domain similarity. The general gesture recognition model is trained using the reference electromyography (EMG) signal and the historical EMG signal to obtain an initial gesture recognition model.

3. The method as described in claim 2, characterized in that, The step of training the general gesture recognition model using the reference electromyography signal and the historical electromyography signal to obtain an initial gesture recognition model includes: The intermediate layer of the general gesture recognition model obtains the source domain intermediate feature vector corresponding to the reference electromyography signal and the target domain intermediate feature vector corresponding to the historical electromyography signal, respectively. Based on the first output layer of the general gesture recognition model, the source domain gesture category probability corresponding to the reference electromyography signal and the main target domain gesture category probability corresponding to the historical electromyography signal are obtained respectively. A training loss function is constructed based on the source domain intermediate feature vector, the target domain intermediate feature vector, the source domain gesture category probability, and the main target domain gesture category probability. The general gesture recognition model is trained using the first-round training loss function to obtain an initial gesture recognition model.

4. The method as described in claim 3, characterized in that, The step of constructing a training loss function based on the source domain intermediate feature vector, the target domain intermediate feature vector, the source domain gesture category probability, and the main target domain gesture category probability includes: Construct a joint maximum mean difference loss function based on the intermediate feature vector of the source domain and the intermediate feature vector of the target domain; A first cross-entropy loss function is constructed using the source domain gesture category probability, and a minimum class confusion loss function is constructed using the main target domain gesture category probability. A training loss function is constructed based on the joint maximum mean difference loss function, the first cross-entropy loss function, and the minimum class confusion loss function.

5. The method as described in claim 4, characterized in that, The step of constructing a minimal class confusion loss function based on the gesture category probabilities of the main target domain includes: The probability of the gesture category in the main target domain is adjusted, and the adjusted probability of the gesture category in the main target domain is reweighted. The class association reduction operation is performed on the reweighted main target domain gesture category probability, and a minimum class confusion loss function is constructed based on the operation result.

6. The method as described in claim 4, characterized in that, The loss function for the first round of training is: in, Let the loss function be the training loss function for that round. Let the first cross-entropy loss function be . Let the joint maximum mean difference loss function be... Let be the minimum class confusion loss function. The first preset weight parameter, This is the second preset weight parameter.

7. The method as described in claim 3, characterized in that, The step of constructing the second cross-entropy loss function using the regularization term includes: The calibration gesture recognition probability corresponding to the calibration electromyography signal of the current user is obtained through the second output layer of the general gesture recognition model. The calibration data label corresponding to the calibration electromyography signal is determined, and a second cross-entropy loss function is constructed based on the calibration gesture recognition probability, the calibration data label, and the regularization term.

8. The method as described in claim 7, characterized in that, The second cross-entropy loss function is: in, Let C be the second cross-entropy loss function, and C be the number of gesture categories. The calibration data label is the one corresponding to the i-th gesture category. The calibration gesture recognition probability is given for the i-th gesture category. For the preset hyperparameters, The parameters of the historical model, These are the current model parameters of the initial gesture recognition model.

9. The method as described in claim 7, characterized in that, The step of training the initial gesture recognition model based on the second cross-entropy loss function and the calibrated electromyography signal of the current user to obtain the target gesture recognition model includes: A confidence test is performed on the gesture category probability of the main target domain to obtain the target domain pseudo-label, and the auxiliary target domain gesture category probability corresponding to the historical electromyography signal is obtained based on the second output layer of the general gesture recognition model. A generalized cross-entropy loss function is constructed based on the target domain pseudo-label and the auxiliary target domain gesture category probability, and a second-round training loss function is constructed based on the generalized cross-entropy loss function and the second cross-entropy loss function; The initial gesture recognition model is trained using the two-round training loss function to obtain the target gesture recognition model.

10. The method as described in claim 9, characterized in that, The generalized cross-entropy loss function is: in, Let C be the generalized cross-entropy loss function, and C be the number of gesture categories. The pseudo-label of the target domain corresponding to the i-th gesture category, The calibration gesture recognition probability is given for the i-th gesture category. These are preset adjustment parameters.

11. The method as described in claim 10, characterized in that, The loss function for the second round of training is: in, Let be the loss function for the second round of training. Let the generalized cross-entropy loss function be... Let the second cross-entropy loss function be... The third weight is preset.

12. The method as described in claim 2, characterized in that, The number of reference electromyography signals is at least two, and the number of target gesture recognition models is the same as the number of reference electromyography signals. The step of performing gesture recognition on the collected electromyography signals of the current user using the target gesture recognition model includes: The electromyography signals of the current user are collected and the gestures are recognized by each of the target gesture recognition models to obtain the initial recognition results corresponding to each target gesture recognition model. The initial recognition results are integrated to obtain the target gesture recognition result.

13. The method according to any one of claims 1 to 12, characterized in that, Before the step of training the general gesture recognition model using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user, the method further includes: Determine if the current user is a first-time user; If so, the calibrated electromyography (EMG) signal of the current user is used as the historical EMG signal of the current user, and the step of training the general gesture recognition model using the calibrated EMG signals of different calibrated users and the historical EMG signal of the current user is performed.

14. A gesture recognition device, characterized in that, The device includes: The parameter acquisition module is used to acquire historical model parameters of the general gesture recognition model, which is obtained through iterative training based on electromyography signals of historical users. A training module is used to train the general gesture recognition model using calibrated electromyography (EMG) signals from different calibrated users and historical EMG signals from the current user, in order to obtain an initial gesture recognition model. The second-round training module is used to train the initial gesture recognition model based on the historical model parameters and the current user's calibrated electromyography signal to obtain the target gesture recognition model. The gesture recognition module is used to perform gesture recognition on the collected electromyography signals of the current user through the target gesture recognition model; The second-round training module is further used to obtain the current model parameters of the initial gesture recognition model, and construct a regularization term based on the current model parameters, preset hyperparameters and the historical model parameters; construct a second cross-entropy loss function through the regularization term, and train the initial gesture recognition model according to the second cross-entropy loss function and the current user's calibrated electromyography signal to obtain the target gesture recognition model.

15. A storage medium, characterized in that, The storage medium stores a gesture recognition program, which, when executed by a processor, implements the gesture recognition method as described in any one of claims 1 to 13.

16. A wearable device, characterized in that, The wearable device includes: a memory, a processor, and a gesture recognition program stored in the memory and executable on the processor, wherein the gesture recognition program, when executed by the processor, implements the gesture recognition method as described in any one of claims 1 to 13.

Images