Classification model updating method and device, electronic equipment and computer program product

Abstract

The invention is suitable for the technical field of classification models, and provides a classification model updating method and device, electronic equipment and a computer program product. The method comprises the following steps: acquiring a first physiological signal of a user in a current session; determining a first accuracy rate of a classification model based on the first physiological signal; if the first accuracy rate is smaller than a first threshold value, acquiring a second physiological signal of the user in the current session, and updating the classification model based on the second physiological signal and a labeling model; and determining the next session of the user as the current session, and returning to execute the step of acquiring the first physiological signal of the user in the current session and the subsequent steps until the first accuracy rate is greater than or equal to the first threshold value. Through the method and the device, the cross-user and cross-session problem can be improved, and the recognition accuracy of the classification model is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of classification models, and in particular, relates to a classification model updating method, device, electronic device and computer program product. Background Art

[0002] The technology of using human physiological signals to identify gestures, biological organisms, emotions, etc. has shown broad application prospects and needs in many fields such as medical rehabilitation and human-computer interaction. However, since physiological signals have data domain offset between different individuals and in different states of the same individual, this poses a challenge to the practical application of the technology. The data domain offset phenomenon mainly has the following two technical problems:

[0003] (1) Cross-Subject Problem: Due to physiological differences between different individuals, even if the same action is performed, the physiological signals generated will be different. This means that a trained algorithm model may experience a significant performance degradation after changing users.

[0004] (2) Cross-Session Problem: A "session" usually refers to an independent collection cycle, that is, the process from wearing a signal collection device to collecting physiological signals to stopping collecting signals and taking off the device. In actual use, even the physiological signals collected by the same user at different times and environmental conditions (i.e., in different sessions) will be different, which may be affected by multiple factors such as changes in the position of electrodes or sensors, changes in the user's physiological state, and deviations in the understanding of the task, thereby causing the recognition system performance to deteriorate. Summary of the invention

[0005] The embodiments of the present application provide a classification model updating method, device, electronic device and computer program product, which can improve cross-user and cross-session problems and improve the recognition accuracy of the classification model.

[0006] In a first aspect, an embodiment of the present application provides a classification model updating method, comprising:

[0007] Acquire a first physiological signal of the user in the current session, where the first physiological signal is a physiological signal marked with a classification label and collected by the signal collection device when the user is wearing the signal collection device;

[0008] Determining a first accuracy rate of a classification model based on the first physiological signal;

[0009] If the first accuracy rate is less than a first threshold, obtaining a second physiological signal of the user in the current session, and updating the classification model based on the second physiological signal and the annotation model, wherein the second physiological signal is an unlabeled physiological signal collected by the signal acquisition device when the user wears the signal acquisition device;

[0010] The next session of the user is determined as the current session, and the step of acquiring the first physiological signal of the user in the current session and subsequent steps are returned to be executed until the first accuracy rate is greater than or equal to the first threshold.

[0011] In an embodiment of the present application, by obtaining a first physiological signal of the user in the current session and determining a first accuracy rate of the classification model based on the first physiological signal, when the first accuracy rate is less than a first threshold, the second physiological signal of the user in the current session can be obtained, the classification model can be updated based on the second physiological signal and the annotation model, and the next session of the user can be determined as the current session, and the step of obtaining the first physiological signal of the user in the current session and subsequent steps are returned to execute, thereby gradually optimizing the classification model through continuous updates in multiple sessions, so that it can maintain a high recognition accuracy rate in different users and different sessions, improve cross-user and cross-session problems, and improve the recognition accuracy rate of the classification model.

[0012] In a second aspect, an embodiment of the present application provides a classification model updating device, comprising:

[0013] A signal acquisition module, used to acquire a first physiological signal of a user in a current session, where the first physiological signal is a physiological signal marked with a classification label and collected by the signal acquisition device when the user is wearing the signal acquisition device;

[0014] An accuracy determination module, used to determine a first accuracy of a classification model based on the first physiological signal;

[0015] a model updating module, configured to obtain a second physiological signal of the user in the current session if the first accuracy rate is less than a first threshold, and update the classification model based on the second physiological signal and the annotation model, wherein the second physiological signal is an unlabeled physiological signal collected by the signal acquisition device when the user is wearing the signal acquisition device;

[0016] The session processing module is used to determine the next session of the user as the current session, and return to execute the signal acquisition module and subsequent modules until the first accuracy rate is greater than or equal to the first threshold.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method described in the first aspect above.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method described in the first aspect above is implemented.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, wherein when the computer program is executed, the method described in the first aspect above is executed.

[0020] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a flow chart of the classification model updating method provided in the embodiment of the present application;

[0023] Figure 2 The embodiment of the present application provides a structural diagram of a CNN-LSTM model;

[0024] Figure 3 is a flowchart of a classification model updating method provided in an embodiment of the present application;

[0025] Figure 4-1 This is the accuracy change chart corresponding to a volunteer;

[0026] Figure 4-2 This is the corresponding accuracy change chart for another volunteer;

[0027] Figure 4-3 This is the corresponding accuracy change chart for another volunteer;

[0028] Figure 4-4 This is the accuracy change chart corresponding to another volunteer;

[0029] Figure 4-5 This is the corresponding accuracy change chart for another volunteer;

[0030] Figure 5 is a structural diagram of a classification model updating device provided in an embodiment of the present application;

[0031] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0033] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0034] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0035] It should be noted that the information collection process (such as physiological signal collection process) / feature extraction process involved in this application is performed with the user's knowledge and permission, that is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

[0036] The physiological signal in this application can be any physiological signal that can be collected by a signal collection device worn by a user, and this application does not limit the specific type of the physiological signal. For example, the physiological signal is a surface electromyography (sEMG) signal or an electroencephalography (EEG) signal.

[0037] The classification model updating method provided in the embodiments of the present application can be applied to application scenarios such as electromyographic gesture recognition, controlling bionic prostheses using sEMG signals, identifying movement intentions using sEMG signals, biometric identification using sEMG signals, EEG signal-based recognition systems, controlling external devices using EEG signals, emotion recognition using EEG signals, and biometric identification using EEG signals.

[0038] Using sEMG signals to control bionic prostheses may mean: after the embodiment of the present application updates the classification model based on the sEMG signal of the hand, gesture recognition is performed based on the updated classification model, and the bionic prosthesis is controlled to execute the corresponding gesture based on the recognized gesture, which can improve the gesture recognition accuracy and the response speed of the bionic prosthesis, and enhance the user experience.

[0039] Using sEMG signals to identify movement intentions may mean: in sports science and rehabilitation medicine, after the embodiment of the present application updates the classification model based on the electromyographic signals of the moving limbs, movement intentions are identified based on the updated classification model, which can identify the user's movement intentions (such as walking, standing, squatting, etc.) and provide personalized rehabilitation training and guidance.

[0040] Using sEMG signals for biometric identification may mean: in the field of security authentication, after the classification model is updated based on the sEMG signal of the wrist or a specific part of the human body, the embodiment of the present application identifies the user's electromyography signal characteristics based on the updated classification model to achieve biometric identification and identity authentication, and the identification results of the classification model are verification passed and verification failed.

[0041] Using EEG signals to control external devices may mean: after the embodiment of the present application updates the classification model based on the EEG signal, the corresponding control instructions are identified based on the updated classification model, and the external device (such as a brain-controlled wheelchair, smart home, etc.) is controlled based on the control instructions.

[0042] Using EEG signals for emotion recognition may mean: after the embodiment of the present application updates the classification model based on the EEG signal, emotion categories such as joy, anger, sorrow, worry, fear, and shock are identified based on the updated classification model, which can be used for mental health assessment and to provide personalized psychological intervention and support.

[0043] Using EEG signals for biometric identification may mean that after the classification model is updated based on the EEG signal, the user's brain wave characteristics are identified based on the updated classification model to achieve identity authentication and security authentication. The identification result of the classification model is verification passed or verification failed.

[0044] For ease of understanding, this embodiment is described by taking electromyographic gesture recognition as an example, but this does not constitute a limitation on the application scenario of the present application.

[0045] Gestures play an important role in people's daily life and communication. Their research and recognition not only have social and cultural significance, but also provide new directions for research in the fields of medicine, robotics, intelligent control and human-computer interaction. At present, a variety of modal signals including sEMG signals, images, optical three-dimensional models, sound waves, acceleration and even photoelectric pulse waves have been applied to gesture recognition research. Among them, sEMG signals have attracted widespread attention due to their advantages such as non-invasive, convenient and safe acquisition process, and have become a hot spot in the current research of gesture recognition technology.

[0046] Gesture recognition technology based on sEMG signals usually collects sEMG signals generated when subjects perform specific gestures by setting electrodes on the skin surface of the forearm and other parts. After these signals are preprocessed and / or feature extracted, artificial intelligence technology is used to train a classification model that can recognize the corresponding gestures. However, although sEMG technology has made significant progress in the field of gesture recognition, it still faces cross-user and cross-session problems, which seriously affect the performance of gesture recognition systems in actual application scenarios.

[0047] In order to deal with the cross-user problem, transfer learning methods are widely used in current research. However, transfer learning methods lack adaptive capabilities (i.e., transfer learning methods usually adopt a one-time update method, and the performance of the calibrated model is prone to degradation in subsequent use due to changes in time and sessions), and the calibration cost is high (i.e., users are required to cooperate in collecting labeled data for a long time, and the calibration cost is high).

[0048] In response to the cross-session problem, existing research has provided two main ideas: one is variable factor identification and recovery (i.e., an automatic electrode channel rearrangement algorithm based on energy differences), and the other is visual information assistance (i.e., introducing a visual depth sensor to assist in real-time gesture discrimination and marking through visual depth information, and helping the sEMG signal classifier model to perform autonomous iteration). However, both ideas have their own limitations. The automatic electrode channel rearrangement algorithm only solves the problem of circumferential electrode position drift, and ignores data offsets caused by other environmental changes (such as temperature, humidity, and axial drift). Visual assistance method: Although it improves the system's adaptability, it increases the complexity of the hardware equipment and the user's usage cost.

[0049] In summary, although the existing solutions have alleviated the cross-user and cross-session problems to a certain extent, they still need to be improved in terms of adaptability and usage cost. The classification model updating method provided in the embodiment of the present application can overcome the shortcomings of the existing technology and improve the overall performance and user experience of the electromyographic gesture recognition system.

[0050] The embodiment of the present application combines traditional machine learning technology and deep learning technology, that is, using traditional machine learning technology as a labeling model and deep learning technology as a classification model, and can take advantage of their respective advantages to dynamically improve the overall performance and adaptive ability of the system. In addition, the embodiment of the present application continuously updates the classification model, so the classification model updating method provided in the embodiment of the present application can also be called a dual-model (i.e., labeling model and classification model) adaptive continuous learning strategy.

[0051] The dual-model adaptive continuous learning strategy provided by the embodiment of the present application can reduce the user's usage cost and improve the user experience. Compared with the transfer learning method, the present application reduces the amount of labeled data collected for calibration. The user only needs to perform a short gesture to complete the calibration, which greatly reduces the user's usage cost. Compared with the visual information auxiliary method, the present application only uses single-modal electromyography data, avoiding additional hardware equipment and complex system integration, simplifying the complexity of the system, and improving the user experience.

[0052] The dual-model adaptive continuous learning strategy provided by the embodiment of the present application has the ability of session adaptation. The present application continuously updates the classification model in multiple session rounds, and continuously improves the generalization performance of the model based on the idea of ​​domain generalization. Unlike the one-time update in the transfer learning method, the present application can continuously optimize the classification model over time, so that it can still maintain a high recognition accuracy when facing new session data.

[0053] A dual-model adaptive continuous learning strategy provided in an embodiment of the present application can improve the overall performance of the system. The present application can achieve a high recognition accuracy rate in the early stage of use by new users by using a small amount of data to train the annotation model and using the annotation model to assist in labeling subsequent real-time electromyographic data in the session. When the classification model calibration is not completed, the present application can also provide high-precision recognition results through the annotation model, making the classification model calibration process transparent to the user, and the user can start using the classification model without waiting for a long time for calibration.

[0054] In summary, the present invention effectively solves the shortcomings of the prior art in cross-user and cross-session problems by proposing a dual-model adaptive continuous learning strategy, and improves the overall performance and user experience of the electromyographic gesture recognition system.

[0055] The classification model updating method provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0056] See also Figure 1 , Figure 1 A flow chart of a classification model updating method provided in an embodiment of the present application is shown. As an example but not a limitation, the method is applied to an electronic device, and the method includes the following steps:

[0057] Step 101: Acquire a first physiological signal of a user in a current session.

[0058] Among them, the first physiological signal is a physiological signal marked with a classification label and collected by the signal acquisition device when the user wears the signal acquisition device.

[0059] The above-mentioned user may be a new user, that is, a user whose physiological signal has not been used to update the classification model before the current moment.

[0060] When the user wears the signal acquisition device, at least one predefined gesture can be performed, and the duration of each gesture is a preset time. On this basis, the electronic device can obtain the first physiological signal from the signal acquisition device. Optionally, the preset time can be set according to the scene requirements, and this application does not limit the specific value of the preset time. For example, the preset time is 4 seconds.

[0061] Step 102: Determine a first accuracy rate of the classification model based on the first physiological signal.

[0062] The accuracy of the classification model can also be understood as the recognition accuracy, which can refer to the proportion of the correct number predicted by the classification model to the total number. The first accuracy is the accuracy determined based on the first physiological signal.

[0063] It should be understood that the classification model has been pre-trained based on discrete data samples before the first session, so that it has basic classification recognition and generalization capabilities. The above discrete data samples may include physiological signal samples of multiple users and labels corresponding to each physiological signal sample. Before pre-training the classification model based on discrete data samples, the discrete data samples may be pre-processed by filtering, normalization, feature extraction, etc.

[0064] This embodiment pre-trains the classification model on discrete data samples, adjusts training hyperparameters (such as learning rate, batch size, etc.), and introduces mechanisms such as dynamic learning rate, so that the classification model can have basic classification recognition and generalization capabilities.

[0065] The first physiological signal is used to test the classification model that has been pre-trained or updated in the previous round of iterations. If the test result of the classification model (i.e., the first accuracy) reaches the first threshold, it indicates that the adaptability of the classification model to the new session data has reached the threshold requirement, and the continuous learning process for the user has been completed. The classification model can be used to classify the user's physiological signals to obtain the corresponding classification results.

[0066] In one embodiment, a Convolutional Neural Networks-Long Short-Term Memory (CNN-LSTM) model can be used as a classification model. The CNN-LSTM model takes into account both signal space and temporal analysis, and is responsible for extracting spatiotemporal features from the user's physiological signals and classifying them. For example, the CNN-LSTM model is used to extract spatiotemporal features from surface electromyography signals and perform gesture classification. The CNN-LSTM model includes three convolutional layers and an LSTM network. The specific structure is as follows: Convolutional network part (i.e., CNN): responsible for extracting spatial features from input physiological signals. The spatial features of physiological signals are extracted through multiple layers of convolution kernels, and the size and number of convolution kernels in each layer are designed differently to capture features of different scales. The pooling layer is used to reduce the data dimension while retaining the most important features. LSTM network: responsible for processing the spatial features output by CNN, capturing the dynamic features of spatial features that change over time through memory units, thereby achieving classification.

[0067] Optionally, a pure CNN model can be used instead of a CNN-LSTM model to improve the model's feature extraction capability by increasing the number of convolutional layers and depth. A recurrent neural network (RNN) can also be used instead of an LSTM network to take advantage of its flexibility in processing time series data and improve the model's ability to capture time-dependent features. A graph neural network (GNN) can also be used instead of a CNN-LSTM model to take advantage of its advantages in graph-structured data and improve the model's performance in complex signal processing.

[0068] Step 103: If the first accuracy rate is less than the first threshold, a second physiological signal of the user in the current session is obtained, and the classification model is updated based on the second physiological signal and the annotation model.

[0069] The second physiological signal is an unlabeled physiological signal collected by the signal collection device when the user wears the signal collection device. Optionally, the first threshold value can be set according to the scene requirements, and the specific value of the first threshold value is not limited in this application. For example, the first threshold value is 90%.

[0070] After the user wears the signal acquisition device and performs the predefined gestures, he can perform some gestures at will. On this basis, the electronic device can obtain the second physiological signal from the signal acquisition device (for example, the electronic device can receive the second physiological signal transmitted by the signal acquisition device in real time).

[0071] In one embodiment, before updating the classification model based on the second physiological signal and the annotation model, the second physiological signal may be pre-processed by real-time filtering, real-time normalization, etc. Among them, the real-time filtering may use a 10Hz fourth-order high-pass filter for filtering low-frequency noise and a 50Hz notch filter for reducing power frequency noise interference. The real-time normalization may use a moving average method (MAM) to perform real-time normalization on the second physiological signal.

[0072] In one embodiment, a k-nearest neighbor (kNN) model that is more sensitive to specific data distribution may be used as a labeling model to provide auxiliary labels for unlabeled data (ie, the second physiological signal).

[0073] Optionally, you can use Support Vector Machine (SVM) instead of kNN model as the annotation model, taking advantage of its good classification performance on small sample data to improve the accuracy of initial annotation. You can also use Random Forest (RF) instead of kNN model, taking advantage of its ensemble learning characteristics to improve the robustness and generalization ability of the model.

[0074] If the first accuracy of the classification model is less than the first threshold, it means that the classification model has a low recognition accuracy for the user's physiological signals. By executing step 103 and subsequent steps, the classification model can be gradually updated so that it can maintain a high recognition accuracy for different users and different sessions.

[0075] In a possible implementation manner, before obtaining the second physiological signal of the user in the current session, the method further includes:

[0076] Training a labeling model based on the first physiological signal;

[0077] The classification model is calibrated based on the first physiological signal.

[0078] Since the first physiological signal is annotated with a label, a labeling model can be trained based on the first physiological signal, and the pre-trained classification model can be calibrated.

[0079] If the first accuracy of the classification model is less than the first threshold, the second physiological signal may be used to train a labeling model for the current round of iterations, and the classification model may be calibrated using the second physiological signal.

[0080] It should be understood that, in contrast to the cross-session data domain shift phenomenon of physiological signals, in the same session, physiological signals basically conform to the same data distribution. Therefore, a machine learning model (i.e., annotated model) that performs well in the current session can be trained using a small amount of physiological signals (i.e., the first physiological signal).

[0081] In a possible implementation, updating the classification model based on the second physiological signal and the annotation model includes:

[0082] Using the annotation model to classify the second physiological signal, and obtain a label of the second physiological signal;

[0083] The classification model is updated based on the second physiological signal and the corresponding label.

[0084] The electronic device can use the trained annotation model to assist in labeling the subsequent real-time second physiological signal in this iteration (ie, the same session) (ie, label the second physiological signal).

[0085] In a possible implementation, before updating the classification model based on the second physiological signal and the corresponding label, the method further includes:

[0086] Determining a second accuracy rate of the classification model based on the second physiological signal and the corresponding label;

[0087] Based on the second physiological signal and the corresponding label, updating the classification model includes:

[0088] If the second accuracy is less than the second threshold, the classification model is updated based on the second physiological signal and the corresponding label, and the step of obtaining the second physiological signal of the user in the current session and subsequent steps are returned to execute until the second accuracy is greater than or equal to the second threshold.

[0089] The second threshold value may be the same as or different from the first threshold value, and this application does not limit this. Optionally, the second threshold value may be set according to the scenario requirements, and this application does not limit the specific value of the second threshold value. For example, the first threshold value is 90%.

[0090] The electronic device can continue to collect the second physiological signal of the user in the current session by returning to execute the step of obtaining the second physiological signal of the user in the current session and subsequent steps, and update the classification model after annotating it with the annotation model until the second accuracy of the classification model is greater than or equal to the second threshold, thereby ending the local iteration.

[0091] In the process of new user usage, this embodiment gradually updates the classification model through a semi-supervised online learning algorithm, which can improve its adaptability to new session data.

[0092] In a possible implementation, the second physiological signal is classified using the annotation model to obtain a label of the second physiological signal, including:

[0093] Processing the second physiological signal using a sliding window algorithm to obtain multiple data windows;

[0094] Determine N data windows as a voting group to obtain multiple voting groups, where N is an integer greater than zero and less than the total number of data windows;

[0095] Use the annotation model to classify the N data windows in each voting group, and obtain the classification results of the N data windows in the corresponding voting group;

[0096] For any voting group, if there is a target classification result in the classification results of the N data windows in the voting group, the target classification result is determined as the label of the voting group, and the target classification result is the classification result whose number is greater than the preset number in the classification results of the N data windows in the corresponding voting group;

[0097] Accordingly, determining a second accuracy rate of the classification model based on the second physiological signal and the corresponding label includes:

[0098] Determining a second accuracy of the classification model based on N data windows and corresponding labels in each voting group of M voting groups that are consecutive in acquisition time, where M is an integer greater than 1 and less than or equal to the total number of voting groups;

[0099] Accordingly, based on the second physiological signal and the corresponding label, the classification model is updated, including:

[0100] When the labels of the M voting groups are the same, the classification model is updated based on the N data windows and corresponding labels in each of the M voting groups.

[0101] This embodiment can improve the stability of the selected data by limiting the labels of M consecutive voting groups in the acquisition time to be the same. For example, when a segment of sEMG signal can be continuously recognized as the same gesture by the annotation model, it can be determined that the stability of this segment of sEMG signal is better and more suitable for updating the classification model.

[0102] As an example but not a limitation, the collected second physiological signal is further processed using a sliding window algorithm, with a sliding window length of 200 milliseconds and a sliding step of 20 milliseconds, which improves data density while formatting the processed data. Finally, 20 consecutive groups of data windows are divided into a voting group for subsequent classification.

[0103] When dividing voting groups, similar ideas as sliding windows and queues are also used. For example, if the current voting group contains data from 1 to 20, after the voting results (i.e. target classification results or labels) are obtained, the "oldest" data 1 is excluded and the latest data 21 is added to form the second voting group. At this time, there are 19 duplicate data in these two voting groups.

[0104] It should be understood that there are duplications between the data windows cut using the sliding window algorithm. For example, when the sliding window length is 200 milliseconds and the step length is 20 milliseconds, 180 milliseconds of data between two adjacent data windows are the same.

[0105] Use the kNN model to classify the data windows within each voting group. If a classification result exceeds 80% (i.e., 16 votes) in a voting group, all 20 data windows of the group are marked accordingly (i.e., labeled). Record the results of 10 consecutive voting groups, and use these data with kNN labeled results (i.e., labels) to test the CNN-LSTM model. If the accuracy is higher than the second threshold (e.g., 90%), the performance of the CNN-LSTM model is considered to be sufficiently close to the kNN model trained in this round of iterations. The CNN-LSTM model will no longer be updated in this round of iterations, and the user will take off the device (the end condition of a single round). Otherwise, proceed to the next step (i.e., updating the deep learning model online).

[0106] In one possible implementation, updating the deep learning model online includes:

[0107] The classification model is updated based on the N data windows and corresponding labels in each of the M voting groups, the first physiological signal, and the offline training samples, where the offline training samples are samples used to pre-train the classification model before the first session.

[0108] As an example but not a limitation, when the above 10 consecutive voting results are consistent, a total of 200 data segments (with duplication) are used to update the CNN-LSTM model. During each update, the electronic device can also extract 10 and 20 data segments of each gesture from the offline data samples and the labeled data collected in each round of iteration, respectively, and add them to the updated material to mitigate possible catastrophic forgetting.

[0109] In a possible implementation, the classification model is a CNN-LSTM model, which includes a CNN and an LSTM network; based on the N data windows and corresponding labels in each of the M voting groups, the first physiological signal, and the offline training sample, the classification model is updated, including:

[0110] The LSTM network is updated based on the N data windows and corresponding labels in each of the M voting groups, the first physiological signal, and the offline training samples.

[0111] The electronic device freezes some parameters of the CNN structure in the CNN-LSTM model during the online learning process and only updates the parameters in the LSTM network. This can effectively capture the dynamic features in the time series and improve the model's adaptability while saving computing resources and retaining the results of pre-training.

[0112] like Figure 2 Shown is a structural diagram of the CNN-LSTM model. Figure 2 Conv1d represents a one-dimensional convolutional layer, BN represents a batch normalization layer, MaxPooling represents a maximum pooling layer, Flatten represents a one-dimensional data operation, FC represents a fully connected layer, and Softmax represents the activation function of the output layer. The parts connected by arrows from left to right represent: input data consisting of L sEMG features (8 channels * 11 features), convolutional layer 1 (convolution kernel size is 3, output dimension is 16), convolutional layer 2 (convolution kernel size is 3, output dimension is 32), convolutional layer 3 (convolution kernel size is 3, output dimension is 64), one-dimensional data after flattening (Flatten), batch normalization layer, LSTM network structure consisting of L LSTM units, and the final five-category classification gesture. The part in the blue dotted box represents the parameters updated in the calibration phase, that is, the parameters of the network part in the box will be updated according to the training data in the calibration phase, and the parameters of the network part outside the box are frozen and do not participate in the update; similarly, the parameters in the yellow dotted box are updated in the online learning phase.

[0113] Step 104: determine the user's next session as the current session.

[0114] The electronic device returns to execute the step of obtaining the first physiological signal of the user in the current session and subsequent steps until the first accuracy rate is greater than or equal to the first threshold. By continuous learning in multiple sessions, the electronic device can gradually optimize the classification model and continuously improve its adaptive ability, so that it can maintain a high recognition accuracy rate for different users and different sessions.

[0115] When the first accuracy of the classification model in the new session is greater than or equal to the first threshold, the construction of the adaptive model is completed, and the classification model can be directly applied to classification recognition or control systems later.

[0116] This embodiment provides a semi-supervised online learning algorithm, which realizes continuous optimization and adaptation of the classification model through steps such as real-time data preprocessing, classification and labeling, and online model updating.

[0117] This embodiment can take all environmental variables into consideration as a whole from a higher level. Based on the idea of ​​domain generalization, the classification model can be better adapted to changes in all environmental factors through continuous learning.

[0118] Domain generalization means allowing the classification model to learn on multiple different data domains to improve the generalization and adaptability of the classification model. Specifically, on the one hand, in the offline pre-training stage, the offline data samples include data from different subjects, different wearing positions (circumferential and axial) and different collection days (for other environments), which can enable the classification model to have a certain generalization ability. On the other hand, the present application can continue to learn between different sessions, that is, to continue to generalize the learning of environmental change factors that may occur between all sessions, and ultimately achieve adaptation to new session data, rather than designing algorithms for a specific change situation.

[0119] like Figure 3 The figure is an example flow chart of the classification model updating method provided in the embodiment of the present application. Figure 3 Take sEMG signal (i.e. sEMG data) as an example for relevant description. The process mainly includes three processes: offline pre-training, semi-supervised online learning and control application. In the offline pre-training process, the electronic device can collect offline sEMG data of each user in the offline user set as an offline data sample based on the offline user set, and perform offline training (i.e. pre-training) on ​​the classification model. In semi-supervised online learning, the electronic device can collect labeled data of new users, and based on these labeled data, the annotation model can be trained (in semi-supervised online learning, in order to distinguish it from the training of the classification model, the training of the annotation model can be called offline training, and the training of the classification model can be called online training), collect unlabeled sEMG data of new users, use the trained annotation model to identify (i.e. classify) and annotate the unlabeled sEMG data, obtain the labeled sEMG data corresponding to the unlabeled sEMG data, and use the labeled sEMG data to train the classification model online. In control applications, through continuous learning in multiple sessions in semi-supervised online learning, the classification model is gradually optimized, and finally a classification model with high recognition accuracy can be obtained. By inputting the re-collected unlabeled sEMG data of new users into the classification model, gesture recognition can be realized. The gesture recognition results obtained can be used in control systems, such as controlling bionic manipulators.

[0120] In an embodiment of the present application, by obtaining a first physiological signal of the user in the current session and determining a first accuracy rate of the classification model based on the first physiological signal, when the first accuracy rate is less than a first threshold, the second physiological signal of the user in the current session can be obtained, the classification model can be updated based on the second physiological signal and the annotation model, and the next session of the user can be determined as the current session, and the step of obtaining the first physiological signal of the user in the current session and subsequent steps are returned to execute, thereby gradually optimizing the classification model through continuous updates in multiple sessions, so that it can maintain a high recognition accuracy rate in different users and different sessions, improve cross-user and cross-session problems, and improve the recognition accuracy rate of the classification model.

[0121] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0122] In order to verify the effectiveness and advantages of the dual-model adaptive continuous learning strategy provided by the embodiment of the present application, a series of experiments were designed, and 5 volunteers were invited to participate in the experiment as new users. The experimental results show that the embodiment of the present application is superior to the existing transfer learning method in terms of gesture classification accuracy and adaptive ability. A standard sEMG acquisition device was used to ensure the consistency of each volunteer wearing the device during the experiment. The experimental process mainly includes the following four steps:

[0123] Step 1, offline pre-training: Use a set of standard offline data sets (i.e., discrete data samples) to train the CNN-LSTM model so that it has basic classification and recognition capabilities and generalization capabilities.

[0124] Step 2, new user calibration: before each new session begins, ensure that the volunteers' hand muscles are fully relaxed and rested, and put on the sEMG acquisition equipment again; after the new session begins, the volunteers perform an accuracy test according to the experimental program prompts in the electronic device, and the experimenter records the initial accuracy of the new session; if the initial accuracy does not exceed the accuracy threshold (90%) set by the experiment, continue the current session and proceed to the next step, otherwise end the current session.

[0125] Step 3, semi-supervised online learning: perform calibration, labeling, online learning and other processes according to the semi-supervised online learning algorithm introduced in the above embodiment; during this period, an accuracy test is performed every 15 seconds until the result exceeds the accuracy threshold, and it is considered that the learning in this session is sufficient, and the session is ended.

[0126] Step 4, multi-session verification: If the volunteer does not complete the entire experiment, he / she will participate in the next experimental session after an interval of at least 1 hour, until the accuracy rate exceeds the threshold for 3 consecutive new sessions, and the experiment for this volunteer is completed.

[0127] During the above experimental process, on average only 33.6 seconds of labeled data need to be collected for each gesture, and a CNN-LSTM model with adaptive capabilities can be obtained through the dual-model adaptive continuous learning strategy provided in the embodiment of the present application.

[0128] Table 1 shows the average accuracy of the CNN-LSTM model obtained using the embodiments of the present application on new session data and the average accuracy of the new session using the transfer learning method.

[0129] Table 1

[0130]

[0131] Experimental results show that the average accuracy of the CNN-LSTM model obtained using the embodiment of the present application on new session data reaches 95.33%, while the average accuracy of the transfer learning method using the same basic model and the same labeled data on new sessions is 82.82%.

[0132] Table 2 shows the comparison of the new session accuracy of each volunteer when using the dual-model adaptive continuous learning strategy and the transfer learning method.

[0133] Table 2

[0134]

[0135] The above experimental results show that the dual-model adaptive continuous learning strategy provided by the embodiment of the present application can effectively and quickly complete the calibration of new users, and is significantly better than the existing transfer learning method in terms of calibration efficiency and model adaptability. By using the embodiment of the present application, the overall performance of the electromyographic gesture recognition system has been significantly improved, and the user experience has also been significantly improved.

[0136] Figure 4-1 to Figure 4-5 The line graph shows the change in accuracy of each volunteer using the dual-model adaptive continuous learning strategy to update the classification model and perform gesture recognition, as well as the comparison with the transfer learning method. Each line graph corresponds to the experimental data record of a volunteer, where the cyan thin solid line represents the change in accuracy of the dual-model adaptive continuous learning strategy in the same iteration (i.e. Figure 4-1 to Figure 4-5 The blue thick solid line indicates the change of the first accuracy of the dual-model adaptive continuous learning strategy in different iterations (i.e. Figure 4-1 to Figure 4-5The acc-cross-iteration in the figure indicates the model's ability to recognize new session data as the number of iterations increases (because the data of this session has not been learned in the first test of each iteration, this result reflects the model's ability to recognize new session data); the orange dotted line corresponds to the blue thick solid line, which indicates the change in the accuracy of the transfer learning method across iterations (i.e. Figure 4-1 to Figure 4-5 The red horizontal line indicates the accuracy threshold of 90% (i.e. Figure 4-1 to Figure 4-5 acc-threshold in). Figure 4-1 to Figure 4-5 The horizontal axis Accuracy represents the accuracy, and the vertical axis Sessions represents the number of sessions (i.e., the number of iterations).

[0137] Corresponding to the classification model updating method described in the above embodiment, Figure 5 A structural schematic diagram of a classification model updating device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0138] Reference Figure 5 , the device comprises:

[0139] The signal acquisition module 501 is used to acquire a first physiological signal of the user in the current session, where the first physiological signal is a physiological signal marked with a classification label and collected by the signal acquisition device when the user wears the signal acquisition device;

[0140] An accuracy determination module 502, configured to determine a first accuracy of a classification model based on the first physiological signal;

[0141] A model updating module 503 is configured to obtain a second physiological signal of the user in the current session if the first accuracy rate is less than a first threshold, and update the classification model based on the second physiological signal and the annotation model, wherein the second physiological signal is an unlabeled physiological signal collected by the signal acquisition device when the user is wearing the signal acquisition device;

[0142] The session processing module 504 is configured to determine the next session of the user as the current session, and return to execute the signal acquisition module 501 and subsequent modules until the first accuracy rate is greater than or equal to the first threshold.

[0143] Optionally, the model updating module 503 includes:

[0144] a classification unit, configured to classify the second physiological signal using the annotation model to obtain a label of the second physiological signal;

[0145] An updating unit is used to update the classification model based on the second physiological signal and the corresponding label.

[0146] Optionally, the model updating module 503 further includes:

[0147] a determining unit, configured to determine a second accuracy rate of the classification model based on the second physiological signal and the corresponding label;

[0148] The above-mentioned updating unit is specifically used for:

[0149] If the second accuracy rate is less than the second threshold, the classification model is updated based on the second physiological signal and the corresponding label, and the model updating module 503 and subsequent modules are returned until the second accuracy rate is greater than or equal to the second threshold.

[0150] Optionally, the above classification unit is specifically used for:

[0151] Processing the second physiological signal using a sliding window algorithm to obtain multiple data windows;

[0152] Determine N data windows as a voting group to obtain multiple voting groups, where N is an integer greater than zero and less than the total number of data windows;

[0153] Using the annotation model to classify the N data windows in each voting group, and obtaining classification results of the N data windows in the corresponding voting group;

[0154] For any of the voting groups, if there is a target classification result among the classification results of the N data windows in the voting group, the target classification result is determined as the label of the voting group, and the target classification result is the classification result whose number is greater than a preset number among the classification results of the N data windows in the corresponding voting group;

[0155] Accordingly, the determining unit is specifically used for:

[0156] Determining a second accuracy of the classification model based on N data windows and corresponding labels in each voting group of M voting groups that are consecutive in acquisition time, where M is an integer greater than 1 and less than or equal to the total number of voting groups;

[0157] Accordingly, the above-mentioned updating unit is specifically used for:

[0158] In the case where the labels of the M voting groups are the same, the classification model is updated based on the N data windows and the corresponding labels in each of the M voting groups.

[0159] Optionally, the updating unit is specifically used for:

[0160] The classification model is updated based on the N data windows and corresponding labels in each of the M voting groups, the first physiological signal, and offline training samples, where the offline training samples are samples used to pre-train the classification model before the first session.

[0161] Optionally, the classification model is a CNN-LSTM model, and the CNN-LSTM model includes a CNN and an LSTM network; and the updating unit is specifically used for:

[0162] The LSTM network is updated based on the N data windows and corresponding labels in each of the M voting groups, the first physiological signal, and the offline training samples.

[0163] Optionally, the model updating module 503 is further used for:

[0164] Training the annotation model based on the first physiological signal;

[0165] The classification model is calibrated based on the first physiological signal.

[0166] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0167] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 6 of this embodiment includes: at least one processor 60 ( Figure 6 Only one is shown in the figure), a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, wherein the processor 60 implements the steps of any of the above-mentioned method embodiments when executing the computer program 62.

[0168] The electronic device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 6 It is only an example of the electronic device 6 and does not constitute a limitation on the electronic device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0169] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0170] In some embodiments, the memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. In other embodiments, the memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 6. Further, the memory 61 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 61 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0171] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0175] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A classification model updating method, characterized in that: include: Acquire a first physiological signal of the user in the current session, where the first physiological signal is a physiological signal marked with a classification label and collected by the signal collection device when the user is wearing the signal collection device; Determining a first accuracy rate of a classification model based on the first physiological signal; If the first accuracy rate is less than a first threshold, obtaining a second physiological signal of the user in the current session, and updating the classification model based on the second physiological signal and the annotation model, wherein the second physiological signal is an unlabeled physiological signal collected by the signal acquisition device when the user wears the signal acquisition device; The next session of the user is determined as the current session, and the step of acquiring the first physiological signal of the user in the current session and subsequent steps are returned to be executed until the first accuracy rate is greater than or equal to the first threshold.

2. The classification model updating method according to claim 1, characterized in that: The updating of the classification model based on the second physiological signal and the annotation model comprises: Using the annotation model to classify the second physiological signal to obtain a label of the second physiological signal; The classification model is updated based on the second physiological signal and the corresponding label.

3. The classification model updating method according to claim 2, characterized in that: Before updating the classification model based on the second physiological signal and the corresponding label, the method further includes: Determining a second accuracy rate of the classification model based on the second physiological signal and the corresponding label; The updating of the classification model based on the second physiological signal and the corresponding label includes: If the second accuracy rate is less than the second threshold, the classification model is updated based on the second physiological signal and the corresponding label, and the step of obtaining the second physiological signal of the user in the current session and subsequent steps are returned to execute until the second accuracy rate is greater than or equal to the second threshold.

4. The classification model updating method according to claim 3, characterized in that: The using the annotation model to classify the second physiological signal to obtain a label of the second physiological signal includes: Processing the second physiological signal using a sliding window algorithm to obtain multiple data windows; Determine N data windows as a voting group to obtain multiple voting groups, where N is an integer greater than zero and less than the total number of data windows; Using the annotation model to classify the N data windows in each voting group, and obtaining classification results of the N data windows in the corresponding voting group; For any of the voting groups, if there is a target classification result among the classification results of the N data windows in the voting group, the target classification result is determined as the label of the voting group, and the target classification result is the classification result whose number is greater than a preset number among the classification results of the N data windows in the corresponding voting group; Accordingly, determining a second accuracy rate of the classification model based on the second physiological signal and the corresponding label includes: Determining a second accuracy of the classification model based on N data windows and corresponding labels in each voting group of M voting groups that are consecutive in acquisition time, where M is an integer greater than 1 and less than or equal to the total number of voting groups; Accordingly, updating the classification model based on the second physiological signal and the corresponding label includes: In the case where the labels of the M voting groups are the same, the classification model is updated based on the N data windows and the corresponding labels in each of the M voting groups.

5. The classification model updating method according to claim 4, characterized in that: The updating of the classification model based on the N data windows and corresponding labels in each of the M voting groups includes: The classification model is updated based on the N data windows and corresponding labels in each of the M voting groups, the first physiological signal, and offline training samples, where the offline training samples are samples used to pre-train the classification model before the first session.

6. The classification model updating method according to claim 5, characterized in that: The classification model is a CNN-LSTM model, which includes a CNN and an LSTM network; updating the classification model based on the N data windows and corresponding labels in each of the M voting groups, the first physiological signal, and the offline training sample, includes: The LSTM network is updated based on the N data windows and corresponding labels in each of the M voting groups, the first physiological signal, and the offline training samples.

7. The classification model updating method according to any one of claims 1 to 6, characterized in that: Before acquiring the second physiological signal of the user in the current session, the method further includes: Training the annotation model based on the first physiological signal; The classification model is calibrated based on the first physiological signal.

8. A classification model updating device, characterized in that: include: A signal acquisition module, used to acquire a first physiological signal of a user in a current session, where the first physiological signal is a physiological signal marked with a classification label and collected by the signal acquisition device when the user is wearing the signal acquisition device; An accuracy determination module, used to determine a first accuracy of a classification model based on the first physiological signal; a model updating module, configured to obtain a second physiological signal of the user in the current session if the first accuracy rate is less than a first threshold, and update the classification model based on the second physiological signal and the annotation model, wherein the second physiological signal is an unlabeled physiological signal collected by the signal acquisition device when the user is wearing the signal acquisition device; The session processing module is used to determine the next session of the user as the current session, and return to execute the signal acquisition module and subsequent modules until the first accuracy rate is greater than or equal to the first threshold.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the electronic device implements the classification model updating method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which enables the classification model updating method according to any one of claims 1 to 7 to be executed when the computer program is executed.

Images