Finger force estimation model training method, device and program product

Abstract

The invention provides a training method and device of a finger force estimation model, equipment, a storage medium and a computer program product, and relates to the technical field of computers. The method comprises the following steps: acquiring multi-user training data; the multi-user training data comprises surface myoelectricity data and finger force data of multiple users in the first user set; based on the multi-user training data, performing model training on the initial model for finger force estimation to obtain a finger force estimation model; obtaining new user training data; and based on a transfer learning strategy of parameter freezing, migrating a first part of parameters learned in a model training process based on the multi-user training data to a model training process based on the new user training data to obtain a finger force estimation model in which a second part of parameters are calibrated. According to the method, the problem that a finger force estimation model is poor in adaptability and generalization ability to new users in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular, the present application relates to a finger force estimation model training method and device, electronic equipment, storage medium and computer program product. BACKGROUND

[0002] The hand is one of the important media for human interaction with the outside world, and through the dexterous control of the hand, the environment of the outside world can be changed. However, unexpected events such as accidents or diseases in life can cause the user to lose part or even all of the hand and wrist function, which brings great inconvenience to the user's life. The surface electromyography data driven control of the exoskeleton rehabilitation hand is an important means to help the hand function disorder user to rebuild the motor function, and the core technology of this means is the hand action intention recognition, specifically, the finger force continuous motion estimation based on surface electromyography data.

[0003] Due to the physiological and anatomical differences of different users, the electromyography interface of multiple user finger force continuous motion training cannot directly adapt to the hand action intention recognition of new users and new actions, and in actual application, it shows serious user dependence. Although this problem of user dependence can be solved by using a large amount of user data to train a finger force estimation model with stronger generalization ability, the collection and labeling of user data are very tedious and time-consuming. Therefore, how to improve the generalization performance of the finger force estimation model in the cross-user and cross-action scene needs to be solved. SUMMARY

[0004] The present application provides a finger force estimation model training method, device, electronic equipment and storage medium, which can solve the problem of poor adaptability and generalization ability of the finger force estimation model in related technologies. The technical solution is as follows:

[0005] According to one aspect of the present application, a finger force estimation model training method, the method comprises: acquiring multi-user training data; the multi-user training data comprises surface electromyography data and finger force data of multiple users in a first user set; based on the multi-user training data, an initial model for finger force estimation is trained to obtain a finger force estimation model; acquiring new user training data; the new user training data comprises surface electromyography data and finger force data of multiple users in a second user set; based on a parameter frozen transfer learning strategy, a first part of parameters learned in the model training process based on the multi-user training data is transferred to the model training process based on the new user training data to obtain the finger force estimation model with second part of parameters completing calibration.

[0006] According to an aspect of the present application, a device for training a finger force estimation model, the device comprising: a first data acquisition module configured to acquire multi-user training data; the multi-user training data comprising surface electromyography data and finger force data of a plurality of users in a first user set; a parameter training module configured to train an initial model for finger force estimation based on the multi-user training data to obtain a finger force estimation model; a second data acquisition module configured to acquire new user training data; the new user training data comprising surface electromyography data and finger force data of a plurality of users in a second user set; and a parameter calibration module configured to migrate, based on a parameter frozen transfer learning strategy, a first part of parameters learned in a model training process based on the multi-user training data to a model training process based on the new user training data to obtain a second part of parameters of the finger force estimation model after calibration.

[0007] According to an aspect of the present application, an electronic device comprising at least one processor and at least one memory, wherein the memory has stored thereon a computer program which, when executed by the processor, implements the method for training a finger force estimation model as described above.

[0008] According to an aspect of the present application, a storage medium having stored thereon a computer program which, when executed by one or more processors, implements the method for training a finger force estimation model as described above.

[0009] According to an aspect of the present application, a computer program product comprising a computer program which, when executed by one or more processors, implements the method for training a finger force estimation model as described above.

[0010] The technical solutions provided by the present application have the beneficial effects that:

[0011] In the technical solution, first, multi-user training data is acquired to perform model training on an initial model for finger force estimation based on the multi-user training data to obtain a finger force estimation model, then new user training data is acquired to perform model training based on a parameter frozen transfer learning strategy to transfer a first part of parameters learned in the model training based on the multi-user training data to the model training based on the new user training data, and finally, a second part of parameters are obtained to complete calibration of the finger force estimation model. Since the multi-user training data refers to surface electromyography and finger force data of multiple users in a first user set, the new user training data refers to surface electromyography and finger force data of multiple users in a second user set, the first user set is different from the second user set, that is, in the training process of the finger force estimation model, not only the hand movements of the multiple users in the first user set are learned, but also the learned knowledge is transferred to the learning process of the hand movements of the multiple users in the second user set based on the parameter frozen transfer learning strategy. Not only is the problem of decreased accuracy of finger force continuous movement estimation when a new user is encountered solved, but also when a new user is encountered in the subsequent application process, model training does not need to be performed again, which is beneficial to shorten the model training time and improve the model training efficiency, thereby effectively solving the problem of poor adaptability and generalization ability of the finger force estimation model when facing a new user in the related art. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0013] Figure 1 is a schematic diagram according to the implementation environment involved in the present application;

[0014] Figure 2 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0015] Figure 3 is a flowchart of a training method of a finger force estimation model according to an exemplary embodiment;

[0016] Figure 4 is a specific implementation schematic diagram of a data preprocessing process according to an exemplary embodiment;

[0017] Figure 5 is a specific implementation schematic diagram of the network structure of a feature extraction module according to an exemplary embodiment;

[0018] Figure 6is a specific implementation schematic diagram of a network structure of a finger force estimation model according to an exemplary embodiment;

[0019] Figure 7 is a specific implementation schematic diagram of a secondary model training process of a finger force estimation model according to an exemplary embodiment;

[0020] Figure 8 is a flowchart of a regression performance evaluation method of a finger force estimation model according to an exemplary embodiment;

[0021] Figure 9 is a flowchart of an application method of a finger force estimation model according to an exemplary embodiment;

[0022] Figure 10 is a structural block diagram of a training device of a finger force estimation model according to an exemplary embodiment;

[0023] Figure 11 is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0024] Embodiments of the present application are described in detail below with reference to the attached drawings, which show by way of example, embodiments in which like reference numerals refer to like elements or elements having the same function throughout the various figures. The embodiments described below are examples only, and are not to be construed as limiting the present application.

[0025] It should be understood by those skilled in the art that, unless specifically stated otherwise, the singular forms "a," "an," and "the" as used herein are intended to include plural forms as well. It should be further understood that the terms "includes", "including", "comprises" and / or "comprising", when used in the specification, mean that there are other features, integers, steps, operations, elements, and / or components that are present, but not specifically named and / or are not present. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intervening elements. In addition, "connected" or "coupled" as used herein can include wireless connection or wireless coupling. The term "and / or" as used herein includes all or any combination of one or more of the associated listed items.

[0026] As described above, due to physiological and anatomical differences of different users, the myoelectric interface of multiple user multiple finger force continuous movement training cannot directly adapt to the hand motion intention recognition of new users and new actions, and in actual application, it shows serious user dependence. Although this problem can be solved by training a finger force estimation model with stronger generalization ability using a large amount of user data, the collection and labeling of user data are very tedious and time-consuming.

[0027] In order to solve the above problems, a deep learning model is used to improve the generalization performance of the model. However, the general deep learning model generally has problems such as large amount of calculation, need of large amount of computing power support, complex model design, etc., and the development of a new model needs to invest a lot of manpower, material resources and time, which will limit the deployment of deep learning model in practical application.

[0028] Further, in order to improve the generalization ability of the deep learning model when facing cross users, the following two schemes are proposed for model parameters:

[0029] The first scheme is to adjust the adaptability of the model through parameter retraining. Although the application scene of fine-tuning is relatively wide, it is easy to cause overfitting and catastrophic forgetting problems, especially when the labeled data in the target domain is scarce, which may cause the loss of information learned in the source domain, affecting the reusability of the model.

[0030] The second scheme is domain adaptation, which aligns the feature spaces of the source domain and the target domain to extract common latent features, so as to achieve more robust cross-user generalization. Similarly, domain adaptation performs poorly when facing significant individual differences, especially when dealing with physiological and motion signals that are significantly affected by individual differences. The general domain adaptation strategy may not achieve the desired results. In addition, the requirement of domain adaptation method for unlabeled data of target user in offline training stage also increases the inconvenience in practical application.

[0031] As can be seen from the above, the related art still has the defect that the finger force estimation model has poor adaptability and generalization ability when facing new users.

[0032] Therefore, the training method of the finger force estimation model provided by the present application can effectively improve the adaptability and generalization ability of the finger force estimation model when facing new users. Correspondingly, the training method of the finger force estimation model is applicable to a training device of the finger force estimation model, and the training device of the finger force estimation model can be deployed in an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, for example, the computer device includes a desktop computer, a notebook computer, a server, etc.

[0033] In order to make the purpose, technical scheme and advantages of the present application more clear, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0034] Figure 1 A schematic diagram of an implementation environment related to an image processing method. It should be noted that this implementation environment is only an example adapted to the present application and should not be considered as providing any limitation on the scope of use of the present application.

[0035] The implementation environment includes a collection end 110, a service end 130, and a user end 150.

[0036] Specifically, the collection end 110 is configured to collect surface electromyography data and finger force data of a user. In a specific embodiment, the collection end 110 includes a finger force sensor, and the user uses one or more fingertips of a dominant hand to press the finger force sensor in nine different force modes, the action dimension covers six degrees of freedom, including five finger bending and thumb abduction, and each action is repeated 6 times, and the maximum and minimum values of the finger force are recorded during the movement of the user's hand to obtain the finger force data of the user; the collection end 110 can also include a 12-channel electromyography collection device, which can record the surface electromyography signals during the movement of the user's hand to obtain the surface electromyography data of the user, which is used for subsequent training of a finger force estimation model, regression performance evaluation, and finger force estimation.

[0037] The service end 130 can be a desktop computer, a notebook computer, a server, or the like electronic device, and can also be a computer cluster composed of multiple servers, or even a cloud computing center composed of multiple servers. The service end 130 is configured to provide background services, for example, the background services include but are not limited to a finger force estimation service calling a finger force estimation model, and the like.

[0038] The service end 130 and the collection end 110 pre-establish a network communication connection through wired or wireless means, and realize data transmission between the service end 130 and the collection end 110 through the network communication connection. The transmitted data includes but is not limited to a finger force estimation model, target user data, and the like.

[0039] The user end 150 can be a desktop computer, a notebook computer, a smart phone, or the like electronic device that can deploy a finger force estimation model.

[0040] In an application scenario, first, the collection end 110 collects surface electromyography data and finger force data of different users, and respectively constructs a first user set and a second user set based on the collected data; then, through the interaction between the collection end 110 and the service end 130, the first user set and the second user set are uploaded to the service end 130 to request the service end 130 to train a finger force estimation model.

[0041] For the service end 130, after receiving the first user set and the second user set, the training of the finger force estimation model can be completed based on the multi-user training data and the new user training data. The training process can include the following steps: obtaining multi-user training data, the multi-user training data including surface electromyography data and finger force data of multiple users in the first user set; based on the multi-user training data, model training is performed on the initial model for finger force estimation to obtain a finger force estimation model; obtaining new user training data, the new user training data including surface electromyography data and finger force data of multiple users in the second user set; the second user set is different from the first user set; based on the parameter frozen transfer learning strategy, the first part of parameters learned in the model training process based on the multi-user training data is migrated to the model training process based on the new user training data to obtain the second part of parameters to complete the calibrated finger force estimation model.

[0042] Further, after the finger force estimation model is completed, the service end 130 can actually deploy it, for example, deploying the finger force estimation model to the user end 150.

[0043] Then, for the user end 150, after the deployment of the finger force estimation model, the finger force estimation model can be called to perform finger force estimation on the target user data, and finally obtain the finger force estimation value of the target user. The target user data includes surface electromyography data and finger force data of the target user.

[0044] It should be noted that the user end 150 can itself be an electronic device with the function of collecting target user data, for example, the electronic device can be a notebook computer configured with a finger force sensor, or the user end 150 can be pre-established a network communication connection with the collection end 110 through wired or wireless means, and obtain the target user data collected by the collection end 110 through the network communication, and the application scenario is not limited in this regard.

[0045] Of course, according to actual operation requirements, the target user data can also be directly uploaded to the service end 130 by the collection end 110 to request the service end 130 to call the finger force estimation service to perform finger force estimation on the target user data, and then the service end 130 sends the finger force estimation value of the target user to the user end 150, thereby completing the finger force estimation of the target user, which is not a specific limitation.

[0046] Please refer to Figure 2 , Figure 2 is a hardware structure diagram of an electronic device according to an exemplary embodiment. The electronic device can be suitable for Figure 1 the service end 130 and the user end 150 in the implementation environment shown in the figure.

[0047] It should be noted that the electronic device is only an example adapted to the present application and should not be considered as providing any limitation to the scope of the present application. The electronic device should also not be interpreted as requiring dependency or necessity of Figure 2 one or more components in the exemplary electronic device 200.

[0048] The hardware structure of the electronic device 200 can vary greatly due to different configurations or performances, such as Figure 2 As shown, the electronic device 200 includes a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0049] Specifically, the power supply 210 is configured to provide operating voltage for each hardware device on the electronic device 200.

[0050] The interface 230 includes at least one wired or wireless network interface 231 configured to interact with external devices. For example, the network interface 231 can be configured to perform Figure 1 As shown, the interaction between the collection end 110 and the service end 130 in the implementation environment, or the interaction between the collection end 110 and the user end 150 in the implementation environment. Figure 1 As shown, the interaction between the collection end 110 and the service end 130 in the implementation environment, or the interaction between the collection end 110 and the user end 150 in the implementation environment.

[0051] Of course, in other examples adapted to the present application, the interface 230 can further include at least one serial-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc., as shown in Figure 2 which is not specifically limited herein.

[0052] The memory 250, as a carrier for resource storage, can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon include an operating system 251, an application program 253, and data 255, etc., and the storage mode can be temporary storage or permanent storage.

[0053] The operating system 251 is configured to manage and control each hardware device on the electronic device 200 and the application program 253, so as to realize the operation and processing of the central processing unit 270 on the mass data 255 in the memory 250, and can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0054] The application program 253 is a computer program formed by computer readable instructions for completing at least one specific work based on the operating system 251, and can include at least one module (e.g., a browser, a media player, a word processor, a spreadsheet, a game, etc.). Figure 2Each module can include corresponding computer readable instructions, respectively (not shown). For example, the training device of the finger force estimation model can be regarded as an application 253 deployed in the electronic device 200.

[0055] The data 255 can be photos, pictures, etc. stored in a disk, and can also be target user data, a first user set, a second user set, a finger force estimation model, etc. stored in the memory 250.

[0056] The central processing unit 270 can include one or more processors, and is configured to communicate with the memory 250 through at least one communication bus, to read the computer program stored in the memory 250, and to realize the operation and processing of the mass data 255 in the memory 250. For example, the training method of the finger force estimation model is completed by reading the application 253 stored in the memory 250 through the central processing unit 270.

[0057] In addition, the present application can also be realized by hardware circuit or hardware circuit combined with software, and therefore, the realization of the present application is not limited to any specific hardware circuit, software and combination of the two.

[0058] Please refer to Figure 3 The embodiment of the present application provides a training method of a finger force estimation model, which is suitable for an electronic device. For example, the electronic device can be Figure 1 The hardware structure of the electronic device can be as shown in Figure 2 .

[0059] In the following method embodiment, in order to facilitate description, the execution subject of each step of the method is taken as an example for description, but this does not constitute a specific limitation.

[0060] As shown in Figure 3 , the method can include the following steps:

[0061] Step 310, obtaining multi-user training data.

[0062] The multi-user training data includes surface electromyography data and finger force data of a plurality of users in a first user set.

[0063] As described above, the surface electromyography data driven control exoskeleton rehabilitation hand is an important means to help users with hand dysfunction to rebuild motor function, and the core technology of this means is hand action intention recognition, specifically, finger force continuous motion estimation based on surface electromyography data. In the embodiment, the surface electromyography data and finger force data of a plurality of users are collected by using a finger force sensor.

[0064] In a possible implementation, the surface electromyography data is from the finger extensor, finger flexor, biceps brachii, triceps brachii, and a circle of muscles 2-6 cm away from the elbow of the user, and is collected by an electromyography collection device at a preset sampling frequency (such as 2000 Hz).

[0065] In a possible implementation, the finger force data is from the fingertips of one or more fingers of the user's dominant hand, and is obtained by pressing the finger force sensor in nine different force modes, with the action dimension covering six degrees of freedom, including five finger flexion and thumb abduction, and each action being repeated 6 times, and the maximum and minimum values of the finger force during the pressing are recorded.

[0066] As for the acquisition of the surface electromyography data and the finger force data of multiple users, the data can be acquired in real time after being collected by the finger force sensor, or can be acquired after being pre-stored after being collected by the finger force sensor. It can also be understood that the surface electromyography data and the finger force data of multiple users can be a set of real-time data collected by the finger force sensor, or a set of historical data stored by the finger force sensor, and the present embodiment does not limit this.

[0067] After the surface electromyography data and the finger force data of multiple users are acquired, a first user set can be constructed, and the surface electromyography data and the finger force data of multiple users in the first user set can be used for model training.

[0068] Further, before using the multi-user training data for model training, the multi-user training data needs to be pre-processed to improve the data quality and the input effect of the model, so as to improve the finger force estimation performance of the model.

[0069] In a possible implementation, Figure 4 The process of pre-processing the surface electromyography data and the finger force data is shown, and the process of pre-processing the data can include the following steps:

[0070] Firstly, the surface electromyography data is denoised, and the denoising includes but is not limited to: using a fourth-order Butterworth band-pass filter to band-pass filter the surface electromyography data, using baseline correction to remove noise of the surface electromyography data, using a 50 Hz notch filter to remove power frequency interference in the surface electromyography data, using a logarithmic scaling algorithm of u-law conversion to amplify the signal to more effectively extract useful information in the surface electromyography data, and the like. Of course, in other embodiments, using the logarithmic scaling algorithm of u-law conversion to amplify the signal can also be considered as signal enhancement processing of the surface electromyography data, which is not specifically limited here.

[0071] Secondly, the denoised sEMG data and the finger force data are normalized respectively to obtain intermediate data of multiple users. The normalization includes but is not limited to root mean square normalization, minimum value normalization, maximum value normalization, etc.

[0072] Thirdly, the intermediate data of multiple users is segmented by setting a sliding window with a window length to obtain multi-channel data of multiple users. That is, the intermediate data of multiple users is segmented into more multi-user training data by setting a sliding window with a window length and sliding at a set distance, which enriches the training data of the model and improves the performance of the model in estimating finger force.

[0073] It is explained that the window length and the set distance of the sliding window can be flexibly adjusted according to the actual needs of the application scenario. For example, in a specific embodiment, the window length of the sliding window is 400 ms and the set distance is 10 ms, which is not a specific limitation.

[0074] In this way, the data quality and the input effect of the model can be effectively improved, which not only improves the performance of the model in estimating finger force, but also effectively improves the operation efficiency of the model.

[0075] Step 330, based on the multi-user training data, the initial model for estimating finger force is trained to obtain a finger force estimation model.

[0076] That is, through the model training based on the multi-user training data, the initial model for estimating finger force converges to the finger force estimation model, which can reflect the mapping relationship between the sEMG signal and the 6-DOF finger force, so as to realize the continuous estimation of the 6-DOF finger force continuous motion. The initial model can be a deep learning model, for example, the deep learning model includes but is not limited to a deep convolutional neural network, a dual-channel convolutional neural network, a Transformer neural network, etc.

[0077] In one possible implementation, the finger force estimation model includes an input module for inputting multi-user training data, a feature extraction module for outputting feature maps, and a full connection module for outputting finger force estimation values.

[0078] In one possible implementation, the feature extraction module can be implemented by a lightweight deep learning model, for example, the feature extraction module can be implemented by a MobileNetV2 model. Taking the MobileNetV2 model as an example, Figure 5The specific network structure of the feature extraction module is shown, which can include multiple reverse residual blocks, each reverse residual layer can include an expansion layer using 1x1 convolution, a depth separable layer using 3x3 convolution, and a projection layer using 1x1 convolution. Among them, the expansion layer is used to expand the channel number of the multi-user training data to map the multi-user training data to a higher dimensional space; the depth separable layer is used to extract spatial features channel by channel to map the multi-user training data to a low-dimensional space, which not only can better extract spatial features and retain more useful information, but also can effectively reduce the amount of calculation; the projection layer is used to compress the channel number to map the multi-user training data back to the original dimension, so as to avoid the information loss caused by the nonlinear activation in the low dimension. In this way, not only can the depth separable convolution idea be continued to decompose the traditional convolution into depth convolution and point-by-point convolution, which significantly reduces the computational demand and parameter size of the model, but also adding residual connection between some layers will help the gradient flow and alleviate the gradient vanishing problem.

[0079] Further, in order to improve the performance of the model in estimating the finger force, in one possible implementation, an attention mechanism layer is embedded in each reverse residual block. In one possible implementation, the attention mechanism layer can be implemented by a lightweight attention mechanism module, for example, the attention mechanism layer can be implemented by a CBAM module. Taking the CBAM module as an example, Figure 6 The specific network structure diagram of the feature extraction module containing the CBAM module is shown, and the attention mechanism layer is connected between the depth separable layer and the projection layer. In this way, by introducing the attention mechanism in the feature extraction process, the feature representation ability of the model can be significantly improved without significantly increasing the computational complexity, which is conducive to improving the performance of the model in estimating the finger force.

[0080] Step 350, obtaining new user training data.

[0081] Among them, the new user training data includes the surface electromyography data and the finger force data of multiple users in the second user set, and the second user set is different from the first user set.

[0082] Similarly, after obtaining the surface electromyography data and the finger force data of multiple users, a second user set different from the first user set can be constructed, and the surface electromyography data and the finger force data of multiple users in the second user set can be used for model calibration and regression performance evaluation.

[0083] Further, before using the new user training data to calibrate the model, the new user training data also needs to be preprocessed to improve the data quality and the input effect of the model, so as to improve the performance of the model in estimating the finger force. It should be noted that the data preprocessing process of the new user training data is basically the same as the data preprocessing process of the multi-user training data described above, and the essence of both is to preprocess the surface electromyography data and the finger force data of the user, which will not be described here.

[0084] In a specific embodiment, the surface electromyography data and the finger force data of a total of 35 users are collected, of which 28 users are used to construct a first user set for model training of the finger force estimation model, with 6 repetitions of each action; and the remaining 7 users are used to construct a second user set for model calibration and regression performance evaluation of the finger force estimation model, with 6 repetitions of each action.

[0085] Step 370, based on the parameter freezing transfer learning strategy, migrates the first part of parameters learned in the model training process based on the multi-user training data to the model training process based on the new user training data to obtain the second part of parameters to complete the calibrated finger force estimation model.

[0086] The parameter migration process implemented by the parameter freezing transfer learning strategy refers to completely freezing the first part of parameters learned in the first model training (related to the multi-user training data), so that the first part of parameters no longer participates in the model training in the second model training (related to the new user training data), and only the second part of parameters is updated in the second model training, so that the finger force estimation model not only retains the information obtained from the old users in the first user set in the first model training, but also adapts to the new users in the second user set, which not only improves the generalization ability of the model, but also reduces the risk of overfitting.

[0087] In a possible implementation, step 370 can include the following steps: based on the parameter freezing transfer learning strategy, storing the first part of parameters learned in the model training process based on the multi-user training data; based on the new user training data, calibrating the second part of parameters of the finger force estimation model with the stored first part of parameters to obtain the second part of parameters to complete the calibrated finger force estimation model.

[0088] In a possible implementation, the first part of parameters refers to the model parameters related to the feature extraction module; and the second part of parameters refers to the model parameters related to the full connection module.

[0089] Figure 7 The process of the finger force estimation model for secondary model training is shown as follows: Figure 7As shown, in the first model training process, all model parameters of the finger force estimation model participate in the model training, and in the second model training process, the first part of the parameters no longer participates in the model training, and only the second part of the parameters is updated, and finally the finger force estimation model after model calibration is obtained.

[0090] Through the above process, in the training process of the finger force estimation model, not only the hand movements of multiple users in the first user set are learned, but also the learned knowledge is transferred to the learning process of the hand movements of multiple users in the second user set based on the parameter freezing transfer learning strategy. Not only does it solve the problem of decreased accuracy of finger force continuous motion estimation when encountering new users, but also does not need to retrain the model when encountering new users in the subsequent application process, which is beneficial to shorten the model training time and improve the model training efficiency, thereby effectively solving the problem of poor adaptability and generalization ability of the finger force estimation model in the related art when facing new users.

[0091] Please refer to Figure 8 In an example embodiment, after step 370, the method can further include the following steps:

[0092] Step 410, obtaining new user test data.

[0093] The new user test data includes surface electromyography data and finger force data of multiple users in the second user set.

[0094] Step 430, based on the new user test data, calculating the Pearson correlation coefficient, the root mean square error and the determination coefficient of the finger force estimation model.

[0095] Step 450, according to the Pearson correlation coefficient, the root mean square error and the determination coefficient of the finger force, the regression performance of the finger force estimation model is evaluated.

[0096] In this embodiment, the Pearson correlation coefficient, the root mean square error and the determination coefficient of the finger force are used as the evaluation criteria for the regression performance evaluation of the finger force estimation model. It should be understood that when the finger force estimation model meets the evaluation criteria, the better the finger force estimation performance of the finger force estimation model, the higher the accuracy of the finger force estimation value.

[0097] Specifically, the regression performance evaluation process can include the following steps:

[0098] 1) Pearson correlation coefficient (CC), CC is a statistical quantity for quantifying the strength and direction of the linear relationship between two continuous variables, which measures the tightness of the data points around the straight line in the scatter plot. It can show the closeness of the hand kinematics continuous estimation value and the corresponding true value. The following is the calculation formula of CC:

[0099]

[0100] The value of CC ranges between -1 and 1, the closer the value of CC is to 1, the stronger the correlation is, the closer the predicted finger motion trajectory is to the actual trajectory, the better the finger force estimation performance of the finger force estimation model is, and the higher the accuracy of the finger force estimation value is.

[0101] 2) Root Mean Square Error (RMSE), RMSE is a commonly used index to evaluate the accuracy of a prediction model, and is usually used to measure the average error size between the continuous estimation value of hand kinematics and the corresponding actual value, the smaller the value is, the better the finger force estimation performance of the finger force estimation model is. The following is the calculation formula of RMSE:

[0102]

[0103] 3) Determination coefficient (R2), R2 is a kind of statistical quantity used to evaluate the fitting degree of a regression model. As a comprehensive evaluation index to measure the overall accuracy of the model, R2 ranges between 0 and 1, and it is defined as the percentage of the change of the actual value explained by the estimation value. The larger the value of R2 is, the better the finger force estimation performance of the finger force estimation model is. The following is the calculation formula of R2:

[0104]

[0105] In the above formulas (1) to (3), N represents the number of users in the new user test data, y pred 、 y real 、 are the estimation value of finger force, the estimation average value of finger force, the actual value of finger force and the actual average value of finger force, respectively.

[0106] Therefore, if the regression performance evaluation fails, it indicates that the current finger force estimation model does not meet the evaluation standard, and the accuracy of the finger force estimation value is not high enough, then returning to step 310 to reacquire the multi-user training data to continue the model training of the finger force estimation model based on the reacquired multi-user training data.

[0107] On the contrary, if the regression performance evaluation succeeds, it indicates that the current finger force estimation model has met the evaluation standard, and the accuracy of the finger force estimation value is high enough, then entering step 450 to obtain the trained finger force estimation model.

[0108] Step 310, acquire multi-user training data to continue the model training of the finger force estimation model.

[0109] Step 470, obtain the trained finger force estimation model.

[0110] That is, for the finger force estimation model evaluated by the regression performance, the mapping relationship between the surface electromyogram signal and the finger force of 6 degrees of freedom can be accurately reflected, so as to have the ability to accurately predict the finger force estimation value. Therefore, after obtaining the surface electromyogram data and the finger force data of the target user, the finger force estimation model evaluated by the regression performance can be called to perform finger force estimation on the surface electromyogram data and the finger force data of the target user, so as to realize continuous estimation of the continuous motion of the finger force of 6 degrees of freedom of the target user.

[0111] Please refer to Figure 9 The embodiment of the present application provides a method for applying a finger force estimation model. The method is suitable for an electronic device. For example, the electronic device can be a Figure 1 As shown in the user end 150 in the implementation environment, the hardware structure of the electronic device can be as shown in Figure 2 .

[0112] In the following method embodiment, in order to facilitate description, the execution subject of each step of the method is taken as an example to be an electronic device, but this does not constitute a specific limitation.

[0113] As shown in Figure 9 The method can include the following steps:

[0114] The finger force estimation model is called to perform finger force estimation on the target user data, and the finger force estimation value of the target user is obtained.

[0115] The target user data includes surface electromyogram data and finger force data of the target user.

[0116] It is explained that the target user data is not only different from the multi-user training data and the new user training data, but also different from the new user test data, that is, the target user is different from any one of the first user set and the second user set. For the finger force estimation model, the finger force estimation of the target user based on the target user data is equivalent to continuous estimation of the multi-degree-of-freedom finger force of a new user.

[0117] The process of continuous estimation of the multi-degree-of-freedom finger force of the new user can include the following steps:

[0118] In step 510, the multi-channel data obtained by preprocessing the target user data is transmitted to the finger force estimation model through the input module.

[0119] Please refer to Figure 6 , Figure 6The network structure of the finger force estimation model is shown, which includes an input module, a feature extraction module, and a full connection module (i.e., an output module). The feature extraction module can include multiple reverse residual layers, each of which can further include an expansion layer, a depth separable layer, an attention mechanism layer (i.e., CBAM), and a projection layer.

[0120] As shown in Figure 6 After the target user data is preprocessed, the processed multi-channel data will be input to the finger force estimation model for subsequent finger force estimation.

[0121] It should be noted that the data preprocessing process of the target user data is basically the same as the data preprocessing process of the multi-user training data described above, and the essence is to preprocess the surface electromyography data and finger force data of the user, which will not be described in detail here. The preprocessing process can include the following steps: denoising the surface electromyography data, and normalizing the denoised surface electromyography data and finger force data respectively to obtain intermediate data of the target user; by setting a sliding window of a window length, the intermediate data of the target user is segmented to obtain multi-channel data of the target user.

[0122] In step 530, the feature map of the multi-channel data is extracted through each reverse residual block in the feature extraction module, and the attention mechanism layer embedded in each reverse residual block enhances the attention to important regions in the feature map during feature extraction, to obtain an enhanced feature map.

[0123] Continuing to refer to Figure 6 In one possible implementation, the reverse residual layer includes an expansion layer, a depth separable layer, and a projection layer. In combination with the network structure of the reverse residual layer shown in Figure 6 The feature map extraction process can include the following steps: inputting the multi-channel data to the expansion layer for channel expansion; in the depth separable layer, based on the channel number of the expanded multi-channel data, the spatial features of the multi-channel data are extracted channel by channel to obtain the feature map corresponding to the multi-channel data; and inputting the feature map to the projection layer for channel compression.

[0124] Continuing to refer to Figure 6 In one possible implementation, the attention mechanism layer is connected between the depth separable layer and the projection layer. In combination with Figure 6The network structure of the attention mechanism layer in the reverse residual layer is shown. The feature map enhancement process can include the following steps: first, in the attention mechanism layer, a channel attention mechanism is introduced to extract the global information of each channel in the feature map, and channel attention weights of each channel are generated by mapping the global information in different dimensions. The mapping specifically includes mapping to a high-dimensional space first, then mapping to a low-dimensional space, and finally mapping back to the original dimension. Second, the channel attention weights of each channel are multiplied with the feature map channel by channel to obtain the enhanced feature map. Third, a spatial attention mechanism is introduced, and a spatial attention map is obtained by pooling processing the enhanced feature map in different dimensions. The pooling processing specifically includes maximum pooling and average pooling on the channel dimension of the enhanced feature map to generate two two-dimensional spatial feature maps, and then a 1x1 convolution layer is used to generate a spatial attention map from the two two-dimensional spatial feature maps. Fourth, the spatial attention map is multiplied with the enhanced feature map pixel by pixel to obtain the enhanced feature map.

[0125] That is, as shown in the finger force estimation model, Figure 6 After the feature extraction module receives the multi-channel data, the multi-channel data is transmitted through the expansion layer, the depth separable layer, the attention mechanism layer, and the projection layer in sequence, and finally the enhanced feature map is obtained and input into the fully connected layer for subsequent finger force estimation.

[0126] Step 550, output the enhanced feature map through the fully connected module to obtain the finger force estimation value of the target user.

[0127] Continuing to refer to Figure 6 Through the fully connected layer, the output can be converted into a six-degree-of-freedom finger force regression prediction value, i.e., the finger force estimation value of the target user.

[0128] Through the above process, the finger force estimation of the new user based on the finger force estimation model is realized, which not only brings higher accuracy to the finger force estimation of the new user, but also has better generalization performance for the new user.

[0129] In different application scenarios, for example, the application scenarios can be intelligent prosthesis control, human-computer interaction, and gesture control device, etc., which can provide more intuitive and portable control and interaction methods for users. In the above application scenarios, the surface electromyography signal can be decoded so that amputees can use their own muscle activities to control intelligent prostheses, simulate natural hand movements, improve operation performance, and provide more natural body extension. The finger force estimation model based on the transfer learning strategy can improve the accuracy of estimating the hand state of new actions of new users. Since it does not need to retrain the model, the deployment ability of the model in the actual use process is improved.

[0130] Meanwhile, the finger force estimation model based on the transfer learning strategy can solve the finger force estimation problem of a multi-user multi-degree-of-freedom complex problem, and can be used for state tracking of a patient in a rehabilitation process in the field of rehabilitation medicine, which helps a rehabilitation therapist to better understand the physical state of the patient and to develop a more personalized and more targeted rehabilitation plan, and to more effectively meet the actual requirements of the patient.

[0131] In addition, based on the transfer learning technology, the model pre-trained using the initial data is promoted to a new user, the generalization is improved, and the time for recalibrating the model is greatly reduced, which greatly reduces the burden of the user.

[0132] It should be understood that, although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0133] The following is an apparatus embodiment of the present application, which can be used to execute the training method of the finger force estimation model involved in the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the method embodiment of the training method of the finger force estimation model involved in the present application.

[0134] Please refer to Figure 10 In the embodiments of the present application, a training device 900 of a finger force estimation model is provided, which includes but is not limited to a first data acquisition module 910, a parameter training module 930, a second data acquisition module 950, and a parameter calibration module 970.

[0135] The first data acquisition module 910 is configured to acquire multi-user training data. The multi-user training data includes surface electromyography data and finger force data of multiple users in a first user set.

[0136] The parameter training module 930 is configured to perform model training on an initial model for finger force estimation based on the multi-user training data, to obtain a finger force estimation model.

[0137] The second data acquisition module 950 is configured to acquire new user training data. The new user training data includes surface electromyography data and finger force data of multiple users in a second user set. The second user set is different from the first user set.

[0138] The parameter calibration module 970 is configured to migrate, based on a parameter frozen transfer learning strategy, a first part of parameters learned in a model training process based on multi-user training data to a model training process based on new user training data, to obtain a second part of parameters to complete calibration of the finger force estimation model.

[0139] It should be noted that the training device of the finger force estimation model provided in the above embodiments is only used as an example to illustrate the division of the above functional modules when training the finger force estimation model. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the training device of the finger force estimation model is divided into different functional modules to complete all or part of the above described functions.

[0140] In addition, the training device of the finger force estimation model and the training method of the finger force estimation model provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments, which will not be repeated here.

[0141] Please refer to Figure 11 In the embodiments of the present application, an electronic device 4000 is provided, which can include a desktop computer, a notebook computer, an electronic device, etc.

[0142] In Figure 11 The electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0143] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 can include a channel for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 can be a PCI (Peripheral Component Interconnect, Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture, Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0144] Optionally, the electronic device 4000 can also include a transceiver 4004, which can be used for data interaction, such as data transmission and / or data reception, etc., between the electronic device and other electronic devices. It should be noted that the transceiver 4004 is not limited to one in actual applications, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0145] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. The processor 4001 can implement or execute the various exemplary logical blocks, modules, and circuits described in connection with the disclosure. The processor 4001 can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0146] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store computer programs in the form of instructions or data structures and that can be accessed by the electronic device 4000, but is not limited thereto.

[0147] The memory 4003 stores a computer program, and the processor 4001 can read the computer program stored in the memory 4003 through the communication bus 4002.

[0148] The computer program is executed by one or more processors 4001 to implement the training method of the finger force estimation model in the above embodiments.

[0149] In addition, the present application provides a storage medium, which stores a computer program, and the computer program is executed by one or more processors to implement the training method of the finger force estimation model as described above.

[0150] The embodiment of the application provides a computer program product comprising a computer program executed by one or more processors to implement the training method of the finger force estimation model as described above.

[0151] Compared with the related art, 1) the parameter freezing-based transfer learning strategy improves the new user finger force estimation accuracy, specifically, the parameter freezing-based transfer learning strategy, that is, the pre-trained model is completely frozen, only the full connection layer is trained, the existing knowledge is transferred to the data of the new user, the model is helped to adapt to the new task faster, the updated model parameters and the test data of the new user are combined, so that the continuous estimation of the multi-degree-of-freedom finger force of the new user is realized, and finally the accuracy of the model finger force estimation is improved; 2) the CBAM module and the MobileNetV2 neural network are used to extract multiple information of the surface electromyography signal, specifically, the MobileNetV2 adopts the reverse residual structure and the depth separable convolution, which can significantly reduce the calculation complexity and the model size while maintaining high accuracy; the CBAM can highlight important features and suppress redundant information through the channel attention and spatial attention mechanisms, so as to improve the attention of the model to important areas, reduce the calculation amount and the parameter size, significantly reduce the demand for computing power, and improve the generalization ability of the model, so as to realize accurate identification of the multi-degree-of-freedom finger force; the finger force estimation model based on MobileNetV2+CBAM has obvious advantages in regression performance evaluation, as shown in Table 1, and has high enough finger force estimation accuracy; 3) the parameter freezing-based transfer learning strategy has a short model correction time, which is beneficial to deployment and implementation, specifically, the parameter freezing-based transfer learning strategy of the application does not need to retrain the large model when a new user is encountered in the use process, the information obtained from the multiple users is retained, so that the model can adapt to the data characteristics of the new user faster, the model deployment time is short, and the application efficiency is high.

[0152] Table 1 Regression performance evaluation of different models in the finger force estimation task

[0153]

[0154] The above only describes some embodiments of the application, and it should be pointed out that, for those skilled in the art, without departing from the principles of the application, some improvements and refinements can be made, which should also be regarded as the protection scope of the application.

Claims

1. A training method for a finger force estimation model, characterized in that, The method comprises: obtaining multi-user training data; the multi-user training data comprises surface electromyography data and finger force data of multiple users in a first user set; based on the multi-user training data, model training is performed on an initial model for finger force estimation to obtain a finger force estimation model; obtaining new user training data; the new user training data comprises surface electromyography data and finger force data of multiple users in a second user set; the second user set is different from the first user set; based on a parameter frozen transfer learning strategy, a first part of parameters learned in the model training process based on the multi-user training data is transferred to the model training process based on the new user training data to obtain a second part of parameters to complete calibration of the finger force estimation model.

2. The method of claim 1, wherein, The parameter frozen transfer learning strategy is used to transfer a first part of parameters learned in the model training process based on the multi-user training data to the model training process based on the new user training data to obtain a second part of parameters to complete calibration of the finger force estimation model, which comprises: based on the parameter frozen transfer learning strategy, storing the first part of parameters learned in the model training process based on the multi-user training data; based on the new user training data, calibrating the second part of parameters of the finger force estimation model having stored the first part of parameters to obtain the finger force estimation model having the second part of parameters completed calibration.

3. The method of claim 2, wherein, The finger force estimation model comprises a feature extraction module for outputting a feature map and a fully connected module for outputting a finger force estimation value; the first part of parameters refers to model parameters related to the feature extraction module; and the second part of parameters refers to model parameters related to the fully connected module.

4. The method of claim 1, wherein, The method further comprises: obtaining new user test data; the new user test data comprises surface electromyography data and finger force data of multiple users in the second user set; based on the new user test data, calculating a Pearson correlation coefficient, a root mean square error and a determination coefficient of finger force about the finger force estimation model; performing regression performance evaluation on the finger force estimation model according to the Pearson correlation coefficient, the root mean square error and the determination coefficient of finger force; if the regression performance evaluation fails, returning to the step of obtaining multi-user training data to continue model training of the finger force estimation model.

5. The method according to any one of claims 1 to 4, characterized in that, The finger force estimation model comprises an input module, a feature extraction module and a fully connected module; The feature extraction module comprises multiple reverse residual blocks, and each reverse residual block has an attention mechanism layer embedded therein; The method further comprises: calling the finger force estimation model to perform finger force estimation on target user data to obtain finger force estimation values of a target user; the target user data comprises surface electromyography data and finger force data of the target user; the calling of the finger force estimation model to perform finger force estimation on target user data to obtain finger force estimation values of a target user comprises: The multi-channel data obtained by processing the target user data is transmitted to the finger force estimation model through the input module; The feature extraction module extracts feature maps from the multi-channel data through each of the reverse residual blocks, and enhances the attention to important regions in the feature maps during feature extraction through the attention mechanism layer embedded in each of the reverse residual blocks, to obtain enhanced feature maps; The enhanced feature maps are output through the full connection module, to obtain the finger force estimation value of the target user.

6. The method of claim 5, wherein, The reverse residual layer includes an expansion layer, a depth separable layer, and a projection layer; The feature extraction module extracts feature maps from the multi-channel data through each of the reverse residual blocks, and enhances the attention to important regions in the feature maps during feature extraction through the attention mechanism layer embedded in each of the reverse residual blocks, to obtain enhanced feature maps; The multi-channel data is input into the expansion layer for channel expansion; In the depth separable layer, spatial features of the multi-channel data are extracted channel by channel based on the number of channels of the expanded multi-channel data, to obtain feature maps corresponding to the multi-channel data; The feature maps are input into the projection layer for channel compression.

7. The method of claim 6, wherein, The attention mechanism layer is connected between the depth separable layer and the projection layer; The feature extraction module extracts feature maps from the multi-channel data through each of the reverse residual blocks, and enhances the attention to important regions in the feature maps during feature extraction through the attention mechanism layer embedded in each of the reverse residual blocks, to obtain enhanced feature maps; In the attention mechanism layer, a channel attention mechanism is introduced to extract global information of each channel of the feature maps, and channel attention weights of each channel are generated by mapping the global information in different dimensions; Each channel attention weight is multiplied with the feature maps channel by channel, to obtain enhanced feature maps; A spatial attention mechanism is introduced to obtain a spatial attention map by performing pooling processing on the enhanced feature maps in different dimensions; The spatial attention map is multiplied with the enhanced feature maps pixel by pixel, to obtain the enhanced feature maps.

8. The method of claim 5, wherein, Before the preprocessing of the target user data, the method further includes: Surface electromyography data and finger force data of the target user are collected respectively; The surface electromyography data is denoised, and the denoised surface electromyography data and the finger force data are normalized respectively, to obtain intermediate data of the target user; The intermediate data of the target user is segmented by setting a sliding window of a window length, to obtain multi-channel data of the target user.

9. A device for training a model for estimating a finger force, characterized by The device includes: A first data acquisition module is configured to acquire multi-user training data; the multi-user training data includes surface electromyography data and finger force data of multiple users in a first user set; A parameter training module is configured to train an initial model for finger force estimation based on the multi-user training data, to obtain a finger force estimation model; A second data acquisition module is configured to acquire new user training data; the new user training data includes surface electromyography data and finger force data of multiple users in a second user set; the second user set is different from the first user set; A parameter calibration module is configured to, based on a parameter frozen transfer learning strategy, transfer a first part of parameters learned in a model training process based on the multi-user training data to a model training process based on the new user training data, to obtain a second part of parameters completing calibration of the finger force estimation model.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by one or more processors, implements the training method of the finger force estimation model according to any one of claims 1 to 8.

Images