Force estimation method and device and electronic equipment

Abstract

The invention provides a force estimation method and device, electronic equipment, a storage medium and a computer program product, and relates to the technical field of computers. The method comprises the following steps: acquiring myoelectricity data; inputting the myoelectricity data into a strength estimation model, and performing multi-scale feature extraction and fusion on the myoelectricity data to obtain fusion features corresponding to the myoelectricity data; the force estimation model comprises a ShuffleNetV2 neural network for enhancing information interaction between the features by using a channel splitting and channel rearrangement technology, and an SE module for constructing a relevance dependency relationship between the features by adopting a self-adaptive weight adjustment mechanism; predicting a target strength mode label corresponding to the myoelectricity data based on fusion features corresponding to the myoelectricity data; and obtaining a force estimation result of the target object based on the target force mode label. The method and the device solve the problem of low accuracy of strength estimation in the prior art.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a force estimation method, apparatus, and electronic device. Background Technology

[0002] The hand is one of the key organs for human interaction with the outside world. Individuals can use fine control of their hands to perform various tasks such as environmental manipulation, object control, and information expression. However, in real life, due to accidental injuries, neurological diseases, or muscle dysfunction, some users may experience varying degrees of loss of hand or wrist function, which seriously affects their quality of life and ability to move independently.

[0003] To restore or enhance users' hand function, exoskeleton rehabilitation hand technology based on surface electromyography (sEMG) signal actuation control has received widespread attention in recent years. These systems can acquire residual electromyographic activity signals from users, identify their movement intentions, and drive external mechanical structures to collaboratively complete target movements, thereby achieving assisted movement or rehabilitation training. Hand movement intention recognition is the core component of this technology, and one of its key tasks is continuous estimation of finger strength based on surface electromyography signals.

[0004] However, the number of action types that can be recognized is currently limited, making it difficult to cover diverse hand movement needs. It also exhibits poor robustness and generalization ability in complex human-computer interaction tasks, making it difficult to adapt to different users or different usage scenarios.

[0005] As can be seen from the above, how to improve the accuracy of force estimation still needs to be addressed. Summary of the Invention

[0006] This application provides a force estimation method, apparatus, electronic device, and storage medium, which can solve the problem of low accuracy in force estimation in related technologies. The technical solutions are as follows:

[0007] According to one aspect of this application, a force estimation method includes: acquiring electromyographic (EMG) data; inputting the EMG data into a force estimation model, extracting and fusing multi-scale features from the EMG data to obtain fused features corresponding to the EMG data; the force estimation model includes a ShuffleNetV2 neural network that uses channel splitting and channel rearrangement techniques to enhance information interaction between features, and an SE module that uses an adaptive weight adjustment mechanism to construct the correlation and dependency relationships between features; predicting a target force pattern label corresponding to the EMG data based on the fused features corresponding to the EMG data; and obtaining a force estimation result for the target object based on the target force pattern label.

[0008] According to one aspect of this application, a force estimation device includes: a data acquisition module for acquiring electromyographic (EMG) data; a feature extraction module for inputting the EMG data into a force estimation model, extracting and fusing multi-scale features from the EMG data to obtain fused features corresponding to the EMG data; the force estimation model includes a ShuffleNetV2 neural network that uses channel splitting and channel rearrangement techniques to enhance information interaction between features, and an SE module that uses an adaptive weight adjustment mechanism to construct the correlation and dependency relationship between features; a label assignment module for predicting a target force pattern label corresponding to the EMG data based on the fused features corresponding to the EMG data; and a force estimation module for obtaining a force estimation result of the target object based on the target force pattern label.

[0009] According to one aspect of this application, an electronic device includes at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, implements the force estimation method as described above.

[0010] According to one aspect of this application, a storage medium having a computer program stored thereon, which, when executed by one or more processors, implements the force estimation method as described above.

[0011] According to one aspect of this application, a computer program product includes a computer program that, when executed by one or more processors, implements the force estimation method as described above.

[0012] The beneficial effects of the technical solution provided in this application are:

[0013] In the above technical solution, the introduction of a multi-scale feature extraction and fusion mechanism can fully explore the deep feature correlations between electromyographic data at different time scales, spatial dimensions, and multiple channels, effectively improving the ability of electromyographic data to represent muscle activity states. Furthermore, by analyzing the fused features and assigning target force pattern labels, clear action context information is provided before entering the force estimation stage, reducing feature confusion between different action types and effectively improving estimation accuracy. The force estimation result of the target object is output in combination with the target force pattern label, which can reflect the user's muscle output level in real time and accurately, thereby effectively solving the problem of low accuracy in force estimation in related technologies. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram based on the implementation environment involved in this application;

[0016] Figure 2 This is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0017] Figure 3 This is a flowchart illustrating a force estimation method according to an exemplary embodiment;

[0018] Figure 4 yes Figure 3 A flowchart of step 330 in one embodiment corresponds to the following example;

[0019] Figure 5 yes Figure 3 The specific model structure diagrams of the first and second networks of the ShuffleNetV2 neural network involved in the corresponding embodiments;

[0020] Figure 6 This is a structural block diagram of a force estimation device according to an exemplary embodiment;

[0021] Figure 7 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0022] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0023] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0024] As mentioned earlier, the number of action types that can be recognized is limited, making it difficult to cover diverse hand movement needs. It also exhibits poor robustness and generalization ability in complex human-computer interaction tasks, making it difficult to adapt to different users or different usage scenarios.

[0025] During the actual research and development and experimentation of the exoskeleton rehabilitation hand system based on surface electromyography signals, the inventors found that when conducting performance tests on the finger movement intention recognition and force estimation modules, the existing methods showed a significant decrease in recognition accuracy when faced with complex hand movements, especially collaborative movements involving multiple fingers simultaneously (such as simultaneous bending of the index and little fingers, or a combination of thumb abduction and index finger bending). This resulted in problems such as large deviations in force estimation values ​​and errors in movement discrimination, making it difficult to meet the requirements of actual rehabilitation training or high-precision human-computer interaction.

[0026] Further analysis revealed that the inventors believe the root cause of the problems lies in the fact that existing methods typically ignore the multimodal and multi-scale nonlinear relationship between electromyographic (EMG) signals and finger movements, and fail to fully consider the semantic information of force patterns implied in EMG signals. This results in a lack of effective structural constraints and contextual guidance during model building. Furthermore, existing solutions often use a uniform model to regress all movements, failing to differentiate the signal characteristics for different movement patterns, leading to insufficient model generalization ability and robustness.

[0027] As can be seen from the above, the relevant technologies still suffer from the drawback of low accuracy in force estimation.

[0028] Therefore, the force estimation method provided in this application can effectively improve the accuracy of force estimation. Accordingly, the force estimation method is applicable to a force estimation device, which can be deployed on an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, such as a desktop computer, a laptop computer, a server, etc.

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0030] Figure 1 This is a schematic diagram of an implementation environment involved in a force estimation method. It should be noted that this implementation environment is merely an example adapted to the present invention and should not be considered as providing any limitation on the scope of the invention.

[0031] The implementation environment includes a data acquisition terminal 110 and a server terminal 130.

[0032] Specifically, the acquisition terminal 110 is used to acquire the user's electromyography (EMG) data. In one specific embodiment, the acquisition terminal 110 may further include a 12-channel EMG acquisition device, which can record the surface EMG signals during the user's hand movements to obtain the user's EMG data for subsequent force estimation.

[0033] In another specific embodiment, the acquisition end 110 includes a pressure linear sensor device. The user uses the fingertips of one or more fingers of the dominant hand to press the finger force sensor in nine different force modes. The movement dimension covers six degrees of freedom, including five-finger flexion and thumb abduction, and each movement is repeated 6 times. The maximum and minimum values ​​of finger force are recorded during the user's hand movement to obtain the user's finger force data.

[0034] Server 130 can be an electronic device such as a desktop computer, laptop computer, or server, or it can be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide backend services, such as, but not limited to, force estimation services.

[0035] The server 130 and the acquisition terminal 110 establish a network communication connection in advance via wired or wireless means, and data transmission between the server 130 and the acquisition terminal 110 is realized through this network communication connection. The transmitted data includes, but is not limited to, electromyography data, etc.

[0036] In one application scenario, through the interaction between the acquisition terminal 110 and the server terminal 130, the acquisition terminal 110 collects electromyographic signals to obtain electromyographic data during the process of the target object making a movement, and uploads the electromyographic data to the server terminal 130 to request the server terminal 130 to provide force estimation service.

[0037] For server 130, after receiving the electromyography (EMG) data uploaded by acquisition terminal 110, it calls the automatic coloring service to input the EMG data into the force estimation model. The server then extracts and fuses multi-scale features from the EMG data to obtain the corresponding fused features. The force estimation model includes a ShuffleNetV2 neural network that uses channel splitting and channel rearrangement techniques to enhance information interaction between features, and an SE module that uses an adaptive weight adjustment mechanism to construct the correlation and dependency relationships between features. Based on the fused features corresponding to the EMG data, the server assigns corresponding target force pattern labels to the predicted EMG data. Based on the target force pattern labels, the server obtains the force estimation result for the target object.

[0038] Please see Figure 2 , Figure 2 This is a hardware structure diagram of an electronic device according to an exemplary embodiment. This electronic device is suitable for... Figure 1 The server 130 in the implementation environment is shown.

[0039] It should be noted that this electronic device is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Furthermore, this electronic device should not be interpreted as requiring or depending on any specific feature. Figure 2 One or more components of the exemplary electronic device 200 shown.

[0040] The hardware structure of electronic device 200 can vary significantly due to differences in configuration or performance, such as... Figure 2 As shown, the electronic device 200 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0041] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 200.

[0042] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. For example, to perform... Figure 1 The diagram shows the interaction between the acquisition terminal 110 and the server terminal 130 in the implementation environment.

[0043] Of course, in other examples adapted in this application, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 2 As shown, this does not constitute a specific limitation.

[0044] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0045] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 200, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0046] Application 253 is a computer program formed by computer-readable instructions based on operating system 251 to perform at least one specific task, and may include at least one module ( Figure 2 (Not shown), each module can contain corresponding computer-readable instructions. For example, the force estimation device can be considered as an application 253 deployed on electronic device 200.

[0047] Data 255 can be photos, pictures, etc. stored on a disk, or electromyography data, etc., stored in memory 250.

[0048] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer programs stored in the memory 250, thereby performing calculations and processing on massive amounts of data 255 stored in the memory 250. For example, a force estimation method may be implemented by the central processing unit 270 reading an application program 253 stored in the memory 250.

[0049] Furthermore, this application can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or combination thereof.

[0050] Please see Figure 3 This application provides a force estimation method, which is applicable to electronic devices, for example, the electronic device may be... Figure 1 The server 130 in the implementation environment is shown. The hardware structure of this electronic device can be as follows: Figure 2 As shown.

[0051] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0052] like Figure 3 As shown, the method may include the following steps:

[0053] Step 310: Obtain electromyographic data.

[0054] First, it should be noted that the exoskeleton rehabilitation hand driven by electromyography (EMG) data is an important means to help users with hand dysfunction rebuild their motor function. The core technology of this means is hand movement intention recognition, specifically referring to the estimation of continuous finger force movement based on surface EMG data. In this embodiment, the user's EMG data is collected using sensors in an EMG acquisition device.

[0055] In one possible implementation, electromyography (EMG) data is obtained from the user's finger extensors, finger flexors, biceps and triceps, as well as a ring of muscles in the forearm 2-6 cm from the elbow, collected by an EMG acquisition device at a pre-set sampling frequency (e.g., 2000 Hz).

[0056] In one possible implementation, estimating a user's finger force based on electromyography (EMG) data requires data preprocessing of the EMG data. This preprocessing process may include the following steps:

[0057] The first step is to denoise the electromyography (EMG) data. This denoising process includes, but is not limited to: applying a fourth-order Butterworth bandpass filter to the surface EMG data; using baseline correction to remove noise from the EMG data; using a 50Hz notch filter to remove power frequency interference from the EMG data; and using a logarithmic scaling algorithm based on the u-law transform to amplify the signal and extract useful information from the EMG data more effectively. Of course, in other embodiments, using a logarithmic scaling algorithm based on the u-law transform to amplify the signal can also be considered as signal enhancement processing of the surface EMG data, and this is not intended to be a specific limitation.

[0058] The second step involves normalizing the denoised electromyography (EMG) data to obtain intermediate data. This normalization process includes, but is not limited to, root mean square normalization, minimum value normalization, and maximum value normalization.

[0059] The third step involves segmenting the intermediate data using a sliding window with a set window length to obtain multi-channel electromyography (EMG) data. In other words, by sliding a window with a set window length across the intermediate data at a set step size, the intermediate data can be segmented into more multi-channel EMG data, which can then be used as input to the force estimation model.

[0060] It should be noted that the set window length and set step size of the sliding window can be flexibly adjusted according to the actual needs of the application scenario. For example, in a specific embodiment, the set window length of the sliding window is 400ms and the set distance is 10ms. This is not a specific limitation.

[0061] This approach effectively improves data quality and model input performance, which not only enhances the model's power estimation performance but also significantly improves computational efficiency.

[0062] Step 330: Input the electromyography (EMG) data into the force estimation model, extract and fuse multi-scale features from the EMG data to obtain the fused features corresponding to the EMG data.

[0063] The force estimation model includes the ShuffleNetV2 neural network, which uses channel splitting and channel rearrangement techniques to enhance the information interaction between features, and the SE module, which uses an adaptive weight adjustment mechanism to construct the correlation and dependency between features.

[0064] First, it should be noted that the ShuffleNetV2 neural network is a lightweight and highly efficient convolutional neural network architecture. The ShuffleNetV2 neural network includes channel splitting, depthwise separable convolution, and channel rearrangement. Among them, channel splitting can effectively reduce computational complexity and memory access costs. Depthwise separable convolution can replace conventional convolution operations, decoupling spatial convolution and channel convolution, improving efficiency, and significantly reducing the number of parameters and computational cost. Channel rearrangement can, after branch convolution, shuffle and rearrange the channels to allow information to cross-mix between different channels, solving the problem of "information silos" between branches and enhancing the flow of information between channels.

[0065] In one embodiment, the overall structure of the ShuffleNetV2 neural network includes basic unit 1, basic unit 2, basic unit 3, and basic unit 4, followed by a convolutional layer and a global pooling layer. Basic unit 1 contains an initial convolutional layer and pooling layer for initial feature extraction, with 24 output channels. Basic units 2, 3, and 4 are each composed of two repeated ShuffleNetV2 basic modules, with 48, 96, and 192 output channels respectively, used to progressively extract deeper features and expand the channel dimension. Finally, the convolutional layer and global pooling layer further integrate the fused features, providing a compact and expressive feature representation for subsequent force estimation.

[0066] The SE module, an attention mechanism module, is used to enhance the feature representation capabilities between channels. The main processing steps of the SE module include squeezing, activation, and recalibration. Squeezing compresses the spatial information of each feature channel into a global feature representation through global average pooling, obtaining the feature response of each feature channel. Activation uses a bottleneck structure composed of two fully connected layers to learn the correlation between feature channels and generate weights for each feature channel. Recalibration applies the learned weights to the original feature map through multiplication, enhancing useful feature channels and suppressing unimportant ones.

[0067] It should be noted that electromyography (EMG) data is usually a multi-channel signal. The activity characteristics of different muscle groups vary in time and intensity. Furthermore, EMG signals contain different levels of physiological activity information at different time scales. Extraction of multi-scale features helps to capture these differences. The fusion of multi-scale features combines local detail information with global contextual information, thereby improving the expressive power of the model.

[0068] Based on this, the ShuffleNetV2 neural network excels at handling channel information interaction, multi-scale features can provide richer channel inputs, the SE module relies on accurate channel relationship modeling, and multi-scale fusion results can help the SE to better assign weights and highlight key features.

[0069] In one possible implementation, such as Figure 4 As shown, step 330 may also include:

[0070] Step 331: In the force estimation model, the ShuffleNetV2 neural network is used to extract spatiotemporal features from the electromyography data to obtain the first feature and the second feature.

[0071] In one possible implementation, the ShuffleNetV2 neural network includes a first network and a second network. The above method may also include: inputting electromyographic data into the first network and the second network respectively to extract features, thereby obtaining the first feature and the second feature.

[0072] The first network is used to extract local spatiotemporal features from the electromyography (EMG) data, and the second network is used to extract global features from the EMG data. The first feature is the local spatiotemporal feature in the EMG data, and the second feature is the global feature in the EMG data.

[0073] In one possible implementation, the above method further includes: performing channel splitting processing on the electromyographic data in the first network, and having the output from the channel splitting received by the first main branch and the first side branch of the first network respectively; concatenating the result of the first side branch with the result of the first main branch in terms of dimension to obtain a first feature; extracting features from the electromyographic data by the second main branch and the second side branch of the second network respectively to obtain a first sub-feature and a second sub-feature; and concatenating the first sub-feature and the second sub-feature to obtain a second feature.

[0074] Figure 5 The diagram shows the specific model structure of the first and second networks of a ShuffleNetV2 neural network, as follows: Figure 5As shown, the ShuffleNetV2 neural network includes a first network and a second network. The first network includes a first main branch and a first side branch, and the second network includes a second main branch and a second side branch. The structures of both the first and second networks are connected to the SE module through splicing and channel rearrangement to complete feature fusion and weighted enhancement processing. This structure can be used for deep feature extraction of intermediate features.

[0075] Channel splitting is a lightweight operation technique in convolutional neural networks. It is mainly used to reduce computation, accelerate network inference, and provide input sources for subsequent multi-branch structures (such as the first main branch and the first side branch).

[0076] For example, if the electromyographic data is a tensor of [B, C, H, W] (B is the batch size, C is the number of channels, H and W are the height and width), and C = 64, the channel splitting process will divide it into two parts. The first part is the first 32 channels, which are input to the first main branch, and the second part is the last 32 channels, which are input to the first test branch.

[0077] Please see back Figure 5 Now combined Figure 5 The first and second networks of the force estimation model are explained. The first network has a stride of 1, which preserves the spatial dimension and does not destroy the local structure, making it suitable for fine-grained modeling. The first main branch consists of three convolutional layers: a 1x1 convolutional layer, a 3x3 depthwise separable convolutional layer, and another 1x1 convolutional layer. The first main branch directly receives the output of the channel splitting operation and performs feature extraction. The first side branch does not process the input and concatenates it with the output of the first main branch to obtain the first feature.

[0078] The second network has a stride of 2, which allows for the extraction of features from a larger receptive field, thus facilitating the acquisition of global contextual information. The second main branch consists of three convolutional layers: a 1x1 convolutional layer, a 3x3 depthwise separable convolutional layer, and another 1x1 convolutional layer. The second side branch consists of two convolutional layers: a 3x3 depthwise separable convolutional layer and a 1x1 convolutional layer. Electromyographic data is input into the second main branch and the second side branch respectively through duplication or splitting for feature extraction, resulting in the first sub-feature and the second sub-feature. The first and second sub-features are then concatenated to obtain the second feature.

[0079] It should be noted that after obtaining the first feature and the second feature, the first feature and the second feature need to be processed by channel rearrangement respectively before being input into the SE module.

[0080] The channel rearrangement operation is used to shuffle and rearrange the channel order after splicing the first main branch and the first side branch (or the second main branch and the second side branch), thereby breaking the independence of channels between branches, enhancing the information interaction capability across channels, and improving the feature fusion effect.

[0081] After splicing and channel rearrangement are completed, the first feature and the second feature are respectively input into the SE modules connected to the first network and the second network respectively.

[0082] Step 333: Use the SE module to perform global channel calibration on the first and second features to obtain the fusion features corresponding to the electromyography data.

[0083] First, it should be noted that the SE module can model the response intensity of each channel through global average pooling and channel attention mechanisms, thereby adaptively enhancing the feature channels related to force estimation, suppressing channel information that is irrelevant to estimation or redundant, and improving the discriminative ability of the overall feature representation.

[0084] In one possible implementation, the above method further includes: inputting the first feature and the second feature into the SE module respectively, so that the SE module performs channel weighting operations on the first feature and the second feature respectively to obtain the first weighted feature and the second weighted feature; and fusing the first weighted feature and the second weighted feature to obtain the fused feature.

[0085] Regarding the SE module, it can explicitly model the correlation and dependency between feature channels and use an adaptive weight adjustment mechanism to dynamically generate the weight coefficients of each feature channel, thereby enhancing the model's ability to capture key features, suppressing the interference of redundant features, and improving the representation learning ability of deep neural networks.

[0086] It is understandable that the SE module adjusts the channel weights of the entire concatenated result, which is beneficial for global optimization of feature representation.

[0087] In one possible implementation, the above method may also include the following steps:

[0088] The first step is to compress the spatial information of each feature channel of the first feature and the second feature into a global feature representation of the channel dimension through a global average pooling operation, thereby obtaining the first response data and the second response data.

[0089] The first response data includes the feature responses of each feature channel in the first feature; the second response data includes the feature responses of each feature channel in the second feature.

[0090] Specifically, the first response data includes a channel response vector obtained by global average pooling of the first feature; the second response data includes a channel response vector obtained by global average pooling of the second feature.

[0091] Through global average pooling, each feature channel of the first / second feature is compressed into a global response value (channel response vector), which represents the overall activation level of the feature channel on the entire feature map. This extracts information about the spatial dimension and transforms it into clues about the importance (i.e., weights) of the channels.

[0092] The second step involves inputting the first response data and the second response data into a bottleneck structure consisting of two fully connected layers to learn the correlation between each feature channel and generate the first weight data and the second weight data respectively.

[0093] The first weight data includes the weight corresponding to each feature channel in the first feature; the second weight data includes the weight corresponding to each feature channel in the second feature.

[0094] It should be noted that the bottleneck structure includes two fully connected layers. The first layer is usually used for dimensionality reduction and extraction of nonlinear relationships (e.g., using ReLU activation), while the second layer is used to restore the original number of channels and generate weight coefficients (e.g., using Sigmoid activation) to explicitly model the dependencies between feature channels.

[0095] The third step is to perform channel weighting on the corresponding feature channels in the first feature based on the first weight data to obtain the first weighted feature.

[0096] It is understandable that, based on the first weight data, each feature channel is assigned a weight, which represents its importance in the current feature map. This is done by multiplying (weighting) with the first feature to highlight valuable channels and suppress redundant channels.

[0097] The fourth step is to perform channel weighting on the corresponding feature channels in the second feature based on the second weight data to obtain the second weighted feature.

[0098] It is understandable that, based on the second weight data, each feature channel is assigned a weight, representing its importance in the current feature map, thereby highlighting valuable channels and suppressing redundant channels by multiplying with the second feature (weighting).

[0099] The fourth step is to fuse the first weighted feature and the second weighted feature to obtain the fused feature.

[0100] In one possible implementation, the fusion process can employ feature concatenation, weighted summation, or other fusion methods to integrate the complementary information of the first weighted feature and the second weighted feature to obtain the final fused feature.

[0101] Under the above embodiments, in the model structure, the SE module, as the tail module of the first network and the second network, performs global channel weighting calibration on the two feature outputs respectively, providing high-quality, optimized channel features for subsequent feature fusion.

[0102] Step 350: Based on the fusion features corresponding to the electromyography data, predict the target force pattern label corresponding to the electromyography data.

[0103] First, it should be noted that fusion features comprehensively characterize the spatial-temporal properties of electromyography (EMG) signals by integrating information from different channels and time scales. Then, using a pre-trained classification model or pattern recognition algorithm, the fusion features can be mapped to predefined force pattern categories, achieving force pattern label assignment for EMG data. The target force pattern label indicates the muscle force state category corresponding to the current EMG data, providing a basis for subsequent force value estimation.

[0104] In one possible implementation, force pattern labels are based on nine different force patterns applied to a linear pressure sensor device by the fingertips of one or more fingers of the user's dominant hand. Specifically, the motion dimension covers six degrees of freedom, including finger flexion and thumb abduction. Each force pattern action is repeated six times to ensure data stability and representativeness.

[0105] The nine different force modes may include, but are not limited to, the following types: light touch pressing mode (light force, similar to tapping), light grip mode (small grip force), medium grip mode (medium intensity grip), heavy grip mode (greater grip force), thumb abduction force mode (force of thumb spreading outward), single finger pressing mode (force applied by only one fingertip), multi-finger coordinated pressing mode (force applied by multiple fingertips simultaneously with coordinated force), continuous stable force mode (continuous application of constant force), and gradually increasing force mode (force gradually increases from small to large). No specific limitations are made here.

[0106] During the pressing process, a linear pressure sensor device collects the force signal applied by the finger, recording the maximum and minimum force values ​​for each action. Based on this, different actions and force intensities are classified, and a corresponding force pattern tag library is constructed.

[0107] To achieve accurate mapping between electromyography (EMG) data and force pattern labels, EMG signals were simultaneously acquired when the fingers performed the nine force pattern movements described above. Through timestamp alignment, the force change data recorded by the pressure sensor was synchronously matched with the EMG signals acquired by the EMG sensing device. Subsequently, based on the fused EMG features at each sampling time point, combined with the corresponding pressure sensor force values, the force pattern label corresponding to the EMG data at that moment was assigned.

[0108] The establishment of this correspondence enables the accurate identification of the current force pattern category of the finger by analyzing the multi-scale fusion features of electromyography (EMG) data, thereby achieving force pattern classification and estimation based on EMG signals.

[0109] Step 370: Based on the target force pattern label, obtain the force estimation result of the target object.

[0110] The target object refers to the user who performs force estimation, that is, the specific individual whose force is estimated by collecting their electromyographic data and combining it with force pattern labels. The force estimation result can reflect the actual force output of the target object's muscles.

[0111] Specifically, based on the target strength pattern label and combined with a pre-established strength mapping model or regression model, the muscle strength of the target object can be quantitatively estimated. Specifically, using the target strength pattern label as input, the corresponding strength estimation function or lookup table algorithm is invoked to calculate and output the target object's strength value in the current action state.

[0112] One possible implementation involves collecting electromyographic (EMG) data and corresponding actual force values ​​from multiple users performing various force patterns (including force output with different amplitudes, directions, and finger combinations). The actual force values ​​are acquired using high-precision pressure or force sensors and used as training labels.

[0113] Therefore, by using machine learning algorithms, such as multinomial regression, support vector regression (SVR), random forest regression, neural networks, or deep learning models (such as MLP, CNN, LSTM, etc.), a mapping model from electromyography data to actual strength values ​​can be established. This model can be trained separately for different target strength pattern labels, or it can be jointly modeled under multi-label conditions.

[0114] Based on this, the regression model parameters for various strength modes can be stored after training. In the actual application stage, the corresponding mapping model can be called according to the target strength mode label to input and predict the real-time collected electromyographic features and output the corresponding strength estimate.

[0115] It should be noted that the force pattern label plays a role in contextual constraint and prior limitation in the mapping process between EMG data and actual force. Since the fusion characteristics of EMG data under different force patterns are significantly different (for example, light touch and strong grip are significantly different in terms of temporal amplitude, spectral density and channel distribution), directly modeling all EMG data in a unified manner can easily lead to a decrease in model estimation accuracy, especially in the case of large errors during movement intersections or transitions.

[0116] By introducing target force pattern labels, the relationship between electromyography (EMG) data and force can be modeled and predicted within a specific pattern subspace. For example, in the "light touch" mode, the corresponding force prediction range is set to a small value range, and the modeling focuses on the response of high-sensitivity, low-amplitude EMG signals.

[0117] In other words, the target strength pattern label is equivalent to imposing a contextual constraint on the electromyography (EMG) data, which allows for the selection of a more suitable strength prediction range for strength regression, significantly improving the targeting and accuracy of EMG data and strength mapping.

[0118] Through the above process, the introduction of a multi-scale feature extraction and fusion mechanism can fully explore the deep feature correlations between electromyographic data at different time scales, spatial dimensions, and multiple channels, effectively improving the ability of electromyographic data to represent muscle activity states. Furthermore, by analyzing the fused features and assigning target force pattern labels, clear action context information is provided before entering the force estimation stage, reducing feature confusion between different action types and effectively improving estimation accuracy. Combined with the target force pattern labels, the force estimation results of the target object are output, which can reflect the user's muscle output level in real time and accurately.

[0119] In an exemplary embodiment, the training process of the force estimation model may include the following steps: obtaining a training dataset, inputting the training dataset into an initial force estimation model, so that the initial force estimation model learns the features of each sample, in order to train the initial force estimation model.

[0120] The training dataset includes training electromyography (EMG) data from multiple training users, and the EMG data carries training strength pattern labels.

[0121] Specifically, training electromyography (EMG) data refers to the raw or preprocessed EMG signal data collected by EMG sensors under standardized movement conditions, while training strength mode labels are used to mark the target strength mode category corresponding to each segment of EMG data.

[0122] Then, the training dataset is input into an initial force estimation model, which can learn the feature distribution patterns of different electromyographic data under different force modes, thereby achieving training optimization of the initial force estimation model.

[0123] After training the initial force estimation model, the above method may further include the following steps: obtaining a test dataset, inputting the test dataset into the force estimation model, estimating the force for each test user, and obtaining predicted force pattern labels; comparing the sample force pattern labels with the predicted force pattern labels, and calculating precision, accuracy, recall, and / or F1 score; evaluating the regression performance of the force estimation model based on precision, accuracy, recall, and / or F1 score; if the regression performance evaluation fails, continuing the training of the force estimation model.

[0124] The test dataset includes electromyography (EMG) data from multiple test users, and the EMG data carries test force pattern labels.

[0125] Precision refers to the proportion of samples that the model predicts to be positive, but which are actually positive. It reflects the accuracy of the model in identifying relevant targets. The method for calculating precision is as follows:

[0126]

[0127] Accuracy refers to the proportion of correct predictions among all test samples; it reflects the overall accuracy of the model's predictions. The method for calculating accuracy is as follows:

[0128]

[0129] Recall refers to the proportion of samples that are actually positive that are correctly predicted as positive. It measures the model's ability to identify positive samples. The method for calculating recall is as follows:

[0130]

[0131] The F1 score is the harmonic mean of precision and recall, used to comprehensively evaluate model performance. The F1 score is calculated as follows:

[0132]

[0133] In the above formula, TP represents the number of samples correctly predicted as positive, FP represents the number of samples incorrectly predicted as positive, TN represents the number of samples correctly predicted as negative, and FN represents the number of samples incorrectly predicted as negative.

[0134] Based on the numerical results of the above evaluation indicators, the regression performance of the force estimation model is evaluated to determine whether it meets the preset accuracy requirements. If the regression performance evaluation does not meet the accuracy requirements (i.e., the evaluation indicators are lower than the preset threshold), the next round of training process is entered to continue to adjust and optimize the parameters of the force estimation model until the model performance meets the requirements.

[0135] For example, in one embodiment, the evaluation metric thresholds for the force estimation model are set as follows: accuracy not less than 90%, precision not less than 88%, recall not less than 85%, and F1 score not less than 0.87. After the force estimation model is trained, the test dataset is input into the trained force estimation model, prediction is performed, and predicted force pattern labels are obtained. The evaluation metrics are calculated by comparing the predicted force pattern labels with the test force labels. Assume that the results of this round of evaluation metrics are as follows: accuracy 87%, precision 85%, recall 83%, and F1 score 0.84. Since multiple evaluation metrics are lower than the preset thresholds, it is determined that the current force estimation model performance does not meet the accuracy requirements, and then the next round of training begins.

[0136] Through the above process, the ShuffleNetV2 neural network used in this method is a supervised nonparametric model, which is easy to implement and can estimate the angles of hand joints with more degrees of freedom. Training the force estimation model using electromyography data from multiple training users ensures good generalization ability and prediction accuracy, improving its practicality and robustness across different users and movement patterns. Through a linkage mechanism between the training and evaluation phases, after the model outputs predicted force pattern labels, the difference between the predicted results and the actual labels can be compared in real time, quantifying the model's classification or regression error. A closed-loop feedback system based on key evaluation metrics such as precision, accuracy, recall, and F1 score facilitates dynamic optimization and rapid iteration of the model.

[0137] The following is a detailed description of a force estimation method in an application scenario. In this scenario, the aforementioned force estimation method is used to identify and evaluate finger force patterns. Fifteen subjects were recruited to complete a total of nine different force pattern movements, including: single-finger flexion from the little finger to the thumb (five types), thumb abduction, simultaneous flexion of the index and little fingers, simultaneous flexion of the ring and middle fingers, and a combination of index finger flexion and thumb abduction.

[0138] In this application scenario, each subject performed the movement six times for each force pattern, and electromyography (EMG) data and finger force data were collected using an EMG acquisition device and a pressure linear sensor. The first four repetitions were used to build a training set to train the force estimation model based on the EMG data; the remaining two repetitions were used to build a test set to evaluate the model's force pattern recognition performance on unseen data. The EMG data reflects the physiological electrical signal activity of each muscle group during the subject's movement; the finger force data represents the actual force exerted by the subject through finger pressure during the movement, and can serve as a reference output label or true force value corresponding to that EMG data sample.

[0139] During the training of the force estimation model, the input consists of electromyographic data collected from subjects performing different force patterns, and the output is the corresponding target force pattern label. The trained model is then applied to the test set, and its four performance metrics—precision, accuracy, recall, and F1 score—are calculated on the test set.

[0140] As shown in Table 1, the experimental results show that the method in this application scenario outperforms the two mainstream finger force pattern recognition algorithms in all four performance indicators. This indicates that the force estimation method used in this application scenario has higher recognition accuracy and robustness in multi-finger action scenarios and can more effectively achieve the finger force pattern recognition task.

[0141] Table 1

[0142]

[0143] In this application scenario, the force estimation model, while introducing the SE module to optimize feature representation, inherits the lightweight architecture advantages of the ShuffleNetV2 neural network, reducing the computational cost and memory footprint of the force estimation model. This makes it easier to deploy and apply on resource-constrained devices, broadening the application scope of the technology. Furthermore, by combining the channel rearrangement and splitting techniques of the ShuffleNetV2 neural network with the channel adaptive calibration capability of the SE module, it achieves efficient mining and optimized fusion of multi-scale characteristics of electromyography features. This fully extracts feature information related to finger force patterns, improves the model's ability to capture dynamic mechanical features, and solves the problem of finger force pattern recognition estimation for complex multi-movement problems. It can be used in the rehabilitation medicine field for tracking the patient's state during rehabilitation, helping rehabilitation therapists better understand the patient's physical condition and develop more personalized and targeted rehabilitation plans, more effectively addressing the patient's actual needs. It also greatly improves sports injury repair techniques, enabling better targeted and comprehensive rehabilitation training for specific muscles.

[0144] This application scenario has multiple applications in the fields of intelligent prosthetic control, human-computer interaction, and gesture control devices, providing users with a more intuitive and portable control and interaction method. This method decodes electromyographic signals, enabling amputees to use their own muscle activity to control intelligent prosthetics, simulating natural hand movements, improving operational performance, and providing a more natural body extension. Users can conveniently control home automation systems, smart devices, smartphones, etc., through electromyographic signals, promoting convenience and intelligence in their lives. In conclusion, this algorithm has a profound impact on the fields of intelligent prosthetic control, human-computer interaction, and gesture control devices, not only providing assistance to people with disabilities but also bringing innovation and convenience to the technological field.

[0145] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0146] The following are embodiments of the apparatus described in this application, which can be used to execute the force estimation method involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the force estimation method involved in this application.

[0147] Please see Figure 6 This application provides a force estimation device 900, including but not limited to: a data acquisition module 910, a feature extraction module 930, a label allocation module 950, and a force estimation module 970.

[0148] Among them, the data acquisition module 910 is used to acquire electromyographic data;

[0149] The feature extraction module 930 is used to input electromyography (EMG) data into the force estimation model, extract and fuse multi-scale features of the EMG data, and obtain the fused features corresponding to the EMG data. The force estimation model includes a ShuffleNetV2 neural network that uses channel splitting and channel rearrangement techniques to enhance the information interaction between features, and an SE module that uses an adaptive weight adjustment mechanism to construct the correlation and dependency relationship between features.

[0150] The label assignment module 950 is used to predict the target force pattern label corresponding to the electromyography data based on the fusion features corresponding to the electromyography data.

[0151] The force estimation module 970 is used to obtain the force estimation result of the target object based on the target force pattern label.

[0152] It should be noted that the force estimation device provided in the above embodiments is only illustrated by the division of the above functional modules when performing force estimation. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the force estimation device will be divided into different functional modules to complete all or part of the functions described above.

[0153] Furthermore, the force estimation device and force estimation method embodiments provided in the above embodiments belong to the same concept, and the specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0154] Please see Figure 7 This application provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc.

[0155] exist Figure 7 In this context, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0156] Data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not indicate that there is only one bus or one type of bus.

[0157] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0158] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0159] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing computer programs having instruction or data structure forms and accessible by electronic device 400, but not limited to these.

[0160] The memory 4003 stores a computer program, and the processor 4001 can read the computer program stored in the memory 4003 through the communication bus 4002.

[0161] The computer program is executed by one or more processors 4001 to implement the force estimation methods in the above embodiments.

[0162] Furthermore, this application provides a storage medium storing a computer program that is executed by one or more processors to implement the force estimation method described above.

[0163] This application provides a computer program product, including a computer program that is executed by one or more processors to implement the force estimation method described above.

[0164] Compared with related technologies, this solution introduces a multi-scale feature extraction and fusion mechanism, which can fully explore the deep feature correlations between electromyography (EMG) data at different time scales, spatial dimensions, and multiple channels. This effectively improves the ability of EMG data to represent muscle activity states. Furthermore, by analyzing the fused features and assigning target force pattern labels, clear action context information is provided before entering the force estimation stage, reducing feature confusion between different action types and effectively improving estimation accuracy. The force estimation results of the target object are output in combination with the target force pattern labels, which can reflect the user's muscle output level in real time and accurately.

[0165] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A force estimation method, characterized in that, include: Acquire electromyography data; The electromyography (EMG) data is input into the force estimation model, and multi-scale features are extracted and fused from the EMG data to obtain the fused features corresponding to the EMG data. The force estimation model includes a ShuffleNetV2 neural network that uses channel splitting and channel rearrangement techniques to enhance the information interaction between features, and an SE module that uses an adaptive weight adjustment mechanism to construct the correlation and dependency relationship between features. Based on the fusion features corresponding to the electromyographic data, predict the target force pattern label corresponding to the electromyographic data; Based on the target force pattern label, the force estimation result of the target object is obtained.

2. The method as described in claim 1, characterized in that, The step of inputting the electromyography (EMG) data into the force estimation model, extracting and fusing multi-scale features from the EMG data to obtain the fused features corresponding to the EMG data includes: In the force estimation model, the ShuffleNetV2 neural network is used to extract spatiotemporal features from the electromyography data to obtain the first feature and the second feature; The SE module is used to perform global channel calibration on the first feature and the second feature to obtain the fusion feature corresponding to the electromyography data.

3. The method as described in claim 2, characterized in that, The ShuffleNetV2 neural network includes a first network and a second network; In the force estimation model, the ShuffleNetV2 neural network is used to extract spatiotemporal features from the electromyography data to obtain a first feature and a second feature, including: The electromyographic data are input into the first network and the second network respectively for feature extraction to obtain the first feature and the second feature; The first network is used to extract local spatiotemporal features from the electromyography (EMG) data, and the second network is used to extract global features from the EMG data; the first feature is a local spatiotemporal feature in the EMG data, and the second feature is a global feature in the EMG data.

4. The method as described in claim 3, characterized in that, The step of inputting the electromyographic data into the first network and the second network respectively for feature extraction to obtain the first feature and the second feature includes: The electromyographic data is processed by channel splitting in the first network, and the output from the channel splitting is received by the first main branch and the first side branch of the first network, respectively. The result of the first side branch is concatenated with the result of the first main branch in terms of dimension to obtain the first feature; In the second main branch and the second side branch of the second network, feature extraction is performed on the electromyography data to obtain the first sub-feature and the second sub-feature; The first sub-feature and the second sub-feature are concatenated to obtain the second feature.

5. The method as described in claim 2, characterized in that, The step of using the SE module to perform global channel calibration on the first feature and the second feature to obtain the fusion feature corresponding to the electromyography data includes: The first feature and the second feature are input into the SE module respectively, so that the SE module performs channel weighting operation on the first feature and the second feature respectively to obtain the first weighted feature and the second weighted feature; The first weighted feature and the second weighted feature are fused together to obtain the fused feature.

6. The method as described in claim 5, characterized in that, The step of inputting the first feature and the second feature into the SE module respectively includes: By performing a global average pooling operation, the spatial information of each feature channel of the first feature and the second feature is compressed into a global feature representation of the channel dimension, respectively, to obtain the first response data and the second response data; the first response data includes the feature response of each feature channel in the first feature; the second response data includes the feature response of each feature channel in the second feature; The first response data and the second response data are respectively input into a bottleneck structure composed of two fully connected layers to learn the correlation between each feature channel and generate the first weight data and the second weight data respectively; the first weight data includes the weight corresponding to each feature channel in the first feature; the second weight data includes the weight corresponding to each feature channel in the second feature. Based on the first weight data, channel weighting is performed on the corresponding feature channels in the first feature to obtain the first weighted feature. Based on the second weight data, channel weighting is performed on the corresponding feature channels in the second feature to obtain the second weighted feature.

7. The method according to any one of claims 1 to 6, characterized in that, After acquiring the electromyographic data, the method further includes: The electromyographic data is denoised, and the denoised electromyographic data is then normalized to obtain intermediate data. By using a sliding window with a set window length, the intermediate data is segmented to obtain multi-channel electromyographic data.

8. The method according to any one of claims 1 to 6, characterized in that, The training process of the force estimation model includes: Obtain a training dataset, which includes training electromyography (EMG) data from multiple training users, and the training EMG data carries training strength pattern labels. The training dataset is input into the initial force estimation model, so that the initial force estimation model learns the features in the training dataset to train the initial force estimation model. After training the initial force estimation model, the method further includes: Obtain a test dataset, which includes electromyography (EMG) data from multiple test users, and the EMG data carries test force pattern labels; The test dataset is input into the force estimation model to estimate the force of each test user and obtain the predicted force pattern label. Based on the comparison between the sample strength pattern label and the predicted strength pattern label, calculate one or more of the following: precision, accuracy, recall and / or F1 score; The regression performance of the force estimation model is evaluated based on one or more of the precision, accuracy, recall and / or F1 score. If the regression performance evaluation fails, the training of the force estimation model continues.

9. A force estimation device, characterized in that, include: The data acquisition module is used to acquire electromyographic data; The feature extraction module is used to input the electromyography (EMG) data into the force estimation model, extract and fuse multi-scale features of the EMG data to obtain the fused features corresponding to the EMG data; the force estimation model includes a ShuffleNetV2 neural network that uses channel splitting and channel rearrangement techniques to enhance the information interaction between features, and an SE module that uses an adaptive weight adjustment mechanism to construct the correlation and dependency relationship between features; The label assignment module is used to predict the target force pattern label corresponding to the electromyographic data based on the fusion features corresponding to the electromyographic data. The force estimation module is used to obtain the force estimation result of the target object based on the target force pattern label.

10. An electronic device comprising at least one processor and at least one memory, wherein, The memory stores a computer program, characterized in that the computer program, when executed by the processor, implements the force estimation method as described in any one of claims 1 to 8.

Images