An upper limb rehabilitation robot motion intention prediction method and system based on image-text multi-modal fine-grained reasoning
By employing a multimodal fine-grained reasoning method combining text and images, along with a VGG16-LSTM network and a multi-head cross-attention mechanism, the problem of low accuracy and poor robustness in single-dimensional electromyography signal prediction was solved. This resulted in high-precision recognition and anti-interference capabilities for upper limb movement intentions, thereby improving the effectiveness of rehabilitation training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HOHAI UNIV
- Filing Date
- 2026-03-12
- Publication Date
- 2026-06-09
AI Technical Summary
Existing single-dimensional electromyography signal input motion prediction models have poor accuracy and robustness in predicting upper limb movement intentions, and cannot fully depict the complex state of patients during the rehabilitation period, thus limiting the scope of application of rehabilitation robots.
We employ a fine-grained reasoning method based on image and text multimodal approaches. This method involves preprocessing surface electromyography signals corresponding to actions by synchronously acquiring them, and then combining this with a VGG16-LSTM dual-branch feature learning network to extract one-dimensional time-frequency domain and two-dimensional coded image features. We utilize a multi-head cross-attention mechanism for feature fusion and classification, and finally use a random forest classifier for action recognition.
It improves the accuracy and robustness of motor intention prediction, enhances the resistance to interference from environmental noise, individual differences and sensor malfunctions, and ensures the safety and reliability of upper limb rehabilitation training.
Smart Images
Figure CN122174018A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot upper limb motion recognition technology, specifically relating to a method and system for predicting the movement intention of an upper limb rehabilitation robot based on fine-grained reasoning using image and text multimodal methods. Background Technology
[0002] Stroke is one of the leading causes of long-term disability worldwide. Clinical data shows that about 80% of stroke survivors will have residual upper limb motor dysfunction, which manifests as limited range of motion in the shoulder, elbow and wrist joints, decreased hand grasping accuracy, muscle spasms and other symptoms.
[0003] Meanwhile, traditional rehabilitation medical resources are currently scarce, and there is a significant shortage of rehabilitation medical personnel. A single therapist must simultaneously care for multiple patients, leading to high rehabilitation pressure and low efficiency. Rehabilitation robots can replace therapists in assisting patients with some repetitive and mechanical rehabilitation tasks, improving the efficiency of upper limb rehabilitation training for stroke patients. When stroke patients possess some degree of voluntary motor ability, detecting their active motor intentions to achieve assisted movement is particularly crucial in rehabilitation training. Identifying patients' active motor intentions through surface electromyography (EMG) signals is a commonly used method, with core steps including signal acquisition and preprocessing, feature extraction, and classifier learning.
[0004] However, traditional research methods require manual extraction of time-domain and frequency-domain features from one-dimensional electromyographic signals. These features are then combined and input into a classifier for learning and classification, and corresponding action labels are assigned. While traditional research methods are simple in principle and have strong interpretability, the signal data input to the model has a single dimension and weak resistance to uncertain interference, resulting in insufficient robustness of the recognition model. This makes it impossible to fully depict the complex state of patients during the recovery period, which significantly limits the scope of application of rehabilitation robots. Summary of the Invention
[0005] This invention proposes a method and system for predicting the movement intention of an upper limb rehabilitation robot based on fine-grained reasoning using image and text multimodal methods. This solves the problems of poor accuracy and robustness of existing single-dimensional electromyography signal input movement prediction models when predicting upper limb movement intentions.
[0006] To achieve the above objectives, the present invention proposes the following technical content: S1: Synchronously acquire surface electromyography (EMG) signals corresponding to the given action instructions, and preprocess the EMG signals; the preprocessing includes signal filtering and noise reduction, active segment detection, and overlapping windowing; the given action instructions include elbow horizontal flexion, elbow horizontal extension, shoulder horizontal adduction, shoulder horizontal extension, forearm internal rotation, and a compound action of forearm internal rotation and elbow horizontal flexion. S2: Extract time-frequency features from the one-dimensional windowed surface electromyography signal in S1; S3: Extract image features from the one-dimensional windowed surface electromyography signal in S1; S4: Construct a VGG16-LSTM dual-branch feature learning network, input one-dimensional time-frequency domain features into the LSTM model and two-dimensional encoded images into the VGG16 model; use real labels of upper limb rehabilitation movements as supervision, optimize the parameters of the two models respectively through backpropagation of loss values, and obtain the trained VGG16 model and LSTM model after training convergence; S5: For newly acquired upper limb surface electromyography (EMG) signals, after preprocessing in S1, time-frequency feature extraction in S2, and image feature extraction in S3, one-dimensional time-frequency domain features and two-dimensional coded images are obtained respectively. These are then input into the model trained in S4 to extract fine-grained features, which are fused using a fine-grained alignment module under a multi-head cross-attention mechanism to obtain a high-dimensional feature vector. After standardizing the data to eliminate dimensional differences, the high-dimensional features are used to calculate the covariance matrix and perform singular value decomposition. Principal components with a cumulative variance contribution rate of 95% are selected as the dimensionality-reduced features. D fusion ; Dimensionally reduced features D fusion The data is fed into a random forest classification system consisting of multiple trained decision trees, resulting in six classification results for the action.
[0007] Further, step S2 includes the following steps: S2.1: The first i The surface electromyography signal of each channel is decomposed into M =5 modal components, the formula is:
[0008] In the formula, Indicates in t At that moment, the i The first channel, the first m One modal component, m ∈[1, M ], and is an integer; M Indicates the total number of modal components; R b ( t ) i Indicates the first preprocessed step in S1 i One channel, t Surface electromyography signals at any given time; S2.2: Perform a Hilbert transform on each modal component to obtain the analytic signal of that modal component, and then define the bandwidth of each modal component using its norm; the formula is:
[0009] In the formula, Indicates the firsti The first channel, the first m Each modal component, t Analyzing the signal at any given moment; Indicates in t At that moment, the i The first channel, the first m One modal component; Indicates to The imaginary part obtained by performing the Hilbert transform; τ Represents the integral variable; S2.3: The analytic signal is modulated and shifted to baseband, and the bandwidth is estimated using the L2 norm of the modulated signal gradient; the formula is:
[0010] In the formula, Indicates the first i The first channel, the first m The bandwidth of each modal component after transformation is estimated. Indicates the first i The first channel m The center frequency of each modal component; Indicates to t Find the partial derivatives; Represents the L2 norm; S2.4: Transform the problem into a variational constraint problem for solution; the formula is: ; In the formula, Indicates minimum bandwidth; S2.5: Combining the formula in S2.4 with the Lagrange operator to introduce a quadratic penalty function to strengthen the reconstruction of the augmented Lagrange function, transforming it into an unconstrained problem, the formula is:
[0011] In the formula, L ( *,*,* () is the augmented Lagrange function; α cf Represented as a quadratic penalty parameter; λ (t) denotes the Lagrange operator; <*> denotes the inner product; The above equation is solved iteratively using the Alternating Direction Multiplier Method (ADMM). After the iteration converges, the output is the first... i Channel, First M Modal components ; S2.6: Calculate the first... i Mean of each channel and all modal components and standard deviation , and serve as a one-dimensional time-domain feature; S2.7: The solution obtained through ADMM iteration is the first... i Channel, First m Modal components Discretize the sampling and divide it into P The first segment; the first i Channel, First m The first modal component n Each segmented signal is denoted as , n ∈[1, P ], and is an integer; Add a window function to each segment, with the following formula:
[0012] In the formula, Indicates the first window after adding a window. n Each segmented signal; w ( n ) indicates the first n Window functions added in segments; Indicates the first i Channel, First m The first modal component n Each segmented signal; S2.8: Perform FFT on the windowed segmented signal, then square the modulus and normalize it to obtain the power spectrum estimate of the segment;
[0013] In the formula, L n Indicates the first n The length of each segmented signal; U This represents the window function energy correction factor; f s Represents frequency variables. , F s Indicates the data sampling frequency; P Indicates the total number of segments; Indicates the first i Channel, First m The modal component, the first n Each segmented signal has frequency variables f s Power spectrum at; S2.9: Calculation P The power spectral mean of each segment; the formula is:
[0014] In the formula, Indicates the first i Channel, First mEach modal component in the frequency variable f s The average power spectrum at that location; S2.10: According to the first i Channel, First m Each modal component in the frequency variable f s Average power spectrum at Find the first... i Channels, all modal components in frequency variable f s mean of the power spectrum at point and standard deviation And it serves as a one-dimensional frequency domain feature.
[0015] Furthermore, step S3 includes the following steps: S3.1: Gram angle field transformation.
[0016] Obtain the i The sequence of one-channel, one-dimensional windowed surface electromyography signals is given by the following formula:
[0017] In the formula, Indicates the first i One channel, one-dimensional windowed surface electromyography signal sequence; Indicates the first K Electromyography signals from each sampling point; surface electromyography signal sequence Normalization yields:
[0018] In the formula, Indicates the first i A normalized sequence of one-channel, one-dimensional windowed surface electromyography signals; This represents the result of normalizing the Kth electromyographic signal; S3.2: Use the arccos function to convert a one-dimensional time series into a polar coordinate system, the formula is:
[0019] In the formula, Indicates the first i The angle cosine value of an electromyographic signal sequence; S3.3: Two Gram angular field matrices are obtained through two encoding methods: angular sum field and angular difference field. Among them, in the Gram angular field matrix of the angle and the field, the first... x Line number y The elements of the column are:
[0020] In the Gram angular field matrix of the angular difference field, the first... x Line number y The elements of the column are:
[0021] S3.4: Concatenate the Gram angular field matrices of the angle and field and the Gram angular field difference, and then obtain the two-dimensional coded image through mapping:
[0022] In the formula, GAF ( d,s ) represents the concatenated matrix, the first... d Line number s The numerical values of the column elements; Image ( d,s ) indicates the first d Line number s A two-dimensional encoded image of a column.
[0023] Further, step S5 includes the following steps: S5.1: Decompose the two-dimensional coded image I and the one-dimensional time-frequency feature E respectively; the formula is:
[0024] In the formula, I fine This shows the image after disassembly. E fine This represents the one-dimensional time-frequency characteristics after decomposition; W split,1 and W split,2 This represents the corresponding decomposition weight matrix; S5.2: Sub-features are mapped to a unified attribute space through a linear layer, and InfoNCE contrastive loss constraints are used to mine discriminative local features; the formula is:
[0025] In the formula, Indicates the activation function; W I and W E Indicates the mapping weights; B I and B E The bias term is represented by sin(*, *), which represents the cosine similarity. B b Indicates batch size; N n Indicates the number of local features; Indicates the temperature coefficient; S5.3: Calculate the disassembled 2D coded image I fine and the one-dimensional time-frequency characteristics after decomposition E fine The similarity matrix of fine-grained sub-features is used to obtain semantic matching weights through Softmax, thereby achieving local feature alignment:
[0026] In the formula, S Represents the disassembled two-dimensional coded image I fine and the one-dimensional time-frequency characteristics after decomposition E fine Cosine similarity; a match Represents semantic matching weights; Softmax(*) represents the activation function; Represents the 2-norm of a matrix; S5.4: Reconstruct one-dimensional time-frequency domain features based on matching weights, making them correspond one-to-one with two-dimensional image features in fine-grained semantics; the formula is:
[0027] In the formula, E fine This represents the one-dimensional time-frequency characteristics after decomposition. Flatten (*) indicates flattening fine-grained high-dimensional features; E fine，aligned This represents the one-dimensional time-frequency characteristic to be flattened; W merge Represents the weight matrix; E aligned This represents the one-dimensional time-frequency domain global features after fine-grained alignment of two-dimensional image features. S5.5: The input two-dimensional image features are used as the query vector, and the aligned one-dimensional time-frequency threshold global features are used as key-value pairs, which are then mapped to the multi-head attention space.
[0028] In the formula, Q represents the query vector, K represents the key vector, V represents the value vector, and I represents the input two-dimensional encoded image features. The weight matrix is a learnable linear transformation. S5.6: Calculate the multi-head attention score, concatenate the multi-head attention outputs, and obtain the enhanced image features; the formula is:
[0029] In the formula, I enhancedIndicates the enhanced image features; W out This represents the attention output matrix; Concat(*) concatenates the attention matrices. Atten This indicates a high level of attention from multiple parties. Represents the 2-norm of a matrix; S5.7: High-dimensional fused features are obtained by combining enhanced image features and finely aligned one-dimensional time-frequency domain global features; the formula is:
[0030] In the formula, and The weights set; F fusion Represents high-dimensional fusion features; S5.8: After standardizing the data to eliminate dimensional differences, the covariance matrix of the high-dimensional features is calculated and singular value decomposition is performed. The principal components with a cumulative variance contribution rate of 95% are selected as the dimensionality-reduced features. D fusion ; Dimensionally reduced features D fusion The data is fed into a random forest classification system consisting of multiple trained decision trees, resulting in six classification results for the action.
[0031] Further, step S1 includes the following steps: S1.1: Filtering and noise reduction; S1.2: Using a moving average combined with a threshold method, active segment detection is performed on the denoised surface electromyography signal to obtain the active signal segments; specifically, the following steps are included: S1.2.1: Set a rectangular sliding window with a size of [size missing]. h For the surface electromyography signal of each channel, calculate h The moving average of 1 data point; the formula is:
[0032] In the formula, R ( t ) i express t Time-of-sampling, the first i Surface electromyography signals of each channel i ∈[1,5], and is an integer; h Indicates the total number of samples; MAS t,i Indicates the current time t、 No. i Moving average of each channel; S1.2.2: Set thresholds based on the resting baseline mean and standard deviation; The formula is:
[0033] In the formula, μ rest,i Indicates the first i The average electromyographic signal of each channel and muscle in a resting state; σ rest,i Indicates the first i Standard deviation of electromyographic signals of individual channels and muscles in a resting state; k orc This is an empirical coefficient; Threshold i Indicates the first i One channel, the threshold of muscle in a resting state; S1.2.3: Filter active segment signals; For the i Each channel compares the moving average with a set threshold for muscle resting state. If the value within the rectangular sliding window... h If the moving average of the data points is greater than the threshold, then the signal of the rectangular sliding window is the active segment signal; if the moving average of the data points is greater than the threshold, then the signal of the rectangular sliding window is the active segment signal. h If the moving average of a number of data points is less than or equal to the threshold, then the signal of the rectangular sliding window is the resting segment signal; retain the active segment signal and discard the resting segment signal. S1.3: An overlapping sliding window is used to apply overlapping windowing to the active segment signal to ensure the continuity of signal time.
[0034] Furthermore, a motion intention prediction system for an upper limb rehabilitation robot based on graph-text multimodal fine-grained reasoning includes: Acquisition module: Used to acquire surface muscle electrical signals and preprocess the acquired surface muscle electrical signals; The first calculation module is used to convert the preprocessed surface electromyography signal into the corresponding Gram angle field map through Gram angle field transformation, extract the time-frequency features of the signal through variational mode decomposition and Welch method, and use VGG16 to extract features from the image and LSTM to extract features from the preprocessed data to obtain feature vectors. The second calculation module is used to input the acquired feature data information into the constructed fine-grained inference module based on the cross-attention mechanism to obtain the cross-modal similarity matrix. After dimensionality reduction and random forest classification, the classification result of the surface electromyography signal is obtained.
[0035] The beneficial effects that can be achieved by adopting the above technologies are: This method for predicting the movement intention of an upper limb rehabilitation robot based on image-text multimodal fine-grained reasoning uses Gram angle field transform to convert a one-dimensional signal into a two-dimensional Gram angle field image. It extracts time-frequency data features of electromyography (EMG) signals and image features, performs cross-modal information fusion, and then inputs the data into a classifier for classification. Experimental results show that, compared with traditional methods that use single-dimensional EMG signal features as input models, this method shows better performance in ablation experiments and achieves satisfactory classification accuracy. Attached Figure Description
[0036] Figure 1 This is a flowchart of a method for predicting movement intentions in upper limb rehabilitation robots; Figure 2 This is a schematic diagram of signal activity segment detection and signal windowing; Figure 3 This is a schematic diagram of VMD signal decomposition; Figure 4 This is a schematic diagram of Gram angle field transformation and splicing of Gram angle fields in each channel; Figure 5 This is the flowchart for the fine-grained relational reasoning module; Figure 6 This is an ablation experiment result diagram comparing the motion intention prediction method of upper limb rehabilitation robots with the traditional feature extraction method; Figure 7 This is a flowchart of the online action recognition and prediction process. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0038] like Figure 1 As shown, a method for predicting the movement intention of an upper limb rehabilitation robot based on fine-grained reasoning using graph-text multimodal methods includes the following steps: S1: Synchronously acquire the surface electromyography (EMG) signals corresponding to the given action instructions, and preprocess the surface EMG signals. There are 6 pre-set movements, including elbow horizontal flexion, elbow horizontal extension, shoulder horizontal adduction, shoulder horizontal extension, forearm internal rotation, and a compound movement of forearm internal rotation and elbow horizontal flexion. Preprocessing includes signal filtering and noise reduction, active segment detection, and overlapping windowing; S1.1: Filtering and noise reduction; Specifically, Butterworth high-pass filtering combined with wavelet thresholding is used for denoising. A third-order Butterworth filter is used to remove frequency components below 10Hz. For wavelet thresholding, the db2 wavelet is used to perform a four-level decomposition of the electromyography (EMG) signal. A fixed threshold and a hard threshold function are combined to complete wavelet thresholding denoising, resulting in the denoised surface EMG signal.
[0039] S1.2: As Figure 2 As shown, a moving average combined with a threshold method is used to detect active segments in the denoised surface electromyography (EMG) signal to obtain the active signal segments; specifically, the following steps are included: S1.2.1: Set a rectangular sliding window with a size of [size missing]. h (Number of data points contained within the window), for the surface electromyography signal of each channel, calculate h Moving average of data points; The formula is:
[0040] In the formula, R ( t ) i express t Time-of-sampling, the first i Surface electromyography signals of each channel i ∈[1,5], and is an integer; h Indicates the total number of samples; MAS t,i Indicates the current time t、 No. i Moving average of each channel.
[0041] S1.2.2: Set thresholds based on the resting baseline mean and standard deviation; The formula is:
[0042] In the formula, μ rest,i Indicates the first i The average electromyographic signal of each channel and muscle in a resting state; σ rest,i Indicates the first i Standard deviation of electromyographic signals of individual channels and muscles in a resting state; k orc This is an empirical coefficient; Threshold i Indicates the first i One channel, the threshold of muscle in a resting state.
[0043] S1.2.3: Filter active segment signals.
[0044] For the iEach channel compares the moving average with a set threshold for muscle resting state. If the value within the rectangular sliding window... h If the moving average of the data points is greater than the threshold, then the signal of the rectangular sliding window is the active segment signal; if the moving average of the data points is greater than the threshold, then the signal of the rectangular sliding window is the active segment signal. h If the moving average of the data points is less than or equal to the threshold, then the signal of the rectangular sliding window is the resting segment signal; the active segment signal is retained, and the resting segment signal is discarded.
[0045] That is, when When the rectangular sliding window is in motion, the signal is the active segment signal; otherwise, it is the resting segment signal.
[0046] S1.3: An overlapping sliding window is used to apply overlapping windowing to the active segment signal to ensure the continuity of signal time. The sliding window length is 200ms, and the overlap rate of adjacent windows is set to 40% of the sliding window length. Finally, several standardized windowed electromyographic signals are obtained, which are used as inputs for bi-branch feature extraction.
[0047] S2: Extract time-frequency features from the one-dimensional windowed surface electromyography signal in S1. Specifically, this includes the following steps: S2.1: The first i The surface electromyography signal of each channel is decomposed into M =5 modal components, such as Figure 3 As shown; the formula is:
[0048] In the formula, Indicates in t At that moment, the i The first channel, the first m One modal component, m ∈[1, M ], and is an integer; M Indicates the total number of modal components; R b ( t ) i Indicates the first preprocessed step in S1 i One channel, t Surface electromyography signals at any given time.
[0049] S2.2: Perform a Hilbert transform on each modal component to obtain the analytic signal of that modal component, and then define the bandwidth of each modal component by the norm; The formula for the Hilbert transform is:
[0050] In the formula, Indicates the first i The first channel, the first mEach modal component, t Analyzing the signal at any given moment; Indicates in t At that moment, the i The first channel, the first m One modal component; Indicates to The imaginary part obtained by performing the Hilbert transform; τ This represents the integral variable.
[0051] S2.3: Modulate the analytic signal to baseband and estimate the bandwidth using the L2 norm of the modulated signal gradient.
[0052] The formula is:
[0053] In the formula, Indicates the first i The first channel, the first m The bandwidth of each modal component after transformation is estimated. Indicates the first i The first channel m The center frequency of each modal component; Indicates to t Find the partial derivatives; This represents the L2 norm.
[0054] S2.4: Transform the problem into a variational constraint problem and solve it. The formula is:
[0055] In the formula, This represents the minimum bandwidth.
[0056] S2.5: Since the constrained optimization problem in S2.4 is difficult to solve directly, a quadratic penalty function is introduced by combining the Lagrange operator to strengthen the reconstruction of the augmented Lagrange function, thus transforming it into an unconstrained problem. The formula is as follows:
[0057] In the formula, L ( *,*,* () is the augmented Lagrange function; α cf Represented as a quadratic penalty parameter; λ (t) denotes the Lagrange operator; <*> denotes the inner product. The above equation is solved iteratively using the Alternating Direction Multiplier Method (ADMM). After convergence, the result is output as the nth product. i Channel, First M Modal components ; S2.6: Calculate the first... i Mean of each channel and all modal components and standard deviation And it serves as a one-dimensional time-domain feature.
[0058] S2.7: The solution obtained through ADMM iteration is the first... i Channel, First m Modal components Discretize the sampling and divide it into P The first segment; the first i Channel, First m The first modal component n Each segmented signal is denoted as , n ∈[1, P ], and is an integer.
[0059] Add a window function to each segment, with the following formula:
[0060] In the formula, Indicates the first window after adding a window. n Each segmented signal; w ( n ) indicates the first n Window functions added in segments; Indicates the first i Channel, First m The first modal component n Each segmented signal.
[0061] S2.8: Perform an FFT on the windowed segmented signal, then square the modulus and normalize it to obtain the power spectrum estimate for that segment:
[0062] In the formula, L n Indicates the first n The length of each segmented signal; U This represents the window function energy correction factor; f s Represents frequency variables; P Indicates the total number of segments; Indicates the first i Channel, First m The modal component, the first n Each segmented signal has frequency variables f s Power spectrum at; S2.9: Calculation P The power spectral mean of each segment. The formula is:
[0063] In the formula, Indicates the first i Channel, First m Each modal component in the frequency variable f s The average power spectrum at that location; , F s This indicates the data sampling frequency.
[0064] S2.10: According to the first i Channel, First m Each modal component in the frequency variable f s Average power spectrum at Find the first... i Channels, all modal components in frequency variable f s mean of the power spectrum at point and standard deviation , and serve as a one-dimensional frequency domain feature.
[0065] S3: Extract image features from the one-dimensional windowed surface electromyography signal in S1. This includes the following steps: S3.1: Gram angle field transformation.
[0066] Obtain the i The sequence of one-channel, one-dimensional windowed surface electromyography signals is given by the following formula:
[0067] In the formula, Indicates the first i One channel, one-dimensional windowed surface electromyography signal sequence; Indicates the first K Electromyography (EMG) signals from each sampling point.
[0068] Surface electromyography signal sequence Normalization yields:
[0069] In the formula, Indicates the first i A normalized sequence of one-channel, one-dimensional windowed surface electromyography signals; This represents the result of normalizing the Kth electromyographic signal.
[0070] S3.2: Use the arccos function to convert a one-dimensional time series into a polar coordinate system, the formula is:
[0071] In the formula, Indicates the first iThe angle cosine value of an electromyographic signal sequence.
[0072] S3.3: Two Gram angular field matrices are obtained through two encoding methods: Angular Sum Field (GASF) and Angular Difference Field (GADF); Among them, in the Gram angular field matrix of the angle and the field, the first... x Line number y The elements of the column are:
[0073] In the Gram angular field matrix of the angular difference field, the first... x Line number y The elements of the column are:
[0074] S3.4: Concatenate the Gram angular field matrices of the angle and field and the Gram angular field of the difference field, and then obtain a two-dimensional coded image through mapping, such as... Figure 4 As shown.
[0075]
[0076] In the formula, GAF ( d,s ) represents the concatenated matrix, the first... d Line number s The numerical values of the column elements; Image ( d,s ) indicates the first d Line number s A two-dimensional encoded image of a column.
[0077] S4: Construct a VGG16-LSTM dual-branch feature learning network, inputting one-dimensional time-frequency domain features into the LSTM model and two-dimensional encoded images into the VGG16 model; using real labels of upper limb rehabilitation movements as supervision, optimize the parameters of the two models respectively through backpropagation of loss values. After training convergence, the trained VGG16 model and LSTM model are obtained.
[0078] S5: As Figure 5 As shown, for newly acquired upper limb surface electromyography signals, after preprocessing in S1, time-frequency feature extraction in S2, and image feature extraction in S3, one-dimensional time-frequency domain features and two-dimensional coded images are obtained respectively. These are then input into the model trained in S4 to extract fine-grained features. These features are then fused through a fine-grained alignment module under a multi-head cross-attention mechanism to obtain a high-dimensional feature vector, ultimately completing the motion intent classification and prediction. The specific steps include: S5.1: Decompose the two-dimensional coded image I and the one-dimensional time-frequency feature E respectively; the formula is:
[0079] In the formula, I fine This shows the image after disassembly. E fine This represents the one-dimensional time-frequency characteristics after decomposition; W split,1 and W split,2 This represents the corresponding decomposition weight matrix.
[0080] S5.2: Sub-features are mapped to a unified attribute space through a linear layer, and InfoNCE contrastive loss constraints are used to mine discriminative local features; the formula is:
[0081] In the formula, Indicates the activation function; W I and W E Indicates the mapping weights; B I and B E The bias term is represented by sin(*, *), which represents the cosine similarity. B b Indicates batch size; N n Indicates the number of local features; Indicates the temperature coefficient; S5.3: Calculate the disassembled 2D coded image I fine and the one-dimensional time-frequency characteristics after decomposition E fine The similarity matrix of fine-grained sub-features is used to obtain semantic matching weights through Softmax, thereby achieving local feature alignment:
[0082] In the formula, S Represents the disassembled two-dimensional coded image I fine and the one-dimensional time-frequency characteristics after decomposition E fine Cosine similarity; a match Represents semantic matching weights; Softmax(*) represents the activation function; This represents the 2-norm of a matrix.
[0083] S5.4: Reconstruct one-dimensional time-frequency domain features based on matching weights, making them correspond one-to-one with two-dimensional image features in fine-grained semantics; the formula is:
[0084] In the formula, E fine This represents the one-dimensional time-frequency characteristics after decomposition. Flatten (*) indicates flattening fine-grained high-dimensional features; E fine，aligned This represents the one-dimensional time-frequency characteristic to be flattened; W merge Represents the weight matrix; E aligned This represents the one-dimensional time-frequency domain global features after fine-grained alignment of two-dimensional image features. S5.5: Map the input 2D image features as the query vector and the aligned 1D time-frequency threshold global features as key / value pairs into the multi-head attention space.
[0085] In the formula, Q Represents the query vector. K Represents the key vector. V Represents a value vector. I Represents the features of the input two-dimensional encoded image. It is a learnable linear transformation weight matrix.
[0086] S5.6: Calculate the multi-head attention score, concatenate the multi-head attention outputs, and obtain the enhanced image features; the formula is:
[0087] In the formula, I enhanced Indicates the enhanced image features; W out This represents the attention output matrix; Concat(*) concatenates the attention matrices. Atten This indicates a high level of attention from multiple parties. This represents the 2-norm of a matrix.
[0088] S5.7: High-dimensional fused features are obtained by combining enhanced image features and finely aligned one-dimensional time-frequency domain global features; the formula is:
[0089] In the formula, and The weights set are all 0.5; F fusion This indicates high-dimensional fusion characteristics.
[0090] S5.8: After standardizing the data to eliminate dimensional differences, the covariance matrix of the high-dimensional features is calculated and singular value decomposition is performed. The principal components with a cumulative variance contribution rate of 95% are selected as the dimensionality-reduced features. D fusion ; Dimensionally reduced features D fusion The data is fed into a random forest classification system consisting of multiple trained decision trees, resulting in six classification results for the action.
[0091] right Figure 6 and Figure 7 The analysis is as follows: Figure 6 This is an ablation experiment result diagram comparing the motion intention prediction method of upper limb rehabilitation robots with the traditional feature extraction method; Figure 7 This is the online action recognition and prediction process; as can be seen from the two figures: Figure 6 In the training of the model using this method, after multiple epochs, the training loss decreased rapidly after about 10 epochs, indicating good convergence and excellent fitting effect. The test loss stabilized at about 0.2 in the later stage, indicating that the model has reliable generalization ability. The training and testing accuracy stabilized at 0.98~1.0 in the later stage, indicating that the model has high accuracy for action recognition tasks.
[0092] Figure 7 In this invention, the overall architecture, through multimodal sensor fusion and hierarchical control architecture design, combined with the high-precision recognition model, effectively improves the robustness of the system to interferences such as environmental noise, individual differences, and sensor failures, ensuring the safety and reliability of the upper limb rehabilitation training process.
[0093] Example 2: A motion intention prediction system for an upper limb rehabilitation robot based on fine-grained reasoning using image-text multimodal methods, comprising: Acquisition module: Used to acquire surface muscle electrical signals and preprocess the acquired surface muscle electrical signals; The first calculation module is used to convert the preprocessed surface electromyography signal into the corresponding Gram angle field map through Gram angle field transformation, extract the time-frequency features of the signal through variational mode decomposition and Welch method, and use VGG16 to extract features from the image and LSTM to extract features from the preprocessed data to obtain feature vectors. The second calculation module is used to input the acquired feature data information into the constructed fine-grained inference module based on the cross-attention mechanism to obtain the cross-modal similarity matrix. After dimensionality reduction and random forest classification, the classification result of the surface electromyography signal is obtained.
[0094] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.
Claims
1. A method for predicting the movement intention of an upper limb rehabilitation robot based on fine-grained reasoning using image-text multimodal methods, characterized in that, Includes the following steps: S1: Synchronously acquire the surface electromyography (EMG) signals corresponding to the given action instructions, and preprocess the surface EMG signals. Preprocessing includes signal filtering and noise reduction, active segment detection, and overlapping windowing; the given motion commands include elbow horizontal flexion, elbow horizontal extension, shoulder horizontal adduction, shoulder horizontal extension, forearm internal rotation, and a compound motion of forearm internal rotation and elbow horizontal flexion. S2: Extract time-frequency features from the one-dimensional windowed surface electromyography signal in S1; S3: Extract image features from the one-dimensional windowed surface electromyography signal in S1; S4: Construct a VGG16-LSTM dual-branch feature learning network, input one-dimensional time-frequency domain features into the LSTM model and two-dimensional encoded images into the VGG16 model; use real labels of upper limb rehabilitation movements as supervision, optimize the parameters of the two models respectively through backpropagation of loss values, and obtain the trained VGG16 model and LSTM model after training convergence; S5: For newly acquired upper limb surface electromyography (EMG) signals, after preprocessing in S1, time-frequency feature extraction in S2, and image feature extraction in S3, one-dimensional time-frequency domain features and two-dimensional coded images are obtained respectively. These are then input into the model trained in S4 to extract fine-grained features, which are fused using a fine-grained alignment module under a multi-head cross-attention mechanism to obtain a high-dimensional feature vector. After standardizing the data to eliminate dimensional differences, the high-dimensional features are used to calculate the covariance matrix and perform singular value decomposition. Principal components with a cumulative variance contribution rate of 95% are selected as the dimensionality-reduced features. D fusion ; Dimensionally reduced features D fusion The random forest classification, consisting of multiple trained decision trees, is used to obtain six classification results for the action.
2. The method for predicting the movement intention of an upper limb rehabilitation robot based on fine-grained reasoning using image-text multimodal methods according to claim 1, characterized in that, Step S2 includes the following steps: S2.1: The first i The surface electromyography signal of each channel is decomposed into M =5 modal components, the formula is: ; In the formula, Indicates in t At that moment, the i The first channel, the first m One modal component, m ∈[1, M ], and is an integer; M Indicates the total number of modal components; R b ( t ) i Indicates the first preprocessed step in S1 i One channel, t Surface electromyography signals at any given time; S2.2: Perform a Hilbert transform on each modal component to obtain the analytic signal of that modal component, and then define the bandwidth of each modal component using its norm; the formula is: ； In the formula, Indicates the first i The first channel, the first m Each modal component, t Analyzing the signal at any given moment; Indicates in t At that moment, the i The first channel, the first m One modal component; Indicates to The imaginary part obtained by performing the Hilbert transform; τ Represents the integral variable; S2.3: The analytic signal is modulated and shifted to baseband, and the bandwidth is estimated using the L2 norm of the modulated signal gradient; the formula is: ； In the formula, Indicates the first i The first channel, the first m The bandwidth of each modal component after transformation is estimated. Indicates the first i The first channel m The center frequency of each modal component; Indicates to t Find the partial derivatives; Represents the L2 norm; S2.4: Transform the problem into a variational constraint problem for solution; the formula is: ； In the formula, Indicates minimum bandwidth; S2.5: Combining the formula in S2.4 with the Lagrange operator to introduce a quadratic penalty function to strengthen the reconstruction of the augmented Lagrange function, transforming it into an unconstrained problem, the formula is: ； In the formula, L ( *,*,* () is the augmented Lagrangian function; α cf Represented as a quadratic penalty parameter; λ (t) denotes the Lagrange operator; <*> denotes the inner product; The above equation is solved iteratively using the Alternating Direction Multiplier Method (ADMM). After the iteration converges, the output is the first... i Channel, First M Modal components ; S2.6: Calculate the first... i Mean of each channel and all modal components and standard deviation , and serve as a one-dimensional time-domain feature; S2.7: The solution obtained through ADMM iteration is the first... i Channel, First m Modal components Discretize the sampling and divide it into P The first segment; the first i Channel, First m The first modal component n Each segmented signal is denoted as , n ∈[1, P ], and is an integer; Add a window function to each segment, with the following formula: ； In the formula, Indicates the first window after adding a window. n Each segmented signal; w ( n ) indicates the first n Window functions added in segments; Indicates the first i Channel, First m The first modal component n Each segmented signal; S2.8: Perform FFT on the windowed segmented signal, then square the modulus and normalize it to obtain the power spectrum estimate of the segment; ； In the formula, L n Indicates the first n The length of each segmented signal; U This represents the window function energy correction factor; f s Represents frequency variables. , F s Indicates the data sampling frequency; P Indicates the total number of segments; Indicates the first i Channel, First m The modal component, the first n Each segmented signal has frequency variables f s The power spectrum at that location; S2.9: Calculation P The power spectral mean of each segment; the formula is: ； In the formula, Indicates the first i Channel, First m Each modal component in the frequency variable f s The average power spectrum at that location; S2.10: According to the first i Channel, First m Each modal component in the frequency variable f s Average power spectrum at Find the first... i Channels, all modal components in frequency variable f s mean of the power spectrum at point and standard deviation And it serves as a one-dimensional frequency domain feature.
3. The method for predicting the movement intention of an upper limb rehabilitation robot based on fine-grained reasoning using image-text multimodal methods according to claim 1, characterized in that, Step S3 includes the following steps: S3.1: Gram angle field transformation; Obtain the i The sequence of one-channel, one-dimensional windowed surface electromyography signals is given by the following formula: ； In the formula, Indicates the first i One channel, one-dimensional windowed surface electromyography signal sequence; Indicates the first K Electromyography signals from each sampling point; surface electromyography signal sequence Normalization yields: ； In the formula, Indicates the first i A normalized sequence of one-channel, one-dimensional windowed surface electromyography signals; This represents the result of normalizing the Kth electromyographic signal; S3.2: Use the arccos function to convert a one-dimensional time series into a polar coordinate system, the formula is: ； In the formula, Indicates the first i The angle cosine value of an electromyographic signal sequence; S3.3: Two Gram angular field matrices are obtained through two encoding methods: angular sum field and angular difference field. Among them, in the Gram angular field matrix of the angle and the field, the first... x Line number y The elements of the column are: ； In the Gram angular field matrix of the angular difference field, the first... x Line number y The elements of the column are: ； S3.4: Concatenate the Gram angular field matrices of the angle and field and the Gram angular field difference, and then obtain the two-dimensional coded image through mapping: ； In the formula, GAF ( d,s ) represents the concatenated matrix, the first... d Line number s The numerical values of the column elements; Image ( d,s ) indicates the first d Line number s A two-dimensional encoded image of a column.
4. The method for predicting the movement intention of an upper limb rehabilitation robot based on fine-grained reasoning using image-text multimodal methods according to claim 1, characterized in that, Step S5 includes the following steps: S5.1: Decompose the two-dimensional coded image I and the one-dimensional time-frequency feature E respectively; the formula is: ； In the formula, I fine This shows the image after disassembly; E fine This represents the one-dimensional time-frequency characteristics after decomposition. W split,1 and W split,2 This represents the corresponding decomposition weight matrix; S5.2: Sub-features are mapped to a unified attribute space through a linear layer, and InfoNCE contrastive loss constraints are used to mine discriminative local features; the formula is: ； In the formula, Indicates the activation function; W I and W E Indicates the mapping weights; B I and B E The bias term is represented by sin(*, *), which represents the cosine similarity. B b Indicates batch size; N n Indicates the number of local features; Indicates the temperature coefficient; S5.3: Calculate the disassembled two-dimensional coded image I fine and the one-dimensional time-frequency characteristics after disassembly E fine The similarity matrix of fine-grained sub-features is used to obtain semantic matching weights through Softmax, thereby achieving local feature alignment: ; In the formula, S Represents the disassembled two-dimensional coded image I fine and the one-dimensional time-frequency characteristics after disassembly E fine Cosine similarity; a match Represents semantic matching weights; Softmax(*) represents the activation function; Represents the 2-norm of a matrix; S5.4: Reconstruct one-dimensional time-frequency domain features based on matching weights, making them correspond one-to-one with two-dimensional image features in fine-grained semantics; the formula is: ; In the formula, E fine This represents the one-dimensional time-frequency characteristics after decomposition. Flatten (*) indicates flattening fine-grained high-dimensional features; E fine，aligned This represents the one-dimensional time-frequency characteristic to be flattened; W merge Represents the weight matrix; E aligned This represents the one-dimensional time-frequency domain global features after fine-grained alignment of two-dimensional image features. S5.5: The input two-dimensional image features are used as the query vector, and the aligned one-dimensional time-frequency threshold global features are used as key-value pairs, which are then mapped to the multi-head attention space. ; In the formula, Q Represents the query vector. K Represents the key vector. V Represents a value vector. I Represents the features of the input two-dimensional encoded image. The weight matrix is a learnable linear transformation. S5.6: Calculate the multi-head attention score, concatenate the multi-head attention outputs, and obtain the enhanced image features; the formula is: ; In the formula, I enhanced Indicates the enhanced image features; W out This represents the attention output matrix; Concat(*) concatenates the attention matrices. Atten This indicates a high level of attention from multiple parties. Represents the 2-norm of a matrix; S5.7: High-dimensional fused features are obtained by combining enhanced image features and finely aligned one-dimensional time-frequency domain global features; the formula is: ; In the formula, and The weights set; F fusion Represents high-dimensional fusion features; S5.8: After standardizing the data to eliminate dimensional differences, the covariance matrix of the high-dimensional features is calculated and singular value decomposition is performed. The principal components with a cumulative variance contribution rate of 95% are selected as the dimensionality-reduced features. D fusion ; Dimensionally reduced features D fusion The random forest classification, consisting of multiple trained decision trees, is used to obtain six classification results for the action.
5. The method for predicting the movement intention of an upper limb rehabilitation robot based on fine-grained reasoning using image-text multimodal methods according to claim 1, characterized in that, Step S1 includes the following steps: S1.1: Filtering and noise reduction; S1.2: Using a moving average combined with a threshold method, active segment detection is performed on the denoised surface electromyography signal to obtain the active signal segments; specifically, the following steps are included: S1.2.1: Set a rectangular sliding window with a size of [size missing]. h For the surface electromyography signal of each channel, calculate h The moving average of 1 data point; the formula is: ; In the formula, R ( t ) i express t Time-of-sampling, the first i Surface electromyography signals of each channel i ∈[1,5], and is an integer; h Indicates the total number of samples; MAS t,i Indicates the current time t、 No. i Moving average of each channel; S1.2.2: Set thresholds based on the resting baseline mean and standard deviation; The formula is: ; In the formula, μ rest,i Indicates the first i The average electromyographic signal of each channel and muscle in a resting state; σ rest,i Indicates the first i Standard deviation of electromyographic signals of individual channels and muscles in a resting state; k orc This is an empirical coefficient; Threshold i Indicates the first i One channel, the threshold of muscle in a resting state; S1.2.3: Filter active segment signals; For the i Each channel compares the moving average with a set threshold for muscle resting state. If the value within the rectangular sliding window... h If the moving average of the data points is greater than the threshold, then the signal of the rectangular sliding window is the active segment signal; if the moving average of the data points is greater than the threshold, then the signal of the rectangular sliding window is the active segment signal. h If the moving average of a number of data points is less than or equal to the threshold, then the signal of the rectangular sliding window is the resting segment signal; retain the active segment signal and discard the resting segment signal. S1.3: An overlapping sliding window is used to apply overlapping windowing to the active segment signal to ensure the continuity of signal time.
6. A motion intention prediction system for an upper limb rehabilitation robot based on fine-grained reasoning using image-text multimodal methods, characterized in that, include: Acquisition module: Used to acquire surface muscle electrical signals and preprocess the acquired surface muscle electrical signals; The first calculation module is used to convert the preprocessed surface electromyography signal into the corresponding Gram angle field map through Gram angle field transformation, extract the time-frequency features of the signal through variational mode decomposition and Welch method, and use VGG16 to extract features from the image and LSTM to extract features from the preprocessed data to obtain feature vectors. The second calculation module is used to input the acquired feature data information into the constructed fine-grained inference module based on the cross-attention mechanism to obtain the cross-modal similarity matrix. After dimensionality reduction and random forest classification, the classification result of the surface electromyography signal is obtained.