Upper arm muscle strength estimation method based on dual-branch self-attention mechanism network

Abstract

The present invention relates to a method for estimating upper arm muscle strength based on a dual-branch self-attention mechanism network, comprising: obtaining a normalized original electrical impedance imaging voltage sequence and a normalized true force sequence; obtaining a two-dimensional electrical impedance image and a frequency domain representation of the voltage sequence; obtaining an electrical impedance image tensor, a voltage frequency domain tensor, and a true force tensor; constructing a dual-branch self-attention mechanism network, inputting the electrical impedance image tensor and the voltage frequency domain tensor into the dual-branch self-attention mechanism network, and outputting a muscle strength estimation result; comparing the muscle strength estimation result with the true force sequence collected by a six-dimensional force sensor. The present invention effectively fills in the deep muscle information missing from other biological signals such as electromyographic signals and muscle sound signals; by extracting more global information to calculate more complex features, the accuracy of detection is improved; and it can more accurately achieve non-destructive upper arm muscle strength estimation, with the advantages of being lightweight, having high estimation accuracy, and having good portability.

Description

Technical Field

[0001] The present invention relates to the technical field of human biological signal processing and muscle strength estimation, and in particular to an upper arm muscle strength estimation method based on a dual-branch self-attention mechanism network. Background Art

[0002] Muscle force estimation can reflect a person's movement intention, muscle fatigue, and muscle health, and is a crucial research topic for biomechanics, human-computer interaction, sports, and rehabilitation medicine. Data acquisition methods can be categorized as either kinematics / dynamics-based or bioinformatics-based. In recent years, due to the rapid development of deep learning technology and its hardware cost advantages, bioinformatics-based methods have become a research hotspot.

[0003] Traditional biometric muscle force estimation methods use human biosignals such as electromyography and muscle sounds. Early attempts to extract handcrafted features from these signals allowed for further classification and recognition. In recent years, deep learning-based methods have significantly improved recognition performance compared to earlier handcrafted feature-based methods. Deep neural networks, particularly convolutional neural networks, are able to effectively extract spatiotemporal features from the data. However, these signals can only detect superficial muscle information, and crosstalk between muscles can negatively impact the estimation results.

[0004] Electrical impedance tomography (EIT), a novel biosensor, records changes in muscle cross-sections by virtue of its inherent two-dimensional image information. Current research on EIT has expanded into numerous fields, including clinical medicine, human-computer interaction, and sensors. In this area, EIT's sensitivity to impedance changes effectively detects changes in human muscle tissue during skeletal muscle contraction, facilitating identification. Therefore, it is necessary to design a novel upper arm muscle force estimation method based on the characteristics of EIT signals. Summary of the Invention

[0005] In order to solve the problem that traditional estimation methods can only detect superficial muscle information, ignore deep muscle information during muscle contraction, and ignore the frequency domain signal characteristics of electrical impedance imaging itself, the purpose of the present invention is to provide a lightweight, high-precision, and non-destructive upper arm muscle strength estimation method based on a dual-branch self-attention mechanism network.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for estimating upper arm muscle strength based on a dual-branch self-attention mechanism network, comprising:

[0007] (1) In one measurement cycle, the original electrical impedance imaging voltage signal generated by the contraction of the upper arm muscles of the human body is measured by the electrical impedance imaging sensor and normalized to obtain the normalized original electrical impedance imaging voltage sequence Among them, M1 is the total number of normalized original electrical impedance imaging voltage sequences recorded in a measurement cycle, i is the index subscript of the normalized original electrical impedance imaging voltage sequence at the current moment, i is an integer and 1≤i≤M1; at the same time, the six-dimensional force sensor is used to collect the forces Fx, Fy, Fz in the three directions and the moments Mz, My, Mz in the three directions generated by the contraction of the upper arm muscles of the human body. Fx, Fy, Fz and Mz, My, Mz together constitute the real force sequence, and the real force sequence is normalized to obtain the normalized real force sequence Where M2 is the total number of true forces recorded in a measurement cycle, j is the index subscript of the true force at the current moment, j is an integer and 1≤j≤M2;

[0008] (2) The normalized original electrical impedance imaging voltage sequence Perform one-step Gauss-Newton image reconstruction and Fourier transform respectively to obtain a two-dimensional electrical impedance image Voltage sequence frequency domain characterization Wherein, M3 is the total number of two-dimensional electrical impedance images recorded in one measurement cycle, v is the index subscript of the two-dimensional electrical impedance image at the current moment, v is an integer and 1≤v≤M3; M4 is the total number of frequency domain representations of the voltage sequence recorded in one measurement cycle, u is the index subscript of the frequency domain representation of the voltage sequence at the current moment, u is an integer and 1≤u≤M4;

[0009] (3) Frequency domain representation of voltage sequence 2D electrical impedance image and the normalized true force series The dataset is composed and divided into batches to obtain the electrical impedance image tensor V of size batchsize×1×64×64 IMG , the voltage frequency domain tensor V of size batchsize×1×28 fft and the true force tensor V of size batchsize×1×6 F , batchsize indicates the amount of data contained in each batch;

[0010] (4) Constructing a dual-branch self-attention mechanism network, which consists of a first branch network, a second branch network and a time-frequency domain feature fusion module; converting the electrical impedance image tensor V IMG , voltage frequency domain tensor V fft Input the dual-branch self-attention mechanism network, and the dual-branch self-attention mechanism network outputs the muscle force estimation result;

[0011] (5) The obtained muscle force estimation results are compared with the actual force sequence collected by the six-dimensional force sensor, and the mean square error is used to reflect the degree of difference between the two.

[0012] The step (1) specifically includes the following steps:

[0013] (1a) Repeat the measurement of the electrical impedance imaging sensor multiple times, and define each measurement from the beginning to the end as a measurement cycle; in each measurement cycle, the excitation current is injected along the voltage on each adjacent electrode, and the response voltage V on each adjacent electrode is obtained. l , N is the number of electrodes and N=8, that is, The 28 response voltage values ​​obtained by 8 electrodes in the adjacent excitation measurement mode at the current moment are set as a frame; after continuous measurement, the original voltage sequence within a measurement cycle is obtained vol(i raw )={V1,V2,...,V 28} is the response voltage value of the current frame; the original voltage sequence collected in each frame is normalized to the maximum and minimum values, and all voltages are linearly shrunk to [0,1]:

[0014]

[0015] Among them, vol is the response voltage value at the current moment, vol max is the maximum response voltage value in the current measurement cycle, vol min is the minimum response voltage value in the current measurement cycle; the normalized original electrical impedance imaging voltage sequence is obtained Set the voltage sequence vol measured in the first frame of each measurement norm (1) is the initial value;

[0016] (1b) In each measurement cycle of the six-dimensional force sensor, the real force sequence collected at the current moment is recorded as force(j) = {F x (j),F y (j),F z (j),M x (j),M y (j),M z (j)}, perform maximum and minimum normalization on the collected real force to obtain the normalized real force sequence

[0017] The step (2) specifically includes the following steps:

[0018] (2a) Reconstruct the 2D electrical impedance image using the one-step Gauss-Newton reconstruction algorithm:

[0019] σ=σ0-[[J(σ0)] T J(σ0)+αdiag(J T J)] -1 [J(σ0)]T [V(σ0)-U]

[0020] Where σ is the conductivity distribution vector, σ0 is the initial conductivity distribution, α is the regularization factor, and J T J is the regularization matrix, V(σ0) is the calculated boundary voltage under the conductivity distribution of σ0, J(σ0) is the Jacobian matrix, and U is the measured boundary voltage;

[0021] After reconstruction, the two-dimensional electrical impedance image is obtained, which is recorded as Where img(v) is the two-dimensional electrical impedance image of the current frame, which is used to reflect the conductivity distribution information of the cross section of the upper arm muscle of the human body at the current moment;

[0022] (2b) The normalized original electrical impedance imaging voltage sequence The frequency domain information contained in is decoupled to obtain the frequency domain representation of the voltage sequence

[0023]

[0024] Among them, M1=M3=M4=M, M is the total number of samples in a measurement cycle; fft(u) is the amplitude spectrum of the frequency domain signal of the original voltage sequence of the current frame, which is used to reflect the frequency domain information of the cross-section of the upper arm muscle of the human body at the current moment.

[0025] In step (4), the first branch network includes an image serialization module and a first multi-head self-attention mechanism module, and the second branch network includes a first fully connected layer and a second multi-head self-attention mechanism module;

[0026] The time-frequency domain feature fusion module includes a first global average pooling layer, a second global average pooling layer, a first RELU activation layer, a second RELU activation layer, a first Sigmoid activation layer, a second Sigmoid activation layer, and second, third, fourth, fifth, sixth, seventh, eighth, and ninth fully connected layers;

[0027] First, the electrical impedance image tensor V IMG Send to the first branch network for processing: the image serialization module converts V IMG The image is divided into patchsize×patchsize small images, where patchsize=16, which means the image is divided into 16 rows and 16 columns. The segmented image is first linearly transformed, and then position coding is added to each part of the linearly transformed image. Finally, the image data of each part after adding position coding is spliced ​​together, and the whole data obtained by splicing is As a sequence, it is sent to the first multi-head self-attention mechanism module for operation; the first multi-head self-attention mechanism module adopts the scaled dot product attention mechanism, and the formula is as follows:

[0028]

[0029] Among them, Q, K and V represent query, key and value respectively, Attention represents the output tensor, Indicates the number of dimensions of matrix Q;

[0030] Will By multiplying the trainable weight matrix to perform linear transformation, three initial vector representations are obtained. The formula is as follows:

[0031] Q=W q V IMG

[0032] K=W k V IMG

[0033] V=W v V IMG

[0034] Among them, W q 、W k and W v Both represent trainable weight matrices;

[0035] Substitute the obtained Q, K and V into formula (1) to calculate the output tensor Attention; IMG After learning independently multiple times, the multiple outputs are concatenated and fed into a fully connected layer of the first multi-head self-attention mechanism module to obtain a feature tensor of size batchsize×patchsize×dim. The first dimension batchsize represents the amount of data contained in each batch, the second dimension patchsize=16, and the third dimension dim=1024. Dim represents the length of each image vector after image serialization. The mean is calculated on the second dimension, and finally a time domain feature tensor U with a size of 8×1024 is obtained. T ;

[0036] The voltage frequency domain tensor V fft Send it to the second branch network for processing: First, the size of V is batchsize×1×28 fft Input the first fully connected layer and convert V fft Expand in the third dimension to obtain a feature tensor of size batchsize×1×1024, which is then fed into the second multi-head self-attention mechanism module to obtain a frequency domain feature tensor U of size 8×1024. F ;

[0037] The obtained time domain feature tensor U T and frequency domain feature tensor U FReshaped into 8×1024×1 and sent to the time-frequency domain feature fusion module: First, the global average pooling operation is performed through the first global average pooling layer and the second global average pooling layer to convert the time domain feature tensor U T and frequency domain feature tensor U F Compress to obtain the compressed time domain feature tensor and frequency domain feature tensor Then and They are sent to the second fully connected layer and the fourth fully connected layer respectively, and then the channel dimension is reduced by the first ReLU activation layer and the second ReLU activation layer respectively, and then sent to the third fully connected layer and the fifth fully connected layer respectively, and the channel dimension is reduced by the first Sigmoid activation layer and the second Sigmoid activation layer respectively. and Each element in the vector is compressed to between 0 and 1 to obtain the weight of each channel and Next, the weight and the time domain feature tensor U T After multiplication, it is sent to the sixth fully connected layer to obtain the time domain channel attention feature tensor The weight and the frequency domain feature tensor U F After multiplication, it is sent to the seventh fully connected layer to obtain the frequency domain channel attention feature tensor At the same time, U T 、U F The downsampled time domain feature tensors are respectively sent to the eighth and ninth fully connected layers to obtain the downsampled time domain feature tensors. And the downsampled frequency domain feature tensor Then, the frequency domain channel attention feature tensor And the downsampled time domain feature tensor After matrix multiplication, the time domain feature tensor after downsampling is obtained. Add together to get the first fused tensor U Fusion1 ; At the same time, the time domain channel attention feature tensor And the downsampled frequency domain feature tensor After matrix multiplication, the frequency domain feature tensor after downsampling is obtained Add together to get the second fusion tensor U Fusion2 ; The first fused tensor U Fusion1 , the second fused tensor U Fusion2 Add together to get an output of size 8×6 The specific formula is as follows:

[0038]

[0039] Where GAP represents global average pooling, W1, W2, W3, and W4 all represent fully connected layers, δ and σ represent the ReLU activation function and the Sigmoid activation function, respectively, and ⊙ is the matrix element multiplication.

[0040] It can be seen from the above technical solution that the beneficial effects of the present invention are: First, the present invention takes the original electrical impedance signal of the upper arm muscle as the research object, effectively filling the deep muscle information missing from other biological signals such as electromyographic signals and muscle sound signals; Second, the present invention extracts the frequency domain signal of the original electrical impedance imaging voltage signal and the two-dimensional electrical impedance image reconstructed by the one-step Newton-Gaussian method from the original electrical impedance signal, and calculates more complex features by extracting more global information, thereby improving the accuracy of detection; Third, the present invention designs a time-frequency domain feature fusion module to fuse the time domain features and frequency domain features of the original electrical impedance imaging voltage signal, and can obtain more time-frequency domain features at the same time; Fourth, compared with the typical neural network algorithm, the present invention can more accurately realize the estimation of the size of the upper arm muscle without damage. At the same time, the present invention has the advantages of lightweight, high estimation accuracy, and good portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the method of the present invention;

[0042] Figure 2 Schematic diagram of the excitation-measurement sequence of the electrical impedance sensor in the present invention;

[0043] Figure 3 Schematic diagram of the muscle strength estimation experimental process in the present invention;

[0044] Figure 4 This is a flow chart of preprocessing the original electrical impedance imaging voltage signal in the present invention;

[0045] Figure 5 A schematic diagram of the structure of the first branch network and the second branch network in the present invention;

[0046] Figure 6 It is a structural diagram of the time-frequency domain feature fusion module in the present invention. DETAILED DESCRIPTION

[0047] like Figure 1 As shown in FIG, a method for estimating upper arm muscle strength based on a dual-branch self-attention mechanism network includes:

[0048] (1) In one measurement cycle, the original electrical impedance imaging voltage signal generated by the contraction of the upper arm muscles of the human body is measured by the electrical impedance imaging sensor and normalized to obtain the normalized original electrical impedance imaging voltage sequence Among them, M1 is the total number of normalized original electrical impedance imaging voltage sequences recorded in a measurement cycle, i is the index subscript of the normalized original electrical impedance imaging voltage sequence at the current moment, i is an integer and 1≤i≤M1; at the same time, the six-dimensional force sensor is used to collect the forces Fx, Fy, Fz in the three directions and the moments Mz, My, Mz in the three directions generated by the contraction of the upper arm muscles of the human body. Fx, Fy, Fz and Mz, My, Mz together constitute the real force sequence, and the real force sequence is normalized to obtain the normalized real force sequence Where M2 is the total number of true forces recorded in a measurement cycle, j is the index subscript of the true force at the current moment, j is an integer and 1≤j≤M2;

[0049] (2) The normalized original electrical impedance imaging voltage sequence Perform one-step Gauss-Newton image reconstruction and Fourier transform respectively to obtain a two-dimensional electrical impedance image Voltage sequence frequency domain characterization Wherein, M3 is the total number of two-dimensional electrical impedance images recorded in one measurement cycle, v is the index subscript of the two-dimensional electrical impedance image at the current moment, v is an integer and 1≤v≤M3; M4 is the total number of frequency domain representations of the voltage sequence recorded in one measurement cycle, u is the index subscript of the frequency domain representation of the voltage sequence at the current moment, u is an integer and 1≤u≤M4;

[0050] (3) Frequency domain representation of voltage sequence 2D electrical impedance image and the normalized true force series The dataset is composed and divided into batches to obtain the electrical impedance image tensor V of size batchsize×1×64×64 IMG , the voltage frequency domain tensor V of size batchsize×1×28 fft and the true force tensor V of size batchsize×1×6 F , batchsize indicates the amount of data contained in each batch;

[0051] (4) Constructing a dual-branch self-attention mechanism network, which consists of a first branch network, a second branch network and a time-frequency domain feature fusion module; converting the electrical impedance image tensor V IMG , voltage frequency domain tensor V fft Input the dual-branch self-attention mechanism network, and the dual-branch self-attention mechanism network outputs the muscle force estimation result;

[0052] (5) The obtained muscle force estimation results are compared with the actual force sequence collected by the six-dimensional force sensor, and the mean square error is used to reflect the degree of difference between the two.

[0053] The step (1) specifically includes the following steps:

[0054] (1a) Figure 2 As shown, the measurement of the electrical impedance imaging sensor is repeated multiple times, and each measurement from the beginning to the end is set as a measurement cycle; in each round of measurement cycle, the excitation current is injected along the voltage on each adjacent electrode in turn, and the response voltage V on each adjacent electrode is obtained. l , N is the number of electrodes and N=8, that is, The 28 response voltage values ​​obtained by 8 electrodes in the adjacent excitation measurement mode at the current moment are set as a frame; after continuous measurement, the original voltage sequence within a measurement cycle is obtained vol(i raw )={V1,V2,...,V 28} is the response voltage value of the current frame; the original voltage sequence collected in each frame is normalized to the maximum and minimum values, and all voltages are linearly shrunk to [0,1]:

[0055]

[0056] Among them, vol is the response voltage value at the current moment, vol max is the maximum response voltage value in the current measurement cycle, vol min is the minimum response voltage value in the current measurement cycle; the normalized original electrical impedance imaging voltage sequence is obtained Set the voltage sequence vol measured in the first frame of each measurement norm (1) is the initial value;

[0057] (1b) Figure 3 As shown, in each measurement cycle of the six-dimensional force sensor, the real force sequence collected at the current moment is recorded as force(j)={F x (j),F y (j),F z (j),M x (j),M y (j),M z (j)}, perform maximum and minimum normalization on the collected real force to obtain the normalized real force sequence

[0058]

[0059] like Figure 4 As shown, the first 100 groups of original electrical impedance imaging voltage signals are preprocessed, and the step (2) specifically includes the following steps:

[0060] (2a) Reconstruct the 2D electrical impedance image using the one-step Gauss-Newton reconstruction algorithm:

[0061] σ=σ0-[[J(σ0)] T J(σ0)+αdiag(J T J)] -1 [J(σ0)] T [V(σ0)-U]

[0062] Where σ is the conductivity distribution vector, σ0 is the initial conductivity distribution, α is the regularization factor, and J T J is the regularization matrix, V(σ0) is the calculated boundary voltage under the conductivity distribution of σ0, J(σ0) is the Jacobian matrix, and U is the measured boundary voltage;

[0063] After reconstruction, the two-dimensional electrical impedance image is obtained, which is recorded as Where img(v) is the two-dimensional electrical impedance image of the current frame, which is used to reflect the conductivity distribution information of the cross section of the upper arm muscle of the human body at the current moment;

[0064] (2b) The normalized original electrical impedance imaging voltage sequence The frequency domain information contained in is decoupled to obtain the frequency domain representation of the voltage sequence

[0065]

[0066] Among them, M1=M3=M4=M, M is the total number of samples in a measurement cycle; fft(u) is the amplitude spectrum of the frequency domain signal of the original voltage sequence of the current frame, which is used to reflect the frequency domain information of the cross-section of the upper arm muscle of the human body at the current moment.

[0067] In step (4), if Figure 5 As shown, the first branch network includes an image serialization module and a first multi-head self-attention mechanism module, and the second branch network includes a first fully connected layer and a second multi-head self-attention mechanism module;

[0068] like Figure 6 As shown, the time-frequency domain feature fusion module includes a first global average pooling layer, a second global average pooling layer, a first RELU activation layer, a second RELU activation layer, a first Sigmoid activation layer, a second Sigmoid activation layer, and second, third, fourth, fifth, sixth, seventh, eighth, and ninth fully connected layers;

[0069] First, the electrical impedance image tensor V IMG Send to the first branch network for processing: the image serialization module converts V IMGThe image is divided into patchsize×patchsize small images, where patchsize=16, which means the image is divided into 16 rows and 16 columns. The segmented image is first linearly transformed, and then position coding is added to each part of the linearly transformed image. Finally, the image data of each part after adding position coding is spliced ​​together, and the whole data obtained by splicing is As a sequence, it is sent to the first multi-head self-attention mechanism module for operation; the first multi-head self-attention mechanism module adopts the scaled dot product attention mechanism, and the formula is as follows:

[0070]

[0071] Among them, Q, K and V represent query, key and value respectively, Attention represents the output tensor, Indicates the number of dimensions of matrix Q;

[0072] Will By multiplying the trainable weight matrix to perform linear transformation, three initial vector representations are obtained. The formula is as follows:

[0073] Q=W q V IMG

[0074] K=W k V IMG

[0075] V=W v V IMG

[0076] Among them, W q 、W k and W v Both represent trainable weight matrices;

[0077] Substitute the obtained Q, K and V into formula (1) to calculate the output tensor Attention; IMG After learning independently multiple times, the multiple outputs are concatenated and fed into a fully connected layer of the first multi-head self-attention mechanism module to obtain a feature tensor of size batchsize×patchsize×dim. The first dimension batchsize represents the amount of data contained in each batch, the second dimension patchsize=16, and the third dimension dim=1024. Dim represents the length of each image vector after image serialization. The mean is calculated on the second dimension, and finally a time domain feature tensor U with a size of 8×1024 is obtained. T ;

[0078] The voltage frequency domain tensor V fftSend it to the second branch network for processing: First, the size of V is batchsize×1×28 fft Input the first fully connected layer and convert V fft Expand in the third dimension to obtain a feature tensor of size batchsize×1×1024, which is then fed into the second multi-head self-attention mechanism module to obtain a frequency domain feature tensor U of size 8×1024. F ;

[0079] The obtained time domain feature tensor U T and frequency domain feature tensor U F Reshaped into 8×1024×1 and sent to the time-frequency domain feature fusion module: First, the global average pooling operation is performed through the first global average pooling layer and the second global average pooling layer to convert the time domain feature tensor U T and frequency domain feature tensor U F Compress to obtain the compressed time domain feature tensor and frequency domain feature tensor Then and They are sent to the second fully connected layer and the fourth fully connected layer respectively, and then the channel dimension is reduced by the first ReLU activation layer and the second ReLU activation layer respectively, and then sent to the third fully connected layer and the fifth fully connected layer respectively, and the channel dimension is reduced by the first Sigmoid activation layer and the second Sigmoid activation layer respectively. and Each element in the vector is compressed to between 0 and 1 to obtain the weight of each channel and Next, the weight and the time domain feature tensor U T After multiplication, it is sent to the sixth fully connected layer to obtain the time domain channel attention feature tensor The weight and the frequency domain feature tensor U F After multiplication, it is sent to the seventh fully connected layer to obtain the frequency domain channel attention feature tensor At the same time, U T 、U F The downsampled time domain feature tensors are respectively sent to the eighth and ninth fully connected layers to obtain the downsampled time domain feature tensors. And the downsampled frequency domain feature tensor Then, the frequency domain channel attention feature tensor And the downsampled time domain feature tensor After matrix multiplication, the time domain feature tensor after downsampling is obtained. Add together to get the first fused tensor U Fusion1 ; At the same time, the time domain channel attention feature tensor And the downsampled frequency domain feature tensor After matrix multiplication, the frequency domain feature tensor after downsampling is obtained Add together to get the second fusion tensor U Fusion2 ; The first fused tensor U Fusion1 , the second fused tensor U Fusion2 Add together to get an output of size 8×6 The specific formula is as follows:

[0080]

[0081] Where GAP represents global average pooling, W1, W2, W3, and W4 represent fully connected layers, δ and σ represent ReLU activation functions and Sigmoid activation functions, respectively, and ⊙ represents matrix element multiplication.

[0082] As shown in Table 1, Table 1 compares the recognition effects of the present invention and multiple upper arm muscle strength estimation methods based on traditional machine learning methods on the test set:

[0083] Table 1

[0084]

[0085] In Table 1, LSTM, VGG16, ResNet18, DensNet121, ShuffleNet_v2_x0_5, MobileNet_v2, EfficientNet_b0, and ConvNeXt are upper arm muscle strength estimation methods based on traditional machine learning methods. By comparing the experimental results in the table, it can be found that the present invention can more accurately realize the estimation of the size of the upper arm muscle strength without damage compared with the upper arm muscle strength estimation method based on the traditional machine learning method. At the same time, the present invention has the advantages of lightweight, high estimation accuracy, and good portability. By taking the original electrical impedance imaging voltage signal as the research object, the present invention effectively fills the deep muscle information missing from other biological signals such as electromyographic signals and muscle sound signals. The present invention extracts more global information from the original electrical impedance signal to calculate more complex features, thereby improving the accuracy of detection. At the same time, the present invention designs a time-frequency domain feature fusion module to fuse the time domain features and frequency domain features of the original electrical impedance imaging voltage signal, so that more time-frequency domain features can be obtained at the same time.

Claims

1. A method for estimating upper arm muscle strength based on a dual-branch self-attention mechanism network, characterized by: The method comprises the following steps in sequence: (1) In one measurement cycle, the original electrical impedance imaging voltage signal generated by the contraction of the upper arm muscles of the human body is measured by the electrical impedance imaging sensor and normalized to obtain the normalized original electrical impedance imaging voltage sequence. Among them, M1 is the total number of normalized original electrical impedance imaging voltage sequences recorded in a measurement cycle, i is the index subscript of the normalized original electrical impedance imaging voltage sequence at the current moment, i is an integer and 1≤i≤M1; at the same time, the six-dimensional force sensor is used to collect the forces Fx, Fy, Fz in the three directions and the moments Mz, My, Mz in the three directions generated by the contraction of the upper arm muscles of the human body. Fx, Fy, Fz and Mz, My, Mz together constitute the real force sequence, and the real force sequence is normalized to obtain the normalized real force sequence Where M2 is the total number of true forces recorded in a measurement cycle, j is the index subscript of the true force at the current moment, j is an integer and 1≤j≤M2; (2) Normalized original electrical impedance imaging voltage sequence Perform one-step Gauss-Newton image reconstruction and Fourier transform respectively to obtain a two-dimensional electrical impedance image Voltage sequence frequency domain characterization Wherein, M3 is the total number of two-dimensional electrical impedance images recorded in one measurement cycle, v is the index subscript of the two-dimensional electrical impedance image at the current moment, v is an integer and 1≤v≤M3; M4 is the total number of frequency domain representations of the voltage sequence recorded in one measurement cycle, u is the index subscript of the frequency domain representation of the voltage sequence at the current moment, u is an integer and 1≤u≤M4; (3) Frequency domain characterization of voltage sequence 2D electrical impedance image and the normalized true force series The dataset is composed and divided into batches to obtain the electrical impedance image tensor V of size batchsize×1×64×64 IMG , the voltage frequency domain tensor V of size batchsize×1×28 fft and the true force tensor V of size batchsize×1×6 F , batchsize indicates the amount of data contained in each batch; (4) Construct a dual-branch self-attention mechanism network, which consists of a first branch network, a second branch network and a time-frequency domain feature fusion module; transform the electrical impedance image tensor V IMG , voltage frequency domain tensor V fft Input the dual-branch self-attention mechanism network, and the dual-branch self-attention mechanism network outputs the muscle force estimation result; (5) Compare the obtained muscle force estimation results with the actual force sequence collected by the six-dimensional force sensor, and use the mean square error to reflect the degree of difference between the two; In step (4), the first branch network includes an image serialization module and a first multi-head self-attention mechanism module, and the second branch network includes a first fully connected layer and a second multi-head self-attention mechanism module; The time-frequency domain feature fusion module includes a first global average pooling layer, a second global average pooling layer, a first RELU activation layer, a second RELU activation layer, a first Sigmoid activation layer, a second Sigmoid activation layer, and second, third, fourth, fifth, sixth, seventh, eighth, and ninth fully connected layers; First, the electrical impedance image tensor V IMG Send to the first branch network for processing: the image serialization module converts V IMG The image is divided into patchsize×patchsize small images, where patchsize=16, which means the image is divided into 16 rows and 16 columns. The segmented image is first linearly transformed, and then position coding is added to each part of the linearly transformed image. Finally, the image data of each part after adding position coding is spliced ​​together, and the whole data obtained by splicing is As a sequence, it is sent to the first multi-head self-attention mechanism module for operation; the first multi-head self-attention mechanism module adopts the scaled dot product attention mechanism, and the formula is as follows: Among them, Q, K and V represent query, key and value respectively, Attention represents the output tensor, Indicates the number of dimensions of matrix Q; Will By multiplying the trainable weight matrix to perform linear transformation, three initial vector representations are obtained. The formula is as follows: Q=W q V IMG K=W k V IMG V=W v V IMG Among them, W q 、W k and W v Both represent trainable weight matrices; Substitute the obtained Q, K and V into formula (1) to calculate the output tensor Attention; IMG After learning independently multiple times, the multiple outputs are concatenated and fed into a fully connected layer of the first multi-head self-attention mechanism module to obtain a feature tensor of size batchsize×patchsize×dim. The first dimension batchsize represents the amount of data contained in each batch, the second dimension patchsize=16, and the third dimension dim=1024. Dim represents the length of each image vector after image serialization. The mean is calculated on the second dimension, and finally a time domain feature tensor U with a size of 8×1024 is obtained. T ; The voltage frequency domain tensor V fft Send it to the second branch network for processing: First, the size of V is batchsize×1×28 fft Input the first fully connected layer and convert V fft Expand in the third dimension to obtain a feature tensor of size batchsize×1×1024, which is then fed into the second multi-head self-attention mechanism module to obtain a frequency domain feature tensor U of size 8×1024. F ; The obtained time domain feature tensor U T and frequency domain feature tensor U F Reshaped into 8×1024×1 and sent to the time-frequency domain feature fusion module: First, the global average pooling operation is performed through the first global average pooling layer and the second global average pooling layer to convert the time domain feature tensor U T and frequency domain feature tensor U F Compress to obtain the compressed time domain feature tensor and frequency domain feature tensor Then and They are sent to the second fully connected layer and the fourth fully connected layer respectively, and then the channel dimension is reduced by the first ReLU activation layer and the second ReLU activation layer respectively, and then sent to the third fully connected layer and the fifth fully connected layer respectively, and the channel dimension is reduced by the first Sigmoid activation layer and the second Sigmoid activation layer respectively. and Each element in the vector is compressed to between 0 and 1 to obtain the weight of each channel and Next, the weight and the time domain feature tensor U T After multiplication, it is sent to the sixth fully connected layer to obtain the time domain channel attention feature tensor The weight and the frequency domain feature tensor U F After multiplication, it is sent to the seventh fully connected layer to obtain the frequency domain channel attention feature tensor At the same time, U T 、U F The downsampled time domain feature tensors are respectively sent to the eighth and ninth fully connected layers to obtain the downsampled time domain feature tensors. And the downsampled frequency domain feature tensor Then, the frequency domain channel attention feature tensor And the downsampled time domain feature tensor After matrix multiplication, the time domain feature tensor after downsampling is Add together to get the first fused tensor U Fusion1 ; At the same time, the time domain channel attention feature tensor And the downsampled frequency domain feature tensor After matrix multiplication, the frequency domain feature tensor after downsampling is obtained Add together to get the second fusion tensor U Fusion2 ; The first fused tensor U Fusion1 , the second fused tensor U Fusion2 Add together to get an output of size 8×6 The specific formula is as follows: Where GAP represents global average pooling, W1, W2, W3, and W4 all represent fully connected layers, δ and σ represent the ReLU activation function and the Sigmoid activation function, respectively, and ⊙ is the matrix element multiplication.

2. The upper arm muscle strength estimation method based on a dual-branch self-attention mechanism network according to claim 1, characterized in that: The step (1) specifically includes the following steps: (1a) Repeat the measurement of the electrical impedance imaging sensor multiple times, and set each measurement from the beginning to the end as a measurement cycle; in each measurement cycle, the excitation current is injected along the voltage on each adjacent electrode in turn, and the response voltage on each adjacent electrode is obtained. N is the number of electrodes and N=8, that is, The 28 response voltage values ​​obtained by 8 electrodes in the adjacent excitation measurement mode at the current moment are set as a frame; after continuous measurement, the original voltage sequence within a measurement cycle is obtained vol(i raw )={V1,V2,...,V 28 } is the response voltage value of the current frame; the original voltage sequence collected in each frame is normalized to the maximum and minimum values, and all voltages are linearly shrunk to [0,1]: Among them, vol is the response voltage value at the current moment, vol max is the maximum response voltage value in the current measurement cycle, vol min is the minimum response voltage value in the current measurement cycle; the normalized original electrical impedance imaging voltage sequence is obtained Set the voltage sequence vol measured in the first frame of each measurement norm (1) is the initial value; (1b) In each measurement cycle of the six-dimensional force sensor, the real force sequence collected at the current moment is recorded as force(j) = {F x (j),F y (j),F z (j),M x (j),M y (j),M z (j)}, perform maximum and minimum normalization on the collected real force to obtain the normalized real force sequence 3. The upper arm muscle strength estimation method based on a dual-branch self-attention mechanism network according to claim 1, characterized in that: The step (2) specifically includes the following steps: (2a) Reconstruct the 2D electrical impedance image using the one-step Gauss-Newton reconstruction algorithm: σ=σ0-[[J(σ0)] T J(σ0)+αdiag(J T J)] -1 [J(σ0)] T [V(σ0)-U] Where σ is the conductivity distribution vector, σ0 is the initial conductivity distribution, α is the regularization factor, and J T J is the regularization matrix, V(σ0) is the calculated boundary voltage under the conductivity distribution of σ0, J(σ0) is the Jacobian matrix, and U is the measured boundary voltage; After reconstruction, the two-dimensional electrical impedance image is obtained, which is recorded as Where img(v) is the two-dimensional electrical impedance image of the current frame, which is used to reflect the conductivity distribution information of the cross section of the upper arm muscle of the human body at the current moment; (2b) Normalized original electrical impedance imaging voltage sequence The frequency domain information contained in is decoupled to obtain the frequency domain representation of the voltage sequence Among them, M1=M3=M4=M, M is the total number of samples in a measurement cycle; fft(u) is the amplitude spectrum of the frequency domain signal of the original voltage sequence of the current frame, which is used to reflect the frequency domain information of the cross-section of the upper arm muscle of the human body at the current moment.

Images