Intelligent artificial limb control and posture detection method and device based on HighHRNet

Abstract

The invention discloses an intelligent artificial limb control and posture detection method and device based on HighHRNet, and relates to the field of computer vision, and the method comprises the steps: S1, obtaining a data set comprising a two-dimensional artificial limb user body image; s2, constructing an intelligent artificial limb control and posture detection high-resolution network based on a HighHRNet; and S3, processing the two-dimensional artificial limb user body image in the data set by using an intelligent artificial limb control and posture detection high-resolution network to obtain a final key point heat map. According to the lightweight intelligent artificial limb control and posture detection high-resolution network disclosed by the invention, the self-distillation module, the channel correlation distillation module, the DSNT module and the deconvolution module are introduced, so that distillation network learning and accurate positioning of key points in human body posture detection are realized; the method has very good recognition accuracy for single-person posture recognition and multi-person posture recognition, and has relatively low calculation parameters and operation times while improving the human body posture recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an intelligent prosthesis control and posture detection method and device based on HigherHRNet. Background Art

[0002] Intelligent prosthetic control and posture detection technology has broad application prospects in the medical field, especially in the rehabilitation and life assistance of amputees. Through precise posture analysis and feedback, intelligent prostheses can respond to patients' movement intentions more sensitively and imitate the movements of natural limbs, thereby significantly improving patients' quality of life. In medical scenarios, posture detection can not only help prostheses achieve dynamic movement adjustments, but also adaptively optimize according to patients' real-time posture changes, support daily tasks such as walking and grasping, and enhance the functionality and comfort of prostheses. In addition, posture detection of intelligent prostheses also provides strong support for the rehabilitation process. Doctors can comprehensively evaluate patients' rehabilitation progress through posture data collected in real time, and adjust rehabilitation training programs based on these data to help patients restore their daily activities more efficiently. This technology not only improves the use of prostheses, but also provides medical teams with a more accurate basis for formulating rehabilitation programs.

[0003] At present, posture detection methods based on deep learning have been widely used in the field of intelligent prostheses, especially HigherHRNet (high-resolution network), which has shown significant advantages in complex motion posture recognition. HigherHRNet can effectively capture the details of human posture and is particularly suitable for intelligent prosthesis control in medical rehabilitation scenarios. However, with the increase in network depth, the demand for computing resources also increases accordingly, which brings challenges to the real-time and computing efficiency of prosthetic systems. At the same time, traditional posture detection methods have certain errors in the positioning of joint coordinates, which makes it difficult to meet the high requirements for the control accuracy of intelligent prostheses. Therefore, how to optimize computing efficiency to meet real-time requirements while ensuring high precision is still a key issue in the development of intelligent prosthetic control technology. Summary of the invention

[0004] In view of the above problems, the present invention proposes an intelligent prosthesis control and posture detection method and device based on HigherHRNet. By constructing a lightweight high-resolution network for human posture detection, the SE attention mechanism, self-distillation module, channel correlation distillation module, deconvolution module and DSNT module are introduced to realize the distillation network learning and precise positioning of key points in human posture detection. It has good recognition accuracy for single-person posture distillation and multi-person posture recognition, and has lower calculation parameters and number of operations while improving the accuracy of human posture recognition.

[0005] On the one hand, the intelligent prosthesis control and posture detection method based on HigherHRNet has the following specific steps:

[0006] S101, obtaining a data set including a two-dimensional human body image of a prosthesis user;

[0007] S102, building a high-resolution network for intelligent prosthesis control and posture detection based on HigherHRNet;

[0008] S103, using the intelligent prosthesis control and posture detection high-resolution network to process the two-dimensional prosthesis user human body image in the data set, obtain the final key point heat map, and complete the posture detection;

[0009] The intelligent prosthesis control and posture detection high-resolution network includes:

[0010] A preprocessing layer, which preprocesses the images in the input data set and inputs the preprocessed images into the multi-dimensional scale fusion part; the preprocessing layer includes an SE attention mechanism;

[0011] The multi-dimensional scale fusion part includes four stages: stage 1, stage 2, stage 3 and stage 4. Each stage outputs feature representation to the self-distillation module, outputs image features from the last layer to the channel correlation distillation module, and outputs the feature fused image from stage 4 to the convolution layer;

[0012] The self-distillation module uses the feature representation of the four stages of multi-dimensional fusion to construct the self-distillation loss;

[0013] Channel correlation distillation module, which uses the image features output by the last layer of the multi-dimensional scale fusion part to construct the channel correlation distillation loss;

[0014] Convolutional layer, which converts the number of channels of the image after the feature fusion of the multi-dimensional scale fusion part into the number of key points in the data set, and outputs the key point set to the DSNT layer;

[0015] The DSNT layer regresses the key point coordinates in the key point set to generate a prediction heat map;

[0016] The deconvolution module concatenates the feature fused image output by the multi-dimensional scale fusion part with the predicted heat map, and sequentially passes through the deconvolution operation, residual block and convolution operation to output the final key point heat map.

[0017] Preferably, the SE attention mechanism is as follows:

[0018] First, the feature map of each channel in the image is compressed into a scalar through global average pooling to obtain a global feature description. The specific calculation formula is as follows:

[0019] ;

[0020] in, Represents the global description scalar of the cth channel; H represents the height of the feature map, and W represents the width of the feature map; Indicates the input feature map in the cth channel pixel value;

[0021] Secondly, the global description is processed through two fully connected layers and activation functions to generate the weight of each channel. The specific calculation formula is as follows:

[0022] ;

[0023] in, represents the recalibrated channel weight vector; and Represent the weight matrices of the fully connected layers respectively; Represents the Sigmoid activation function; ReLU activation function.

[0024] Finally, the obtained channel weights are remapped back to the feature map of each channel, and the features of each channel are reweighted by element-by-element multiplication. The specific calculation formula is as follows:

[0025] ;

[0026] in, represents the feature map of the cth channel after recalibration by the SE module; represents the calibration weight of the cth channel; Represents the feature map of the cth channel of the input feature map.

[0027] Preferably, the construction of channel correlation distillation loss is specifically as follows:

[0028] First, calculate the image features of the last layer output of the multi-dimensional scale fusion part The channel correlation matrix is ​​calculated as follows:

[0029]

[0030] in, Represents the high-level feature extractor for the input value Output features of express The image features of the mth channel in the image features; express The image features of the nth channel in the image features; represents the feature mapping function; represents the inner product function, which is used to measure the correlation between the channel dimensions of the input image;

[0031] Secondly, obtain the same Feature images of the same size, i.e., feature representations of the output of stage 4 The characteristic image is then used to calculate the channel correlation matrix. The calculation formula is as follows:

[0032] ;

[0033] in, Represents the low-level feature extractor for the input value Output features of express The image features of the mth channel in the image features; express The image features of the nth channel in the image features;

[0034] Finally, calculate and The L2 norm between them completes the channel correlation distillation loss , the calculation formula is as follows:

[0035]

[0036] in, represents the L2 norm.

[0037] Optimized, overall loss of high-resolution networks for intelligent prosthetic control and posture detection The calculation formula is as follows:

[0038]

[0039] in, represents the network loss without the self-distillation module and the channel correlation distillation module, A heatmap representing the kth keypoint of the network prediction heatmap, Represents the heat map of the kth key point of the true value heat map, K represents the number of key points, represents the mean square error loss function; Represents the overall loss of the self-distillation module; They represent weight coefficients respectively.

[0040] Preferably, the deconvolution module is specifically as follows:

[0041] The feature fused image output by the multi-dimensional scale fusion part is concatenated with the predicted heat map and used as the input of a 4×4 transposed convolution for deconvolution. Then, the input feature map is upsampled through the learning of BatchNorm and RELU. After the transposed convolution operation, 4 residual blocks are added to refine the upsampled feature map, and another convolution operation is performed to output the final key point heat map.

[0042] Preferably, the formula for the transposed convolution operation is as follows:

[0043]

[0044] Among them, Out represents the output size after transposed convolution; In represents the size of the input feature map; stride represents the step size of the convolution; padding represents the number of zero padding added around the input feature map; kernel_size represents the size of the convolution kernel.

[0045] Specifically, the convolution operation is performed again as follows:

[0046]

[0047] in, Represents the final key point heat map generated using the Gaussian function; Represents any pixel position; Represents the coordinates of correctly labeled key points; hyperparameters represents a predetermined spatial variance; represents the natural exponential function.

[0048] On the other hand, the intelligent prosthesis control and posture detection device based on HigherHRNet includes the following:

[0049] An image acquisition module, used to acquire a data set including a two-dimensional human body image of a prosthesis user;

[0050] Intelligent prosthesis control and posture detection high-resolution network building module, used to build intelligent prosthesis control and posture detection high-resolution network based on HigherHRNet;

[0051] The final key point heat map acquisition module is used to process the two-dimensional prosthesis user body images in the data set using the intelligent prosthesis control and posture detection high-resolution network to obtain the final key point heat map and complete posture detection;

[0052] The intelligent prosthesis control and posture detection high-resolution network includes:

[0053] A preprocessing layer, which preprocesses the images in the input data set and inputs the preprocessed images into the multi-dimensional scale fusion part; the preprocessing layer includes an SE attention mechanism;

[0054] The multi-dimensional scale fusion part includes four stages: stage 1, stage 2, stage 3 and stage 4. Each stage outputs feature representation to the self-distillation module, outputs image features from the last layer to the channel correlation distillation module, and outputs the feature fused image from stage 4 to the convolution layer;

[0055] The self-distillation module uses the feature representation of the four stages of multi-dimensional fusion to construct the self-distillation loss;

[0056] Channel correlation distillation module, which uses the image features output by the last layer of the multi-dimensional scale fusion part to construct the channel correlation distillation loss;

[0057] Convolutional layer, which converts the number of channels of the image after the output feature fusion of stage 4 into the number of key points in the data set, and outputs the key point set to the DSNT layer;

[0058] The DSNT layer regresses the key point coordinates in the key point set to generate a prediction heat map;

[0059] The deconvolution module concatenates the feature fused image output by the multi-dimensional scale fusion part with the predicted heat map, and sequentially passes through the deconvolution operation, residual block and convolution operation to output the final key point heat map.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] (1) The present invention optimizes the HigherHRNet network and adds a DSNT module for calculating the coordinates of key points through regression and generating heat maps. In traditional methods, the coordinates of key points generated by the Gaussian function are usually integers. Even if the heat map is restored without error, errors in the key point pixels will be introduced, affecting the precise positioning of the key points. Compared with the original network that uses the Gaussian function to generate heat maps, the DSNT module in the present invention performs better when processing low-resolution heat maps. The DSNT module not only retains the precise position and overall spatial information of the key points of the human body, but also effectively eliminates the pixel errors caused by the heat map generated by the Gaussian function, thereby improving the learning ability of the distillation network and the precise positioning effect of the key points.

[0062] (2) The present invention introduces a lightweight high-resolution network specifically for the control and posture detection of intelligent prostheses. While ensuring high-precision posture recognition, the network significantly reduces the calculation parameters and the number of operations, enabling it to run efficiently on resource-constrained devices. For example, intelligent prostheses usually rely on portable devices for real-time calculations, which have high requirements for processing speed and energy consumption. The lightweight network of the present invention reduces the computational complexity by optimizing the structure, so that under limited hardware conditions, the intelligent prosthesis can still respond quickly and accurately to the user's posture changes and provide a smooth control experience. This design not only improves the applicability of the system, but also ensures the stability and reliability of the prosthesis in daily use.

[0063] (3) The present invention introduces the SE attention mechanism in the preprocessing part to further improve the accuracy and responsiveness of the intelligent prosthesis control and posture detection system. The SE attention mechanism can highlight the feature information that is critical to posture recognition by adaptively adjusting the importance of different feature channels, while suppressing irrelevant or interfering information. In the intelligent prosthesis control scenario, the movements of various parts of the human body are complex and diverse. Through the SE attention mechanism, the system can more accurately identify and process key posture information, thereby improving the accuracy of posture detection. This mechanism not only helps to optimize the reaction speed of the prosthesis when performing complex movements, but also significantly enhances the control accuracy and adaptability of the prosthesis, ensuring the stability and flexibility of the system in a variety of environments.

[0064] (4) The present invention introduces a self-distillation module to further improve the performance of the intelligent prosthesis control and posture detection network. Through the self-distillation mechanism, the network not only has extremely high accuracy in single-person posture recognition, but also significantly improves the recognition ability in multi-person posture scenarios. This is particularly important for the control system of intelligent prostheses because users may interact with others or perform complex movements in different environments. The self-distillation module helps the network better extract and learn the posture features of multiple people, thereby ensuring that even in a multi-person environment, the intelligent prosthesis can accurately recognize the user's movements and make corresponding adjustments. This enhanced recognition performance provides technical support for the precise control of the prosthesis and improves the recognition accuracy and response sensitivity of the overall system.

[0065] (5) The present invention introduces a channel correlation distillation module to further optimize the accuracy and efficiency of the intelligent prosthesis control and posture detection system. Through this module, the network can capture the correlation information between different feature channels, thereby improving the accuracy of overall posture recognition. In intelligent prosthesis control, there are complex relationships between the movements of different joints and limbs. The channel correlation distillation module enhances the network's understanding and extraction capabilities of multi-dimensional posture features by learning these correlation information. This module not only improves the data processing effect of a single channel, but also optimizes the response of the prosthesis to complex movements, allowing the intelligent prosthesis to make adjustments more accurately and flexibly when facing changing postures and movements. The introduction of this module has greatly improved the control accuracy and user experience of intelligent prostheses.

[0066] (6) The deconvolution module of the present invention can effectively improve the accuracy of spatial information recovery and enhance the detail reconstruction of low-resolution feature maps in intelligent prosthetic control and posture detection. By optimizing the upsampling process, the accuracy of posture estimation can be improved, and the details of human body movements can be captured and restored more accurately, thereby improving the accuracy and robustness of prosthetic control. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The present invention is further described in detail below in conjunction with the accompanying drawings;

[0068] Figure 1 is a flow chart of an intelligent prosthesis control and posture detection method based on HigherHRNet according to an embodiment of the present invention;

[0069] Figure 2 A schematic diagram of the structure of a high-resolution network for intelligent prosthesis control and posture detection of an intelligent prosthesis control and posture detection method based on HigherHRNet according to an embodiment of the present invention;

[0070] Figure 3 4 is a structural block diagram of an intelligent prosthesis control and posture detection device based on HigherHRNet according to an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The present invention is further described below through specific implementation modes.

[0072] like Figure 1 As shown in the figure, the intelligent prosthesis control and posture detection method based on HigherHRNet is as follows:

[0073] S101, obtaining a data set including two-dimensional human body images of a prosthesis user.

[0074] S102, build a high-resolution network for intelligent prosthesis control and posture detection based on HigherHRNet.

[0075] Among them, the high-resolution network for intelligent prosthetic control and posture detection includes a preprocessing layer, a multi-dimensional scale fusion part, a self-distillation module, a channel correlation distillation module, a convolutional layer and a DSNT layer.

[0076] The preprocessing layer consists of the SE attention mechanism, two convolutional layers and a nonlinear activation layer RELU, and the SE attention mechanism is introduced to process the input image, as follows:

[0077] First, the feature map of each channel is compressed into a scalar through global average pooling to obtain a global feature description. The specific calculation formula is as follows:

[0078] ;

[0079] in, represents the global description vector of the cth channel, H and W represent the height and width of the feature map respectively, Indicates the input feature map in the cth channel pixel value;

[0080] Secondly, after obtaining the global description of each channel, the global description is processed through two fully connected layers and activation functions to generate the weight of each channel. The specific calculation formula is as follows:

[0081] ;

[0082] in, represents the recalibrated channel weight vector, W1 and W2 represent the weight matrix of the fully connected layer, represents the Sigmoid activation function, ReLU activation function.

[0083] Finally, the obtained channel weights are remapped back to the feature map of each channel, and the features of each channel are reweighted by element-by-element multiplication. The specific calculation formula is as follows:

[0084] ;

[0085] in, represents the feature map of the cth channel after recalibration by the SE module, represents the calibration weight of the cth channel, Represents the feature map of the cth channel of the input feature map.

[0086] This embodiment introduces the SE attention mechanism in the preprocessing part to further improve the accuracy and responsiveness of the intelligent prosthesis control and posture detection system. The SE attention mechanism can highlight the feature information that is critical to posture recognition by adaptively adjusting the importance of different feature channels, while suppressing irrelevant or interfering information. In the intelligent prosthesis control scenario, the movements of various parts of the human body are complex and diverse. Through the SE attention mechanism, the system can more accurately identify and process key posture information, thereby improving the accuracy of posture detection. This mechanism not only helps to optimize the reaction speed of the prosthesis when performing complex movements, but also significantly enhances the control accuracy and adaptability of the prosthesis, ensuring the stability and flexibility of the system in a variety of environments.

[0087] After the image processed by the SE attention mechanism passes through the remaining modules of this part, the number of channels will change from the original unknown number of channels C to 64. The preprocessed image is directly input into the multi-dimensional scale fusion part; the convolution operation f(x) of the preprocessing part is expressed as:

[0088]

[0089] Where X represents the input image, and represents the weights in the first and second convolution layers, and represents the bias in the first and second convolution layers, Represents the RELU activation function.

[0090] Multidimensional scaling fusion part:

[0091] The implementation of the multi-dimensional scale fusion part is as follows:

[0092] The output feature of the residual block in the multi-resolution module is expressed as:

[0093]

[0094] Where i represents the stage, j represents the branch, k represents the sequence number of the residual block, M represents the number of input channels, and N represents the number of output channels;

[0095] Output of the first residual block in stage 1 It is expressed as:

[0096]

[0097] Among them, conv represents the convolution operation, represents the RELU activation function; therefore, the output of the multi-resolution module in stage 1 It is expressed as:

[0098] ;

[0099] Since there is only one branch in stage 1 and no multi-scale fusion is performed, a new branch is generated in stage 1. The operations are:

[0100] ;

[0101] in, Indicates the separator, represents the input of the first parallel branch of the multi-resolution module in stage 2, represents the input of the second parallel branch of the multi-resolution module in stage 2;

[0102] The last multi-resolution module in stage 2 performs multi-scale fusion and convolution process to generate new branches It is expressed as:

[0103]

[0104] The convolution operation is performed using a convolution kernel of size 3×3, and upsampling is performed using the nearest neighbor interpolation method with a convolution kernel size of 1×1. represents the input of the first parallel branch of the multi-resolution module in stage 3, represents the input of the second parallel branch of the multi-resolution module in stage 3, represents the input of the third parallel branch of the multi-resolution module in stage three;

[0105] The feature representation of each stage of the multi-dimensional scaling fusion part is as follows:

[0106] ;

[0107] in Represents the high-dimensional features of stage one, represents the high-dimensional features of the second stage, Represents the high-dimensional features of the third stage, Represents the high-dimensional features of the fourth stage. The sizes of each high-dimensional feature are (B, 64, 48, 32), where B represents the batch size of the input image data, 64 represents the image feature channel dimension, and 48 and 32 represent the width and height of the image feature, respectively;

[0108] The multi-resolution modules in the multi-dimensional scale fusion part all contain high-dimensional features. The high-dimensional features of the last layer are expressed as , and high-dimensional features at different stages , , , The dimensions are the same, (B, 64, 48, 32).

[0109] Self-distillation module:

[0110] The implementation of the self-distillation module is as follows:

[0111] calculate and Between The norm completes the first stage of self-distillation, and its calculation formula is as follows:

[0112] ;

[0113] in, express norm;

[0114] The calculation formulas for self-distillation in the second, third and fourth stages are as follows:

[0115] ;

[0116] Overall loss of self-distillation module The calculation formula is as follows:

[0117] ;

[0118] This embodiment introduces a self-distillation module to further improve the performance of the intelligent prosthesis control and posture detection network. Through the self-distillation mechanism, the network not only has extremely high accuracy in single-person posture recognition, but also significantly improves the recognition ability in multi-person posture scenarios. This is especially important for the control system of intelligent prostheses, because users may interact with others or perform complex movements in different environments. The self-distillation module helps the network better extract and learn the posture features of multiple people, thereby ensuring that even in a multi-person environment, the intelligent prosthesis can accurately recognize the user's movements and make corresponding adjustments. This enhanced recognition performance provides technical support for the precise control of the prosthesis and improves the recognition accuracy and response sensitivity of the overall system.

[0119] The implementation of the channel correlation distillation module is as follows:

[0120] First, calculate The channel correlation matrix , and its calculation formula is as follows:

[0121] ;

[0122] in, express The image features of the mth channel in the image features, express The image features of the nth channel in the image features, Represents the feature mapping function, which is used to map a two-dimensional feature vector to a vector of length hw. hw refers to the product of the height (h) and width (w) of the feature map, that is, the total number of pixels in the feature map, which represents the length when converting two-dimensional image data into a one-dimensional vector. represents the inner product function, which is used to measure the correlation between the channel dimensions of the input image;

[0123] Secondly, obtain the same Feature images of the same size, i.e., extract The characteristic image is then used to calculate the channel correlation matrix. The calculation formula is as follows:

[0124] ;

[0125] in, express The image features of the mth channel in the image features; express The image features of the nth channel in the image features.

[0126] Finally, calculate and The L2 norm between them completes the channel correlation distillation , and its calculation formula is as follows:

[0127] ;

[0128] in, Represents the high-level feature extractor for the input value The output features of Represents the low-level feature extractor for the input value The output features of .

[0129] This embodiment introduces a channel correlation distillation module to further optimize the accuracy and efficiency of the intelligent prosthesis control and posture detection system. Through this module, the network can capture the correlation information between different feature channels, thereby improving the accuracy of overall posture recognition. In intelligent prosthesis control, there are complex relationships between the movements of different joints and limbs. The channel correlation distillation module enhances the network's understanding and extraction capabilities of multi-dimensional posture features by learning these correlation information. This module not only improves the data processing effect of a single channel, but also optimizes the response of the prosthesis to complex movements, so that the intelligent prosthesis can make adjustments more accurately and flexibly when facing changing postures and movements. The introduction of this module has greatly improved the control accuracy and user experience of intelligent prostheses.

[0130] The specific implementation of the convolution layer and DSNT layer is as follows:

[0131] The 1×1 convolution layer converts the number of channels of the image into the number of key points K in the dataset, and then the DSNT module regresses the key point coordinates to generate a heat map;

[0132] The process of key point regression in the DSNT module is as follows:

[0133] Construct matrices X and Y. The matrix construction formula is as follows:

[0134] ;

[0135] ;

[0136] Where a=1,...,m, b=1,...,n; a represents the number of matrix rows; b represents the number of matrix columns; X ab represents the element in the ath row and bth column of matrix X; Y ab represents the element in the ath row and bth column of the matrix Y; m represents the number of matrix rows of the regularized key point coordinate feature map, and n represents the number of matrix columns of the regularized key point coordinate feature map; the set Z represents the K key point sets to be output at the end of the network, and the size of each key point feature map is m×n. The softmax function is used to regularize Z, and the regularized set is represented as ,in is the feature map of the kth key point coordinates after regularization, represented by a matrix of size m×n;

[0137] The formula for generating heat maps by the DSNT module is as follows:

[0138] ;

[0139] in, and They respectively represent the coordinates of the key points generated after the F-normal form calculation;

[0140] The operation of the deconvolution module is as follows:

[0141] First, the feature map output by the backbone network and the heat map generated after the DSNT layer are concatenated as the input of a 4×4 transposed convolution for deconvolution to generate a high-quality feature map with a resolution twice that of the input feature map. Then, BatchNorm and RELU are used to learn to upsample the input feature map, and 4 residual blocks are further added after the transposed convolution to refine the upsampled feature map.

[0142] The formula for the transposed convolution operation is as follows:

[0143] ;

[0144] Among them, Out represents the output size after transposed convolution, In represents the size of the input feature map, stride represents the step size of the convolution, padding represents the number of zero padding added around the input feature map, and kernel_size represents the size of the convolution kernel.

[0145] Finally, perform another convolution operation and use the Gaussian function to generate the heat map formula as follows:

[0146] ;

[0147] in, represents any pixel position, represents correctly labeled key points, hyperparameters Represents a predetermined spatial variance.

[0148] The deconvolution module of this embodiment can effectively improve the accuracy of spatial information recovery and enhance the detail reconstruction of low-resolution feature maps in intelligent prosthetic control and posture detection. By optimizing the upsampling process, the accuracy of posture estimation can be improved, and the details of human body movements can be captured and restored more accurately, thereby improving the accuracy and robustness of prosthetic control.

[0149] Overall loss of intelligent prosthetic control and posture detection method based on HigherHRNet , and its calculation formula is as follows:

[0150] ;

[0151] in, is the original network loss, and Represent the heatmaps of the network prediction heatmap and the true value heatmap of the kth key point, represents the mean square error loss function, Represents the weight coefficient.

[0152] S103, using the intelligent prosthesis control and posture detection high-resolution network to process the two-dimensional prosthesis user body images in the data set, obtain the final key point heat map, and complete the posture detection.

[0153] This embodiment optimizes the HigherHRNet network and adds a DSNT module for calculating the coordinates of key points through regression and generating a heat map. In traditional methods, the key point coordinates generated by the Gaussian function are usually integers. Even if the heat map is restored without errors, errors in the key point pixels will be introduced, affecting the precise positioning of the key points. Compared with the original network that uses a Gaussian function to generate a heat map, the DSNT module in the present invention performs better when processing low-resolution heat maps. The DSNT module not only retains the precise position and overall spatial information of the key points of the human body, but also effectively eliminates the pixel errors caused by the generation of the heat map by the Gaussian function, thereby improving the learning ability of the distillation network and the precise positioning effect of the key points.

[0154] This embodiment introduces a lightweight high-resolution network specifically for the control and posture detection of intelligent prostheses. While ensuring high-precision posture recognition, the network significantly reduces the calculation parameters and the number of operations, enabling it to run efficiently on resource-constrained devices. For example, intelligent prostheses usually rely on portable devices for real-time calculations, and therefore have high requirements for processing speed and energy consumption. The lightweight network of the present invention reduces the computational complexity by optimizing the structure, so that under limited hardware conditions, the intelligent prosthesis can still respond quickly and accurately to the user's posture changes and provide a smooth control experience. This design not only improves the applicability of the system, but also ensures the stability and reliability of the prosthesis in daily use.

[0155] like Figure 3 As shown, the present invention also discloses an intelligent prosthesis control and posture detection device based on HigherHRNet, comprising:

[0156] An image acquisition module 301 is used to acquire a data set including a two-dimensional human body image of a prosthesis user;

[0157] Intelligent prosthesis control and posture detection high-resolution network construction module 302, used to construct an intelligent prosthesis control and posture detection high-resolution network based on HigherHRNet;

[0158] The final key point heat map acquisition module 303 is used to process the two-dimensional prosthesis user human body image in the data set using the intelligent prosthesis control and posture detection high-resolution network to obtain the final key point heat map and complete the posture detection;

[0159] The intelligent prosthesis control and posture detection high-resolution network includes:

[0160] A preprocessing layer, which preprocesses the images in the input data set and inputs the preprocessed images into the multi-dimensional scale fusion part; the preprocessing layer includes an SE attention mechanism;

[0161] The multi-dimensional scale fusion part includes four stages: stage 1, stage 2, stage 3 and stage 4. Each stage outputs feature representation to the self-distillation module, outputs image features from the last layer to the channel correlation distillation module, and outputs the feature fused image from stage 4 to the convolution layer;

[0162] The self-distillation module uses the feature representation of the four stages of multi-dimensional fusion to construct the self-distillation loss;

[0163] Channel correlation distillation module, which uses the image features output by the last layer of the multi-dimensional scale fusion part to construct the channel correlation distillation loss;

[0164] Convolutional layer, which converts the number of channels of the image after the output feature fusion of stage 4 into the number of key points in the data set, and outputs the key point set to the DSNT layer;

[0165] The DSNT layer regresses the key point coordinates in the key point set to generate a prediction heat map;

[0166] The deconvolution module concatenates the feature fused image output by the multi-dimensional scale fusion part with the predicted heat map, and sequentially passes through the deconvolution operation, residual block and convolution operation to output the final key point heat map.

[0167] The specific implementation of the intelligent prosthesis control and posture detection device based on HigherHRNet is the same as the intelligent prosthesis control and posture detection method based on HigherHRNet, and will not be repeated in this embodiment.

[0168] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. An intelligent prosthesis control and posture detection method based on HigherHRNet, characterized in that: The steps include: S101, obtaining a data set including a two-dimensional human body image of a prosthesis user; S102, building a high-resolution network for intelligent prosthesis control and posture detection based on HigherHRNet; S103, using the intelligent prosthesis control and posture detection high-resolution network to process the two-dimensional prosthesis user human body image in the data set, obtain the final key point heat map, and complete the posture detection; The intelligent prosthesis control and posture detection high-resolution network includes: A preprocessing layer, which preprocesses the images in the input data set and inputs the preprocessed images into the multi-dimensional scale fusion part; the preprocessing layer includes an SE attention mechanism; The multi-dimensional scale fusion part includes four stages: stage 1, stage 2, stage 3 and stage 4. Each stage outputs feature representation to the self-distillation module, outputs image features from the last layer to the channel correlation distillation module, and outputs the feature fused image from stage 4 to the convolution layer; The self-distillation module uses the feature representation of the four stages of multi-dimensional fusion to construct the self-distillation loss; The channel correlation distillation module uses the image features output by the last layer of the multi-dimensional scale fusion part to construct the channel correlation distillation loss; Convolutional layer, which converts the number of channels of the image after the output feature fusion of stage 4 into the number of key points in the data set, and outputs the key point set to the DSNT layer; The DSNT layer regresses the key point coordinates in the key point set to generate a prediction heat map; The deconvolution module concatenates the feature fused image output by the multi-dimensional scale fusion part with the predicted heat map, and sequentially passes through the deconvolution operation, residual block and convolution operation to output the final key point heat map.

2. The intelligent prosthesis control and posture detection method based on HigherHRNet according to claim 1, characterized in that: The SE attention mechanism is as follows: First, the feature map of each channel in the image is compressed into a scalar through global average pooling to obtain a global feature description. The specific calculation formula is as follows: ； in, Represents the global description scalar of the cth channel; H represents the height of the feature map, and W represents the width of the feature map; Indicates the input feature map in the cth channel pixel value; Secondly, the global description is processed through two fully connected layers and activation functions to generate the weight of each channel. The specific calculation formula is as follows: ； in, represents the recalibrated channel weight vector; and Represent the weight matrices of the fully connected layers respectively; Represents the Sigmoid activation function; ReLU activation function. Finally, the obtained channel weights are remapped back to the feature map of each channel, and the features of each channel are reweighted by element-by-element multiplication. The specific calculation formula is as follows: ； in, represents the feature map of the cth channel after recalibration by the SE module; represents the calibration weight of the cth channel; Represents the feature map of the cth channel of the input feature map.

3. The intelligent prosthesis control and posture detection method based on HigherHRNet according to claim 1, characterized in that: The construction of channel correlation distillation loss is specifically as follows: First, calculate the image features of the last layer output of the multi-dimensional scale fusion part The channel correlation matrix is ​​calculated as follows: ； in, Represents the high-level feature extractor for the input value Output features of express The image features of the mth channel in the image features; express The image features of the nth channel in the image features; represents the feature mapping function; represents the inner product function, which is used to measure the correlation between the channel dimensions of the input image; Secondly, obtain the same Feature images of the same size, i.e., feature representations of the output of stage 4 The characteristic image is then used to calculate the channel correlation matrix. The calculation formula is as follows: ； in, Represents the low-level feature extractor for the input value Output features of express The image features of the mth channel in the image features; express The image features of the nth channel in the image features; Finally, calculate and The L2 norm between them completes the channel correlation distillation loss , the calculation formula is as follows: ； in, represents the L2 norm.

4. The intelligent prosthesis control and posture detection method based on HigherHRNet according to claim 3 is characterized in that: Overall loss of high-resolution networks for intelligent prosthetic control and posture detection The calculation formula is as follows: ； in, represents the network loss without the self-distillation module and the channel correlation distillation module, A heatmap representing the kth keypoint of the network prediction heatmap, Represents the heat map of the kth key point of the true value heat map, K represents the number of key points, represents the mean square error loss function; Represents the overall loss of the self-distillation module; They represent weight coefficients respectively.

5. The intelligent prosthesis control and posture detection method based on HigherHRNet according to claim 1, characterized in that: The deconvolution module is specifically as follows: The feature fused image output by the multi-dimensional scale fusion part is concatenated with the predicted heat map and used as the input of a 4×4 transposed convolution for deconvolution. Then, the input feature map is upsampled through the learning of BatchNorm and RELU. After the transposed convolution operation, 4 residual blocks are added to refine the upsampled feature map, and another convolution operation is performed to output the final key point heat map.

6. The intelligent prosthesis control and posture detection method based on HigherHRNet according to claim 5, characterized in that: The formula for the transposed convolution operation is as follows: ； Among them, Out represents the output size after transposed convolution; In represents the size of the input feature map; stride represents the step size of the convolution; padding represents the number of zero padding added around the input feature map; kernel_size represents the size of the convolution kernel.

7. The intelligent prosthesis control and posture detection method based on HigherHRNet according to claim 5, characterized in that: The convolution operation is specifically expressed as follows: ； in, Represents the final key point heat map generated using the Gaussian function; Represents any pixel position; Represents the coordinates of correctly labeled key points; hyperparameters represents a predetermined spatial variance; represents the natural exponential function.

8. An intelligent prosthesis control and posture detection device based on HigherHRNet, comprising the following: An image acquisition module, used to acquire a data set including a two-dimensional human body image of a prosthesis user; Intelligent prosthesis control and posture detection high-resolution network building module, used to build intelligent prosthesis control and posture detection high-resolution network based on HigherHRNet; The final key point heat map acquisition module is used to process the two-dimensional prosthesis user body images in the data set using the intelligent prosthesis control and posture detection high-resolution network to obtain the final key point heat map and complete posture detection; in, The intelligent prosthesis control and posture detection high-resolution network includes: A preprocessing layer, which preprocesses the images in the input data set and inputs the preprocessed images into the multi-dimensional scale fusion part; the preprocessing layer includes an SE attention mechanism; The multi-dimensional scale fusion part includes four stages: stage 1, stage 2, stage 3 and stage 4. Each stage outputs feature representation to the self-distillation module, outputs image features from the last layer to the channel correlation distillation module, and outputs the feature fused image from stage 4 to the convolution layer; The self-distillation module uses the feature representation of the four stages of multi-dimensional fusion to construct the self-distillation loss; Channel correlation distillation module, which uses the image features output by the last layer of the multi-dimensional scale fusion part to construct the channel correlation distillation loss; Convolutional layer, which converts the number of channels of the image after the output feature fusion of stage 4 into the number of key points in the data set, and outputs the key point set to the DSNT layer; The DSNT layer regresses the key point coordinates in the key point set to generate a prediction heat map; The deconvolution module concatenates the feature fused image output by the multi-dimensional scale fusion part with the predicted heat map, and sequentially passes through the deconvolution operation, residual block and convolution operation to output the final key point heat map.

Images