Human acupoint recognition method based on human region segmentation and semantic attention enhancement

Abstract

This invention discloses a method for acupoint recognition based on human body region segmentation and semantic attention enhancement, comprising: acquiring RGB and depth images of the human body; extracting image features using a backbone extraction network; generating mask images of different human structures using a human body region segmentation network; performing multiplicative modulation on the image features using the mask images to obtain image fusion features; then passing these features through a convolutional layer to obtain preliminary acupoint recognition results based on human body region segmentation; acquiring prior knowledge of acupoints; using a dual attention mechanism and based on the prior knowledge of acupoints and image fusion features to obtain channel attention weights and spatial attention weights; and using the channel attention weights and spatial attention weights to perform semantic attention enhancement processing on the preliminary acupoint recognition results to obtain the final acupoint recognition results. This invention can effectively suppress problems such as cross-regional false detection and confusion of acupoints in similar regions, thereby improving the accuracy of acupoint localization.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and intelligent traditional Chinese medicine, specifically involving a method for human acupoint recognition based on human body region segmentation and semantic attention enhancement. Background Technology

[0002] In recent years, computer vision and key point detection technologies have developed rapidly, and more and more methods are being applied to acupoint localization. However, while these methods can achieve feature learning from RGB images and promote the development of acupoint localization, they still have the following drawbacks: First, using only RGB information lacks depth perception capabilities, resulting in large localization errors when dealing with different body parts; second, there are problems such as false detection across limbs and regions, which can easily lead to serious medical risks; third, they ignore the semantic information of acupoints, making it difficult to effectively distinguish acupoints with highly similar appearances, resulting in significant category confusion. In short, the existing methods still face insufficient robustness and reliability in real-world scenarios, making them difficult to directly apply to medical devices. Summary of the Invention

[0003] This invention provides a method for human acupoint recognition based on human body region segmentation and semantic attention enhancement, which can effectively suppress cross-regional false detection and confusion of acupoints in similar regions, and improve the accuracy of acupoint localization.

[0004] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0005] A method for identifying acupoints on the human body based on human region segmentation and semantic attention enhancement includes:

[0006] S1. Obtain the RGB and depth images of the human body, extract image features using the backbone extraction network, and generate mask images of different human body structures using the human body region segmentation network.

[0007] S2, multiplicative modulation of image features using a mask image is used to obtain image fusion features, and then a convolutional layer is used to obtain preliminary acupoint recognition results based on human body region segmentation;

[0008] S3: Obtain prior knowledge of acupoints, use a dual attention mechanism, and obtain channel attention weights and spatial attention weights based on prior knowledge of acupoints and image fusion features;

[0009] S4 utilizes channel attention weights and spatial attention weights to perform semantic attention enhancement processing on the preliminary acupoint recognition results, obtaining the final acupoint recognition results.

[0010] Furthermore, the backbone extraction network used to extract image features specifically adopts ResNet101, and the first layer input convolution is changed from three channels to four channels to receive the concatenated input of RGB image and depth image in the channel dimension; the implementation process is represented as follows:

[0011] , ;

[0012] ;

[0013] ;

[0014] in, and These represent the input RGB image and depth image, respectively. Representative to and Perform channel splicing. The backbone extracts features from the input image of the network. It is a four-channel backbone extraction network. The image features extracted by the backbone extraction network represent the output of the image. The number of output channels extracted from the backbone network. and These represent the height and width of the input image, respectively.

[0015] Furthermore, the image features are multiplicatively modulated using the mask image to obtain the image fusion features, represented as:

[0016] ;

[0017] In the formula, The backbone network extracts image features from the output. Mask images of different human structures generated by a human body region segmentation network. For the Hadama product operation, Image fusion features obtained by multiplicative modulation;

[0018] A preliminary acupoint recognition result based on human body region segmentation is obtained through a convolutional layer, represented as follows:

[0019] ;

[0020] In the formula, The weight parameters represent the weights of a 1×1 convolutional layer. , The number of output channels extracted from the backbone network. and These are the height and width of the input image, respectively. Number of acupoint categories; This is a preliminary acupoint regression heatmap, serving as an initial result for acupoint identification.

[0021] Furthermore, step S3 uses a dual attention mechanism to process the prior knowledge of acupoints and the image fusion features, specifically as follows:

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] In the formula, This represents prior knowledge of acupoints. Number of acupoint categories This refers to the dimension of prior knowledge about acupoints. Number of channels for image fusion features This represents a fully connected layer used to linearly transform the prior knowledge text of acupoints, mapping it to the same dimension as the image fusion features. This refers to the prior knowledge of acupoints after linear transformation mapping; For global average pooling operators, As a multi-head attention mechanism, This represents the query vector in MHA. This represents a specific pixel on the image fusion feature map. The deep semantic features contained therein; This represents the channel attention weights obtained by global average pooling of the image fusion feature X. This indicates the use of a multi-head attention mechanism. Image fusion features X and mapped acupoint prior knowledge Spatial attention weights are obtained through processing.

[0028] Furthermore, step S4 performs semantic attention enhancement processing on the preliminary acupoint identification results, specifically as follows:

[0029] ;

[0030] In the formula, Represents the sigmoid activation function. Indicates channel attention weights. Represents spatial attention weights. This indicates the preliminary results of acupoint identification. This indicates the final acupoint identification result.

[0031] Further, step S1 uses a human body region segmentation network to generate mask images of different human body structures, including: first, using a feature extraction network to extract features from the RGB image of the human body; then, feeding the extracted features into a pre-trained mask generation network to generate a mask. The body region mask of the channel; then... The body region mask image of each channel is upsampled to a resolution consistent with the features of the fused image to obtain a clustering mask; finally, the clustering mask is mapped to a number of channels consistent with the features of the fused image; specifically represented as follows:

[0032] ;

[0033] ;

[0034] ;

[0035] In the formula, Represents the input RGB image, The RGB image feature extraction network represents the mask generation channel. Represents the RGB image features extracted from the mask generation channel; This represents a pre-trained mask generation network. This represents an upsampling operation. This represents the clustering mask obtained from upsampling; For 1×1 convolution, This represents the sigmoid activation function. This represents the final generated mask image; These represent the number of channels, height, and width of the image features extracted by the backbone extraction network, respectively.

[0036] This invention discloses a method for human acupoint recognition based on human region segmentation and semantic attention enhancement. It modifies the feature extraction channel of the backbone extraction network, achieving early fusion of RGB and depth information. A lightweight human region mask generator is used to suppress false detections across limbs and regions. Furthermore, a semantic feature fusion module is utilized to embed prior knowledge of acupoints, and a dual attention mechanism is used to semantically enhance the preliminary regression results. Finally, a normalized heatmap of each acupoint is output. Compared with existing human acupoint recognition methods, this method has the following technical advantages:

[0037] (1) Overcoming the shortcomings of low accuracy when using traditional ResNet101 as the backbone feature extraction network and the inability to combine depth information for prediction when focusing only on RGB images, this invention improves the input channel convolution of traditional networks to achieve early fusion of RGB information and depth information, thereby improving the ability of acupoint recognition models to mine acupoint location information.

[0038] (2) The human body region mask generation network and semantic attention enhancement module proposed in this invention strengthen the interaction between visual features and semantic features, effectively suppress cross-regional false detection and confusion of acupoints in similar regions, and improve the accuracy of acupoint positioning.

[0039] (3) It has strong practicality and generalization ability. The present invention has achieved good results on the current human acupoint dataset H-APDT and has important clinical application value. Attached Figure Description

[0040] Figure 1 This is a structural diagram of the human acupoint recognition model described in the embodiments of this application. Detailed Implementation

[0041] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.

[0042] This embodiment provides a method for recognizing human acupoints based on human body region segmentation and semantic attention enhancement, such as... Figure 1 As shown, it includes:

[0043] S1: Obtain RGB and depth images of the human body, extract image features using a backbone extraction network, and generate mask images of different human body structures using a human body region segmentation network.

[0044] This embodiment constructs and trains a human acupoint recognition model, which mainly includes a backbone extraction network, a human region segmentation network, and a re-attention mechanism module. First, a human acupoint dataset is collected and divided proportionally into training, validation, and test sets. Then, the training set is used to train the human acupoint recognition model, the validation set is used to validate the accuracy of the trained model, and the test set is used to test the generalization accuracy of the validated model. Finally, a human acupoint recognition model with the required generalization accuracy can be used for acupoint recognition in this invention.

[0045] In a specific embodiment, the H-APDT dataset includes 2612 sets of RGB and depth images, each set comprehensively covering 178 key acupoints commonly used in clinical acupuncture. By filtering the image data and using labeling, a dataset of 2612 fully labeled human acupoint images, each containing a bounding box and 178 key acupoints, was obtained. The dataset was then divided into training, validation, and test sets in an 8:1:1 ratio, with a resolution of 480×640.

[0046] The input image undergoes random cropping, center cropping, random augmentation, and data augmentation operations, including horizontal flipping, vertical flipping, and rotation. Some hyperparameters during the training of the human acupoint recognition model are as follows: the optimizer uses Adam, with an initial learning rate of 0.0001, a batch size of 4, and a total training duration of 50 epochs. The loss is reduced when it does not decrease within 10 epochs. Training requires an Ubuntu 16.04 LTS system or later, with PyTorch 1.6 and Python 3.6 or later. The hardware platform includes an Nvidia GeForce RTX 4060 GPU with CUDA 11.7, a CPU with at least 64GB of RAM, and a solid-state drive with at least 512GB of storage.

[0047] In this embodiment, the backbone extraction network used to extract image features specifically adopts ResNet101, and the first layer input convolution is changed from three channels to four channels to receive the concatenated input of RGB image and depth image in the channel dimension; the implementation process is represented as follows:

[0048] , ;

[0049] ;

[0050] ;

[0051] in, and These represent the input RGB image and depth image, respectively. Representative to and Perform channel splicing. The backbone extracts features from the input image of the network. It is a four-channel backbone extraction network. The image features extracted by the backbone extraction network represent the output of the image. The number of output channels extracted from the backbone network. and These represent the height and width of the input image, respectively. In this embodiment, the number of channels... .

[0052] This invention introduces a spatial attention mask constructed from body structural region information before acupoint location identification to improve the predictive ability of acupoint locations. In this embodiment, a human body region segmentation network is used to generate mask images of different human structures, including: firstly, using a feature extraction network to extract features from the RGB image of the human body; then, feeding the extracted features into a pre-trained mask generation network to generate a mask. A body region mask image with channels, where each channel represents the clustering information of different body regions; then... The body region mask image of each channel is upsampled to a resolution consistent with the features of the fused image to obtain a clustering mask; finally, the clustering mask is mapped to a channel number consistent with the features of the fused image. Specifically, this is represented as follows:

[0053] ;

[0054] ;

[0055] ;

[0056] In the formula, Represents the input RGB image, The RGB image feature extraction network represents the mask generation channel. Represents the RGB image features extracted from the mask generation channel; This represents a pre-trained mask generation network. This represents an upsampling operation. This represents the clustering mask obtained from upsampling; For 1×1 convolution, This represents the sigmoid activation function. This represents the final generated mask image; These represent the number of channels, height, and width of the image features extracted by the backbone extraction network, respectively.

[0057] S2 uses a mask image to perform multiplicative modulation on the image features to obtain image fusion features, and then passes through a convolutional layer to obtain preliminary acupoint recognition results based on human body region segmentation.

[0058] To effectively suppress false detections across regions, this invention uses a human body region segmentation masking network to obtain the final mask image representing the body region. Extracting image features from the backbone extraction network. Multiplicative modulation yields image fusion features, represented as follows:

[0059] ;

[0060] In the formula, The backbone network extracts image features from the output. Mask images of different human structures generated by a human body region segmentation network. For the Hadama product operation, These are the image fusion features obtained through multiplicative modulation.

[0061] A preliminary acupoint recognition result based on human body region segmentation is obtained through a convolutional layer, represented as follows:

[0062] ;

[0063] In the formula, The weight parameters represent the weights of a 1×1 convolutional layer. , The number of output channels extracted from the backbone network. and These are the height and width of the input image, respectively. Number of acupoint categories; This is a preliminary acupoint regression heatmap, serving as an initial result for acupoint identification. In this embodiment, the number of acupoint categories is... .

[0064] S3: Obtain prior knowledge of acupoints, use a dual attention mechanism, and obtain channel attention weights and spatial attention weights based on prior knowledge of acupoints and image fusion features.

[0065] The dual attention mechanism establishes a mapping relationship between feature representation and prior knowledge by embedding semantic information of acupoints, enabling the human acupoint recognition model to not only rely on the appearance features of images, but also to use the structural information of acupoints in semantic information for reasoning.

[0066] Specifically, a dual attention mechanism is used to process prior knowledge of acupoints and image fusion features, as follows:

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] ;

[0072] In the formula, This represents prior knowledge of acupoints. Number of acupoint categories This refers to the dimension of prior knowledge about acupoints; Each row represents a digital feature of prior knowledge about an acupoint, used to guide the model on "which acupoint to look for" in the input image; The number of channels for image fusion features; This represents a fully connected layer used to linearly transform the prior knowledge text of acupoints, mapping it to the same dimension as the image fusion features. This refers to the prior knowledge of acupoints after linear transformation mapping; For global average pooling operators, As a multi-head attention mechanism, This represents the query vector in MHA. This represents a specific pixel on the image fusion feature map. The deep semantic features contained therein; This represents the channel attention weights obtained by global average pooling of the image fusion feature X. This indicates the use of a multi-head attention mechanism. Image fusion features X and mapped acupoint prior knowledge Spatial attention weights are obtained through processing.

[0073] S4 utilizes channel attention weights and spatial attention weights to perform semantic attention enhancement processing on the preliminary acupoint recognition results, obtaining the final acupoint recognition results.

[0074] Step S4 performs semantic attention enhancement processing on the preliminary acupoint identification results, specifically as follows:

[0075] ;

[0076] In the formula, Represents the sigmoid activation function. Indicates channel attention weights. Represents spatial attention weights. This indicates the preliminary results of acupoint identification. This indicates the final acupoint identification result.

[0077] To facilitate understanding of the technical effects of this invention, a comparison of the application of this invention and conventional methods is provided below:

[0078] Table 1 presents ablation experiments verifying the effectiveness of each module in this embodiment. The main comparison focuses on the impact of different modules on the average precision (PCK). Other indicators are auxiliary metrics, representing: CLAcc (the percentage of correct acupoint existence judgments); Mean-PixErr (the minimum pixel difference of predicted points); and Max-PixErr (the maximum pixel difference of predicted acupoints). As shown in Table 1, on the H-APDT dataset, the proposed modules achieve a significant improvement in precision, with PCK improvements of 8.1%, 3.8%, and 1.7%, respectively. Furthermore, the proposed method also demonstrates significant advantages in auxiliary metrics. Table 1 also shows that the method for keypoint detection in RGB-D images using Human Region Segmentation (HRSM) and Semantic Attention Enhancement (SAE) achieves substantial performance improvements, demonstrating the effectiveness of this invention.

[0079] .

[0080] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.

Claims

1. A method for recognizing acupoints on the human body based on human body region segmentation and semantic attention enhancement, characterized in that, include: S1. Obtain the RGB and depth images of the human body, extract image features using the backbone extraction network, and generate mask images of different human body structures using the human body region segmentation network. Step S1 uses a human region segmentation network to generate mask images of different human structures, including: first, using a feature extraction network to extract features from the RGB image of the human body; then, feeding the extracted features into a pre-trained mask generation network to generate a mask. The body region mask of the channel; then... The body region mask image of each channel is upsampled to a resolution consistent with the features of the fused image to obtain a clustering mask; finally, the clustering mask is mapped to a number of channels consistent with the features of the fused image. S2, multiplicative modulation of image features using a mask image is used to obtain image fusion features, and then a convolutional layer is used to obtain preliminary acupoint recognition results based on human body region segmentation; S3: Obtain prior knowledge of acupoints, use a dual attention mechanism, and obtain channel attention weights and spatial attention weights based on prior knowledge of acupoints and image fusion features; S4 utilizes channel attention weights and spatial attention weights to perform semantic attention enhancement processing on the preliminary acupoint recognition results, obtaining the final acupoint recognition results.

2. The method for identifying human acupoints according to claim 1, characterized in that, The backbone network used for extracting image features specifically employs ResNet101, with the first-layer input convolution changed from three channels to four channels to receive the concatenated input of RGB and depth images in the channel dimension; the implementation process is represented as follows: ， ； ； ； in, and These represent the input RGB image and depth image, respectively. Representative to and Perform channel splicing. The backbone extracts features from the input image of the network. It is a four-channel backbone extraction network. The image features extracted by the backbone extraction network represent the output of the image. The number of output channels extracted from the backbone network. and These represent the height and width of the input image, respectively.

3. The method for identifying human acupoints according to claim 1, characterized in that, Image fusion features are obtained by multiplicatively modulating image features using a mask image, and are represented as follows: ； In the formula, The backbone network extracts image features from the output. Mask images of different human structures generated by a human body region segmentation network. For the Hadama product operation, Image fusion features obtained by multiplicative modulation; A preliminary acupoint recognition result based on human body region segmentation is obtained through a convolutional layer, represented as follows: ； In the formula, The weight parameters represent the weights of a 1×1 convolutional layer. , The number of output channels extracted from the backbone network. and These are the height and width of the input image, respectively. Number of acupoint categories; This is a preliminary acupoint regression heatmap, serving as an initial result for acupoint identification.

4. The method for identifying human acupoints according to claim 1, characterized in that, Step S3 uses a dual attention mechanism to process prior knowledge of acupoints and image fusion features, specifically as follows: ； ； ； ； ； In the formula, This represents prior knowledge of acupoints. Number of acupoint categories This refers to the dimension of prior knowledge about acupoints. The number of channels for image fusion features; This represents a fully connected layer used to linearly transform the prior knowledge text of acupoints, mapping it to the same dimension as the image fusion features. This refers to the prior knowledge of acupoints after linear transformation mapping; For global average pooling operators, As a multi-head attention mechanism, This represents the query vector in MHA. This represents a specific pixel on the image fusion feature map. The deep semantic features contained therein; This represents the channel attention weights obtained by global average pooling of the image fusion feature X. This indicates the use of a multi-head attention mechanism. Image fusion features X and mapped acupoint prior knowledge Spatial attention weights are obtained through processing.

5. The method for identifying human acupoints according to claim 1, characterized in that, Step S4 performs semantic attention enhancement processing on the preliminary acupoint identification results, specifically as follows: ； In the formula, Represents the sigmoid activation function. Indicates channel attention weights. Represents spatial attention weights. This indicates the preliminary results of acupoint identification. This indicates the final acupoint identification result.

6. The method for identifying human acupoints according to claim 1, characterized in that, Step S1 uses a human body region segmentation network to generate mask images of different human body structures, specifically as follows: ； ； ； In the formula, Represents the input RGB image, The RGB image feature extraction network represents the mask generation channel. Represents the RGB image features extracted from the mask generation channel; This represents a pre-trained mask generation network. This represents an upsampling operation. This represents the clustering mask obtained from upsampling; For 1×1 convolution, This represents the sigmoid activation function. This represents the final generated mask image; These represent the number of channels, height, and width of the image features extracted by the backbone extraction network, respectively.

Images