A Hand Acupoint Detection Method Based on Image Fusion and Attention Mechanism

By combining image fusion and attention mechanisms with Kinect depth camera and traditional Chinese medicine measurement methods, a graph neural network model was designed to solve the problems of accuracy and efficiency in acupuncture point detection, and to achieve high-precision hand acupoint localization.

CN117911264BActive Publication Date: 2026-11-17UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410242704.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-04
Publication Date
2026-11-17
Estimated Expiration
2044-03-04

AI Technical Summary

Technical Problem

In existing technologies, acupuncture point detection relies on human experience, which is inefficient and inaccurate. Deep learning algorithms have seen limited development in the field of combined color image and depth map detection, lacking depth image processing and noise filtering, resulting in inaccurate hand acupoint localization.

Method used

A hand acupoint detection method based on image fusion and attention mechanism is adopted. Images are acquired through Kinect depth camera, camera parameters are calibrated, depth map noise is removed, and hand acupoints are detected by combining bone measurement method and finger measurement method with graph neural network model with attention mechanism. Color image and depth map information are fused to extract key points and acupoints.

Benefits of technology

It achieves high-precision and efficient acupoint detection, combining traditional Chinese medicine theory with artificial intelligence to improve diagnostic efficiency and speed, and avoids random errors in a single algorithm.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117911264B_ABST
    Figure CN117911264B_ABST
Patent Text Reader

Abstract

This invention discloses a hand acupoint detection method based on image fusion and attention mechanisms, which improves the efficiency and accuracy of acupoint localization during acupuncture. It applies keypoint detection methods from deep learning to acupoint detection and localization, and employs neural networks to improve detection speed and real-time performance. A method for fusing depth and color images is proposed, improving the completeness of input data information, localization accuracy, and the model's generalization and fitting ability. A graph neural network based on an attention mechanism is proposed, which can analyze and calculate key parts of the image during input image processing, exhibiting high environmental adaptability. Finally, a method combining bone measurement and finger measurement methods from traditional Chinese medicine is proposed to correct acupoints using the theory of measurement, enhancing localization accuracy and technical persuasiveness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of image processing deep learning technology and traditional Chinese medicine acupuncture positioning theory, and in particular to the design of a hand acupoint detection method based on image fusion and attention mechanism. Background Technology

[0002] Acupuncture is a science that uses the theories of meridians and acupoints, along with needling techniques, to prevent disease. Acupuncture is a general term encompassing both moxibustion and acupuncture techniques. Needling involves inserting needles into the patient's body at specific angles under the guidance of Traditional Chinese Medicine (TCM) theory, and then stimulating specific points on the body using different needling techniques to treat diseases. Accurate identification of acupoints is crucial before needle insertion; the accuracy of acupoint identification plays a decisive role in disease treatment. Acupuncture has a long history, accumulated through ancient medical practice. As an important TCM therapy, traditional acupuncture is widely used worldwide. Specifically, the human body has 361 regular acupoints, with approximately one hundred on the hand, including more than ten important acupoints such as Dazhui, Hegu, and Neiguan.

[0003] In current technology, the artificial determination of acupoints based on traditional medical theory is entirely determined by acupuncturists based on relevant medical knowledge and clinical experience. However, acupuncturists need years of study and internship experience to correctly identify each acupoint, resulting in low diagnostic efficiency, high manpower and time costs, and a certain degree of inaccuracy in acupoint location and needle placement.

[0004] Depth images are widely used in object detection. Acquired depth images are typically ordinary color images with a depth map added. In 3D computer graphics, a depth map is an image or image channel containing information about the distances to the surfaces of objects in the scene relative to the viewpoint. Object detection using depth images can achieve more accurate results. However, most current deep learning object detection algorithms are based on two-dimensional detection and primarily use color images as input. Development in combining depth and color images for detection is limited, and there is a lack of depth image processing and noise filtering. In the field of combining deep learning with traditional medicine, acupoint detection using deep learning algorithms sometimes simply uses color images, while other times it uses both color and depth images as inputs, trains two models, and then directly adds the detection results from both models. However, the human hand is generally quite flat, and both color and depth images have missing information; considering only one image is insufficient for accurate acupoint localization.

[0005] Therefore, it is necessary to find a method for detecting hand acupoints that can make full use of image information and combine it with traditional acupuncture theory. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a technical solution for hand acupoint detection based on image fusion and an attention mechanism. This solution includes input image processing, depth map noise filtering, fusion of bone and finger measurement methods from traditional Chinese medicine theory, measurement positioning theory, fusion of color and depth images, a graph neural network model based on an attention mechanism for hand acupoint detection, and the design and training of this model. The attention mechanism designed in this invention is based on geometric and appearance information.

[0007] The technical solution of this system is: a hand acupoint detection method based on image fusion and attention mechanism, which includes:

[0008] Step 1: Acquire and correct the dataset;

[0009] Step 1.1: Obtain complete images of the hands and arms of multiple patients with different genders, ages, heights, and weights using a Kinect depth camera equipped with a color camera and a depth sensor.

[0010] Step 1.2: Camera calibration;

[0011] To accurately locate acupoints, the Kinect depth camera needs to be calibrated to determine the geometric relationship and parameters between them, ensuring that it can accurately capture and measure key points of hand structure and acupoints. The Kinect projection parameters are calibrated quickly and accurately by estimating the Kinect's resolution, while also correcting the relative pose between the color camera and the depth camera, and converting the Kinect parallax unit to metric depth.

[0012] Step 1.3: The calibration setup includes a checkerboard image of 11 horizontal squares and 8 vertical squares (25 mm in size) for calibrating the color image camera and depth sensor; as shown... Figure 1 As shown, multiple color and depth images were captured at an average distance of 1 to 1.5 m in three different directions. The rotation angle between consecutive images was set to 20 degrees. Furthermore, a planar shooting calibration image was captured to correct camera distortion, as detailed below. Figure 2 As shown; a series of correction parameters were obtained through the above operations;

[0013] Step 2: Align the color image with the depth image: Align the coordinates of the color image with the coordinates of the depth image using the reprojection matrix below:

[0014]

[0015] X = a1x + a2y + a3 - a7xX - a8yX

[0016] Y = a4x + a5y + a6 - a7xY - a8yY

[0017] in, This represents the color image pixels when the depth values ​​in the camera coordinate system are normalized, where X and Y represent the depth values ​​at the x and y coordinates, respectively. This represents the depth image pixels when the depth values ​​in the camera coordinate system are normalized, where x and y represent the horizontal and vertical coordinates of the depth image. This represents the transformation matrix from color image pixels to depth image pixel values.

[0018] Step 3: Remove noise from the depth map information: In this embodiment of the invention, an adaptive directional filter is used to remove noise and fill holes in the depth image;

[0019] Step 3.1: Calculate the gradient of the image: Considering 8 directions, apply the Sobel operator or other gradient operators to the depth image to calculate the gradient magnitude and direction of each pixel, as shown in the following formula:

[0020] G k (i,j)=|I(i,j)-I(i+dk,j+ek)|

[0021] Where (i,j) represents the position of the pixel, G k (i,j) represents the gradient of pixel (i,j) in direction k, I(i,j) represents the pixel value at position (i,j) in the depth image, and dk and ek represent the horizontal and vertical components of the gradient operator in direction k, respectively.

[0022] Step 3.2: Determine the local orientation of each pixel: For each pixel, calculate its local orientation based on its gradient; the direction with the largest gradient magnitude is found by using the angle of the gradient (or the ratio between the x and y components), which is called the gradient direction, and the formula is as follows:

[0023] D(i,j)=argmax(G k (i,j))

[0024] Step 3.3: Group pixels with similar orientations; group pixels with similar orientations in the image into different orientation groups; this helps to distinguish between random depth values ​​that may be noise and actual continuous structures;

[0025] Step 3.4: Filter for each direction; perform directional filtering on each pixel based on the dominant direction; that is, only consider neighborhood pixels similar to the dominant direction, and suppress directions different from the dominant direction. The weight calculation formula for each pixel p is as follows:

[0026]

[0027] Filtered pixel value calculation:

[0028] D(i,j)=(1 / K)*Σp∈N(i,j)W(i,j,p)*I(p)

[0029] K = Σp∈N(i,j)W(i,j,p)

[0030] Where W(i,j,p) represents the weight of p at position (i,j), G(i,j) represents the pixel value at position (i,j), G(p) represents the gradient magnitude of the center pixel, D(p) represents the direction of the center pixel, T represents the threshold of the direction difference, N(i,j) represents the total number of pixels in the neighborhood centered at (i,j), I(p) represents the value of the center pixel p in the original input image, and K represents the normalization coefficient;

[0031] Step 3.5: For pixels at the holes, interpolate and fill them by referencing the grayscale values ​​of nearby valid pixels. The specific formula is as follows:

[0032] D(i,j)=(1 / M)*Σq∈V(i,j)D(q)

[0033] Where V(i,j) represents the set of effective pixels within the hole centered at (i,j), and M is the number of effective pixels;

[0034] Step 4: Dataset creation; The RGB depth images of the patients were captured by placing their arms and hands on a flat white background surface, with their hands parallel to the background surface, freely extended, and fingers spread apart; Under these conditions, a dataset of approximately 2000 images (color and depth images) of the hands of 200 healthy patients was constructed, with an image resolution of 640×480.

[0035] Step 4.1: Determine "cun1" according to the bone measurement method: The bone measurement method is a method of locating acupoints according to proportions. Specifically, the distance from the wrist crease to the elbow crease is recorded as twelve cun. The distance between the elbow crease and the wrist crease is calculated by the two detected line segments and the bone measurement is calculated to obtain the length of one cun, which is recorded as 0cun1, where 0 represents patient number 0.

[0036] Step 4.1.1: Cut out the arm area based on the color space;

[0037] Convert the color image obtained by the camera to the YCbCr space, where chrominance and luminance are separable. Therefore, luminance is first extracted in this color space, leaving the Cb and Cr components, in which case skin color clustering performs well; thus, in the two-dimensional CbCr color space, the artificial threshold method is used to separate the skin color regions and perform a binarization operation, then the image is processed using opening operation and closing operation to eliminate part of the noise and smooth the boundaries, and finally the arm region is segmented using the maximum connected domain;

[0038] Step 4.1.2: Determine the wrist and elbow inflection points according to the arm region; acquire the inflection points of the wrist and the elbow based on an improved curvature-based inflection point detection method;

[0039] Step 4.1.3: Determine the circumscribed matrix;

[0040] Perform rectangular fitting on the arm portion from the wrist to the elbow inflection point according to the inflection points and the extracted arm region, find the minimum circumscribed matrix, and determine the forearm range;

[0041] Step 4.1.4: Determine the length of cun 1;

[0042] Perform edge detection according to the circumscribed matrix, where the upper and lower lines of the rectangle are the positions of the elbow crease and the wrist crease, and the irrelevant ranges are eliminated through Hough transform to determine the length from the elbow crease to the wrist crease, then divide this length by 12 to obtain the length of cun 1;

[0043] Step 4.2: Determine the length of "cun 2" according to the finger bone proportional cun measurement method;

[0044] Measure the length of the second phalangeal segment of the patient's middle finger and the width of the patient's thumb interphalangeal joint, and take the average of the two to obtain cun 2, denoted as 0cun2, where 0 represents the No. 0 patient;

[0045] Step 4.3: Take the average of cun 1 and cun 2 to obtain the final length unit "cun" used for determining acupoints, which can effectively avoid the occurrence of accidental errors; then use a Python script to label the data: in combination with the "cun" length of each patient obtained above, determine each acupoint on the hand by a professional; label the color image as 0lcolor.png, where 0l represents the left hand of the No. 0 patient, 0rcolor.png, where 0r represents the right hand of the No. 0 patient, and the depth image as 0ldepth, which is finally made into a PascalVOC format dataset; then divide the dataset into a training set and a test set at a ratio of 8:2, and manually label the hand acupoints on each image using the LabelImg tool;

[0046] Step 5: Starting from this step, the formal hand acupoint detection work is carried out, wherein the overall detection and positioning flowchart is as shown in Figure 1Hand region extraction: To reduce complexity and improve runtime, the object of interest is detected, and the foreground mask is obtained by subtracting the hand pixels from the image using background subtraction. A Gaussian mixture model is used to distinguish between the background and hand regions. The initial conditions are as follows:

[0047] μ0=I0

[0048] σ0 = <default value>

[0049] Among them, I t This represents the intensity of the pixel at time t. The default value is 1. The mean pixel intensity at time t is μ. t and variance They will all be updated according to the following formula:

[0050] μ t =ρI t +(1-ρ)μ t-1

[0051]

[0052] ρ = 0.01

[0053] Where ρ is the size of the time window, and d is the Euclidean distance between the mean and value of each pixel, and then the pixel is divided into background or hand according to the formula below and the specific pixel value:

[0054]

[0055] Where k is a set constant value, which is 2.3;

[0056] Step 6: Define key points of the hand structure based on the geometric and anatomical structure of the hand; this includes 5 fingertips, 10 phalanges, 5 finger base key points, and one palm base key point, for a total of 21 key points of the hand;

[0057] Step 7: Fuse the depth map and color map to obtain the fused image, and then obtain the point cloud distribution of the fused image:

[0058] Step 7.1: First, represent the points in the point cloud of the fused image as P = {[p1,a1],[p1,a2],...,[p...} N ,a N ]}∈R 6 , where p i = [x,y,z] represents the position of the point, a i = [r, g, b] represents the pixel at that point. From the fused point cloud P of a given point, a directed graph G = (V, E) is computed based on its spatial neighbors, where V = {1, 2, ..., N}. Let N represent the set of vertices and the set of directed edges, respectively, and let N represent the number of points in the point cloud.

[0059] Step 7.2: Construct the directed graph G = (V, E) as a k-nearest neighbor graph of P in geometric space. Self-loops are added to the graph, which means that each vertex has its own edge. Let N(i) = {j: (i, j) ∈ E} ∪ {i} be the set of neighbors of vertex i; let H = {h1, h2, ..., h N} represents the input feature set, as shown in the formula below:

[0060]

[0061]

[0062] in, This represents the edge features at position (i,j) at time t. Representing vertex features, f t () represents a function that calculates the edge features between vertices, ρ t () indicates that edge features are clustered, g t () Update vertices using aggregated edge features;

[0063] Step 8: Build a graph neural network based on attention to extract edge features;

[0064] First, an attention mechanism is employed, emphasizing the most important key parts of the fused image. This attention mechanism helps the graph neural network to more effectively extract and learn features from the most crucial areas of the image based on its geometric features. Specifically, the attention mechanism is based on... Figure 4 The attention weights for the edges of the set shapes, appearances, and features listed in the table represent their contribution or importance to the output representation. The specific formulas are as follows:

[0065]

[0066] Where || represents a join operation. p represents a multilayer perceptron. j This indicates the position of the point in the colored point cloud, a j c represents the image pixel value of that point in the color point cloud. i p represents the corresponding point i Feature weights;

[0067] Step 9: Then, use the softmax function to normalize all choices of j, which means normalizing the attention values ​​of each node and edge to between 0 and 1. The result is... This indicates relative importance; the specific formula is as follows:

[0068]

[0069] Where, N i p i The neighborhood set;

[0070] Step 10: The output obtained from the attention map neural network is further input into the fully connected layer to extract more abstract features;

[0071] Fully connected layers, also known as densely connected layers or simply fully connected layers, are a fundamental layer type in deep learning neural networks. Combining attention-based neural networks (GNNs) with fully connected layers allows for feature fusion and dimensionality reduction of the features extracted by the GNN, while simultaneously enhancing the model's expressive power. The output dimension of the GNN determines the dimension of the connected layers; using multi-dimensional fully connected layers can effectively extract image features across multiple dimensions.

[0072] Step 11: Use a Multilayer Perceptron (MLP) for modeling. The multilayer nonlinear transformation of the MLP allows for fitting more complex functions, enhancing the model's predictive expressive power. The MLP can map the feature vectors learned in the graph attention network to the final predicted value or category, establishing the mapping relationship between the graph neural network's learned features and the prediction target. Finally, the output of the attention-based graph neural network obtained through the MLP is:

[0073]

[0074] Step 12: During the training of the attention-based graph neural network, the model is trained 50 times with a batch size of 8. RAdam and Lookahead are used as optimizers, and cosine scheduling decay is used to finally obtain the trained attention-based graph neural network model.

[0075] Step 13: During the reasoning process, the trained attention-based graph neural network is used to detect key points and acupoints of the hand structure;

[0076] The reasoning process is also a process from feature vectors to the final prediction of key points of hand structure and acupoints of the hand. The feature vectors of the hand image extracted by the attention-based graph neural network are input into two parallel MLP branches respectively. These two MLP branches perform predictions of rotation and translation respectively. Finally, the two branches are fused to obtain the detection of key points of hand structure and acupoints.

[0077] Step 13.1: In the translation branch, a voting scheme is used to obtain the predicted output. First, the vector from feature point p to center point c is obtained according to MLP, where v represents the unit vector from the point to the center, as shown in the following formula:

[0078]

[0079] Then, a random sampling consensus algorithm is used to vote and obtain the center coordinates;

[0080] Step 13.2: In the rotation branch section, a regression scheme is used to obtain the predicted output, predicting the output at the rotation anchor point. The deviation is ΔR i Then the final rotation is obtained. Where ΔR i Restricted to Nearby, the regression loss is:

[0081]

[0082] Where, q i and It corresponds to R i and Quaternion representation,

[0083] Step 13.3: The final uncertain scores for key points and acupoints of the hand structure are as follows:

[0084]

[0085] Where, d i To account for the loss after normalization, the minimum uncertainty score is selected as the final output during the final detection process. That is, the location most likely to be the key point of the hand structure and the acupoint is selected as the output, which is represented as "Coor_Key" and "Coor_Acu1" respectively.

[0086] Step 14: Some acupoints on the hand are closely related to the location of key points in the hand structure. For example, the Daling acupoint on the hand is 5 cun above the wrist crease, the Guanyuan acupoint on the middle finger is 7 cun above the wrist crease, and the Yangchi acupoint on the middle finger is 8 cun above the wrist crease. Here, the wrist crease is the parallel line indicated by the palm root node. Relatively speaking, the location of key points in the hand structure is obtained more accurately than that of acupoints. Based on the palm root node obtained in Step 12.3, combined with the cun in Step 4.3, the locations of acupoints such as Daling and Guanyuan are re-determined. The re-determined acupoints are represented as "Coor_Acu2". The average of this value with the acupoint location "Coor_Acu1" obtained in Step 12.3 is taken to obtain the accurate output of the hand acupoints, which is represented as "Coor_Acu".

[0087] This invention, featuring a color and depth image fusion technology, can fully understand and utilize information from the patient's hand in the image to deduce the distribution of acupoints on the hand. Furthermore, it combines traditional Chinese medicine theory with artificial intelligence by integrating bone and finger measurement methods, while avoiding the random errors inherent in single algorithms. Simultaneously, the attention mechanism in this invention effectively extracts the geometric and shape features of the image, allowing the graph neural network to focus its feature extraction and learning on the most important parts of the image. In summary, this invention effectively utilizes image information, integrating traditional Chinese medicine theory and artificial intelligence algorithms, to significantly improve diagnostic efficiency and speed while maintaining high accuracy in acupoint detection. Attached Figure Description

[0088] Figure 1 This is a flowchart illustrating the overall process of acupoint detection and localization in this invention.

[0089] Figure 2 This is a schematic diagram of the camera being calibrated according to the present invention.

[0090] Figure 3 This is a schematic diagram of the graph neural network structure based on the attention mechanism of the present invention.

[0091] Figure 4 This is a structural diagram of the attention mechanism in this invention. Specific implementation methods

[0092] The technical details of embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0093] Example 1: Figure 1 This is a structural block diagram of the overall process of acupoint detection and positioning in this invention, as shown below. Figure 1 As shown, it includes:

[0094] S1: Acquire the original image, including the depth image and the color image, process the color image and then fuse it with the depth image.

[0095] S11: Specifically, the first step is to align the color RGB image with the depth image to ensure that the coordinates of corresponding pixels in the two images are aligned.

[0096] S12: Then, the aligned color image is processed to remove noise. Specifically, an adaptive directional filter is used to remove noise and fill holes in the image.

[0097] S13: Then, skin color segmentation is performed on the color image based on the color space to cut out the hand area. The segmented image is then fused with the depth image to generate a three-dimensional spatial image and obtain the point cloud representation of the fused image.

[0098] S2: Further, for the color image of the cut-out hand area, “cun1” and “cun2” are determined by the bone measurement method and the finger measurement method respectively, and the two are averaged to obtain the size of “cun” required in the measurement theory.

[0099] S3: Feed the fused image into a graph neural network based on an attention mechanism.

[0100] Specifically, the attention mechanism enables our graph neural network to focus feature learning on the most important parts of the image, especially key points of the hand structure and the concave and convex areas of the hand. This graph neural network uses multiple attention GNN networks and fully connected layers to fully utilize the depth and color information of the image, extracting hand features in multiple dimensions, and determining key points of the hand structure and acupoints based on the extracted features, denoted as "Coor_Key" and "Coor_Acu1" respectively.

[0101] S4: Some acupoints on the hand are obtained by combining the theory of proportion with key points of the hand structure.

[0102] Specifically, acquiring key points of the hand structure is more precise than acquiring acupoints. Some acupoints on the hand are closely related to the location of these key points, such as the Daling acupoint on the hand, which is 5 cun above the wrist crease; the Guanyuan acupoint on the middle finger, which is 7 cun above the wrist crease; and the Yangchi acupoint on the middle finger, which is 8 cun above the wrist crease. Based on the key points of the hand structure output by the graph neural network in S3 and the "cun" value calculated in S2, some acupoints on the hand are relocated and represented as "Coor_Acu2".

[0103] S5: Merge the hand acupoints obtained from the proportion theory with those obtained from the network model. Specifically, take the average of "Coor_Acu2" and "Coor_Acu1" to obtain the updated hand acupoint coordinates, denoted as "Coor_Acu", and output it as the final result.

[0104] Example 2: As Figure 2 The diagram shown illustrates the spatial geometric relationship when the camera captures data according to the present invention.

[0105] This invention requires the use of a Kinect depth camera with a color camera and a depth sensor to obtain images of the patient's hand. Before using the Kinect, the camera needs to be calibrated to ensure that it can accurately acquire hand image information.

[0106] Specifically, such as Figure 2As shown, to correct camera distortion, a planar calibration image was captured, and multiple color and depth images were captured at average distances of 1 to 1.5 m in three different directions. The rotation angle between consecutive images was set to 20 degrees. A series of correction parameters were obtained through the above operations. The camera's projection parameters were calibrated based on these correction parameters, and the relative pose between the color and depth cameras was also corrected.

[0107] Example 3: As Figure 3 The diagram shows the graph neural network structure based on the attention mechanism of the present invention. This neural network can extract geometric and appearance features from the fused image, and comprehensively utilize the extracted features to determine key points of the hand structure and acupoints of the hand.

[0108] S1: The feature extraction part of this graph neural network consists of multiple attention GNN networks combined with multiple fully connected layers of different sizes. Furthermore, the entire network contains five attention GNN networks and five fully connected layers. NX64 represents a fully connected layer with 64 neurons; NX128 represents a fully connected layer with 128 neurons, and so on.

[0109] S2: The high-order feature representations of the graph are extracted by the multi-layer attention GNN network and fully connected layers. These features can be further refined by the multi-layer perceptron (MLP) to obtain more abstract and relevant feature representations of the prediction target. Specifically, the features extracted by the three fully connected layers (NX128, NX256, NX512) are extracted and fused using concatenation and pooling to achieve feature fusion across different dimensions. The fused features are then input into two parallel MLP branches. These two parallel MLP branches predict the rotation direction and the translation direction, respectively. The prediction for the rotation branch is obtained using a regression method, and the prediction for the translation branch is obtained using a voting method. The prediction results of the two branches are combined to finally obtain the outputs of the hand structure keypoint "Coor_Acu1" and the hand acupoint "Coor_Key".

[0110] Example 4: Figure 4 The diagram shows the structure of the attention mechanism in this invention. The attention GNN network receives relative geometry, appearance, and features as input, and then normalizes them into attention weights.

[0111] Specifically, the relative geometry, appearance, and other features of the hand in the fusion image are encoded into node feature matrices and edge feature matrices, which are then input into the attention layer. The attention GNN network generates attention weights for each node and edge through dot product attention, multiplicative attention, etc., representing their contribution or importance to the output representation. Finally, the attention values ​​of each node and edge are normalized to between 0 and 1 through the softmax function to obtain the relative importance expression.

Claims

1. A method for detecting hand acupoints based on image fusion and attention mechanisms, the method comprising: Step 1: Acquire and correct the dataset; Step 1.1: Obtain complete images of the hands and arms of multiple patients with different genders, ages, heights, and weights using a Kinect depth camera equipped with a color camera and a depth sensor. Step 1.2: Camera calibration; To accurately locate acupoints, the Kinect depth camera needs to be calibrated to determine the geometric relationship and parameters between them, ensuring that it can accurately capture and measure key points of hand structure and acupoints. The Kinect projection parameters are calibrated quickly and accurately by estimating the Kinect's resolution, while also correcting the relative pose between the color camera and the depth camera, and converting the Kinect parallax unit to metric depth. Step 1.3: Calibrate the color imaging camera and depth sensor; Step 2: Align the color image with the depth image: Align the coordinates of the color image with the coordinates of the depth image using the reprojection matrix below: X = a1x + a2y + a3 - a7xX - a8yX Y = a4x + a5y + a6 - a7xY - a8yY in, This represents the color image pixels when the depth values ​​in the camera coordinate system are normalized, where X and Y represent the depth values ​​at the x and y coordinates, respectively. This represents the depth image pixels when the depth values ​​in the camera coordinate system are normalized, where x and y represent the horizontal and vertical coordinates of the depth image. A transformation matrix representing the transformation values ​​from color image pixels to depth image pixel values; Step 3: Remove noise from the depth map information; Step 4: Dataset creation; Step 4.1: Determine "cun1" according to the bone measurement method: The bone measurement method is a method of locating acupoints according to proportions. Specifically, the distance from the wrist crease to the elbow crease is recorded as twelve cun. The distance between the elbow crease and the wrist crease is calculated by the two detected line segments and the bone measurement is calculated to obtain the length of one cun, which is recorded as 0cun1, where 0 represents patient number 0. Step 4.1.1: Cut out the arm area based on the color space; The color images obtained by the camera are converted to the YCbCr color space. Since the chromaticity and luminance of the YCbCr color space can be separated, the luminance is extracted first in this color space, leaving the CbCr component. At this time, the skin color clustering is better. Then, in the two-dimensional CbCr color space, the skin color region is separated by artificial thresholding and binarization is performed. Then, opening and closing operations are used to process the image to eliminate some noise and smooth the boundaries. Finally, the arm region is segmented by the maximum connected region. Step 4.1.2: Determine the inflection points of the wrist and elbow based on the arm region; obtain the inflection points of the wrist and elbow based on the improved bending inflection point detection method; Step 4.1.3: Determine the circumscribed matrix; Based on the inflection point and the extracted arm region, a rectangle is fitted to the arm portion from the wrist to the elbow inflection point to find the minimum outer matrix and determine the forearm range. Step 4.1.4: Determine the length of inch 1; Edge detection is performed based on the circumscribed matrix, where the upper and lower lines of the rectangle represent the positions of the elbow crease and wrist crease. Irrelevant regions are eliminated by Hough transform to determine the length from the elbow crease to the wrist crease. This length is then divided by 12 to obtain the length of inch 1. Step 4.2: Determine the length of "cun 2" according to the finger measurement method; Measure the length of the second phalanx of the middle finger of the patient and the width of the thumb knuckle of the patient, and average the two to obtain cun2, which is denoted as 0cun2, where 0 represents the 0th patient; Step 4.3: Average cun1 and cun2 to obtain the final length unit "cun" for determining acupoints, and in combination with the obtained length of "cun" for each patient mentioned above, each acupoint on the hand is determined by a professional; Step 5: Extract the hand region; Step 6: Define hand structure keypoints according to the geometric structure and anatomical structure of the hand; Step 7: Fuse the depth map and the color map to obtain a fused image, and obtain the point cloud distribution of the fused image accordingly: Step 7.1: First, represent the points in the point cloud of the fused image as P = {[p1,a1],[p1,a2],...,[p...} N ,a N ]}∈R 6 , where p i = [x,y,z] represents the position of the point, a i = [r, g, b] represents the pixel at that point. From the fused point cloud P of a given point, a directed graph G = (V, E) is computed based on its spatial neighbors, where V = {1, 2, ..., N}. Let N represent the set of vertices and the set of directed edges, respectively, and let N represent the number of points in the point cloud. Step 7.2: Construct the directed graph G = (V, E) as a k-nearest neighbor graph of P in geometric space. Self-loops are added to the graph, which means that each vertex has its own edge. Let N(i) = {j: (i, j) ∈ E} ∪ {i} be the set of neighbors of vertex i; let H = {h1, h2, ..., h N } represents the input feature set, as shown in the formula below: in, This represents the edge features at position (i,j) at time t. Representing vertex features, f t (·) represents a function that calculates the edge features between vertices, ρ t (·) indicates that edge features are clustered, g t (·) Update vertices using aggregated edge features; Step 8: Establish a graph neural network based on attention to extract edge features; The specific formula is as follows: Where || represents a join operation. p represents a multilayer perceptron. j This indicates the position of the point in the colored point cloud, a j c represents the image pixel value of that point in the color point cloud. i p represents the corresponding point i Feature weights; Step 9: Then, use the softmax function to normalize all choices of j, which means normalizing the attention values ​​of each node and edge to between 0 and 1. The result is... This indicates relative importance; the specific formula is as follows: Where, N i p i The neighborhood set; Step 10: The output obtained by the above attention-based graph neural network is further input into a fully connected layer to extract more abstract features; Step 11: A multi-layer perceptron (MLP) is used for modeling, and the final output of the attention-based graph neural network obtained through the multi-layer perceptron is: Step 12: Train the attention-based graph neural network to obtain a trained attention-based graph neural network model; Step 13: During the inference process, the trained attention-based graph neural network is used to detect hand structure keypoints and acupoints; The inference process is also a process of mapping from feature vectors to the final predicted hand structure keypoints and hand acupoints. The hand image feature vectors extracted by the attention-based graph neural network are respectively input into two parallel MLP branches, which respectively perform prediction for the rotation branch and the translation branch, and finally the two branches are fused to obtain the detection of hand structure keypoints and acupoints; Step 13.1: In the translation branch part, a voting scheme is adopted to obtain the prediction output. First, the vector from the feature point p to the center point c is obtained according to the MLP, where v represents the unit vector from the point to the center, as shown in the following formula: Then the random sample consensus algorithm is used for voting to obtain the center coordinates; Step 13.2: In the rotation branch section, a regression scheme is used to obtain the predicted output, predicting the output at the rotation anchor point. The deviation is ΔR i Then the final rotation is obtained. Where ΔR i Restricted to Nearby, the regression loss is: Where, q i and It corresponds to R i and Quaternion representation, Step 13.3: The final uncertainty scores of the hand structure keypoints and acupoints are as follows: Where, d i To account for the loss after normalization, in the final detection process, the minimum uncertainty score is selected as the final output, that is, the location most likely to be the key point of the hand structure and the acupoint is selected as the output, which is represented as "Coor_Key" and "Coor_Acu1" respectively. The hand Daling acupoint is 5 cun above the wrist crease, the middle finger Guanyuan acupoint is 7 cun above the wrist crease, and the middle finger Yangchi acupoint is 8 cun above the wrist crease, where the wrist crease is the parallel line indicated by the palm root node; According to the palm root node obtained in step 12.3, combined with the "cun" in step 4.3, the positions of acupoints such as the Daling acupoint and the middle finger Guanyuan acupoint are re-determined, the re-determined acupoints are denoted as "Coor_Acu2", and averaged with the acupoint position "Coor_Acu1" obtained in step 12.3, so as to finally obtain the accurate output of the hand acupoints, and the final output is denoted as "Coor_Acu".

2. The hand acupoint detection method based on image fusion and attention mechanism as described in claim 1, characterized in that, The specific method of the aforementioned 3 is: Step 3.1: Calculate the gradient of the image; Calculate the gradient magnitude and direction of each pixel point, the formula is as follows: G k (i,j)=|I(i,j)-I(i+dk,j+ek)| Where (i,j) represents the position of the pixel, G k (i,j) represents the gradient of pixel (i,j) in direction k, I(i,j) represents the pixel value at position (i,j) in the depth image, and dk and ek represent the horizontal and vertical components of the gradient operator in direction k, respectively. Step 3.2: Determine the local direction of each pixel point: for each pixel, calculate its local direction based on its gradient; the formula is as follows: D(i,j)=argmax(G k (i,j)) Step 3.3: Group pixels with similar directions; divide pixels with similar directions in the image into one group to form different direction groups; Step 3.4: Filter for each direction; perform directional filtering on each pixel based on the dominant direction; that is, only consider neighborhood pixels similar to the dominant direction, and suppress directions different from the dominant direction. The weight calculation formula for each pixel p is as follows: Filtered pixel value calculation: D(i,j)=(1 / K)*Σp∈N(i,j)W(i,j,p)*I(p) K = Σp∈N(i,j)W(i,j,p) Where W(i,j,p) represents the weight of p at position (i,j), G(i,j) represents the pixel value at position (i,j), G(p) represents the gradient magnitude of the center pixel, D(p) represents the direction of the center pixel, T represents the threshold of the direction difference, N(i,j) represents the total number of pixels in the neighborhood centered at (i,j), I(p) represents the value of the center pixel p in the original input image, and K represents the normalization coefficient; Step 3.5: For pixels at the holes, interpolate and fill them by referencing the grayscale values ​​of nearby valid pixels. The specific formula is as follows: D(i,j)=(1 / M)*Σq∈V(i,j)D(q) Where V(i,j) represents the set of valid pixels within the hole centered at (i,j), and M is the number of valid pixels.

3. The hand acupoint detection method based on image fusion and attention mechanism as described in claim 1, characterized in that, In step 4.3, the color images are labeled as 0lcolor.png, where 0l represents the left hand of patient 0, and 0rcolor.png, where 0r represents the right hand of patient 0. The depth images are labeled as 0ldepth, and the final dataset is made in PascalVOC format. It is then divided into training and testing sets in an 8:2 ratio, and the acupoints on the hands are manually labeled for each image using the LabelImg tool.

4. The hand acupoint detection method based on image fusion and attention mechanism as described in claim 1, characterized in that, The specific method for step 5 is as follows: To reduce complexity and improve runtime, the object of interest is detected, and a foreground mask is obtained by subtracting the hand pixels from the image using background subtraction. A Gaussian mixture model is used to distinguish between the background and hand regions. The initial conditions are as follows: μ0=I0 σ0 = <default value> Among them, I t This represents the intensity of the pixel at time t. The default value is 1. The mean pixel intensity at time t is μ. t and variance They will all be updated according to the following formula: m t =ρI t +(1-p)m t-1 Where ρ is the size of the time window, and d is the Euclidean distance between the mean and value of each pixel, and then the pixel is divided into background or hand according to the formula below and the specific pixel value: Where k is a set constant value.

5. The hand acupoint detection method based on image fusion and attention mechanism as described in claim 1, characterized in that, The key points of the hand structure in step 6 include 5 fingertips, 10 finger joints, 5 key points at the base of the fingers, and one key point at the base of the palm, for a total of 21 key points of the hand.

6. The hand acupoint detection method based on image fusion and attention mechanism as described in claim 2, characterized in that, In step 3.1, the gradients in eight directions are calculated, and the Sobel operator is applied to the depth image.

7. The hand acupoint detection method based on image fusion and attention mechanism as described in claim 4, characterized in that, in, ρ = 0.01, k = 2.3.

Citation Information

Patent Citations

  • Acupoint positioning method based on double depth cameras

    CN110930374A

  • Back acupuncture point identification method and device, electronic equipment and readable storage medium

    CN116403241A