A machine vision-based back acupoint positioning method

Abstract

The application discloses a back acupoint positioning method based on machine vision, first, two-dimensional RGB image data containing contour, posture and other information of a human body and three-dimensional point cloud data containing depth information of the back of the human body are acquired; shoulder feature points are extracted from the two-dimensional RGB image data; hip feature points are extracted from the three-dimensional point cloud data; the extracted two-dimensional shoulder feature points and hip feature points are fused; model training is performed; coordinates of all the fused feature points are input into a GBDT algorithm for regression training to obtain a regression model of standard acupoint coordinates; standard standard acupoint coordinates are obtained through the regression model; a calculation formula of other back acupoint coordinates is established based on the standard standard acupoint coordinates, and other back acupoint coordinates are calculated; after the model training is completed, manual calibration is no longer needed, and reference is provided for subsequent various massage operations.

Description

Technical Field

[0001] This invention relates to the field of back acupoint location technology, specifically a method for locating back acupoints based on machine vision. Background Technology

[0002] Locating acupoints on the back is one of the challenges in the research of TCM massage robots. Traditional TCM massage typically relies on the doctor's experience and the TCM theory of "body measurement," which uses surface features of the joints to determine the specific location of acupoints. However, these traditional methods require on-site palpation to determine coordinates, and are heavily influenced by the doctor's experience and on-the-spot decisions. Currently, the number of TCM massage therapists with comprehensive theoretical knowledge of TCM is limited, and relying solely on manpower cannot meet the market demand for TCM massage.

[0003] Over the past decade, numerous scholars have conducted research on acupoint location. These studies can be broadly categorized into two types: marker recognition and coordinate regression. Marker recognition determines acupoint coordinates by identifying markers on the human back; depending on the type of marker, it can be further divided into physical marker methods and optical marker methods. Since 2012, Lu Shouyin et al. from Shandong Jianzhu University have researched a TCM massage robot based on a gantry structure. This robot achieves precise massage mode switching, but manual marking is required before each use to determine acupoint coordinates. Lin Xuehua et al. from Fuzhou University applied optical positioning methods to locate acupoint coordinates in 2014. However, their optical markers only play an auxiliary role in judgment; the final acupoint location still relies on manual judgment. These methods are generally susceptible to interference and greatly influenced by the environment and the operator's own experience. Coordinate regression, on the other hand, does not require manual marking during massage operations. Liu Zhaotong et al. used a person's height, weight, and known acupoint coordinates as training parameters to obtain a positioning model for predicting target acupoint coordinates, achieving relatively accurate acupoint location results; however, the prediction still requires known acupoint coordinates as input. Existing researchers have developed a human acupoint localization system based on a PSO-BP neural network. This system pre-obtains reference acupoint coordinates through manual annotation and then trains the system to obtain the coordinates of target acupoints. Building upon this, further researchers have employed deep learning algorithms to predict acupoint coordinates, improving the prediction accuracy. However, training deep learning models requires a large amount of data, and the collection of acupoint data relies on manual calibration, which is challenging. Preparing large datasets is extremely difficult and cannot meet the automation requirements of intelligent massage robots. Therefore, we propose an improvement: a machine vision-based method for locating back acupoints. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0005] This invention discloses a method for locating acupoints on the back based on machine vision, comprising the following steps:

[0006] Step 1: Obtain 2D RGB image data containing information such as the outline and posture of the human body, and 3D point cloud data containing depth information of the human back;

[0007] Step 2: Extract shoulder feature points from 2D RGB image data; extract hip feature points from 3D point cloud data;

[0008] Step 3: Fuse the extracted 2D shoulder feature points and hip feature points;

[0009] Step 4: Perform model training. Input the coordinates of all fused feature points into the GBDT algorithm for regression training to obtain the regression model of the reference acupoint coordinates.

[0010] Step 5: Obtain the standard reference acupoint coordinates through the regression model. Based on the standard reference acupoint coordinates, establish the calculation formula for other back acupoint coordinates and calculate the coordinates of other back acupoints.

[0011] As a preferred technical solution of the present invention, the method used in step 2 to extract shoulder feature points from two-dimensional RGB image data is to extract features using the OpenPose human pose detection algorithm, with the three key points of the human skeleton—the left shoulder node, the neck and back node, and the right shoulder node—as two-dimensional feature points.

[0012] As a preferred technical solution of the present invention, the method of extracting buttock feature points from three-dimensional point cloud data in step 2 is as follows: a 3D camera is used to capture three-dimensional point cloud data of the human back. The intersection of the line connecting the highest points of the two buttocks on both sides of the human body and the posterior midline is used as the three-dimensional feature point. The y-axis of the 3D camera is kept parallel to the long side of the massage bed. Therefore, the position of the posterior midline of the human body can be determined based on the number of points on both sides of the point cloud y-axis. The z-axis coordinates of the points on both sides are traversed and compared to obtain the coordinates of the highest points of the two buttocks on both sides. Finally, the three-dimensional point cloud feature points are calculated.

[0013] As a preferred technical solution of the present invention, the method for fusing the extracted two-dimensional shoulder feature points and hip feature points is to convert the 3D camera coordinate system into the RGB camera coordinate system by rotating and translating the rigid body, and then converting the three-dimensional coordinates into two-dimensional coordinates according to the perspective projection relationship, and fusing the three-dimensional hip feature points and two-dimensional shoulder feature points into the two-dimensional coordinate system.

[0014] As a preferred technical solution of the present invention, the method of model training is to integrate the coordinates of the fused buttock feature points and shoulder feature points into a vector as input, and input it into the gradient boosting decision tree machine learning algorithm to establish a regression model of the reference acupoint coordinates.

[0015] As a preferred embodiment of the present invention, step 5 specifically involves inputting the newly acquired feature point data into the model to obtain the reference acupoint coordinates, where the reference acupoint coordinates are P. 大椎穴 P 腰阳关穴 Based on the distribution pattern of vertebrae in the spinal region and the coordinates of the reference acupoints, the formula for calculating the coordinates of other acupoints is as follows:

[0016]

[0017] The beneficial effects of this invention are:

[0018] This machine vision-based method for locating back acupoints involves the following steps: First, acquiring 2D RGB image data containing information such as the human body's contour and posture, and 3D point cloud data containing depth information of the human back; extracting shoulder feature points from the 2D RGB image data; extracting hip feature points from the 3D point cloud data; fusing the extracted 2D shoulder and hip feature points; training the model by inputting the coordinates of all fused feature points into a GBDT algorithm for regression training to obtain a regression model for the baseline acupoint coordinates; obtaining standard baseline acupoint coordinates through the regression model, and establishing calculation formulas for other back acupoint coordinates based on the standard baseline acupoint coordinates to calculate the coordinates of other back acupoints; after model training is complete, manual calibration is no longer required, providing a reference for subsequent massage operations. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0020] Figure 1 This is a flowchart illustrating a method for locating acupoints on the back based on machine vision, according to the present invention.

[0021] Figure 2 This is a network framework diagram of OpenPose, a method for locating acupoints on the back based on machine vision, according to the present invention.

[0022] Figure 3 This is a schematic diagram of the GBDT algorithm flow for a machine vision-based method for locating acupoints on the back according to the present invention.

[0023] Figure 4This is a schematic diagram showing the locations of the Dazhui and Yaoyangguan acupoints in the back acupoints of the present invention, which is based on machine vision.

[0024] Figure 5 This is a schematic diagram showing the relative positional relationship between acupoints on the Bladder Meridian of Foot Taiyang and the Governing Vessel, based on a machine vision-based method for locating acupoints on the back according to the present invention.

[0025] Figure 6 This is a schematic diagram of the back acupoint localization result of a back acupoint localization method based on machine vision according to the present invention. Detailed Implementation

[0026] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0027] Example: Figure 1 , Figure 2 , Figure 3 As shown, this invention provides a machine vision-based method for locating acupoints on the back. In Traditional Chinese Medicine (TCM) meridian theory, the main meridians on the back that are easily accessible for automated massage include the Governing Vessel (Du Mai) located along the spine on the posterior midline, and the Bladder Meridian located 1.5 and 3 bone-widths lateral to the spine. The acupoint location reference on the Bladder Meridian is the same as that on the Governing Vessel, perpendicular to the spine. Therefore, the coordinates of the acupoints on the Governing Vessel can be used as a reference to locate the coordinates on the Bladder Meridian. Furthermore, the Governing Vessel, located along the spine, has abundant surface features, providing numerous two-dimensional and three-dimensional landmarks for identification, making it easier to determine the location and improving the accuracy and stability of automated acupoint positioning. Among the 12 acupoints on the Governing Vessel on the back, the Dazhui (GV14) and Yaoyangguan (GV3) acupoints are the two most easily located and have the most obvious surface landmarks. Figure 4 As shown. Because the Dazhui (GV14) and Yaoyangguan (GV3) acupoints are located at the beginning and end of the Du meridian on the back, respectively, and the distance between the acupoints is relatively far, they can serve as the positioning reference for other acupoints on the Du meridian. Therefore, the acupoint positioning scheme of this invention is mainly divided into two steps. The first step is to calculate and regress the specific coordinates of the Dazhui (GV14) and Yaoyangguan (GV3) acupoints by means of the contour of the human body surface and body surface information. The second step is to infer the coordinates of other acupoints on the Du meridian and the Bladder meridian based on the known positional relationship between the two reference acupoints and the back acupoints.

[0028] Step 1: Acquire 2D RGB image data containing human contour and posture information, and 3D point cloud data containing depth information of the human back; Step 2: Extract shoulder feature points from the 2D RGB image data; extract hip feature points from the 3D point cloud data; The method used in Step 2 to extract shoulder feature points from the 2D RGB image data is to use the OpenPose human pose detection algorithm for feature extraction, using three key points in the human skeleton—the left shoulder node, the neck and back node, and the right shoulder node—as 2D feature points; 2D feature points are mainly determined based on the outer contour of the human body, so this study references feature point extraction in pose recognition, using the human shoulder key points extracted by the OpenPose human pose estimation network as 2D feature points for acupoint regression. The OpenPose network framework structure diagram is shown below. Figure 2 As shown, OpenPose is a top-down human pose detection algorithm. It first detects keypoints in the image and then connects them to form a skeleton. Compared to bottom-up methods, which first detect the human body's bounding box and then the keypoints within it, the top-down approach is less affected by the background, requires a smaller receptive field, has lower computational complexity, higher detection efficiency, and stronger stability in single-person pose detection. OpenPose adds new Part Affinity Fields (PAFs) to the previous Convolutional Pose Machine (CPM). These features determine the relationship between candidate regions by calculating the line integral between each pixel and the line connecting keypoints. This preserves not only the positional information of the limb support domain but also the orientation information. Combined with the detection of the keypoints themselves, it can more effectively output pose estimation results.

[0029] Step 2, which involves extracting buttock feature points from 3D point cloud data, utilizes the fact that the 3D data contains depth information about the human back, reflecting its undulations and providing additional information to support the determination of acupoint coordinates on top of the 2D image. The 3D feature points are primarily determined based on the back depth information reflected in the point cloud data. However, due to system errors in the depth acquisition equipment, the point cloud is affected by the environment during acquisition, resulting in outliers and point oscillations. Therefore, preprocessing such as filtering and smoothing of the point cloud is necessary to ensure the accuracy of the subsequently calculated 3D feature points. The method involves capturing 3D point cloud data of the human back using a 3D camera. The intersection of the line connecting the highest points of both buttocks and the posterior midline is used as the 3D feature point. The y-axis of the 3D camera is kept parallel to the long side of the massage bed during capture. Therefore, the position of the posterior midline can be determined based on the number of points on both sides of the point cloud along the y-axis. The z-axis coordinates of the points on both sides are then compared to obtain the coordinates of the highest points of both buttocks, ultimately calculating the 3D point cloud feature points.

[0030] Step 3: Fuse the extracted two-dimensional shoulder feature points and hip feature points; The method for fusing the extracted two-dimensional shoulder feature points and hip feature points is to rotate and translate the rigid body to convert the 3D camera coordinate system into the RGB camera coordinate system, and then convert the three-dimensional coordinates into two-dimensional coordinates according to the perspective projection relationship, and fuse the three-dimensional hip feature points and two-dimensional shoulder feature points into the two-dimensional coordinate system.

[0031] Step 4: Perform model training. Input the coordinates of all fused feature points into the GBDT algorithm for regression training to obtain the regression model of the reference acupoint coordinates.

[0032] The method for training the model is to integrate the coordinates of the fused buttock feature points and shoulder feature points into a vector as input, and then input it into the gradient boosting decision tree machine learning algorithm to establish a regression model of the reference acupoint coordinates.

[0033] To obtain the coordinates of the target acupoint by fully utilizing two-dimensional and three-dimensional feature points, this invention integrates the coordinates of all feature points into a vector as input and outputs the coordinates of the target acupoint for regression. Due to limitations in the size of the collected dataset, this invention employs the Gradient Boosting Decision Tree (GBDT) algorithm, which is more suitable for small sample computations. As a member of the ensemble learning Boosting class, GBDT is essentially a combination of gradient descent and CART regression trees, and is an iterative decision tree algorithm. Figure 3As shown, each iteration generates a weak learner. Each learner is trained based on the residuals of the previous learner, continuously approximating the target value by fitting the residuals of the previous problem. After training the regression model using the obtained training set, the newly collected feature point data can be input into the model to predict the acupoint coordinates.

[0034] Step 5: Obtain the standard reference acupoint coordinates through the regression model. Based on the standard reference acupoint coordinates, establish the calculation formula for other back acupoint coordinates and calculate the coordinates of other back acupoints.

[0035] The specific operation of step 5 is to input the newly collected feature point data into the model to obtain the reference acupoint coordinates, where the reference acupoint coordinates are P. 大椎穴 P 腰阳关穴 ,

[0036] The Governing Vessel (Du Mai) has a total of 12 acupoints distributed on the back, namely:

[0037] (1) Dazhui point: In the spinal region, in the depression below the spinous process of the 7th cervical vertebra, on the posterior midline;

[0038] (2) Taodao acupoint: In the spinal region, in the depression below the spinous process of the first thoracic vertebra, on the posterior midline;

[0039] (3) Shenzhu point: In the spinal region, in the depression below the spinous process of the 3rd thoracic vertebra, on the posterior midline;

[0040] (4) Shendao Point: Located in the spinal region, in the depression below the spinous process of the 5th thoracic vertebra, on the posterior midline;

[0041] (5) Lingtai point: In the spinal region, in the depression below the spinous process of the 6th thoracic vertebra, on the posterior midline;

[0042] (6) Zhiyang point: In the spinal region, in the depression below the spinous process of the 7th thoracic vertebra, on the posterior midline;

[0043] (7) Jinshuo point: In the spinal region, in the depression below the spinous process of the 9th thoracic vertebra, on the posterior midline;

[0044] (8) Zhongshu point: In the spinal region, in the depression below the spinous process of the 10th thoracic vertebra, on the posterior midline;

[0045] (9) Jizhong point: In the spinal region, in the depression below the spinous process of the 11th thoracic vertebra, on the posterior midline;

[0046] (10) Xuanshu point: In the spinal region, in the depression below the spinous process of the first lumbar vertebra, on the posterior midline;

[0047] (11) Mingmen point: In the spinal region, in the depression below the spinous process of the second lumbar vertebra, on the posterior midline;

[0048] (12) Yaoyangguan: In the spinal region, in the depression below the spinous process of the 4th lumbar vertebra, on the posterior midline.

[0049] It can be seen that all the above acupoints are distributed in the depression below the spinous process of the spine, and there is a certain linear relationship. The coordinate calculation formulas for other acupoints can be obtained based on the distribution pattern of vertebrae in the spine and the coordinates of the reference acupoints.

[0050]

[0051] In the formula, P represents the coordinates of a specific acupoint. According to the acupoint location standards, the acupoints on the Bladder Meridian are located on both sides of the body, symmetrical along the posterior midline, and can correspond to the acupoints on the Governing Vessel along the spine. Specifically, as follows... Figure 5 The relationship between them is shown.

[0052] For individual bladder meridian acupoints without corresponding direct points, linear interpolation can be used to estimate their location based on known bladder meridian coordinates. The bladder meridian has two branches perpendicular to the spine, located 1.5 cun and 3 cun lateral to the Governing Vessel (Du Mai), respectively. The most commonly used methods are finger cun and bone cun. Finger cun is primarily used to locate acupoints near the ends of bones; using this method to locate acupoints in the middle of bones will result in accumulated errors. Bone cun is more versatile than finger cun, and its measurement method varies in different areas of the body. Furthermore, bone cun offers higher reliability and accuracy than finger cun. According to relevant TCM texts, there are multiple methods for measuring bone cun in the back and waist region.

[0053] The distance from the spinous process of the seventh cervical vertebra to the lumbar vertebra can be converted to 17 bone measurements. According to acupoint standards, the area below the spinous process of the seventh cervical vertebra is the clinical landmark for the Dazhui acupoint, while the Xuanshu acupoint on the Du meridian is located in the depression below the spinous process of the first lumbar vertebra, representing the boundary between the lumbar and thoracic vertebrae. Therefore, the distance from Dazhui to Xuanshu can be used to approximately estimate the distance from the spinous process of the seventh cervical vertebra to the lumbar vertebra, thus deriving the approximate length of the bone measurements. Therefore, the coordinates of other acupoints on the Du meridian can be calculated based on the baseline acupoint coordinates, and the coordinates of acupoints on the Bladder meridian can also be calculated. A schematic diagram of the locations of all target acupoints on the back is shown below. Figure 6 As shown in the diagram, the names of acupoints are replaced with numbers, and the correspondence between the numbers and acupoints is as follows: Figure 5 As shown.

[0054] To evaluate the localization results of the trained model, a test set was input into the model, and the error was calculated based on the model output and calibration results. Our model's single-joint coordinate localization error (MPAPPE) is calculated. MPAPPE refers to the average Euclidean distance between the estimated coordinates of each joint and the true coordinates on the ground.

[0055]

[0056] Where N represents the size of the test set. px(i) is the coordinate of the i-th model output acupoint, and px(i) is the calibration coordinate of the i-th acupoint.

[0057] The location error for the Dazhui acupoint was 4.71 mm, while the location error for the Yaoyangguan acupoint was 16.43 mm. The number of decision trees in the random forest was set to 1000, with the rest set to default parameters. Results showed that GBDT had a smaller error compared to other regression algorithms, providing a more accurate mapping between back keypoints and acupoints. MAE refers to the average Euclidean distance between the estimated coordinates and the calibrated coordinates at each joint.

[0058]

[0059] Where N represents the size of the test set. It is a function that calculates the average positional error of each acupoint.

[0060] The coordinates of the reference acupoints on the Governing Vessel (Du Mai) can be obtained using the positioning model described in the previous section. As the acupoint positioning scheme shows, based on the reference acupoint coordinates and empirical formulas relating the vertebrae of the human spine, the coordinates of acupoints on the Governing Vessel, excluding Dazhui (GV14) and Yaoyangguan (GV3), can be calculated. After obtaining the coordinates of the acupoints on the Governing Vessel, the specific coordinates of the acupoints on the Bladder Meridian can be calculated based on the relative positional relationship between the acupoints on the Bladder Meridian and the Governing Vessel. Acupoints on the first lateral line are located 1.5 bone degrees lateral to the Governing Vessel in the direction perpendicular to the spine, while those on the second lateral line are located 3 bone degrees lateral to the Governing Vessel. For some acupoints on the Bladder Meridian that do not have a direct counterpart on the Governing Vessel, the anterior-posterior average interpolation method can be used to approximate their specific coordinates using the average coordinates of adjacent acupoints in the meridian sequence. The calculation method is as follows.

[0061] P mid =(P front +P back ) / 2.

[0062] In the formula, P mid Let P be the coordinates of the acupoint to be solved. front and P back These represent the coordinates of the anterior and posterior acupoints adjacent to the acupoint to be solved, respectively.

[0063] This machine vision-based method for locating back acupoints involves the following steps: First, acquiring 2D RGB image data containing information such as the human body's contour and posture, and 3D point cloud data containing depth information of the human back; extracting shoulder feature points from the 2D RGB image data; extracting hip feature points from the 3D point cloud data; fusing the extracted 2D shoulder and hip feature points; training the model by inputting the coordinates of all fused feature points into a GBDT algorithm for regression training to obtain a regression model for the baseline acupoint coordinates; obtaining standard baseline acupoint coordinates through the regression model, and establishing calculation formulas for other back acupoint coordinates based on the standard baseline acupoint coordinates to calculate the coordinates of other back acupoints; after model training is complete, manual calibration is no longer required, providing a reference for subsequent massage operations.

[0064] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for locating acupoints on the back based on machine vision, characterized in that: Includes the following steps, Step 1: Obtain 2D RGB image data containing the outline and posture information of the human body, and 3D point cloud data containing the depth information of the human back. Step 2: Extract shoulder feature points from 2D RGB image data; extract hip feature points from 3D point cloud data; Step 3: Fuse the extracted 2D shoulder feature points and hip feature points; Step 4: Perform model training. Input the coordinates of all fused feature points into the GBDT algorithm for regression training to obtain the regression model of the reference acupoint coordinates. Step 5: Obtain the standard reference acupoint coordinates through a regression model. Based on the standard reference acupoint coordinates, establish calculation formulas for other back acupoint coordinates, and calculate the coordinates of other back acupoints. The method for training the model is to integrate the coordinates of the fused buttock feature points and shoulder feature points into a vector as input, and then input it into the gradient boosting decision tree machine learning algorithm to establish a regression model of the reference acupoint coordinates.

2. The method for locating acupoints on the back based on machine vision according to claim 1, characterized in that, The method used in step 2 to extract shoulder feature points from two-dimensional RGB image data is to use the OpenPose human pose detection algorithm to extract features, and the three key points of the human skeleton, namely the left shoulder node, the neck and back node, and the right shoulder node, are used as two-dimensional feature points.

3. The method for locating acupoints on the back based on machine vision according to claim 1, characterized in that, The method for extracting buttock feature points from 3D point cloud data in step 2 is as follows: 3D point cloud data is captured by a 3D camera on the back of the human body. The intersection of the line connecting the highest points of the two buttocks and the posterior midline is used as the 3D feature point. The y-axis of the 3D camera is kept parallel to the long side of the massage bed during the capture. Therefore, the position of the posterior midline of the human body can be determined based on the number of points on both sides of the y-axis of the point cloud. The z-axis coordinates of the points on both sides are traversed and compared to obtain the coordinates of the highest points of the two buttocks. Finally, the 3D point cloud feature points are calculated.

4. The method for locating acupoints on the back based on machine vision according to claim 3, characterized in that, The method for fusing the extracted two-dimensional shoulder and hip feature points is to rotate and translate the rigid body to convert the 3D camera coordinate system into the RGB camera coordinate system, and then convert the three-dimensional coordinates into two-dimensional coordinates according to the perspective projection relationship, and then fuse the three-dimensional hip feature points and two-dimensional shoulder feature points into the two-dimensional coordinate system.

5. The method for locating acupoints on the back based on machine vision according to claim 1, characterized in that, The specific operation of step 5 is to input the newly collected feature point data into the model to obtain the reference acupoint coordinates, which are: , Based on the distribution pattern of vertebrae in the spinal region and the coordinates of the reference acupoints, the formula for calculating the coordinates of other acupoints is as follows: 。

Images