Robot vision recognition method and physiotherapy system based on visual detection of back acupoint key points and acupoint spatial coordinate positioning

Abstract

The present invention relates to a robot visual recognition method and physical therapy system based on visual detection of back acupuncture point key points and acupuncture point spatial coordinate positioning. First, back acupuncture point images of several patients and volunteers are collected, and the collected images are divided into training sets and verification sets after acupuncture point annotation. The HRNet neural network is modified to adapt to the back acupuncture point key point detection task. The modified HRNet neural network is trained to construct a neural network model for back acupuncture point key point detection. The network is deployed to a D435i depth camera on a robot to realize self-collection and adaptive back acupuncture point key point detection. 3D vision technology is used to convert the two-dimensional acupuncture point key points predicted by the HRNet neural network into three-dimensional coordinates in space. The predicted three-dimensional acupuncture point coordinates are used to perform posture estimation on the robot and motion planning simulation and physical deployment for simulating different physical therapy tasks, thereby realizing the robot's self-collection and self-identification of acupuncture point physical therapy tasks.

Description

Technical Field

[0001] The present invention relates to the fields of machine vision, deep learning and three-dimensional vision technology, and in particular to a robot vision recognition method and a physical therapy system based on visual detection of back acupuncture point key points and acupuncture point spatial coordinate positioning. Background Art

[0002] Acupuncture points on the back are commonly used in traditional Chinese medicine (TCM) acupuncture, massage, and tuina. They play a vital role in regulating Qi and blood, and improving the function of internal organs. By stimulating and massaging the correct acupuncture points, pain relief, physical fitness, and disease prevention can be achieved.

[0003] Currently, physical therapy and rehabilitation for acupuncture points on the back require physicians to first identify each individual's acupuncture points before proceeding with the relevant physical therapy and rehabilitation treatment. However, identifying and distinguishing key acupuncture points requires specialized knowledge, which not only takes a long time to identify acupuncture points but also requires a long time for the associated physical therapy. This not only requires a large number of professional physicians to carry out repetitive work, but also significantly increases the hospital's labor costs.

[0004] With the development of artificial intelligence algorithms, neural network-based visual algorithms can be used to construct targeted visual key point detection tasks. At the same time, neural network visual algorithms have good generalization capabilities to adapt to the detection of back acupoints on the human body of different genders, body shapes and postures. They can well replace doctors to identify back acupoints, and at the same time can improve the accuracy of acupoint recognition for non-professionals and reduce the difficulty of acupoint recognition.

[0005] With the continuous advancement of robotics technology, modern robots not only possess exceptional collaboration and flexibility, but also significantly improve production efficiency and automation. In particular, the integration of computer vision and 3D vision technologies has enabled robots to achieve more advanced applications in a variety of fields, replacing or even assisting humans in completing complex labor tasks. The integration of these technologies enables robots to more accurately perceive and understand their surroundings, enabling more complex and sophisticated operations.

[0006] Back acupoint rehabilitation is complex and time-consuming, limiting its application and popularity, confining it to specific locations like traditional Chinese medicine hospitals. Consequently, there's a lack of intelligent robots capable of performing back acupoint rehabilitation in place of physicians. Combining computer vision and 3D vision technology could allow robots to effectively replace physicians in some of these tasks. Summary of the Invention

[0007] This paper takes the key points of acupuncture points on the human back as the research object, and uses machine vision and 3D vision technology to provide a new method for detecting the three-dimensional coordinate points of human acupuncture points, thereby solving the problem of robot positioning and physical treatment of back acupuncture points. By combining the existing HRNet key point detection network and 3D vision technology, the HRNet network can accurately predict the coordinate points of the two-dimensional image of the acupuncture points on the human back. The depth camera then realizes the conversion of the two-dimensional coordinate points to three-dimensional coordinate points, and estimates the robot posture based on the three-dimensional coordinates, ultimately facilitating the six-degree-of-freedom robotic arm to implement motion planning and complete the corresponding physical treatment tasks.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A robot vision recognition method based on visual detection of back acupuncture point key points and acupuncture point spatial coordinate positioning includes the following steps:

[0010] Step 1: Build a data sample library, collect back target images of multiple patients and volunteers in different environments, different lighting conditions, and different postures, and annotate the back acupoint key points to build a back acupoint key point dataset;

[0011] Step 2: Classify the data set collected in step 1 into two categories: training set and validation set; and process the labeled data to facilitate input into the HRNet network for training and validation;

[0012] Step 3: Adjust the key point evaluation index OKS and the number of HRNet detection head predictions for the human back acupoint prediction, so that the HRNet key point detection network is suitable for the task of human back acupoint key point detection; and perform mosaic data augmentation, Gaussian noise data augmentation, and adaptive scaling on the dataset before inputting the data into the network;

[0013] Step 4: Deploy the trained HRNet network to the depth camera, enabling the robot to self-collect and adaptively detect acupuncture point key points on human back photos;

[0014] Step 5: Calibrate the depth camera to obtain the conversion relationship between the world coordinate system and the camera coordinate system. Then, perform hand-eye calibration of the depth camera and the robot in the form of eye-on-hand to solve the conversion relationship between the camera coordinate system and the robot end coordinate system.

[0015] Step 6: Convert the two-dimensional coordinates detected by the camera to predict the three-dimensional coordinates of the back acupuncture points;

[0016] Step 7: For different back acupoint therapy tasks, the robot posture is estimated to implement visual servo positioning motion planning for different therapy tasks.

[0017] The present invention also constructs a physical therapy system, including a robot and a physical therapy device. The two-dimensional coordinates of the acupuncture points on the back of the human body are predicted through a neural network, and the two-dimensional coordinates of the acupuncture points are converted into three-dimensional coordinates using a depth camera. The robot is used to simulate the motion process of controlling the physical therapy task, and the physical therapy device implements the physical therapy task.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] (1) The present invention adjusts the existing HRNet so that the network is suitable for the task of detecting key points of acupuncture points on the back of the human body and improves the model training stability and convergence speed.

[0020] (2) The present invention uses the generalization capability of HRNet to adapt to the back acupoint detection of human bodies of different genders, body shapes and postures.

[0021] (3) The present invention uses validation set data that did not participate in training to verify the generalization ability of the model.

[0022] (4) The present invention deploys the modified HRNet network to the D435i depth camera on the robot to realize self-collection and adaptive back acupuncture point key point detection.

[0023] (5) The present invention combines computer vision technology and 3D vision technology to not only predict the key points of the back acupuncture points, but also to determine the three-dimensional coordinate points of the back acupuncture points.

[0024] (6) The present invention uses the method of constructing a spatial plane to estimate the motion posture of the physiotherapy robot, so that the robot can be applied to back acupuncture point physiotherapy tasks, and the robot has the ability to generalize to more diverse and complex physiotherapy tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Flowchart of the method for building a back acupuncture therapy robot based on a vision algorithm.

[0026] Figure 2 This is the change diagram of the HRNet network training loss function after adjustment.

[0027] Figure 3 The mAP change diagram for the adjusted HRNet network training.

[0028] Figure 4 This is a map of the acupuncture points on the back of the human body that is used for verification.

[0029] Figure 5 It is the annotation map of the network output results on the input image.

[0030] Figure 6 Schematic diagram of a back acupoint therapy robot equipped with a depth camera.

[0031] The meanings of the reference numbers in the figure are: 1-robot base, 2-Realman RM-65 six-degree-of-freedom robotic arm, 3-Intel D435i depth camera, 4-camera connecting plate, 5-connecting male and female connectors, 6-physical therapy device.

[0032] Figure 7 This is a schematic diagram of the female connector for connecting the physical therapy device.

[0033] Figure 8 This is a schematic diagram of the male connector for connecting the physiotherapy device.

[0034] Figure 9 This is a schematic diagram of the male and female connectors for connecting the physiotherapy device.

[0035] Figure 10 Schematic diagram of the RM-65 robot hand-eye calibration.

[0036] Figure 11 This is a simulation diagram of the back acupoint therapy robot motion planning based on visual algorithm.

[0037] Figure 12 This is the workflow diagram of the back acupoint therapy robot based on visual algorithm. DETAILED DESCRIPTION

[0038] like Figure 1 As shown, the present invention provides a robot visual recognition method based on visual detection of back acupuncture point key points and acupuncture point spatial coordinate positioning, and the implementation steps include:

[0039] Step 1: Build a data sample library, collect back target images of multiple patients and volunteers in different environments, different lighting, and different postures, and mark the key points of the back acupuncture points (common physical therapy tasks) to build a back acupuncture point key point dataset.

[0040] Step 2: Classify the data set collected in step 1 into two categories: training set and validation set; and process the labeled data to facilitate input into the HRNet network for training and validation.

[0041] Step 3: Adjust the key point evaluation index OKS and the number of HRNet detection head predictions for the human back acupoint prediction, so that the HRNet key point detection network is suitable for the task of human back acupoint key point detection; and perform Mosaic data enhancement, Gaussian noise data enhancement and adaptive scaling on the dataset before inputting the data into the network.

[0042] Step 4: Deploy the trained HRNet network to the depth camera to enable the robot to self-collect and adaptively detect acupoint key points on human back photos.

[0043] Step 5: Calibrate the depth camera to obtain the conversion relationship between the world coordinate system and the camera coordinate system; then, complete the hand-eye calibration of the depth camera and the robot in the form of eye-on-hand to solve the conversion relationship between the camera coordinate system and the robot end coordinate system.

[0044] Step 6: Convert the two-dimensional coordinates detected by the camera to predict the three-dimensional coordinates of the back acupuncture points.

[0045] First, the two-dimensional image in the depth camera is input into the network to perform two-dimensional prediction of the acupuncture points on the back of the human body. According to the conversion relationship between the world coordinate system and the camera coordinate system, the two-dimensional coordinates of the two-dimensional image are converted into three-dimensional coordinates in the camera coordinate system. Then, according to the conversion relationship between the camera coordinate system and the robot end coordinate system, the three-dimensional coordinates in the camera coordinate system are converted into three-dimensional coordinates in the robot end coordinate system. Finally, according to the forward kinematics of the robot, the conversion relationship between the end coordinate system and the base coordinate system is used to convert the three-dimensional coordinates in the robot end coordinate system into the three-dimensional coordinates in the robot base coordinate system.

[0046] Step 7: For different back acupoint therapy tasks, the robot posture is estimated to implement visual servo positioning motion planning for different therapy tasks.

[0047] The present invention is further described below in conjunction with embodiments and drawings.

[0048] In step 1 of this example, a diverse dataset of human back acupoints was constructed using research from Anhui University of Traditional Chinese Medicine. Target images were collected from multiple patients and volunteers in various environments, lighting conditions, and postures. Common acupoints for physical therapy tasks were selected from the collected back images and annotated. The acupoints involved are: Dazhui, Jianjing, Tianzong, Jianshu, Jianzhen, Dazhui, Fengmen, Feishu, Jueyinshu, Xinshu, Geshu, Ganshu, Danshu, Pishu, Weishu, Sanjiaoshu, Shenshu, Dachangshu, and Gaohuang. Except for Dazhui, all other acupoints are symmetrical, for a total of 37 annotated points.

[0049] In step 2 of this example, the dataset collected in step 1 is classified into two categories: training set and validation set, with a ratio of 8:2. The labeled data is then processed to facilitate input into the HRNet network for training and validation.

[0050] In step 3 of this example, the initial HRNet neural network model is trained using the training set, and the model performance is verified using the validation set to obtain a trained and effective model. However, since HRNet is primarily used for human posture key point detection and human acupoint key point detection, it is necessary to derive HRNet internal parameters and adjust the HRNet learning strategy to make HRNet suitable for human back acupoint key point detection. The specific derivation and adjustment are as follows:

[0051] OKS is a commonly used evaluation metric for key point detection. However, its parameters are mainly applicable to the task of detecting key points on the human body. Therefore, it is necessary to adjust the value of σ in OKS to adapt to the task of detecting key points on the human back. The specific processing method is as follows:

[0052] Step 3.1: OKS is a commonly used evaluation indicator for key point detection. The OKS calculation formula is as follows:

[0053]

[0054] in, is the square of the distance between the true coordinates and the predicted coordinates, S represents the square root of the area of ​​the back in the image, but different key points have different difficulties in the prediction process of the neural network, so simply using distance and area as evaluation indicators is not enough, so σ is introduced to avoid the deviation of the prediction difficulty of different key points, δ is the Kronecker function, when manually annotated, each key point has three attributes, v = 0 means that the key point is invisible and the key point position cannot be inferred, v = 1 means that the key point is visible, v = 2 means that the key point is invisible, but the key point position can be inferred, and then the Kronecker function is defined as:

[0055]

[0056] Finally, it is normalized and the OKS value range is limited to 0-1, so that the distance, back area, and manual annotation offset are consistent with the increase and decrease of the OKS score and the above analysis.

[0057] Step 3.2: Back acupoint key point detection task. Different acupoints have different levels of difficulty in prediction. Therefore, 100 images were selected from the dataset in step 1 above. Students and professors of Traditional Chinese Medicine were asked to annotate the images with acupoints. The professors’ annotated data was used as the standard to calculate the value of σ, which represents the difficulty of predicting different acupoints. The calculation steps are as follows:

[0058] First compare the coordinates of each point:

[0059]

[0060] Among them, (x′ pi -x pi ) 2 +(y′ pi -y pi ) 2 The square of the distance between the professor and the student’s key points, w p h p are the width and height of the image respectively.

[0061] Recalculate The average value of:

[0062]

[0063] Where M is the number of pictures.

[0064] Final calculation The standard deviation of:

[0065]

[0066] The value shown in formula (5) is the key point normalization factor of the corresponding point.

[0067] Step 3.3: Replace the multi-step learning rate decay in HRNet with cosine annealing learning rate decay. Since the model parameters are very unstable in the early stage of new task training, a smaller learning rate should be selected at the beginning, rather than the large to small learning rate in the multi-step learning rate. The cosine annealing learning rate decay equation is constructed using the characteristics of the cosine function as follows:

[0068]

[0069] Among them, η t is the learning rate at the tth round, η min is the lower limit of the learning rate, η max is the initial learning rate, T cur is the current epoch number, T max is the preset maximum number of rounds.

[0070] At the same time, the image needs to be further enhanced to facilitate the deployment of the network to the D435i camera. The specific processing methods for Mosaic data enhancement and Gaussian noise data enhancement are as follows:

[0071] Step 3.4: Perform Mosaic data enhancement. Randomly select four photos from the data in step 1, perform random scaling, random cropping, and random arrangement, and then perform border stitching to stitch the four photos into a complete image. Taking the stitched photo in the upper left corner as an example, select the center point (x c ,y c ) Align the lower right corner of the upper left photo with the center of the entire image for stitching. There are two cases: one is that the stitched image does not exceed the size of the entire image. The coordinates of the upper left corner of the stitched image in the entire image are (x c -w,y c -h), the overall coordinates of the spliced ​​image are (x c -w,y c -h,x c ,yc The other is that the stitched image exceeds the size of the whole image. The coordinates of the upper left corner of the stitched image in the whole image are (0,0). The upper left corner image needs to be cropped. The cropping coordinates are (wx c ,hy c ,w,h), the overall coordinates of the spliced ​​image are (0,0,x c ,y c ), the processing methods for other directions are similar to the upper left picture, and the key points are translated accordingly.

[0072] Step 3.5: Perform Gaussian noise data augmentation. Since the camera used to capture the human back data is different from the D435i depth camera deployed on the robot, Gaussian noise is added during training to ensure deployment results. This simulates random perturbations in devices such as image sensors, thereby improving training results. Gaussian noise is based on a normal distribution, and the noise value N is defined by the following formula:

[0073] N~N(μ,σ 2 )(7)

[0074] Among them, μ is the mean representing the center point of the noise, σ is the standard deviation representing the discreteness of the noise value, and these noises are added to the random pixels of the image.

[0075] Step 3.6: Perform adaptive scaling. In step 3.4 and when the network image is input, the image will be scaled. The image is scaled by filling the image with the least gray edge to avoid image distortion during the scaling process.

[0076] Step 3.7: Modify the HRNet detection head. The original number of key points used by HRNet for human posture estimation is 17, while the number of our acupoint key points is 37. Therefore, it is necessary to modify the heatmaps predicted by HRNet from 17 to 37. The loss is calculated for each of the 37 heatmaps and the corresponding acupoint labels. Then, the model is optimized through back propagation to train a network suitable for back acupoint keypoint detection.

[0077] After adjusting HRNet and enhancing the image, the loss value changes and mAP changes obtained after network training are as follows: Figure 2 、 3 As shown in the figure, it shows that the network can achieve relatively fast convergence in the back acupoint key point detection task training, and the mAP tends to be stable after 200 rounds of training, indicating that the network tends to be stable during the training process and can achieve good results.

[0078] Select an image data from the validation set such as Figure 4As shown, it is input into the modified HRNet network for prediction, and the prediction results are plotted on the original image as shown Figure 5 As shown, Figure 4 and Figure 5 By comparison, the predicted acupoints have smaller errors than the labeled data, indicating that the trained modified HRNet network achieves better results in detecting the key points of 37 back acupoints.

[0079] In step 4 of this example, a platform was built using the Realman RM-65 six-degree-of-freedom robotic arm to conduct experiments, detect key acupuncture points on the human body, locate acupuncture points in three dimensions, and complete path planning for robotic physiotherapy tasks. Specifically:

[0080] Build as Figure 6 The experimental platform shown in the figure mainly includes: robot base 1, Reelman RM-65 six-degree-of-freedom robot arm 2, Intel D435i depth camera 3, camera connection board 4, physical therapy device 6 and its connecting male and female connectors 5. Among them, the Reelman RM-65 six-degree-of-freedom robot arm is mainly used to simulate the motion process of physical therapy tasks, the Intel D435i depth camera obtains two-dimensional images and depth images, and realizes the conversion of two-dimensional coordinates of acupuncture points into three-dimensional coordinates. The connecting male and female connectors of the physical therapy device are mainly used to quickly replace different physical therapy devices, such as Figure 7-9 The figure shows an example of connecting the male and female connectors of a physical therapy device.

[0081] In step 5 of this example, the robot vision system needs to be calibrated to obtain the transformation relationship between the world coordinate system and the camera coordinate system, as well as the transformation relationship between the robot end coordinate system and the camera coordinate system. Vision system calibration includes calibration of the RGB camera in the D435i depth camera and hand-eye calibration in the eye-on-hand mode. The quality of the robot vision system calibration results will directly affect the robot physical therapy visual servoing task. The specific calibration steps are as follows:

[0082] Step 5.1: First determine the transformation relationship between the world coordinate system and the camera coordinate system.

[0083] The relationship between the pixel coordinate system and the image coordinate system is as follows:

[0084]

[0085] Among them, u and v are pixel coordinates, and x and y are image coordinates.

[0086] The relationship between the image coordinate system and the camera coordinate system is as follows:

[0087]

[0088] P(x p ,y p ,f) is a point in the image coordinate system, M(xM ,y M ,z M ) is a point in the camera coordinate system, where f represents the focal length. The relationship between the camera coordinate system and the world coordinate system is as follows:

[0089]

[0090] Substituting the above formula into the equation, we can get:

[0091]

[0092] Among them, Z M is the value of the z-axis in the camera coordinate system, Represents the camera internal parameter, f in the internal parameter matrix x f y u0 v0 are the intrinsic parameters of the camera; Represents the camera extrinsic parameters, which only change when the camera pose changes.

[0093] Step 5.2: Determine the transformation relationship between the camera coordinate system and the RM-65 end-user coordinate system. According to the eye-in-hand calibration method, the transformation matrix from the D435i camera coordinate system to the RM-65 robot end-user coordinate system can be obtained. Figure 10 Here, base represents the base coordinate system of the RM-65 robot; end represents the coordinate system of the robot end; cam represents the coordinate system of the D435i camera fixed on the flange of the RM-65 end; and board represents the coordinate system of the calibration board.

[0094] First, assume that there is a point P0 in the board coordinate system. Using the extrinsic parameter matrix T1 of the calibrated D435i RGB camera, transform P0 to P1 in the cam coordinate system. Then, using the desired hand-eye calibration matrix (transformation matrix X), transform P1 to P2 in the end coordinate system. Based on the known parameters of the RM-65 robot, transform P2 to P3 in the base coordinate system. Establish the following relationship:

[0095] T3XT1P0=P3(12)

[0096] Then move the robotic arm. For the same point, the coordinate values ​​of P0 and P3 remain unchanged, and only the coordinate values ​​of P1 and P2 change. The above relationship is as follows:

[0097] T′3XT′1P0=P3 (13)

[0098] Using the calibration board, the camera's external parameter matrix can be solved, and the robot's transformation matrix can be obtained through the robot's own parameters, that is, T3 ′-1 T3X=XT′1T1 -1 Where T′3-1 T3 and T′1T1 -1 All can be solved, and the formula can be expanded to obtain:

[0099]

[0100] Transform the problem into a solution and Solving the problem can obtain the rotation matrix between the camera coordinate system and the RM-65 terminal coordinate system. and translation vectors

[0101] In step 6 of this example, after detecting the key acupuncture points on the back of the human body, the 2D pixel coordinates (u, v) of the acupuncture points are obtained, and the depth z of the pixel coordinates is obtained using the D435i depth sensor. Combining the hand-eye conversion matrix obtained above and the robot's own joint conversion matrix, the 2D pixel coordinates of the acupuncture points are converted to 3D spatial coordinates (x, y, z) in the robot's base coordinate system. The specific calculation method is as follows:

[0102] Step 6.1: Convert the 2D pixel coordinates (u, v) to the x coordinate system of the camera M y M z M , the formula is as follows:

[0103]

[0104] Step 6.2: The x coordinate of the known acupuncture point in the camera coordinate system M y M z M Finally, the coordinate points in the camera coordinate system are transformed to the coordinate points in the robot base coordinate system through the robot's own joint transformation matrix and the hand-eye transformation matrix obtained above. The formula is as follows:

[0105]

[0106] Among them, T b is the robot joint transformation matrix, T c is the hand-eye transformation matrix.

[0107] Therefore, the two-dimensional coordinates of the 37 acupoints predicted by HRNet can be converted into three-dimensional coordinates in the robot base coordinate system.

[0108] In step 7 of this example, different physical therapy tasks may require the robot to perform posture estimation and motion planning for simulating different physical therapy tasks. The specific posture estimation method and motion planning are as follows:

[0109] Step 7.1: Perform pose estimation for each acupoint therapy task. The three-dimensional coordinates of the 37 acupoints in the robot base coordinate system have been obtained from step 6. To ensure the stability and reliability of the pose solution before and after the prediction, the minimum pixel distance between adjacent 500ms images for acupoint prediction is calculated when the robot arm is stationary. If the distance exceeds the minimum distance, re-prediction is required. The formula is as follows:

[0110] (a[0]-b[0]) 2 +(a[1]-b[1]) 2 (17)

[0111] Further stabilize the positions of the acupoints to prevent the misalignment of the predicted acupoints before and after, which will lead to excessive errors in the posture solution.

[0112] Step 7.2: Taking the estimation of the Dazhui acupoint posture as an example, select the acupoint closest to the Dazhui acupoint and the next treatment task acupoint, compare the distances between the adjacent acupoints, and select the acupoint with the smallest distance and the next treatment task acupoint, which should generally be the left Dazhui and the right Dazhui. Given the Dazhui coordinate point P (x0, y0, z0), the left Dazhui coordinate point Q (x1, y1, z1), and the right Dazhui coordinate point R (x2, y2, z2), the plane equation of the estimated plane calculated by these three points is:

[0113]

[0114] To facilitate the robot's motion planning for different physiotherapy task simulations, the unit vector of the line connecting the next physiotherapy task acupoint and the posture estimation acupoint is selected as the x-axis direction in the plane:

[0115]

[0116] Therefore, the rotation matrix of the end effector in the base coordinate system is for:

[0117]

[0118] Different physical therapy tasks may require some offset in the direction perpendicular to the back. For example, moxibustion therapy does not require the actuator to contact the skin. Assuming the offset is d (that is, the distance between the end effector and the plane), the translation vector of the end effector is:

[0119]

[0120] Therefore, the pose matrix of the end effector is obtained

[0121]

[0122] Step 7.3: Motion planning for different physiotherapy task simulations. Here, we take moxibustion therapy as an example, where the robot simulates the physician performing moxibustion therapy and performs trajectory planning. The end effector posture obtained in step 7.2 is combined with a fifth-order polynomial for motion planning to ensure the stability and efficiency of the physiotherapy task. The fifth-order polynomial is in the form of:

[0123] θ(t)=A0+A1t+A2t 2 +A3t 3 +A4t 4 +A5t 5 (twenty three)

[0124] The boundary conditions are:

[0125]

[0126] Among them, t0 is the starting time, t f The robot will go from the ready state (prepared for moxibustion) to the specific moxibustion technique trajectory through trajectory planning. Different moxibustion techniques require different operating characteristics, movement speeds, accelerations, and path curves, so different speeds, accelerations, and path curves need to be set for different techniques.

[0127] Step 7.4: If Figure 11 As shown, the three-dimensional coordinates of the acupoints, robot posture estimation and manipulation simulation obtained in the above steps are imported into the simulation program for simulation to determine whether the manipulation simulation for different therapy tasks is reasonable. After the simulation is correct, the position, velocity and acceleration obtained by the motion planning are deployed to the robot to realize the motion planning for manipulation simulation for different therapy tasks and realize the visual servo positioning motion planning of the robot for different therapy tasks.

[0128] In summary, the basic workflow of the back acupuncture point visual detection and spatial coordinate positioning based on robot positioning is as follows: Figure 12 As shown, the depth camera captures the acupoint images on the human back, and the images are input into the HRNet neural network for prediction. The prediction generates 37 acupoint prediction heat maps corresponding to the 37 acupoints that need to be predicted. The predicted acupoints are transformed into the camera coordinate system and the robot coordinate system to realize the prediction of the three-dimensional coordinate points of the acupoints. Finally, the posture estimation of the acupoints is performed to estimate the physical therapy posture required for each acupoint and perform motion planning for the corresponding physical therapy tasks. The machine vision and three-dimensional vision technology are used to realize the robot's self-identification of acupoint physical therapy tasks, and realize the construction of the physical therapy system.

[0129] The above content is merely an example and explanation of the concept of the present invention. Those skilled in the art may make various modifications or additions to the described specific embodiments or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should all fall within the scope of protection of the present invention.

Claims

1. A robot vision recognition method based on back acupuncture point key point visual detection and acupuncture point spatial coordinate positioning, characterized in that: The steps include: Step 1: Build a data sample library, collect back target images of multiple patients and volunteers in different environments, different lighting conditions, and different postures, and annotate the back acupoint key points to build a back acupoint key point dataset; Step 2: Classify the data set collected in step 1 into two categories: training set and validation set; and process the labeled data to facilitate input into the HRNet network for training and validation; Step 3: Adjust the key point evaluation index OKS and the number of HRNet detection head predictions for the human back acupoint prediction, so that the HRNet key point detection network is suitable for the task of human back acupoint key point detection; and perform mosaic data augmentation, Gaussian noise data augmentation, and adaptive scaling on the dataset before inputting the data into the network; Step 4: Deploy the trained HRNet network to the depth camera, enabling the robot to self-collect and adaptively detect acupuncture point key points on human back photos; Step 5: Calibrate the depth camera to obtain the conversion relationship between the world coordinate system and the camera coordinate system. Then, perform hand-eye calibration of the depth camera and the robot in the form of eye-on-hand to solve the conversion relationship between the camera coordinate system and the robot end coordinate system. Step 6: Convert the two-dimensional coordinates detected by the camera to predict the three-dimensional coordinates of the back acupuncture points; Step 7: For different back acupoint therapy tasks, the robot posture is estimated to implement visual servo positioning motion planning for different therapy tasks.

2. The robot visual recognition method according to claim 1, wherein: In step 3, the specific steps for adjusting the key point evaluation index OKS for the prediction of acupuncture points on the back of the human body are as follows: OKS is a commonly used evaluation metric for key point detection. However, its parameters are mainly applicable to the task of detecting key points on the human body. Therefore, it is necessary to adjust the value of σ in OKS to adapt to the task of detecting key points on the human back. The specific processing method is as follows: Step 3.1: OKS is a commonly used evaluation indicator for key point detection. The OKS calculation formula is as follows: in, is the square of the distance between the true coordinates and the predicted coordinates, S represents the square root of the area of ​​the back in the image, but different key points have different difficulties in the prediction process of the neural network, so simply using distance and area as evaluation indicators is not enough, so σ is introduced to avoid the deviation of the prediction difficulty of different key points, δ is the Kronecker function, when manually annotated, each key point has three attributes, v = 0 means that the key point is invisible and the key point position cannot be inferred, v = 1 means that the key point is visible, v = 2 means that the key point is invisible, but the key point position can be inferred, and then the Kronecker function is defined as: Finally, it was normalized and the OKS value range was limited to 0-1, so that the distance, back area, and manual annotation offset were consistent with the increase and decrease of the OKS score and the above analysis; Step 3.2: Back acupoint key point detection task. Different acupoints have different levels of difficulty in prediction. Therefore, several images are selected from the dataset in step 1 above. Students and professors of Traditional Chinese Medicine are asked to annotate the images with acupoints. The professors’ annotated data are used as the standard to calculate the value of σ, which represents the difficulty of predicting different acupoints. The calculation steps are as follows: First compare the coordinates of each point: Among them, (x′ pi -x pi ) 2 +(y′ pi -y pi ) 2 The square of the distance between the professor and the student’s key points, w p h p are the width and height of the image respectively; Recalculate The average value of: Where M is the number of pictures; Final calculation The standard deviation of: The value shown in formula (5) is the key point normalization factor of the corresponding point obtained; Step 3.3: Replace the multi-step learning rate decay in HRNet with cosine annealing learning rate decay. Since the model parameters are very unstable in the early stage of new task training, a smaller learning rate should be selected at the beginning, rather than the large to small learning rate in the multi-step learning rate. The cosine annealing learning rate decay equation is constructed using the characteristics of the cosine function as follows: Among them, η t is the learning rate at the tth round, η min is the lower limit of the learning rate, η max is the initial learning rate, T cur is the current epoch number, T max is the preset maximum number of rounds; Step 3.4: Perform Mosaic data enhancement. Randomly select four photos from the data in step 1, perform random scaling, random cropping, and random arrangement, and then perform border stitching to stitch the four photos into a whole picture. Take the stitched photo in the upper left corner as an example, select the center point (x c ,y c ) Align the lower right corner of the upper left photo with the center of the entire image for stitching. There are two cases: one is that the stitched image does not exceed the size of the entire image. The coordinates of the upper left corner of the stitched image in the entire image are (x c -w,y c -h), the overall coordinates of the spliced ​​image are (x c -w,y c -h,x c ,y c ); The other is that the stitched picture exceeds the size of the whole image. The coordinates of the upper left corner of the stitched picture in the whole picture are (0, 0). The upper left corner picture needs to be cropped. The cropping coordinates are (wx c ,hy c , w, h), the overall coordinates of the spliced ​​image are (0, 0, x c ,y c ), the processing methods for the other directions are similar to the upper left picture, and the key points are translated accordingly; Step 3.5: Perform Gaussian noise data enhancement. Since the camera used to capture the human back data is different from the depth camera model deployed on the robot, Gaussian noise is added during training to ensure the deployment effect. This simulates the random perturbations in the image sensor device and improves the training effect. Gaussian noise is based on normal distribution. The noise value N is defined by the following formula: N~N(μ,σ 2 ) (7) Among them, μ is the mean value representing the center point of the noise, σ is the standard deviation representing the discrete degree of the noise value, and these noises are added to the random pixels of the image; Step 3.6: Perform adaptive scaling. In step 3.4 and when the network image is input, the image will be scaled. The image is scaled by filling the image with the least gray edges to avoid image distortion during the scaling process. Step 3.7: Modify the HRNet detection head. The original number of key points used by HRNet for human posture estimation is 17, while the number of our acupoint key points is 37. Therefore, it is necessary to modify the heatmaps predicted by HRNet from 17 to 37. The loss is calculated for each of the 37 heatmaps and the corresponding acupoint labels. Then, the model is optimized through back propagation to train a network suitable for back acupoint keypoint detection.

3. The robot visual recognition method according to claim 2, wherein: Step 5: To obtain the transformation relationship between the world coordinate system and the camera coordinate system, and the transformation relationship between the robot end coordinate system and the camera coordinate system, the robot vision system needs to be calibrated. The vision system calibration includes the calibration of the RGB camera in the depth camera and the hand-eye calibration of the eye-on-hand form. The quality of the robot vision system calibration results will directly affect the visual servoing task of robotic physical therapy. The specific calibration steps are as follows: Step 5.1: First determine the transformation relationship between the world coordinate system and the camera coordinate system; The relationship between the pixel coordinate system and the image coordinate system is as follows: Among them, u, v are pixel coordinates, x, y are image coordinates; The relationship between the image coordinate system and the camera coordinate system is as follows: P(x p ,y p ,f) is a point in the image coordinate system, M(x M ,y M ,z M ) is a point in the camera coordinate system, where f represents the focal length; the relationship between the camera coordinate system and the world coordinate system is as follows: Substituting the above formula into the equation, we can get: Among them, Z M is the value of the z-axis in the camera coordinate system, Represents the camera internal parameter, f in the internal parameter matrix x f y u0 v0 are the intrinsic parameters of the camera; Represents the camera extrinsic parameters, which only change when the camera's pose changes; Step 5.2: Determine the transformation relationship between the camera coordinate system and the robot end coordinate system; the transformation matrix from the depth camera coordinate system to the robot end coordinate system can be obtained based on the eye-in-hand calibration method; First, assume that there is a point P0 in the board coordinate system. Use the extrinsic parameter matrix T1 of the RGB camera in the calibrated depth camera to transform P0 to P1 in the cam coordinate system. Then, combined with the hand-eye calibration matrix to be determined, P1 is further transformed to P2 in the end coordinate system. Based on the known robot parameters, P2 is transformed to P3 in the base coordinate system. The following relationship is established: T3XT1P0=P3(12) Then move the robotic arm. For the same point, the coordinate values ​​of P0 and P3 remain unchanged, and only the coordinate values ​​of P1 and P2 change. The above relationship is as follows: T3XT1′P0=P3(13) Using the calibration board, the camera's external parameter matrix can be solved, and the robot's transformation matrix can be obtained through the robot's own parameters, that is, In the formula and All can be solved, and the formula can be expanded to obtain: Transform the problem into a solution and Solving this problem can obtain the rotation matrix between the camera coordinate system and the robot end coordinate system. and translation vectors 4. The robot visual recognition method according to claim 3, wherein: Step 6 The specific steps are: After detecting the key acupuncture points on the back of the human body, the 2D pixel coordinates (u, v) of the acupuncture points are obtained, and the depth z of the pixel coordinates is obtained using the depth sensor in the depth camera. Combining the obtained hand-eye conversion matrix and the robot's own joint conversion matrix, the 2D pixel coordinates of the acupuncture points are converted to 3D spatial coordinates (x, y, z) in the robot's base coordinate system. The specific calculation method is as follows: Step 6.1: Convert the 2D pixel coordinates (u, v) to the x coordinate system of the camera M y M z M , the formula is as follows: Step 6.2: The x coordinate of the known acupuncture point in the camera coordinate system M y M z M Finally, the coordinate points in the camera coordinate system are transformed to the coordinate points in the robot base coordinate system through the robot's own joint transformation matrix and the hand-eye transformation matrix obtained above. The formula is as follows: Among them, T b is the robot joint transformation matrix, T c is the hand-eye transformation matrix; Therefore, the two-dimensional coordinates of the 37 acupoints predicted by HRNet can be converted into three-dimensional coordinates in the robot base coordinate system.

5. The robot visual recognition method according to claim 4, wherein: Step 7 The specific steps are: Step 7.1: Perform posture estimation for each acupoint therapy task. The three-dimensional coordinates of the therapy acupoints in the robot base coordinate system have been obtained from step 6. To ensure the stability and reliability of the posture solution before and after the prediction, calculate the minimum pixel distance between adjacent 500ms images for acupoint prediction when the robot arm is stationary. If the distance exceeds the minimum distance, re-prediction is required. The formula is as follows: (a[0]-b[0]) 2 +(a[1]-b[1]) 2 (17) Further stabilize the positions between the acupoints to prevent the misalignment of the predicted acupoints before and after, which will lead to excessive errors in the posture solution. Step 7.2: Select the acupoint closest to the task acupoint and the next treatment task acupoint, compare the distances between adjacent acupoints, select the acupoint with the smallest distance and the next treatment task acupoint, and construct the spatial plane equation as follows: d=-(ax0+by0+cz0) To facilitate the robot's motion planning for different physiotherapy task simulations, the unit vector of the line connecting the next physiotherapy task acupoint and the posture estimation acupoint is selected as the x-axis direction in the plane: Therefore, the rotation matrix of the end effector in the base coordinate system is for: For different physical therapy tasks, some offset may be required in the direction perpendicular to the back. For example, for moxibustion therapy, the actuator does not need to contact the skin. Assuming the offset is d, the translation vector of the end effector is: Therefore, the pose matrix of the end effector is obtained Step 7.3: Motion planning for different physical therapy task simulations; the end effector's end posture obtained in step 7.2 is combined with a fifth-order polynomial for motion planning to ensure the stability and efficiency of the physical therapy task. The fifth-order polynomial is in the form of: θ(t)=A0+A1t+A2t 2 +A3t 3 +A4t 4 +A5t 5 (23) The boundary conditions are: Among them, t0 is the starting time, t f The robot will go from the preparation state to the specific therapy technique through trajectory planning. Different therapy techniques require different operation characteristics, movement speeds, accelerations, and path curves. Therefore, different speeds, accelerations, and path curves need to be set for different techniques. Step 7.4: Import the three-dimensional coordinates of the acupoints, robot posture estimation and manipulation simulation obtained in the above steps into the simulation program for simulation to determine whether the manipulation simulation for different physical therapy tasks is reasonable. After the simulation is correct, deploy the position, velocity and acceleration obtained by the motion planning to the robot to realize the motion planning for manipulation simulation for different physical therapy tasks and realize the visual servo positioning motion planning of the robot for different physical therapy tasks.

6. A physical therapy system implementing the robot visual recognition method according to any one of claims 1 to 5, characterized in that: It includes a robot and a mounted physiotherapy device. It uses a neural network to predict the two-dimensional coordinates of the acupuncture points on the human back, and uses a depth camera to convert the two-dimensional coordinates of the acupuncture points into three-dimensional coordinates. The robot is used to simulate the motion process of controlling the physiotherapy task, and the physiotherapy device implements the physiotherapy task.

Images