Robot visual identification method and physiotherapy system based on back acupuncture point key point visual detection and acupuncture point space coordinate positioning

Abstract

The invention relates to a robot visual identification method and physiotherapy system based on back acupoint key point visual detection and acupoint space coordinate localization, and the method comprises the steps: firstly collecting back acupoint images of a plurality of patients and volunteers, carrying out the acupoint labeling of the collected images, and dividing the images into a training set and a verification set; correcting the HRNet neural network to adapt to a back acupuncture point key point detection task; training the corrected HRNet neural network, and constructing a back acupuncture point key point detection neural network model; deploying the network to a D435i depth camera on the robot to realize self-acquisition and self-adaptive back acupuncture point key point detection; converting the two-dimensional acupoint key points predicted by the HRNet neural network into three-dimensional coordinates in a space by using a 3D visual technology; the predicted acupoint three-dimensional coordinates are used for carrying out posture estimation on the robot and motion planning simulation and real object deployment simulated by different physiotherapy task techniques, and self-collection and self-recognition acupoint physiotherapy tasks of the robot are achieved.

Description

Technical Field

[0001] The present invention relates to the fields of machine vision, deep learning and three-dimensional vision technology, and in particular to a robot vision recognition method and a physical therapy system based on visual detection of back acupuncture point key points and acupuncture point spatial coordinate positioning. Background Art

[0002] The acupuncture points on the back of the human body are often used in traditional Chinese medicine acupuncture, massage and massage. They play an important role in regulating the body's qi and blood and improving the function of internal organs. By stimulating and massaging the correct acupuncture points, pain can be relieved, physical fitness can be strengthened, and diseases can be prevented.

[0003] At present, the physical therapy and rehabilitation tasks for the acupuncture points on the back of the human body require doctors to first determine the acupuncture points for each person, and then carry out relevant physical therapy and rehabilitation treatment. However, the judgment and identification of key points of acupuncture points requires certain professional knowledge. Not only does the acupuncture point judgment take a long time, but the related physical therapy tasks also require a long working time. Not only does it require a large number of professional doctors to perform repetitive work, but it also greatly increases the hospital's employment costs.

[0004] With the development of artificial intelligence algorithms, visual algorithms based on neural networks can be used to construct targeted visual key point detection tasks. At the same time, the visual algorithms based on neural networks have good generalization capabilities to adapt to the detection of back acupoints of people with different genders, body shapes and postures. They can well replace doctors to identify back acupoints, and at the same time can improve the accuracy of acupoint recognition for non-professionals and reduce the difficulty of acupoint recognition.

[0005] And with the continuous advancement of robotics technology, modern robots not only have excellent collaboration and flexibility, but also can significantly improve production efficiency and automation levels. In particular, combined with computer vision technology and 3D vision technology, robots can achieve more advanced applications in multiple fields, replacing or even assisting humans in completing complex labor tasks. The integration of these technologies enables robots to perceive and understand the surrounding environment more accurately, thereby performing more complex and delicate operations.

[0006] Back acupoint physiotherapy is complex and time-consuming, which limits its application and popularity, and makes rehabilitation physiotherapy related treatment tasks can only be carried out in specific places such as traditional Chinese medicine hospitals. Therefore, there is a lack of an intelligent robot that can replace doctors for back acupoint physiotherapy in the market. Combining computer vision technology and 3D vision technology can enable robots to better replace doctors to complete part of the physiotherapy work. Summary of the invention

[0007] The present invention takes the key points of acupuncture points on the back of the human body as the research object, and uses machine vision and three-dimensional vision technology to provide a new method for detecting the three-dimensional coordinate points of acupuncture points on the human body, so as to solve the problem of robot positioning and physical therapy of acupuncture points on the back. The existing HRNet key point detection network and 3D vision technology are combined to realize the HRNet network to accurately predict the coordinate points of the two-dimensional image of the acupuncture points on the back of the human body, and then the depth camera realizes the conversion of the two-dimensional coordinate points to the three-dimensional coordinate points, and estimates the robot posture according to the three-dimensional coordinates, which finally facilitates the six-degree-of-freedom robot arm to realize motion planning and complete the corresponding physical therapy tasks.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A robot vision recognition method based on back acupuncture point key point visual detection and acupuncture point spatial coordinate positioning includes the following steps:

[0010] Step 1: Build a data sample library, collect back target images of multiple patients and volunteers in different environments, different lighting and different postures, and mark the key points of the back acupoints to build a data set of key points of the back acupoints;

[0011] Step 2: Classify the data set collected in step 1 into two categories: training set and validation set; and process the labeled data to facilitate input into the HRNet network for training and validation;

[0012] Step 3: The key point evaluation index OKS and the number of HRNet detection head predictions are adjusted for the prediction of acupuncture points on the back of the human body, so that the HRNet key point detection network is suitable for the key point detection task of acupuncture points on the back of the human body; and the data set is subjected to Mosaic data enhancement, Gaussian noise data enhancement and adaptive scaling before the data is input into the network;

[0013] Step 4: Deploy the trained HRNet network to the depth camera to enable the robot to self-collect and adaptively detect key acupuncture points on human back photos;

[0014] Step 5: Calibrate the depth camera to obtain the conversion relationship between the world coordinate system and the camera coordinate system; then, complete the hand-eye calibration of the depth camera and the robot in the form of eyes on hands to solve the conversion relationship between the camera coordinate system and the robot end coordinate system;

[0015] Step 6: Transform the two-dimensional coordinates detected by the camera to predict the three-dimensional coordinates of the back acupuncture points;

[0016] Step 7: For different back acupoint therapy tasks, the robot posture is estimated to realize the visual servo positioning motion planning of the robot for different therapy tasks.

[0017] The present invention also constructs a physical therapy system, including a robot and a physical therapy device. The two-dimensional coordinates of the acupuncture points on the back of the human body are predicted by a neural network, and the two-dimensional coordinates of the acupuncture points are converted into three-dimensional coordinates using a depth camera. The robot is used to simulate the motion process of controlling the physical therapy task, and the physical therapy device implements the physical therapy task.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0019] (1) The present invention adjusts the existing HRNet so that the network is suitable for the task of detecting key points of acupuncture points on the back of the human body and improves the stability and convergence speed of model training.

[0020] (2) The present invention uses the generalization ability of HRNet to adapt to the back acupoint detection of the human body of different genders, body shapes and postures.

[0021] (3) The present invention uses validation set data that did not participate in training to verify the generalization ability of the model.

[0022] (4) The present invention deploys the modified HRNet network to the D435i depth camera on the robot to achieve self-collection and adaptive back acupuncture point key point detection.

[0023] (5) The present invention combines computer vision technology and 3D vision technology to not only predict the key points of the back acupuncture points, but also determine the three-dimensional coordinate points of the back acupuncture points.

[0024] (6) The present invention uses the method of constructing a spatial plane to estimate the motion posture of the physiotherapy robot, so that the robot can be applied to back acupuncture point physiotherapy tasks, and the robot has the ability to generalize to more diverse and complex physiotherapy tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Flowchart of the method for building a back acupuncture therapy robot based on a vision algorithm.

[0026] Figure 2 This is the change diagram of the HRNet network training loss function after adjustment.

[0027] Figure 3 This is the mAP change diagram for the adjusted HRNet network training.

[0028] Figure 4 This is a map of the annotations of the acupuncture points on the back of the human body in the verification set.

[0029] Figure 5 It is the annotation map of the network output result on the input image.

[0030] Figure 6 Schematic diagram of a back acupoint therapy robot with a depth camera.

[0031] The meanings of the reference numerals in the figure are: 1- robot base, 2- Reilman RM-65 six-degree-of-freedom robotic arm, 3- Intel D435i depth camera, 4- camera connecting plate, 5- connecting male and female heads, 6- physical therapy device.

[0032] Figure 7 This is a schematic diagram of the female connector for connecting the physical therapy device.

[0033] Figure 8 This is a schematic diagram of the male connector for connecting the physical therapy device.

[0034] Fig. 9 This is a schematic diagram of the male and female connectors for connecting the physiotherapy device.

[0035] Fig.10 Schematic diagram of hand-eye calibration for the RM-65 robot.

[0036] Fig.11 This is a simulation diagram of the motion planning of the back acupoint therapy robot based on visual algorithm.

[0037] Fig.12 This is the workflow diagram of the back acupoint therapy robot based on visual algorithm. DETAILED DESCRIPTION

[0038] like Figure 1 As shown, the present invention provides a robot visual recognition method based on back acupuncture point key point visual detection and acupuncture point spatial coordinate positioning, and the implementation steps include:

[0039] Step 1: Build a data sample library, collect back target images of multiple patients and volunteers in different environments, different lighting and different postures, and mark the key points of the back acupuncture points (common physical therapy tasks) to build a data set of key points of the back acupuncture points.

[0040] Step 2: Classify the data set collected in step 1 into two categories: training set and validation set; and process the labeled data to facilitate input into the HRNet network for training and validation.

[0041] Step 3: Adjust the key point evaluation index OKS and the number of HRNet detection head predictions for the prediction of acupoints on the back of the human body, so that the HRNet key point detection network is suitable for the key point detection task of acupoints on the back of the human body; and perform Mosaic data enhancement, Gaussian noise data enhancement and adaptive scaling on the data set before inputting the data into the network.

[0042] Step 4: Deploy the trained HRNet network to the depth camera to enable the robot to self-collect and adaptively detect acupuncture key points on human back photos.

[0043] Step 5: Calibrate the depth camera to obtain the conversion relationship between the world coordinate system and the camera coordinate system; then, complete the hand-eye calibration of the depth camera and the robot in the form of eyes on the hand, and solve the conversion relationship between the camera coordinate system and the robot end coordinate system.

[0044] Step 6: Transform the two-dimensional coordinates detected by the camera to predict the three-dimensional coordinates of the back acupuncture points.

[0045] First, the two-dimensional image in the depth camera is input into the network to make two-dimensional predictions of the acupuncture points on the back of the human body. According to the conversion relationship between the world coordinate system and the camera coordinate system, the two-dimensional coordinates of the two-dimensional image are converted into three-dimensional coordinates in the camera coordinate system. Then, according to the conversion relationship between the camera coordinate system and the robot terminal coordinate system, the three-dimensional coordinates in the camera coordinate system are converted into three-dimensional coordinates in the robot terminal coordinate system. Finally, according to the positive kinematics of the robot, the three-dimensional coordinates in the robot terminal coordinate system are converted into three-dimensional coordinates in the robot base coordinate system using the conversion relationship between the terminal coordinate system and the base coordinate system.

[0046] Step 7: For different back acupoint therapy tasks, the robot posture is estimated to realize the visual servo positioning motion planning of the robot for different therapy tasks.

[0047] The present invention is further described in detail below in conjunction with embodiments and drawings.

[0048] In step 1 of this example, a diverse dataset of human back acupoints is constructed based on Anhui University of Traditional Chinese Medicine, and target images are collected from multiple patients and volunteers in different environments, different lighting, and different postures. The collected back photos are selected from common acupoints for physical therapy tasks for annotation, including Dazhui, Jianjing, Tianzong, Jianshu, Jianzhen, Dazhui, Fengmen, Feishu, Jueyinshu, Xinshu, Geshu, Ganshu, Danshu, Pishu, Weishu, Sanjiaoshu, Shenshu, Dachangshu, Gaohuang. Except for Dazhui, all other acupoints are symmetrical, with a total of 37 acupoints marked.

[0049] In step 2 of this example, the data set collected in step 1 is classified into two categories: training set and validation set, with a ratio of 8:2. The labeled data is processed to facilitate input into the HRNet network for training and validation.

[0050] In step 3 of this example, the initial HRNet neural network model is trained using the training set, and the model performance is verified using the validation set to obtain a trained and effective model. However, since HRNet is mainly used for key point detection of human posture and key points of human acupuncture points, it is necessary to derive the internal parameters of HRNet and adjust the HRNet learning strategy so that HRNet can adapt to the key point detection of human back acupuncture points. The specific derivation and adjustment are as follows:

[0051] OKS is a commonly used evaluation index for key point detection, but its parameters are mainly applicable to the key point detection task of the human body. Therefore, it is necessary to adjust the value of σ in OKS in a targeted manner to adapt to the key point detection task of the human back acupuncture points. The specific processing method is as follows:

[0052] Step 3.1: OKS is a commonly used evaluation index for key point detection. The OKS calculation formula is as follows:

[0053]

[0054] in, is the square of the distance between the real coordinates and the predicted coordinates, S represents the square root of the area of ​​the back in the image, but different key points have different difficulties in the prediction process of the neural network, so simply using distance and area as evaluation indicators is not enough, so σ is introduced to avoid the deviation of the prediction difficulty of different key points, δ is the Kronecker function, when manually annotated, each key point has three attributes, v = 0 means that the key point is invisible and the key point position cannot be inferred, v = 1 means that the key point is visible, v = 2 means that the key point is invisible, but the key point position can be inferred, and then the Kronecker function is defined as:

[0055]

[0056] Finally, it is normalized and the value range of OKS is limited to 0-1, so that the distance, back area, and manual annotation offset are consistent with the increase and decrease of OKS score and the above analysis.

[0057] Step 3.2: Back acupuncture point key point detection task. Different acupuncture points have different prediction difficulties. Therefore, 100 images are selected from the data set in step 1 above. Students and professors of traditional Chinese medicine are asked to mark the acupuncture points of the images respectively. The value of σ is calculated based on the data marked by the professors to represent the prediction difficulty of different acupuncture points. The calculation steps are as follows:

[0058] First compare the coordinates of each point:

[0059]

[0060] Among them, (x′ pi -x pi ) 2 +(y′ pi -y pi ) 2 is the square of the distance between the professor and the student’s key points, w p h p are the width and height of the image respectively.

[0061] Recalculate The average value of:

[0062]

[0063] Where M is the number of pictures.

[0064] Final calculation The standard deviation of:

[0065]

[0066] The value shown in formula (5) is the key point normalization factor of the corresponding point obtained.

[0067] Step 3.3: Replace the multi-step learning rate decay in HRNet with cosine annealing learning rate decay. Since the model parameters are very unstable in the early stage of new task training, a smaller learning rate should be selected at the beginning, rather than from large to small in the multi-step learning rate. The cosine annealing learning rate decay equation is constructed using the characteristics of the cosine function as follows:

[0068]

[0069] Among them, η t is the learning rate at the tth round, η min is the lower limit of the learning rate, η max is the initial learning rate, T cur is the current epoch number, T max is the preset maximum number of rounds.

[0070] At the same time, the image needs to be further enhanced to facilitate the deployment of the network into the D435i camera. The specific processing methods for Mosaic data enhancement and Gaussian noise data enhancement are as follows:

[0071] Step 3.4: Perform Mosaic data enhancement. Randomly select four photos from the data in step 1, perform random scaling, random cropping, and random arrangement, and then perform border stitching to stitch the four photos into a whole picture. Take the stitched photo in the upper left corner as an example. Select the center point (x c ,y c ) Align the lower right corner of the upper left photo with the center of the overall picture for stitching. There are two cases: one is that the stitched picture does not exceed the size of the overall image. The coordinates of the upper left corner of the stitched picture in the overall picture are (x c -w,y c -h), the overall coordinates of the spliced ​​image are (x c -w,y c -h,x c ,yc The other is that the stitched image exceeds the size of the overall image. The coordinates of the upper left corner of the stitched image in the overall image are (0,0). The upper left corner image needs to be cropped. The cropping coordinates are (wx c ,hy c ,w,h), the overall coordinates of the stitched image are (0,0,x c ,y c ), the processing methods for other directions are similar to the upper left picture, and the key points are translated accordingly.

[0072] Step 3.5: Perform Gaussian noise data enhancement processing. Since the camera used to shoot the human back data is different from the D435i depth camera deployed on the robot, in order to ensure the deployment effect, Gaussian noise is added during training to simulate the random disturbance in devices such as image sensors to improve the training effect. Gaussian noise is noise based on normal distribution, and the noise value N is defined by the following formula:

[0073] N~N(μ,σ 2 )(7)

[0074] Among them, μ is the mean representing the center point of the noise, σ is the standard deviation representing the discreteness of the noise value, and these noises are added to the random pixels of the image.

[0075] Step 3.6: Perform adaptive scaling. In step 3.4 and when the network image is input, the image will be scaled. The image is scaled by filling the image with the least gray edge to avoid image distortion during the image scaling process.

[0076] Step 3.7: Modify the detection head of HRNet. The original number of key points used by HRNet for human posture estimation is 17, while the number of our acupoint key points is 37. Therefore, it is necessary to modify the heat map predicted by HRNet from 17 to 37, calculate the loss for each of the 37 heat maps and the corresponding acupoint annotations, and then optimize the model through back propagation to train a network suitable for back acupoint key point detection.

[0077] After adjusting HRNet and enhancing the image, the loss value changes and mAP changes obtained after network training are shown in the figure below. Figure 2 , 3 As shown in the figure, it shows that the network can achieve relatively fast convergence in the training of the back acupuncture point key point detection task, and the mAP tends to be stable after 200 rounds of training, which means that the network tends to be stable during the training process and can achieve better results.

[0078] Select a picture data from the validation set such as Figure 4As shown, it is input into the modified HRNet network for prediction, and the prediction result is plotted on the original image as shown Figure 5 As shown, Figure 4 and Figure 5 By comparison, the predicted acupoints have smaller errors than the labeled data, indicating that the trained modified HRNet network achieves better results in detecting the key points of 37 back acupoints.

[0079] In step 4 of this example, the platform is built using the Realman RM-65 six-degree-of-freedom robotic arm to conduct experiments, realize the key point detection of human acupuncture points and the three-dimensional coordinate positioning of acupuncture points, and complete the path planning of the robot physiotherapy task, specifically:

[0080] Build as Figure 6 The experimental platform shown in the figure mainly includes: robot base 1, Realman RM-65 six-degree-of-freedom robot arm 2, Intel D435i depth camera 3, camera connection board 4, physical therapy instrument 6 and its connecting male and female heads 5. Among them, the Realman RM-65 six-degree-of-freedom robot arm is mainly used to simulate the motion process of physical therapy tasks, the Intel D435i depth camera obtains two-dimensional images and depth images, and realizes the conversion of two-dimensional coordinates of acupoints to three-dimensional coordinates, and the connecting male and female heads of the physical therapy instrument are mainly used to quickly replace different physical therapy instruments, such as Figure 7-9 The figure shows an example of connecting the male and female connectors of a physical therapy device.

[0081] In step 5 of this example, in order to obtain the conversion relationship between the world coordinate system and the camera coordinate system and the conversion relationship between the robot end coordinate system and the camera coordinate system, the robot vision system needs to be calibrated. The visual system calibration includes the calibration of the RGB camera in the D435i depth camera and the hand-eye calibration in the form of eyes on hands. The quality of the robot vision system calibration result will directly affect the robot physical therapy visual servoing task. The specific calibration steps are as follows:

[0082] Step 5.1: First determine the conversion relationship between the world coordinate system and the camera coordinate system.

[0083] The relationship between the pixel coordinate system and the image coordinate system is as follows:

[0084]

[0085] Among them, u, v are pixel coordinates, and x, y are image coordinates.

[0086] The relationship between the image coordinate system and the camera coordinate system is as follows:

[0087]

[0088] P(x p ,y p ,f) is a point in the image coordinate system, M(xM ,y M ,z M ) is a point in the camera coordinate system, where f represents the focal length. The relationship between the camera coordinate system and the world coordinate system is as follows:

[0089]

[0090] Substituting the above formula into the equation, we get:

[0091]

[0092] Among them, Z M is the value of the z-axis in the camera coordinate system, Represents the camera internal parameters, f in the internal parameter matrix x f y u0 v0 are the intrinsic parameters of the camera; Represents the camera extrinsic parameters, which only change when the camera's pose changes.

[0093] Step 5.2: Determine the conversion relationship between the camera coordinate system and the RM-65 end coordinate system. According to the eye-in-hand calibration method, the conversion matrix from the D435i camera coordinate system to the RM-65 robot end coordinate system can be obtained. Fig.10 Here, base represents the base coordinate system of the RM-65 robot; end represents the coordinate system of the end of the robot; cam represents the coordinate system of the D435i camera fixed on the end flange of the RM-65; and board is the coordinate system where the calibration board is located.

[0094] First, assume that there is a point P0 in the board coordinate system. Use the extrinsic parameter matrix T1 of the calibrated D435i RGB camera to transform P0 to P1 in the cam coordinate system. Then, combined with the hand-eye calibration matrix to be determined (transformation matrix X), P1 is further transformed to P2 in the end coordinate system. Based on the known parameters of the RM-65 robot itself, P2 is transformed to P3 in the base coordinate system. The following relationship is established:

[0095] T3XT1P0=P3(12)

[0096] Then move the robot arm. For the same point, the coordinate values ​​of P0 and P3 remain unchanged, and only the coordinate values ​​of P1 and P2 change. The above relationship is as follows:

[0097] T′3XT′1P0=P3 (13)

[0098] The calibration board can be used to solve the camera's external parameter matrix, and the robot's transformation matrix can be obtained through the robot's own parameters, that is, T3 ′-1 T3X=XT′1T1 -1 Where T′3-1 T3 and T′1T1 -1 All can be solved. Expanding this formula yields:

[0099]

[0100] Transform the problem into a solution and Solving this problem can obtain the rotation matrix between the camera coordinate system and the RM-65 terminal coordinate system. and translation vector

[0101] In step 6 of this example, after detecting the key points of the acupuncture points on the back of the human body, the 2D pixel coordinates (u, v) of the acupuncture points are obtained, and the depth z of the pixel coordinates is obtained using the D435i depth sensor; combined with the hand-eye conversion matrix obtained in the previous solution and the robot's own joint conversion matrix, the 2D pixel coordinates of the acupuncture points are converted to 3D spatial coordinates (x, y, z) in the robot base coordinate system. The specific calculation method is as follows:

[0102] Step 6.1: Convert the 2D pixel coordinates (u, v) to the x coordinate system of the camera M y M z M , the formula is as follows:

[0103]

[0104] Step 6.2: Find the x coordinate of the known acupuncture point in the camera coordinate system M y M z M Then, the coordinate points in the camera coordinate system are transformed to the coordinate points in the robot base coordinate system through the robot's own joint transformation matrix and the hand-eye transformation matrix required above. The formula is as follows:

[0105]

[0106] Among them, T b is the robot joint transformation matrix, T c is the hand-eye transformation matrix.

[0107] Therefore, the two-dimensional coordinates of the 37 acupoints predicted by HRNet can be converted into three-dimensional coordinates in the robot base coordinate system.

[0108] In step 7 of this example, different physical therapy tasks may require posture estimation of the robot and motion planning for simulating different physical therapy task techniques. The specific posture estimation method and motion planning are as follows:

[0109] Step 7.1: Perform posture estimation for each acupoint therapy task. The three-dimensional coordinates of 37 acupoints in the robot base coordinate system have been obtained from step 6. To ensure the stability and reliability of the posture solution before and after the prediction, the minimum pixel distance for acupoint prediction of the image acquired 500ms adjacent to each other is calculated when the robot arm is stationary. If the distance is exceeded, re-prediction is required. The formula is as follows:

[0110] (a[0]-b[0]) 2 +(a[1]-b[1]) 2 (17)

[0111] Further stabilize the positions of the acupoints to prevent the misalignment of the forward and backward predicted acupoints from causing excessive errors in the solved posture.

[0112] Step 7.2: Taking the estimation of Dazhui acupoint posture as an example, select the acupoint closest to Dazhui and the next acupoint for physical therapy to compare the distances between adjacent acupoints, and select the acupoint with the smallest distance and the next acupoint for physical therapy, which should generally be the left Dazhui and the right Dazhui. Given the Dazhui coordinate point P (x0, y0, z0), the left Dazhui coordinate point Q (x1, y1, z1) and the right Dazhui coordinate point R (x2, y2, z2), the plane equation of the estimated plane is calculated by the three points:

[0113]

[0114] In order to facilitate the robot's motion planning for different physiotherapy task simulations, the unit vector of the line connecting the next physiotherapy task acupoint and the posture estimation acupoint is selected as the x-axis direction in the plane:

[0115]

[0116] Therefore, the rotation matrix of the end effector in the base coordinate system is for:

[0117]

[0118] For different physical therapy tasks, some offsets may be required in the direction perpendicular to the back. For example, moxibustion physical therapy tasks do not require the actuator to contact the skin. Assuming the offset is d (i.e., the distance between the end effector and the plane), the translation vector of the end effector is:

[0119]

[0120] Therefore, the pose matrix of the end effector is obtained

[0121]

[0122] Step 7.3: Motion planning for different physiotherapy tasks. Here, we take moxibustion therapy as an example, where the robot needs to simulate the physician to perform moxibustion therapy for trajectory planning. The end-effector posture obtained in step 7.2 is combined with a fifth-order polynomial for motion planning to ensure the stability and efficiency of the physiotherapy task. The fifth-order polynomial is in the form of:

[0123] θ(t)=A0+A1t+A2t 2 +A3t 3 +A4t 4 +A5t 5 (twenty three)

[0124] The boundary conditions are:

[0125]

[0126] Among them, t0 is the starting time, t f The robot will go from the ready state (prepared for moxibustion) to the specific moxibustion technique trajectory planning through trajectory planning. Different moxibustion techniques require different operation characteristics, movement speeds, accelerations, and path curves, so different speeds, accelerations, and path curves need to be set for different techniques.

[0127] Step 7.4: Fig.11 As shown, the three-dimensional coordinates of the acupoints, the robot posture estimation and the manipulation simulation obtained in the above steps are imported into the simulation program for simulation to determine whether the manipulation simulation for different therapy tasks is reasonable. After the simulation is correct, the position, velocity and acceleration obtained by the motion planning are deployed to the robot to realize the motion planning for the manipulation simulation for different therapy tasks and the visual servo positioning motion planning of the robot for different therapy tasks.

[0128] In summary, the basic workflow of the back acupuncture point visual detection and spatial coordinate positioning based on robot positioning is as follows: Fig.12 As shown in the figure, the depth camera collects the acupoint images on the back of the human body, and inputs the images into the HRNet neural network for prediction. The prediction generates 37 acupoint prediction heat maps corresponding to the 37 acupoints that need to be predicted. The predicted acupoints are transformed into the camera coordinate system and the robot coordinate system to realize the prediction of the three-dimensional coordinate points of the acupoints. Finally, the acupoint posture is estimated to estimate the physiotherapy posture required for each acupoint and perform motion planning for the corresponding physiotherapy tasks. Machine vision and three-dimensional vision technology are used to realize the robot's self-identification of acupoint physiotherapy tasks and realize the construction of the physiotherapy system.

[0129] The above contents are merely examples and explanations of the concept of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A robot vision recognition method based on back acupuncture point key point visual detection and acupuncture point spatial coordinate positioning, characterized in that: The steps include: Step 1: Build a data sample library, collect back target images of multiple patients and volunteers in different environments, different lighting and different postures, and mark the key points of the back acupoints to build a data set of key points of the back acupoints; Step 2: Classify the data set collected in step 1 into two categories: training set and validation set; and process the labeled data to facilitate input into the HRNet network for training and validation; Step 3: The key point evaluation index OKS and the number of HRNet detection head predictions are adjusted for the prediction of acupuncture points on the back of the human body, so that the HRNet key point detection network is suitable for the key point detection task of acupuncture points on the back of the human body; and the data set is subjected to Mosaic data enhancement, Gaussian noise data enhancement and adaptive scaling before the data is input into the network; Step 4: Deploy the trained HRNet network to the depth camera to enable the robot to self-collect and adaptively detect key acupuncture points on human back photos; Step 5: Calibrate the depth camera to obtain the conversion relationship between the world coordinate system and the camera coordinate system; then, complete the hand-eye calibration of the depth camera and the robot in the form of eyes on hands to solve the conversion relationship between the camera coordinate system and the robot end coordinate system; Step 6: Convert the two-dimensional coordinates detected by the camera to predict the three-dimensional coordinates of the back acupuncture points; Step 7: For different back acupoint therapy tasks, the robot posture is estimated to realize the visual servo positioning motion planning of the robot for different therapy tasks.

2. The robot visual recognition method according to claim 1, characterized in that: In step 3, the specific steps for adjusting the key point evaluation index OKS for the prediction of acupuncture points on the back of the human body are as follows: OKS is a commonly used evaluation index for key point detection, but its parameters are mainly applicable to the key point detection task of the human body. Therefore, it is necessary to adjust the value of σ in OKS in a targeted manner to adapt to the key point detection task of the human back acupuncture points. The specific processing method is as follows: Step 3.1: OKS is a commonly used evaluation index for key point detection. The OKS calculation formula is as follows: in, is the square of the distance between the real coordinates and the predicted coordinates, S represents the square root of the area of ​​the back in the image, but different key points have different difficulties in the prediction process of the neural network, so simply using distance and area as evaluation indicators is not enough, so σ is introduced to avoid the deviation of the prediction difficulty of different key points, δ is the Kronecker function, when manually annotated, each key point has three attributes, v = 0 means that the key point is invisible and the key point position cannot be inferred, v = 1 means that the key point is visible, v = 2 means that the key point is invisible, but the key point position can be inferred, and then the Kronecker function is defined as: Finally, it was normalized and the OKS value range was limited to 0-1, so that the distance, back area, and manual annotation offset were consistent with the increase and decrease of the OKS score and the above analysis; Step 3.2: Back acupuncture point key point detection task. Different acupuncture points have different prediction difficulties. Therefore, several pictures are selected from the data set in step 1 above. Students and professors of traditional Chinese medicine are asked to mark the acupuncture points of the images respectively. The value of σ is calculated based on the data marked by the professors to represent the prediction difficulty of different acupuncture points. The calculation steps are as follows: First compare the coordinates of each point: Among them, (x′ pi -x pi ) 2 +(y′ pi -y pi ) 2 is the square of the distance between the professor and the student’s key points, w p h p are the width and height of the image respectively; Recalculate The average value of: Where M is the number of pictures; Final calculation The standard deviation of The value shown in formula (5) is the key point normalization factor of the corresponding point obtained; Step 3.3: Replace the multi-step learning rate decay in HRNet with cosine annealing learning rate decay. Since the model parameters are very unstable in the early stage of new task training, a smaller learning rate should be selected at the beginning, rather than from large to small in the multi-step learning rate. The cosine annealing learning rate decay equation is constructed using the characteristics of the cosine function as follows: Among them, η t is the learning rate at the tth round, η min is the lower limit of the learning rate, η max is the initial learning rate, T cur is the current epoch number, T max is the preset maximum number of rounds; Step 3.4: Perform Mosaic data enhancement. Randomly select four photos from the data in step 1, perform random scaling, random cropping, and random arrangement, and then perform border stitching to stitch the four photos into a whole picture. Take the stitched photo in the upper left corner as an example, select the center point (x c ,y c ) Align the lower right corner of the upper left photo with the center of the overall picture for stitching. There are two cases: one is that the stitched picture does not exceed the size of the overall image. The coordinates of the upper left corner of the stitched picture in the overall picture are (x c -w,y c -h), the overall coordinates of the spliced ​​image are (x c -w,y c -h,x c ,y c ); the other is that the stitched image exceeds the size of the overall image. The coordinates of the upper left corner of the stitched image in the overall image are (0, 0). The upper left corner image needs to be cropped. The cropping coordinates are (wx c ,hy c , w, h), the overall coordinates of the spliced ​​image are (0, 0, x c ,y c ), the processing methods for other directions are similar to the upper left corner picture, and the key points are translated accordingly; Step 3.5: Perform Gaussian noise data enhancement processing. Since the camera used to shoot the human back data is different from the depth camera model deployed on the robot, in order to ensure the deployment effect, Gaussian noise is added during training to simulate the random disturbance in the image sensor device to improve the training effect; Gaussian noise is noise based on normal distribution, and the noise value N is defined by the following formula: N~N(μ,σ 2 ) (7) Among them, μ is the mean value representing the center point of the noise, σ is the standard deviation representing the discreteness of the noise value, and these noises are added to the random pixels of the image; Step 3.6: Perform adaptive scaling. In step 3.4 and when the network image is input, the image will be scaled. The image is scaled by filling the image with the least gray edge to avoid image distortion during the image scaling process. Step 3.7: Modify the detection head of HRNet. The original number of key points used by HRNet for human posture estimation is 17, while the number of our acupoint key points is 37. Therefore, it is necessary to modify the heat map predicted by HRNet from 17 to 37, calculate the loss for each of the 37 heat maps and the corresponding acupoint annotations, and then optimize the model through back propagation to train a network suitable for back acupoint key point detection.

3. The robot visual recognition method according to claim 2, characterized in that: Step 5 The specific steps are: In order to obtain the conversion relationship between the world coordinate system and the camera coordinate system and the conversion relationship between the robot end coordinate system and the camera coordinate system, the robot vision system needs to be calibrated; the visual system calibration includes the calibration of the RGB camera in the depth camera and the hand-eye calibration in the form of eye on hand; the quality of the robot vision system calibration result will directly affect the visual servoing task of robot physiotherapy. The specific calibration steps are as follows: Step 5.1: First determine the conversion relationship between the world coordinate system and the camera coordinate system; The relationship between the pixel coordinate system and the image coordinate system is as follows: Among them, u, v are pixel coordinates, x, y are image coordinates; The relationship between the image coordinate system and the camera coordinate system is as follows: P(x p ,y p ,f) is a point in the image coordinate system, M(x M ,y M ,z M ) is a point in the camera coordinate system, where f represents the focal length; the relationship between the camera coordinate system and the world coordinate system is as follows: Substituting the above formula into the equation, we get: Among them, Z M is the value of the z-axis in the camera coordinate system, Represents the camera internal parameters, f in the internal parameter matrix x f y u0 v0 are the intrinsic parameters of the camera; Represents the camera extrinsic parameters, which only change when the camera's pose changes; Step 5.2: Determine the conversion relationship between the camera coordinate system and the robot end coordinate system; according to the eye-on-hand calibration method, the conversion matrix from the depth camera coordinate system to the robot end coordinate system can be obtained; First, assume that there is a point P0 in the board coordinate system. Use the external parameter matrix T1 of the RGB camera in the calibrated depth camera to transform P0 to P1 in the cam coordinate system. Then, combine the hand-eye calibration matrix to be obtained to further transform P1 to P2 in the end coordinate system. According to the known robot parameters, transform P2 to P3 in the base coordinate system. Establish the following relationship: T3XT1P0=P3(12) Then move the robot arm. For the same point, the coordinate values ​​of P0 and P3 remain unchanged, and only the coordinate values ​​of P1 and P2 change. The above relationship is as follows: T3XT1′P0=P3(13) The calibration board can be used to solve the camera's external parameter matrix, and the robot's transformation matrix can be obtained through the robot's own parameters, that is, In the formula and All can be solved. Expanding this formula yields: Transform the problem into a solution and Solving this problem can obtain the rotation matrix between the camera coordinate system and the robot end coordinate system. and translation vectors 4. The robot visual recognition method according to claim 3, characterized in that: Step 6 The specific steps are: After detecting the key points of the acupuncture points on the back of the human body, the 2D pixel coordinates (u, v) of the acupuncture points are obtained, and the depth z of the pixel coordinates is obtained using the depth sensor in the depth camera; the 2D pixel coordinates of the acupuncture points are converted to 3D spatial coordinates (x, y, z) in the robot base coordinate system by combining the obtained hand-eye conversion matrix and the robot's own joint conversion matrix. The specific calculation method is as follows: Step 6.1: Convert the 2D pixel coordinates (u, v) to the x coordinate system of the camera M y M z M , the formula is as follows: Step 6.2: Find the x coordinate of the known acupuncture point in the camera coordinate system M y M z M Then, the coordinate points in the camera coordinate system are transformed to the coordinate points in the robot base coordinate system through the robot's own joint transformation matrix and the hand-eye transformation matrix required above. The formula is as follows: Among them, T b is the robot joint transformation matrix, T c is the hand-eye transformation matrix; Therefore, the two-dimensional coordinates of the 37 acupoints predicted by HRNet can be converted into three-dimensional coordinates in the robot base coordinate system.

5. The robot visual recognition method according to claim 4, characterized in that: Step 7 The specific steps are: Step 7.1: Perform posture estimation for each acupoint therapy task; the three-dimensional coordinates of the therapy acupoints in the robot base coordinate system have been obtained from step 6. To ensure the stability and reliability of the posture solution before and after the prediction, the minimum pixel distance for acupoint prediction of the image acquired in adjacent 500ms is calculated when the robot arm is stationary. If the distance is exceeded, re-prediction is required. The formula is as follows: (a[0]-b[0]) 2 +(a[1]-b[1]) 2 (17) Further stabilize the positions of the acupoints to prevent the misalignment of the predicted acupoints before and after, which may lead to excessive errors in the posture solution; Step 7.2: Select the acupoint closest to the task acupoint and the next physical therapy task acupoint to compare the distances between adjacent acupoints, select the acupoint with the smallest distance and the next physical therapy task acupoint, and construct the spatial plane equation as follows: d=-(ax0+by0+cz0) In order to facilitate the robot's motion planning for different physiotherapy task simulations, the unit vector of the line connecting the next physiotherapy task acupoint and the posture estimation acupoint is selected as the x-axis direction in the plane: Therefore, the rotation matrix of the end effector in the base coordinate system is for: For different physical therapy tasks, some offsets may be required in the direction perpendicular to the back. For example, for moxibustion physical therapy tasks, the actuator does not need to contact the skin. Assuming the offset is d, the translation vector of the end effector is: Therefore, the pose matrix of the end effector is obtained Step 7.3: Motion planning for different physiotherapy task simulations; The end-effector posture obtained in step 7.2 is combined with a fifth-order polynomial for motion planning to ensure the stability and efficiency of the physiotherapy task. The fifth-order polynomial is in the form of: θ(t)=A0+A1t+A2t 2 +A3t 3 +A4t 4 +A5t 5 (23) The boundary conditions are: Among them, t0 is the starting time, t f The robot will go from the preparation state to the specific therapy method through trajectory planning. Different therapy methods require different operation characteristics, movement speeds, accelerations, and path curves. Therefore, different speeds, accelerations, and path curves need to be set for different methods. Step 7.4: Import the three-dimensional coordinates of the acupoints, robot posture estimation and manipulation simulation obtained in the above steps into the simulation program for simulation, and determine whether the manipulation simulation for different physical therapy tasks is reasonable. After the simulation is correct, deploy the position, velocity and acceleration obtained by the motion planning to the robot, and realize the motion planning for manipulation simulation for different physical therapy tasks, and realize the visual servo positioning motion planning of the robot for different physical therapy tasks.

6. A physical therapy system implementing the robot visual recognition method according to any one of claims 1 to 5, characterized in that: It includes a robot and a mounted physiotherapy device. It predicts the two-dimensional coordinates of the acupuncture points on the back of the human body through a neural network, and uses a depth camera to realize the conversion of the two-dimensional coordinates of the acupuncture points into three-dimensional coordinates. The robot is used to simulate the motion process of controlling the physiotherapy task, and the physiotherapy device implements the physiotherapy task.

Images