Two-step method acupoint accurate positioning method based on human-computer interaction

Abstract

The invention discloses a two-step method acupoint accurate positioning method based on human-computer interaction, and belongs to the field of Kangbao acupoint recognition. The method comprises the steps that a human body acupoint data set is collected; preprocessing the collected human body acupoint data set and dividing the collected human body acupoint data set into a training set, a verification set and a test set; an improved mmpose network model is constructed; constructing a human body acupoint detection model; testing the test set by using the human body acupoint detection model; uploading the tested human body acupuncture point detection model to a cloud server, photographing a patient and obtaining RGB pictures and point cloud data; predicting and calculating through a human body acupuncture point detection model to obtain acupuncture point coordinates of the patient in a world coordinate system; the mechanical arm obtains the acupuncture point coordinates under the world coordinate system and moves to the corresponding acupuncture point coordinates; a mechanical arm located at the acupuncture point coordinates tests massage and corrects the acupuncture points of the patient; and massage treatment is performed on the patient. According to the invention, human-computer interaction is combined with a traditional two-step method, so that the detection efficiency and precision of human body acupoint detection are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of health care acupoint identification, and in particular to a two-step acupoint precise positioning method based on human-computer interaction. Background Art

[0002] Chinese massage is not just a simple kneading of muscles. More importantly, it is to accurately find and stimulate the corresponding acupuncture points to achieve the purpose of unblocking meridians and treating diseases. However, the distribution of acupuncture points in the human body is complex, and the location and sensitivity of acupuncture points are different for each person, which requires Chinese massage therapists to have extremely high professional qualities and rich practical experience. This high difficulty makes many people discouraged, resulting in a shortage of Chinese massage talents. In addition, the study of Chinese massage also needs to be combined with modern medical knowledge to understand the knowledge of human anatomy, physiology, pathology, etc., so as to better provide treatment services for patients. This comprehensive medical training requirement makes the training cycle of Chinese massage talents long and difficult, further exacerbating the problem of talent gap. There is a huge gap in the number of professional physiotherapy massage talents.

[0003] At present, most of the human acupoint massage in my country still adopts the traditional manual massage method, which often requires a lot of human resources. In addition, acupoint massage is greatly affected by subjective factors. The manual massage techniques are also affected by subjective factors and it is difficult to reach a unified standard. In recent years, with the development of deep learning technology, machine learning methods have gradually been applied to the health care industry. Compared with the traditional manual massage method, the acupoint massage method based on deep learning has obvious advantages, fast detection speed and high accuracy, and does not require a lot of human resources. Machine vision is of great significance to intelligent manufacturing. By introducing advanced information science and technology such as machine vision technology into the traditional massage service industry, it can fully realize its advantages in acupoint recognition and other aspects, and carry out automatic recognition and classification of acupoints in the process of acupoint recognition, and accurately locate them in real time. However, in the existing technology, the detection efficiency and accuracy of human acupoints need to be further improved. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a two-step acupoint precise positioning method based on human-computer interaction, which can significantly improve the efficiency and accuracy of human acupoint detection.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A two-step acupoint precise positioning method based on human-computer interaction, comprising the following steps:

[0007] Step 1, collecting human acupoint data sets;

[0008] Step 2, after preprocessing and labeling the collected human acupoint data set, the human acupoint data set is divided into a training set, a validation set, and a test set;

[0009] Step 3, build the mmpose network model, improve the mmpose network model, and obtain an improved mmpose network model;

[0010] Step 4, using the training set to train the improved mmpose network model and adding the bone measurement calculation method to the prediction model to obtain a human acupoint detection model;

[0011] Step 5, using a human acupoint detection model to test the test set;

[0012] Step 6: Upload the tested human acupoint detection model to the cloud server, and use the massage software deployed on the iPad to control the depth camera to take photos of the patient and obtain RGB images and point cloud data;

[0013] Step 7, uploading the acquired RGB image and point cloud data to the cloud server, and predicting and calculating the acupoint coordinates in the patient's world coordinate system through the human acupoint detection model;

[0014] Step 8: The robot arm obtains the acupuncture point coordinates in the world coordinate system calculated by the cloud server and moves to the corresponding acupuncture point coordinates;

[0015] Step 9, the seven axes of the robotic arm located at the acupoint coordinates apply a slowly increasing force downward to simulate the process of a human hand gradually applying force, and at the same time voice ask the patient about the pain level, including no pain, a little pain, and very pain. If the very painful level is not recognized, the seven axes of the robotic arm rotate 60 degrees clockwise and repeat the force application, inquiry and recognition actions. When the patient's very painful voice is recognized, the robotic arm automatically reads and saves the current acupoint coordinates;

[0016] Step 10, performing massage therapy on the patient according to the corrected acupoint coordinates.

[0017] A further improvement of the technical solution of the present invention is that in step 1, collecting the human acupoint data set refers to collecting human acupoints of people of different ages and body shapes to obtain the human acupoint data set.

[0018] A further improvement of the technical solution of the present invention is that: in step 2, preprocessing the human acupoint data set means adjusting the size of each image in the human acupoint data set to 256×256;

[0019] Labeling the human acupoint dataset refers to using the image labeling tool Labelme to label the acupoints of each image in the human acupoint dataset after preprocessing;

[0020] The ratio of the training set, the validation set, and the test set is set to 8:1:1.

[0021] A further improvement of the technical solution of the present invention is that in step 3, improving the mmpose network model means that the Backbone of the mmpose network model adopts CSPNet.

[0022] A further improvement of the technical solution of the present invention is that in step 4, the training epoch is set to 400, the batch size is set to 8, and the image size is set to 256×256.

[0023] A further improvement of the technical solution of the present invention is that in step 5, it specifically includes: using the training set and the official mmpose network model for training, obtaining the weight file of the trained mmpose network model, and then obtaining the acupoint error value on the test set through the official mmpose network model; using the training set and the human acupoint detection model for training, obtaining the weight file of the human acupoint detection model, and then obtaining the acupoint error value on the test set through the human acupoint detection model.

[0024] A further improvement of the technical solution of the present invention is that in step 6, the front-end and back-end of the massage software are written in the go language in the Linux system, the SDK for taking pictures with the depth camera and obtaining RGB pictures and point cloud data is written in the back-end, and a photo button is designed on the front-end. After clicking the photo button, the camera takes a photo and obtains the patient's RGB picture and point cloud data.

[0025] A further improvement of the technical solution of the present invention is that in step 7, it specifically includes: using Django to build a cloud server and a human acupoint detection model, uploading the acquired RGB images and point cloud data to the cloud server, and predicting and calculating the acupoint coordinates in the patient's world coordinate system through the human acupoint detection model.

[0026] A further improvement of the technical solution of the present invention is that in step 9, it specifically includes: placing the SDK of the force control of the robotic arm at the rear end, controlling the robotic arm to apply a slowly increasing force in the negative direction of the Z axis to simulate the process of gradual force application by human hands, and at the same time adding a voice recognition module at the rear end to ask the patient about the pain levels including no pain, a little pain, and very pain. When no voice indicating very painful pain is recognized, the robotic arm rises along the Z axis, and its seven axes rotate 60 degrees clockwise, and repeats the force application, inquiry and recognition actions. When the voice recognition module in the robotic arm recognizes the voice indicating very painful pain, the robotic arm automatically reads and saves the current coordinates.

[0027] A further improvement of the technical solution of the present invention is that in step 10, it specifically includes: predicting and locating the acupoint coordinates by using the improved mmpose network model, then calculating the remaining acupoint coordinates by the bone measurement method, and finally further determining the acupoint coordinates by the dichotomy method, and obtaining accurate acupoint information through multiple acupoint coordinate corrections, thereby providing more accurate acupoint coordinate information for robotic arm massage.

[0028] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is:

[0029] 1. The two-step acupoint precise positioning method based on human-computer interaction proposed in the present invention aims to solve the problems that traditional manual massage is greatly affected by subjective factors, and the manual massage techniques are also affected by subjective factors and it is difficult to reach a unified standard. The two-step acupoint precise positioning method based on human-computer interaction is used to identify acupoints of people of different ages and body shapes. Compared with the current traditional acupoint massage, the recognition accuracy is improved and the massage efficiency of medical staff is improved.

[0030] 2. The present invention adopts the top-down method of the mmpose network model to first perform target detection, detect the target area, and then perform key point detection. This method can be used to efficiently and accurately identify acupoints.

[0031] 3. The present invention adds a bone measurement calculation method to the prediction model. After accurately identifying and locating acupoints, the bone measurement calculation method is used to calculate other acupoints. Adding the bone measurement calculation method to the prediction model takes into account both the spatial characteristic information relationship of the acupoints and the relative position information of the acupoints, thereby achieving an improvement in the acupoint detection algorithm.

[0032] 4. The present invention introduces a voice recognition module into the robotic arm to realize the human-computer interaction function with the patient, implements the two-step method of acupoint recognition, and improves the accuracy of acupoint recognition in the target area. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 is a flow chart of a two-step acupoint precise positioning method based on human-computer interaction provided in an embodiment of the present invention;

[0035] Figure 2 Schematic diagram of the CSPNet network structure in an embodiment of the present invention;

[0036] Figure 3 It is an overall project flow chart in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0038] The present invention is further described in detail below with reference to the accompanying drawings and embodiments:

[0039] like Figure 1 As shown, a two-step acupoint precise positioning method based on human-computer interaction includes the following steps:

[0040] Step 1, collecting human acupoint data sets;

[0041] Specifically, human acupoints of people of different ages and body shapes are collected to form a human acupoint data set. Human acupoints include acupoints on the front chest, back and other parts. In this embodiment, the collection of human chest acupoints is taken as an example.

[0042] Step 2, after preprocessing and labeling the collected human chest acupoint data set, the human chest acupoint data set is divided into a training set for training, a verification set for verification, and a test set for testing;

[0043] Specifically include:

[0044] Preprocess the human chest acupuncture point dataset and adjust the size of each image in the human chest acupuncture point dataset to 256×256;

[0045] Use the image annotation tool Labelme to annotate the acupoints of each image in the preprocessed human chest acupoints dataset to obtain an annotation file in json format;

[0046] The data set required for mmpose training is divided into training set, validation set and test set in a ratio of 8:1:1, and the data is converted into the coco format data set required for mmpose training.

[0047] Step 3, build the mmpose network model, improve the mmpose network model, and obtain an improved mmpose network model;

[0048] Specifically, the official mmpose network model is improved by using a top-down approach. First, target detection is performed. After the target area is detected, key point detection is performed. The key point detection network structure model includes: input layer, Backbone, Neck, and Prediction. Backbone uses CSPNet. Figure 2 Each stage of the CSPNet shown consists of a partial dense block and a partial transition layer. In the partial dense block, the feature map of the base layer is divided into two parts x0 = [x0', x0"] according to the channel. x0' is directly connected to the tail of the stage, while x0" passes through a dense block. In the partial transition layer, the first transition layer takes the output [x0", x1, ..., xk] of all layers as input to obtain the output xT, and then concatenates [x0', xT] as the input of the second transition layer to obtain the final output xU. CSPNet reduces the computational bottleneck and memory cost. By integrating the gradient changes into the feature map from beginning to end, the accuracy can be guaranteed while reducing the amount of calculation. The improved mmpose network model improves the detection capability and speed of mmpose.

[0049] Step 4, using the training set to train the improved mmpose network model and adding the bone measurement calculation method to the prediction model to obtain a human acupoint detection model;

[0050] Specifically, when using the training set to train the mmpose network model, the epoch is set to 400, the batch size is set to 8, and the image size is set to 256×256. Use the training set and the official mmpose network model for training to obtain the weight file of the trained mmpose network model. Use the training set and the human acupoint detection model for training to obtain the weight file of the trained human acupoint detection model.

[0051] Step 5, using the human acupoint detection model in step 4 to test the test set;

[0052] Specifically, it includes: using the weight file and test set of the trained mmpose network model for testing to obtain its human acupoint error value; using the weight file and test set of the trained human acupoint detection model for testing to obtain its human acupoint error value, as shown in Table 1.

[0053] Table 1 Human acupoint error table

[0054]

[0055]

[0056] Table 1 continued

[0057]

[0058] In the above table, the unit of the error value of the human acupoint is cm, where the traditional type is the official mmpose network model, and the improved type is the human acupoint detection model of the present invention. Among them, RN22 is Tiantu point, RN21 is Xuanji point, RN17 is Tanzhong point, RN16 is Zhongting point, KI24 is Lingxu point, KI23 is Shenfeng point, LA14 is Qimen point, RN14 is Juque point, RN13 is Shangwan point, RN12 is Zhongwan point, RN10 is Xiawan point, RN9 is Shuifen point, RN8 is Shenque point, KI21 is Youmen point, KI20 is Futong point, KI19 is Yindu point, KI17 is Shangqu point, KI16 is Mangyu point, ST19 is Burong point, ST20 is Chengman point, ST21 is Liangmen point, ST23 is Taiyi point, ST25 is Tianshu point, SP15 is Daheng point, ST171 is Ruzhong point (left), and ST172 is Ruzhong point (right).

[0059] It can be seen from the data in Table 1 that the error value of the human acupoint detection model of the present application on the test set is lower than the error value of the official mmpose network model on the test set. This shows that the acupoints obtained by the two-step acupoint precise positioning method based on human-computer interaction provided by the present invention are more accurate.

[0060] Step 6, upload the human acupoint detection model tested in step 5 to the cloud server, and use the massage software deployed on the iPad to control the depth camera to take pictures of the patient and obtain RGB pictures and point cloud data;

[0061] Specifically, it includes: using the go language to write the front-end and back-end of the massage software in the Linux system, writing the SDK for taking photos with the depth camera and obtaining RGB images and point cloud data in the back-end, and designing a photo button on the front-end. After clicking the photo button, the camera takes a photo and obtains the patient's RGB image and point cloud data.

[0062] Step 7, uploading the RGB image and point cloud data of the patient obtained in step 6 to the cloud server, and predicting and calculating the acupuncture point coordinates of the patient in the world coordinate system through the human acupuncture point detection model;

[0063] Specifically, it includes: using Django to build a cloud server and a human acupoint detection model, uploading the patient's RGB image and point cloud data obtained in step 6 to the cloud server, and predicting and calculating the patient's acupoint coordinates in the world coordinate system through the human acupoint detection model.

[0064] Step 8: The robotic arm obtains the patient's acupuncture point coordinates in the world coordinate system calculated by the cloud server, and moves to the corresponding acupuncture point coordinates;

[0065] Step 9, the seven-axis of the robotic arm located at the acupuncture point coordinates applies a slowly increasing force downward to simulate the process of human hands gradually applying force, and at the same time asks the patient about the pain level by voice (the pain levels include no pain, a little pain and very pain). When the very painful level is not recognized, the seven-axis of the robotic arm rotates 60 degrees clockwise and repeats the above actions. When the patient's very painful voice is recognized, the robotic arm automatically reads and saves the current coordinates.

[0066] Specifically, the robot arm force control SDK is placed at the back end, and the robot arm is controlled to apply a slowly increasing force in the negative direction of the Z axis to simulate the process of gradual force applied by human hands. At the same time, a voice recognition module is added to the back end to ask the patient about the pain level (no pain, a little pain, very pain). When no voice indicating very painful pain is recognized, the robot arm rises along the Z axis, and its seven axes rotate 60 degrees clockwise, and the above actions are repeated. When the voice recognition module in the robot arm recognizes the voice indicating very painful pain, the robot arm automatically reads and saves the current acupuncture point coordinates.

[0067] Step 10, performing massage therapy on the patient according to the acupoint coordinates corrected in step 9.

[0068] like Figure 3 As shown, specifically including: predicting and locating acupoint coordinates by adopting the improved mmpose network model, then calculating the coordinates of other acupoints by bone measurement method, and finally further determining the acupoint coordinates by dichotomy, and obtaining accurate acupoint information by multiple acupoint coordinate corrections, so as to provide more accurate acupoint coordinate information for robotic arm massage.

[0069] In summary, the present invention can significantly improve the detection efficiency and accuracy of human acupoints by providing a two-step acupoint precise positioning method based on human-computer interaction.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A two-step acupoint precise positioning method based on human-computer interaction, characterized in that: The following steps are involved: Step 1, collecting human acupoint data sets; Step 2, after preprocessing and labeling the collected human acupoint data set, the human acupoint data set is divided into a training set, a validation set, and a test set; Step 3, build the mmpose network model, improve the mmpose network model, and obtain an improved mmpose network model; Step 4, using the training set to train the improved mmpose network model and adding the bone measurement calculation method to the prediction model to obtain a human acupoint detection model; Step 5, using a human acupoint detection model to test the test set; Step 6: Upload the tested human acupoint detection model to the cloud server, and use the massage software deployed on the iPad to control the depth camera to take photos of the patient and obtain RGB images and point cloud data; Step 7, uploading the acquired RGB image and point cloud data to the cloud server, and predicting and calculating the acupoint coordinates in the patient's world coordinate system through the human acupoint detection model; Step 8: The robot arm obtains the acupuncture point coordinates in the world coordinate system calculated by the cloud server and moves to the corresponding acupuncture point coordinates; Step 9, the seven axes of the robotic arm located at the acupoint coordinates apply a slowly increasing force downward to simulate the process of a human hand gradually applying force, and at the same time voice ask the patient about the pain level, including no pain, a little pain, and very pain. If the very painful level is not recognized, the seven axes of the robotic arm rotate 60 degrees clockwise and repeat the force application, inquiry and recognition actions. When the patient's very painful voice is recognized, the robotic arm automatically reads and saves the current acupoint coordinates; Step 10, performing massage therapy on the patient according to the corrected acupoint coordinates.

2. The two-step acupoint precise positioning method based on human-computer interaction according to claim 1 is characterized by: In step 1, collecting the human acupoint data set refers to collecting human acupoints of people of different ages and body shapes to obtain the human acupoint data set.

3. The two-step acupoint precise positioning method based on human-computer interaction according to claim 1 is characterized by: In step 2, preprocessing the human acupoint dataset refers to adjusting the size of each image in the human acupoint dataset to 256×256; Labeling the human acupoint dataset refers to using the image labeling tool Labelme to label the acupoints of each image in the human acupoint dataset after preprocessing; The ratio of the training set, the validation set, and the test set is set to 8:1:

1.

4. The two-step acupoint precise positioning method based on human-computer interaction according to claim 1 is characterized by: In step 3, improving the mmpose network model means that the Backbone of the mmpose network model adopts CSPNet.

5. The two-step acupoint precise positioning method based on human-computer interaction according to claim 1 is characterized by: In step 4, the training epoch is set to 400, the batch size is set to 8, and the image size is set to 256×256.

6. The two-step acupoint precise positioning method based on human-computer interaction according to claim 1 is characterized by: In step 5, it specifically includes: using the training set and the official mmpose network model for training, obtaining the weight file of the trained mmpose network model, and then obtaining the acupoint error value on the test set through the official mmpose network model; using the training set and the human acupoint detection model for training, obtaining the weight file of the human acupoint detection model, and then obtaining the acupoint error value on the test set through the human acupoint detection model.

7. The two-step acupoint precise positioning method based on human-computer interaction according to claim 1 is characterized by: In step 6, the front-end and back-end of the massage software are written in the Go language in the Linux system, the SDK for taking photos with the depth camera and obtaining RGB images and point cloud data is written in the back-end, and a photo button is designed on the front-end. After clicking the photo button, the camera takes a photo and obtains the patient's RGB image and point cloud data.

8. The two-step acupoint precise positioning method based on human-computer interaction according to claim 1, characterized in that: In step 7, it specifically includes: using Django to build a cloud server and a human acupoint detection model, uploading the acquired RGB images and point cloud data to the cloud server, and predicting and calculating the acupoint coordinates in the patient's world coordinate system through the human acupoint detection model.

9. The two-step acupoint precise positioning method based on human-computer interaction according to claim 1, characterized in that: In step 9, specifically including: placing the robot arm force control SDK at the back end, controlling the robot arm to apply a slowly increasing force in the negative direction of the Z axis to simulate the process of gradual force application by human hands, and at the same time adding a voice recognition module at the back end to ask the patient about the pain levels including no pain, a little pain, and very pain. When no voice indicating very painful pain is recognized, the robot arm rises along the Z axis, and its seven axes rotate 60 degrees clockwise, and repeats the force application, inquiry and recognition actions. When the voice recognition module in the robot arm recognizes the voice indicating very painful pain, the robot arm automatically reads and saves the current coordinates.

10. The two-step acupoint precise positioning method based on human-computer interaction according to claim 1, characterized in that: In step 10, it specifically includes: predicting and locating the acupoint coordinates by using the improved mmpose network model, then calculating the coordinates of the remaining acupoints by the bone measurement method, and finally further determining the acupoint coordinates by the dichotomy method, and obtaining accurate acupoint information through multiple acupoint coordinate corrections, so as to provide more accurate acupoint coordinate information for robotic arm massage.

Images