Multi-mode massage positioning method and system for human body acupuncture points

Abstract

The invention discloses a multi-modal massage positioning method and system for human body acupuncture points, and mainly relates to the technical field of data multi-modal fusion. Comprising the steps of collecting multi-modal data of a user and constructing a virtual human body model; human body key points are identified based on the multi-modal data; mapping the three-dimensional key points in a depth camera coordinate system to an RGB image coordinate system through coordinate transformation; predicting positions of key acupuncture points of the human body by using a machine learning algorithm; the predicted acupuncture point coordinates are converted into a robot coordinate system through hand-eye calibration; recognizing a massage intention of the user based on a multi-modal intention understanding algorithm; and finally outputting the massage position information to the massage robot to execute massage operation. The intelligent massage robot has the beneficial effects that accurate positioning, multi-modal natural interaction and intelligent intention understanding of human body acupuncture points are realized, and the positioning accuracy and the use convenience of the massage robot are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of data multimodal fusion technology, specifically a multimodal massage positioning method and system for human acupoints. Background Technology

[0002] The global population is aging rapidly, and the younger generation faces increasingly heavy work and life pressures, making it difficult to adequately care for the growing elderly population. At the same time, due to declining physical function, the elderly commonly suffer from various chronic pains. These two factors combined drive an urgent need for healthcare services. Massage robots are popular among the elderly due to their ease of use, and have attracted considerable research attention due to their advantages in alleviating high labor costs and fatigue associated with manual assistance.

[0003] Although intelligent massage robots have been widely adopted, it remains challenging to address the difficulties elderly people face in using massage services and to naturally complete massage tasks with them through existing human-computer interaction and collaborative algorithms. This is mainly due to the following reasons: (1) The interaction mode of intelligent massage systems is singular, the interaction method is not natural enough, and it is very sensitive to information from environmental factors; (2) The system ignores the special characteristics of the elderly population: due to the decline in physical function, the elderly may not be able to express themselves in a standardized manner, making it difficult for the system to accurately understand their intentions and affecting the realization of the target function; (3) Intelligent massage systems cannot independently complete the key task of massage positioning and usually require the support of assistive technologies or assistants, which is not practical in a home environment.

[0004] Therefore, there is an urgent need for a multimodal massage positioning method and system for human acupoints to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal massage positioning method and system for human acupoints, which realizes precise positioning of human acupoints, multimodal natural interaction and intelligent intent understanding, and significantly improves the positioning accuracy and ease of use of massage robots.

[0006] To achieve the above objectives, the present invention employs the following technical solution: On one hand, the present invention provides a multimodal massage positioning method for human acupoints, comprising the following steps: Collect multimodal data from users, including RGB images and depth images; construct a virtual human body model and map the massage positions selected by the user on the virtual model to the real human body; Human body key points are identified based on multimodal data, including shoulder key points and hip key points; The 3D key points in the depth camera coordinate system are mapped to the RGB image coordinate system through coordinate transformation; Machine learning algorithms are used to predict the location of key acupoints on the human body; the predicted acupoint coordinates are then converted to the robot coordinate system through hand-eye calibration. Identifying users' massage intentions based on multimodal intent understanding algorithms; Finally, the massage position information is output to the massage robot to perform the massage operation.

[0007] Preferably, the specific steps for constructing the virtual human body model and realizing virtual-real mapping include: A virtual human body model was built using Blender software, and human-computer interaction functions were implemented using the Block3d visual programming tool; the user selected massage points on the virtual human body model. The system records the coordinates of this point; it also identifies the coordinates of the Dazhui (DU1) and Yaoyangguan (DU3) acupoints on the virtual human body. and Combined with the points selected by the user Calculate corresponding points on the real human body using the principle of triangle similarity. The calculation formula is as follows: ; in, , These are the coordinates of DU1 and DU3 on a real human body, respectively. , This represents the distance between points in the virtual model.

[0008] Preferably, the key point identification specifically includes: The RGB images were processed using the OpenPose network to obtain shoulder keypoints: the back shoulder keypoint, the left shoulder keypoint, and the right shoulder keypoint. The hip keypoints were identified using depth images from the Kinect 2.0 camera, and the intersection of the highest point of both hips and the midline of the back of the body was selected as reference points. Finally, the data of the four keypoints were obtained and input into the GBDT regression algorithm for training to predict the positions of DU1 and DU3.

[0009] Preferably, the specific steps of the coordinate transformation are as follows: 3D points in the depth camera coordinate system Transformed to a 2D point in the RGB camera coordinate system using a rigid body transformation matrix. The transformation formula is: ; in, Let be a rotation matrix. It is a translation vector. , RGB cameras and Focal length in direction.

[0010] Preferably, the specific steps of the hand-eye calibration are as follows: By calibrating the coordinates of the chessboard corner points in the camera coordinate system and the robot coordinate system, the alignment matrix is ​​solved using the ICP algorithm, and the transformation formula is as follows: ; in, For camera coordinate system coordinates, These are the coordinates in the robot's coordinate system.

[0011] Preferably, the multimodal intent understanding algorithm employs an improved Naive Bayes classifier, introducing dynamic information entropy to optimize classification performance. The classifier formula is as follows: ; in, According to the category of massage area, For the first The results of modality recognition, For the first The entropy value of a mode is calculated using the following formula: ; in, For the first The set of probabilities for each category of a modality.

[0012] Preferably, the modality includes a gesture channel. finger pointing Candidate massage locations and voice channel The first three types are obtained through the visual channel, while the speech is obtained through the speech channel.

[0013] Preferably, it also includes automatically excluding a certain modality from the classification calculation when data for that modality is missing.

[0014] On the other hand, the present invention also provides a multimodal massage positioning system for human acupoints, used to implement the multimodal massage positioning method for human acupoints as described above, including: A multimodal data acquisition module is used to acquire RGB images and depth images; The virtual mapping module is used to construct a virtual human body and enable user interaction. Key point recognition module, used to extract key points of the shoulders and hips; The coordinate transformation module is used to map depth coordinates to the RGB image plane; The acupoint location module is used to predict acupoint locations using the GBDT algorithm. The hand-eye calibration module is used to transform coordinates to the robot coordinate system; The intent understanding module is used to identify user intents using a dynamic entropy Bayes classifier. The control output module is used to drive the massage robot to perform massage operations.

[0015] Preferably, it also includes a user interface module that allows users to select massage locations on a virtual human body via mobile phones or tablets, and displays the intent recognition results and massage execution status in real time.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Improve the accuracy and naturalness of massage positioning: By using a virtual human body model to intuitively map the user's body features, the user can directly select the massage location in the virtual interface. The system combines two-dimensional visual key points with three-dimensional depth information, which significantly improves the positioning accuracy of key acupoints (such as Dazhui and Yaoyangguan acupoints).

[0017] 2. Enhance the intelligence and adaptability of human-computer interaction: By introducing a multimodal intent understanding algorithm based on dynamic information entropy, and combining various interaction methods such as gestures, voice, and pointing, the system can adaptively understand user intent and maintain stable recognition performance even when some modal data is missing.

[0018] 3. Achieve efficient fusion and robust processing of multimodal data: By employing the GBDT regression model for key point prediction and combining coordinate transformation and hand-eye calibration techniques, a precise conversion from the image coordinate system to the robot operation coordinate system is achieved, thereby improving the reliability and applicability of the system in actual operation.

[0019] 4. Optimize system response speed and computational efficiency: The proposed dynamic entropy-adjusted Bayesian classifier has low time complexity and adopts a two-dimensional storage structure to ensure efficient operation of the system in real-time interactive scenarios.

[0020] 5. Adapting to the special needs of the elderly population: The system takes into full account the characteristics of elderly people, such as irregular behavior and declining physical function. Through multi-channel information complementarity and virtual-real mapping mechanism, it reduces the dependence on standardized user operation and improves the system's ease of use and inclusiveness.

[0021] 6. Promote the practical application of smart home massage devices: This invention solves the problem of existing massage robots relying on assistants or external equipment, enabling independent and accurate acupoint positioning and massage tasks in a home environment, and has good prospects for widespread application. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0023] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0024] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0025] Example: This embodiment uses Blender software to build a virtual human body model and utilizes the Block3d visual programming tool to implement human-computer interaction functions on the model.

[0026] like Figure 1 As shown, this embodiment provides a multimodal massage positioning method for human acupoints, including the following steps: Collect multimodal data from users, including RGB images and depth images; construct a virtual human body model and map the massage positions selected by the user on the virtual model to the real human body; Human body key points are identified based on multimodal data, including shoulder key points and hip key points; The 3D key points in the depth camera coordinate system are mapped to the RGB image coordinate system through coordinate transformation; Machine learning algorithms are used to predict the location of key acupoints on the human body; the predicted acupoint coordinates are then converted to the robot coordinate system through hand-eye calibration. Identifying users' massage intentions based on multimodal intent understanding algorithms; Finally, the massage position information is output to the massage robot to perform the massage operation.

[0027] like Figure 2 As shown, this embodiment also provides a multimodal massage positioning system for human acupoints, including: A multimodal data acquisition module is used to acquire RGB images and depth images; The virtual mapping module is used to construct a virtual human body and enable user interaction. Key point recognition module, used to extract key points of the shoulders and hips; The coordinate transformation module is used to map depth coordinates to the RGB image plane; The acupoint location module is used to predict acupoint locations using the GBDT algorithm. The hand-eye calibration module is used to transform coordinates to the robot coordinate system; The intent understanding module is used to identify user intents using a dynamic entropy Bayes classifier. The control output module is used to drive the massage robot to perform massage operations; It also includes a user interface module, allowing users to select massage locations on a virtual human body via mobile phones or tablets, and displaying the intent recognition results and massage execution status in real time. In the virtual human body system, obtain the position coordinates of DU1 and DU3 and mark them as... Users can select desired massage locations on a virtual human body via the screen of a mobile phone, tablet, or other electronic device. The system records this location as a point. To acquire key points on the back that approximate the skeletal features of the Dazhui (DU1) and Yaoyangguan (DU3) acupoints, this embodiment selects shoulder key points on the upper back and hip key points on the lower back. Given that the Kinect 2.0 camera provides more information than a regular RGB camera, different data are used for key point estimation based on the characteristics of different key points. For most individuals, the height difference between the cervical spinous processes and the skin around the neck is limited, so RGB images are sufficient to determine shoulder key points. This embodiment utilizes the OpenPose human pose estimation network to process the RGB images, using common key points provided by OpenPose, such as the shoulders and hips, as benchmarks to help locate specific acupoints adjacent to these large key points. For example, this embodiment uses shoulder key points to help determine the location near the Dazhui (DU1) acupoint. Since OpenPose may not be accurate enough in some cases, especially regarding acupoint location in the waist and legs, this embodiment combines depth images provided by the Kinect 2.0 camera to supplement RGB image information, enhancing the accuracy of DU3 acupoint location. This combined use enables more precise identification and location of key acupoints. This technology acquires three key points around the shoulders from the input image: the rear shoulder key point, the left shoulder key point, and the right shoulder key point. It accurately identifies and locates key pose points of the human body in an image, including shoulder key points. For the shoulder region, the color and texture information provided by the RGB image is sufficient to distinguish the boundaries between skin and clothing, which is a crucial factor in key point localization. Furthermore, the structure of the shoulder region is relatively simple, requiring no complex depth information for identification.

[0028] Because OpenPose has relatively weak detection capabilities for the waist and legs, especially when the lower thighs are not within the frame, it struggles to output stable hip keypoints. To obtain reference keypoints near the lower back, this embodiment uses depth images to determine skeletal features. The sacral cleft is an important somatic landmark of the hip, located on the line connecting the left and right sacral angles. To obtain more stable hip nodes, this embodiment selects the intersection of the highest points of both hips and the posterior midline of the body, making the depth features more significant. By utilizing depth information to obtain upper back keypoints from the depth image, these keypoints are then input into the GBDT regression algorithm for training to obtain the localization model.

[0029] Based on the above processing, four keypoints were obtained from images in different dimensions. These four keypoints are the rear shoulder keypoint, left shoulder keypoint, and right shoulder keypoint obtained from the RGB image, and the upper back keypoint obtained from the depth image using depth information. To achieve dataset unification, all data needs to be mapped to the same image. Since RGB images lack planar height information, and the resolution of the RGB lens is slightly higher than that of the depth lens, the three-dimensional coordinates are projected onto the two-dimensional RGB image coordinates, and all four keypoints are displayed on the two-dimensional RGB image. The coordinate transformation mainly includes two steps. First, the depth camera coordinate system is converted to the RGB camera coordinate system through rigid body rotation and translation. Then, the three-dimensional coordinates are reduced to two-dimensional coordinates according to the perspective projection relationship. The specific conversion formula is as follows.

[0030]

[0031] In the formula These are the three-dimensional coordinates in the depth camera coordinate system. These are two-dimensional coordinates in the RGB camera coordinate system. For rigid body rotation matrix, Rigid body translation vector; For RGB cameras Focal length of direction For RGB cameras Focal length in direction.

[0032] The model conversion formula is as follows:

[0033] In the formula and It's on the left shoulder. shaft and Key points of axis coordinates and yes shaft and Shoulder key points in axis coordinates, and yes and Coordinates of the key point on the right shoulder of the axis. and Suppose and The hip keypoints are located in axial coordinates, and the Kinect 2.0 camera is used to capture the height of the plane. This is the design. The coordinates of the hip keypoints in the point cloud are shown in Equation 3.2. The purpose of Equation 3.2 is to convert the coordinates of the 3D depth information captured by the depth camera into a format compatible with the 2D coordinate system of the RGB camera. This conversion effectively maps the coordinates of 3D keypoints in the depth image onto the 2D RGB image plane, allowing data from different sensors to be integrated and processed on the same image analysis platform. This conversion is necessary because it allows this embodiment to combine depth information with visual information from the RGB image.

[0034] This embodiment uses the GBDT algorithm to process the acquired four keypoint data and predict the positions of DU1 and DU3. This embodiment constructs a training dataset containing the coordinates of keypoints with known acupoint markings. This data originates from professional anthropometric measurements. The spatial coordinates of the four keypoints processed by Formula 3.3 are input into the GBDT algorithm. In GBDT, each tree corresponds to a decision process, where nodes represent tests on the keypoint coordinates, and data is assigned to the left and right subtrees based on the test results. For example, a tree might test whether the x-coordinate of the left shoulder keypoint is less than a certain threshold at the top-level node; based on the yes or no answer, the sample is sent to different branches. Each branch represents a further query on a specific subset of the dataset, gradually refining to the leaf nodes, i.e., the final key acupoint prediction.

[0035] The robot system lacks interoperability, and the keypoint coordinates obtained through regression models cannot be directly used in the collaborative robot coordinate system. Therefore, hand-eye calibration becomes a necessary prerequisite for subsequent massage operations. This embodiment uses an eye-hand calibration method, taking the calibrated chessboard as the feature object. After obtaining the coordinates of the chessboard corner points in two different coordinate systems, this embodiment uses the ICP alignment algorithm to solve for the alignment matrix. The rigid body transformation formula between the two coordinate systems is as follows:

[0036] In the formula For camera coordinate system coordinates, Let be the robot's coordinate system. From Formula 3.3, the coordinates of the key acupoints DU1 and DU3 in the real world can be obtained, denoted as follows: In virtual human systems, Three reference points can form a triangle, with side lengths denoted as AP, AB, and BP. The side lengths of the triangle can be calculated using the Euclidean distance formula. The distance.

[0037] Massage points in the real world can be determined based on the principle of triangle similarity. The position is calculated as shown in Formula 3.4.

[0038]

[0039] In terms of massage location intent reasoning, this invention first employs a Naive Bayes classifier. Naive Bayes was chosen because of its simple logic, stable algorithm, and relatively stable classification performance when the data exhibits different characteristics. Especially when the relationships between dataset attributes are relatively independent, the Naive Bayes classification algorithm demonstrates good performance. This embodiment improves the Naive Bayes classifier model by introducing dynamic information entropy. Here, information entropy refers to a measure of the degree of disorder, i.e., the degree of disorder in a single-modality recognition model. When the classification probabilities of the current modality are relatively similar, it indicates that the classification effect is unclear; while higher disorder leads to an increase in entropy, and vice versa. Therefore, this algorithm uses information entropy to weaken information sources with high disorder and emphasize information sources with low disorder, ultimately forming the classifier formula (3.5).

[0040]

[0041] In formula 3.5, The massage points are categorized into 10 body part names. For each current single mode The identified attribute values, among which The range of values ​​is the modal type. ,in These represent the gesture channel

[35] , finger pointing

[36] , candidate massage position, and voice channel, respectively. The first three are obtained through the visual channel, while the voice is obtained through the voice channel. For category The prior probability, For attributes The conditional probability.

[0042] Entropy value in formula (3.5) The calculation method is as follows:

[0043] This represents the degree of confusion in the identification of each channel, calculated using the entropy formula (3.6). It is a set. The elements in the set are the probability values ​​of each category in the modality i recognition result, and the length of the set is the number of categories for this modality; for example, the current gesture recognition set. Determine the gesture as a gesture , , The probability values ​​are respectively =0.18、 =0.61、 =0.21, then the set Then the entropy value of the gesture channel at the current moment can be obtained. This is similar to calculating the real-time dynamic entropy values ​​of the other four modes.

[0044] Equation (3.5) represents a Naive Bayes classifier incorporating dynamic information entropy. It's worth noting that the time complexity of Naive Bayes is O(kn), where n is the number of attributes, while the time complexity of the entropy formula is O(mc), where c is the number of channels. Therefore, the time complexity of Equation (3.5) is O(kn + mc). It is particularly noteworthy that Equation (3.5) employs two-dimensional storage, resulting in lower time and space complexity.

[0045] Meanwhile, this embodiment assumes that if one or more channels of data are missing at the current moment, the missing modalities can be excluded when the classifier calculates the probability. Experiments show that, because this algorithm uses multi-channel information to complement and verify each other, removing information from missing channels will not affect the inference of the user's correct intent. The following describes the process of incorporating a Naive Bayes classifier with dynamic entropy adjustment in the massage point localization algorithm. First, when the user is detected to be in place in the application scenario, the system is awakened to enter the information acquisition state. The input is the current RGB image and depth image. The position of the user selected by the virtual human body system is input into the virtual-real mapping algorithm, and the system segments the user's gesture information, finger pointing, skeletal point information, and key acupoint information. Next, the above content is input into the classifier for intent prediction, and the probability value of each intent is obtained. Then, the intent part with the highest probability is selected as the inference result at this moment.

[0046] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A multi-modal massage positioning method for human acupoints, characterized in that, The method comprises the following steps: Collecting multi-modal data of a user, including RGB images and depth images; Constructing a virtual human model and mapping the massage positions selected by the user on the virtual model to a real human body; Identifying key points of the human body, including shoulder key points and hip key points, based on the multi-modal data; Mapping the three-dimensional key points in the depth camera coordinate system to the RGB image coordinate system through coordinate transformation; Predicting the positions of key acupoints of the human body by using a machine learning algorithm; converting the predicted acupoint coordinates to the robot coordinate system through hand-eye calibration; Identifying the massage intention of the user based on a multi-modal intention understanding algorithm; Finally, outputting the massage position information to the massage robot to perform the massage operation.

2. The multi-modal massage positioning method for human acupoints according to claim 1, characterized in that, The specific steps of constructing the virtual human model and realizing the virtual-real mapping include: A virtual human model is constructed using Blender software, and a human-computer interaction function is realized through Block3d visual programming tool; a user selects a massage position point on the virtual human model , and the system records the point coordinates; by identifying the coordinates of the Dui point DU1 and the Yaoyangguan point DU3 on the virtual human body and , in combination with the point selected by the user , corresponding points on the real human body are calculated through the triangle similarity principle , and the calculation formula is as follows: ； wherein, , are the coordinates of DUs 1 and 3 on the real human body, respectively, , is the distance between the points in the virtual model.

3. The multi-modal massage positioning method for human acupoints according to claim 1, characterized in that, The key point identification specifically includes: Processing the RGB image through the OpenPose network to obtain the shoulder key points: the back shoulder key point, the left shoulder key point and the right shoulder key point; identifying the hip key points through the depth image of the Kinect2.0 camera, selecting the intersection of the highest points of the two hips and the back midline as the reference point; finally obtaining four key point data and inputting them to the GBDT regression algorithm for training to predict the positions of DU1 and DU3.

4. The multi-modal massage positioning method for human acupoints according to claim 1, characterized in that, The specific steps of the coordinate transformation are: Transforming a three-dimensional point in the depth camera coordinate system to a two-dimensional point in the RGB camera coordinate system by a rigid transformation matrix Transforming a three-dimensional point in the depth camera coordinate system to a two-dimensional point in the RGB camera coordinate system by a rigid transformation matrix The transformation formula is: ； wherein, is a rotation matrix, is a translation vector, , are the focal lengths of the RGB camera in and directions, respectively.

5. The multi-modal massage positioning method for human acupoints according to claim 1, characterized in that, The specific steps of the hand-eye calibration are: By calibrating the coordinates of the chessboard corners in the camera coordinate system and the robot coordinate system, the alignment matrix is solved using the ICP algorithm, and the transformation formula is: ； wherein, is the camera coordinate system coordinate, is the robot coordinate system coordinate.

6. The multi-modal massage positioning method for human acupoints according to claim 1, characterized in that, The multi-modal intention understanding algorithm uses an improved Naive Bayes classifier, introduces dynamic information entropy to optimize the classification effect, and the classifier formula is: ； wherein, is a massage part category, is a recognition result of the first modal, is an entropy value of the first modal, and the calculation formula is: ； wherein, is a first set of modal classification probabilities.

7. The multi-modal massage positioning method for human acupoints according to claim 6, characterized in that, The modalities include a gesture channel , a finger pointing , a candidate massage position and a speech channel , where the first three are acquired through a vision channel and the speech through a speech channel.

8. The multi-modal massage positioning method for human acupoints according to claim 1, characterized in that, It also includes automatically excluding a modal from participating in classification calculation when the modal data is missing.

9. A multi-modal massage positioning system for human acupoints, for implementing the multi-modal massage positioning method for human acupoints according to any one of claims 1-8, characterized in that, It comprises: A multi-modal data acquisition module for obtaining RGB images and depth images; A virtual mapping module for constructing a virtual human body and realizing user interaction; A key point identification module for extracting shoulder and hip key points; A coordinate conversion module for mapping depth coordinates to the RGB image plane; An acupoint positioning module for predicting acupoint positions through the GBDT algorithm; A hand-eye calibration module for converting coordinates to the robot coordinate system; An intention understanding module for identifying user intentions through a dynamic entropy Bayesian classifier; A control output module for driving the massage robot to perform the massage operation.

10. The multi-modal massage positioning system for human acupoints according to claim 9, wherein, It also includes a user interface module that supports users to select massage positions on the virtual human body through mobile phones and tablet devices, and to display the intention recognition results and massage execution status in real time.

Images