Self-adaptive massage robot system and method based on visual identification and three-dimensional modeling

Abstract

The invention provides a self-adaptive massage robot system and method based on visual identification and three-dimensional modeling, and the system comprises a three-dimensional scanning module which is used for obtaining a three-dimensional model of a body part of a user; the visual identification module is used for identifying a visual mark area preset by a user or a doctor on the body part of the user; and the central processing unit is connected with the three-dimensional scanning module and the visual identification module, and is used for fusing the visual mark area with the three-dimensional model to determine a massage target area and generate a massage track based on the massage target area. According to the intelligent massage system, professional judgment of doctors, real-time feedback of users and accurate execution of machines are combined, an intelligent massage solution with man-machine cooperation and adaptive optimization is formed, and unification of precision, personalization and safety is effectively achieved.

Description

Technical Field

[0001] This invention relates to an adaptive massage robot system and method based on visual recognition and 3D modeling. Background Technology

[0002] With societal development, health problems such as muscle strain and chronic pain are becoming increasingly common, leading to a continuous growth in market demand for professional and personalized massage and physiotherapy equipment. Against this backdrop, massage robot technology has emerged, aiming to simulate the massage techniques of professional therapists through automated equipment, thereby reducing labor costs and improving service consistency.

[0003] However, existing massage robot systems still suffer from several key technological bottlenecks, which severely restrict their application effectiveness and user experience, such as; Most existing systems rely on pre-programmed, fixed massage trajectories and points, which cannot adapt to the varying muscle strain areas (i.e., "trigger points") of different users. Although some studies have attempted to find stiff areas through pressure sensor feedback, this method is essentially "blind exploration," inefficient, and has limited positioning accuracy. It cannot "see" and confirm the specific strain target area before operation, just like a human doctor. The massage depth (intensity) of existing devices is usually a fixed value or a few simple preset levels, unable to be dynamically adjusted according to the user's real-time, subjective comfort. Since different users, and even different parts of the same user's body, have vastly different tolerances to pressure, this rigid intensity control mode is prone to insufficient intensity and poor effect, or excessive pressure. Excessive force can cause soft tissue damage, posing significant safety hazards and resulting in a poor user experience. The human body, particularly the back, has complex curves, and the optimal massage posture varies depending on the specific muscle groups being exposed and relaxed. Existing robotic systems lack intelligent posture guidance based on target area location, making it difficult for their robotic arm trajectories to achieve optimal fit with the human body's curves in all postures, thus diminishing the massage effect. Current technology heavily relies on manual teaching or simple labeling, lacking the intelligent diagnostic capabilities to proactively analyze and identify physiological abnormalities. The system cannot serve as a professional auxiliary diagnostic tool, providing operators with references to strained areas based on objective imaging evidence. Its function remains at the "execution" level, rather than being an integrated "perception-decision-execution" intelligent system. Summary of the Invention

[0004] This invention provides an adaptive massage robot system and method based on visual recognition and 3D modeling, which can effectively solve the above-mentioned problems.

[0005] This invention is implemented as follows: An adaptive massage robot system based on visual recognition and 3D modeling, including The 3D scanning module is used to acquire 3D models of the user's body parts; The visual recognition module is used to identify visually marked areas on the user's body that are preset by the user or doctor. The central processing unit, connected to the 3D scanning module and the visual recognition module, is used to fuse the visually marked area with the 3D model to determine the massage target area and generate a massage trajectory based on the massage target area. A massage execution module, connected to the central processing unit, is used to perform massage operations according to the massage trajectory; The user feedback module is used to receive real-time feedback signals on the user's massage depth. The central processing unit adjusts the massage depth of the massage execution module in real time based on the feedback signal received by the user feedback module.

[0006] An adaptive massage method based on visual recognition and 3D modeling includes the following steps: S1: Obtain a preliminary 3D model of the user's body parts through the 3D scanning module; S2: Identify visually marked areas on the user's body parts and / or acquire visual images of the body parts through the visual recognition module; S3: Merge the visually marked area with the preliminary three-dimensional model to determine the massage target area; and / or, analyze the appearance image through an AI recognition model to automatically identify the suspected strain area as the massage target area; S4: Generate a massage trajectory based on the massage target area; S5: Control the massage execution module to perform massage operations according to the massage trajectory and based on an initial massage depth; S6: During the massage, it receives real-time feedback signals from the user and dynamically adjusts the massage depth based on these signals.

[0007] The beneficial effects of this invention are: This invention achieves precise and objective spatial positioning of personalized massage target areas through the fusion of 3D scanning and visual recognition. Simultaneously, the system introduces a closed-loop depth adjustment mechanism based on real-time user feedback, enabling the massage intensity to dynamically adapt to the user's subjective feelings and tolerance, fundamentally improving the safety and comfort of the operation. Ultimately, the system combines the doctor's professional judgment, the user's real-time feedback, and the machine's precise execution to form a human-machine collaborative, adaptive, and optimized intelligent massage solution, effectively achieving a unity of precision, personalization, and safety. Attached Figure Description

[0008] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0009] Figure 1 This is a system overall structure block diagram of the present invention.

[0010] Figure 2 This is the overall flowchart of the method of the present invention.

[0011] Figure 3 This is a flowchart of the visual recognition and target area determination process of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0013] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0014] Reference Figure 1-3 As shown, the adaptive massage robot system based on visual recognition and 3D modeling includes... The 3D scanning module is used to acquire 3D models of the user's body parts. It should be noted that this module can use any of the following: a structured light camera (such as the Intel RealSense D415), a binocular stereo vision system (such as the ZED 2 camera), or a time-of-flight (ToF) camera. This application does not impose any restrictions, and the appropriate option can be selected based on specific needs.

[0015] Specifically, the system controls the camera to capture images of the user's back from multiple angles (such as rotating around the user or rotating on the user's own body). Dense 3D point cloud data is generated using the SFM (Structure from Motion) algorithm or the camera's built-in SDK. Subsequently, the point cloud is converted into a triangular mesh model that can be used for path planning using the Poisson reconstruction algorithm (existing technology, which will not be elaborated on here). At the same time, it is necessary to associate the acquired RGB color textures and map them onto the 3D model to form a colored 3D model.

[0016] A visual recognition module is used to identify visually marked areas on the user's body that have been preset by the user or doctor. This module also acquires visual images of the body parts. The central processing unit has a built-in trained AI recognition model that analyzes the visual images to identify physiological characteristics related to muscle strain. Based on these physiological characteristics, it automatically marks suspected strain areas on the 3D model. It should be noted that the visual recognition module can be implemented using a high-definition USB camera mounted at the end of a robotic arm.

[0017] The process involves converting the acquired RGB image to the HSV (Hue, Saturation, Lightness) color space (HSV space is more suitable for color-based segmentation). Simultaneously, based on the specific paint color used (e.g., bright fluorescent green), a corresponding threshold range is set in the HSV space (e.g., H: [40, 80], S: [100, 255], V: [100, 255]). The binary mask of the marked region can be extracted using the cv2.inRange() function (e.g., using the OpenCV library). A contour-finding algorithm is then used on the binary mask to obtain the pixel-level boundaries of the marked region. Finally, using camera calibration parameters and hand-eye calibration techniques, these two-dimensional pixel coordinates are transformed into three-dimensional spatial coordinates in the robot's base coordinate system.

[0018] Furthermore, in implementation, the real-time RGB images captured by the end-effector camera of the robotic arm are first preprocessed using the same methods as during training (size normalization, pixel value normalization). The preprocessed images are then fed into the deployed, trained U-Net model. After forward propagation, the model outputs a probability value (between 0 and 1) for each pixel belonging to the "strained area". Next, a threshold (e.g., 0.5) is set for the model's output probability map to generate a binary mask. Then, morphological operations (such as opening and closing operations) are used to remove small noise points and fill small holes inside the area, resulting in a smooth and connected strained area mask. Finally, a contour extraction algorithm is used to obtain the precise boundary of the area. Finally, through the previously completed hand-eye calibration, the transformation matrix from the camera coordinate system to the robot's base coordinate system can be obtained. Combined with the camera intrinsic parameters, the contour points of the strained area in the two-dimensional image can be transformed into three-dimensional space through back projection. Subsequently, these three-dimensional point clouds are registered with the human body mesh model obtained through three-dimensional scanning, thereby accurately marking the "massage target area" on the three-dimensional model.

[0019] It is important to note that hand-eye calibration is mandatory upon initial system installation or when the robotic arm / camera position changes. Specifically, a standard calibration method using either "eye outside the hand" or "eye on the hand" is employed. A calibration board of known size (e.g., a checkerboard) is used. By moving the robotic arm to multiple poses, multiple sets of images and corresponding robotic arm base coordinates are acquired. A fixed transformation matrix between the camera coordinate system and the robotic arm base coordinate system is then calculated. During coordinate mapping, for each contour point (u, v) identified in the image, its 3D coordinates P_cam in the camera coordinate system are calculated using the formula P_cam = d * K^{-1} * [u, v, 1]^T, combined with its corresponding depth map or depth value d calculated through binocular vision. The camera intrinsic parameter matrix K is used to transform P_cam to the robot's base coordinate system using the hand-eye calibration matrix. Subsequently, this is transformed to the robot's base coordinate system using the hand-eye calibration matrix. Finally, these 3D point clouds are precisely registered with the previously obtained 3D scanned human body mesh model using the Iterative Closest Point (ICP) algorithm, thus marking the target area on the 3D model.

[0020] The physiological characteristics associated with muscle strain include at least one of the following: localized changes in skin color, localized capillary distribution, abnormal muscle contour morphology, and differences in body surface temperature distribution; specifically: The scientific principle behind localized changes in skin color: Chronic muscle strain or fasciitis can lead to poor local blood circulation and accumulation of metabolic products, causing aseptic inflammation. This may manifest on the skin surface as: dullness, bluishness, or purplishness: indicating potential blood stasis and obstructed venous return from capillaries; localized flushing or redness: indicating potential acute inflammatory response and congestion at the arterial end of capillaries. AI models can quantify these color differences by analyzing the hue and saturation values ​​of specific regions in the HSV color space of RGB images. For example, the model can learn that the statistical distribution (such as mean and variance) of pixels in the H and S channels of strained areas differs significantly from that of normal skin areas.

[0021] The scientific principle behind the distribution of local capillaries: The density and morphology of capillaries near trigger points (exhaustion points) may change. Visually, this may manifest as: telangiectasia: fine, reddish, thread-like lines appear on the skin surface; subcutaneous petechiae or ecchymoses: red blood cells leak out due to capillary rupture, forming red or purplish-red spots on the skin. For images captured by high-definition cameras, algorithms such as image sharpening and edge enhancement can highlight these subtle texture features. AI models (especially CNN models, which excel at capturing texture) can learn the unique texture patterns formed by these dilated capillaries or petechiae, thus distinguishing them from normal skin.

[0022] The scientific principle behind abnormal muscle contours: Excessive tension, spasm, or fibrosis of muscles can lead to changes in their physical shape. Common physical manifestations include: muscle depressions: caused by the tension of muscle fibers pulling on surrounding tissues; localized swelling or bulges: due to inflammatory exudation or compensatory muscle hypertrophy; and abnormal contours resembling "cords" or "nodules." Using a mesh model obtained through a 3D scanning module, the system can calculate the curvature map of the human back. In areas of strain, the Gaussian curvature or average curvature will differ significantly from the surrounding smooth areas (e.g., negative curvature in depressions and positive curvature in bulges). The AI ​​model can combine RGB texture and 3D geometric curvature features to jointly identify these subtle morphological abnormalities.

[0023] The scientific principle behind the differences in body surface temperature distribution: Changes in metabolic activity and blood circulation in the strained area lead to a difference in surface temperature between it and surrounding tissues. In most cases, chronically strained areas appear as low-temperature zones due to poor blood flow, while acute inflammatory areas may appear as high-temperature zones. An infrared thermal imaging camera can be integrated as part of a visual recognition module. This camera can directly measure the body surface temperature and generate a thermal map. An AI model can analyze this thermal map, automatically segmenting continuous areas with significant temperature differences from the surrounding environment (e.g., a temperature difference exceeding 0.5°C) and using these as important indicators of suspected strained areas.

[0024] In general, a single feature may not be sufficient for a diagnosis, but multi-feature fusion can greatly improve the reliability of identification. For example, an area that simultaneously exhibits dark color (feature 1), a palpable cord-like texture (feature 3), and a low local temperature (feature 4) is a strong indicator of chronic muscle strain. The AI ​​model of this invention can process these multimodal features (color images, 3D geometry, thermal imaging data) simultaneously or selectively. By fusing these features for decision-making, the system can make comprehensive observations and judgments like an experienced doctor, thereby highlighting the most probable suspected strain areas on the 3D model, providing doctors with intuitive auxiliary diagnostic information, or providing precise target areas for automated massage.

[0025] The central processing unit, connected to the 3D scanning module and the visual recognition module, is used to fuse the visually marked area with the 3D model to determine the massage target area and generate a massage trajectory based on the massage target area. The central processing unit also includes a posture guidance module, which outputs corresponding posture adjustment instructions to the user based on the position information of the massage target area on the body part. It should be noted that the central processing unit is a software program running on a local computer (such as a high-performance industrial control computer) or an embedded system.

[0026] Specifically, during implementation, the 3D marker point set obtained by the visual recognition module and the mesh model obtained by the 3D scanning module are precisely registered using the Iterative Closest Point (ICP) algorithm, thereby accurately marking the "massage target area" in the 3D model, and employing the isoparametric line method or scan line method. That is, on the 3D curved surface (massage target area), a series of path lines (V direction) are generated at fixed intervals along a parametric direction (e.g., U direction). To keep the massage head vertical, the normal vector of each path point needs to be calculated, and the posture of the massage head is aligned with this normal vector. The series of path lines includes a sequence of path points with positions (X, Y, Z) and postures (Rx, Ry, Rz). The posture guidance module is a lookup table stored internally by the system. It divides the human back into several regions (e.g., "left scapula area," "lower back"), and associates each region with a preset posture instruction (e.g., "Please cross your left arm across your chest and place it on your right shoulder"). The user is guided through the UI interface (screen animation) and TTS (text-to-speech) system. The central processing unit adjusts the massage depth based on the feedback signal in the following ways: it initializes a preset conservative massage depth; after receiving an incrementing signal triggered by the user, it controls the massage execution module to gradually increase the massage depth in preset steps.

[0027] The central processing unit also records and stores the optimal massage depth for a specific user and / or a specific massage target area, and calls this depth as the initial value in subsequent massages.

[0028] The massage execution module, connected to the central processing unit, is used to perform massage operations according to the massage trajectory. Specifically, before generating the massage trajectory, the system calls a preset human anatomy restricted area library. This library defines three-dimensional spatial areas where strong pressure should not be applied, such as the spinous processes of the spine, the edges of the scapula, and the kidney region. In the trajectory planning algorithm, it first checks whether the planned path points collide with or are too close to these restricted areas. If so, the point is automatically removed from the trajectory or its pressure value is set to zero, ensuring that the robotic arm only performs pressureless gliding movements when approaching these areas, thereby eliminating the possibility of injury to the user.

[0029] The user feedback module is used to receive real-time feedback signals on the user's massage depth; the user feedback module can be a handheld physical button, a virtual control on a touch screen, or a voice recognition unit.

[0030] The central processing unit adjusts the massage depth of the massage execution module in real time based on the feedback signal received by the user feedback module.

[0031] It should be noted that the feedback device can be a simple wireless remote control with only a "+" button. Specifically, during system initialization, the massage depth is set to 5 mm (a conservative depth that only contacts the skin). The robotic arm begins to move along the trajectory. When the user feels the pressure is insufficient, they press the "+" button. The system receives an interrupt signal, and the main program increases the current massage depth variable by a preset step, for example, depth = current depth + 1 mm. The robotic arm controller receives the new path point with the increased depth and immediately replans the trajectory, using the new depth at the next path point.

[0032] Furthermore, in addition to the "+" button, the user feedback module also has a prominent and easily accessible "emergency stop" button. Pressing this button will send a high-priority interrupt signal to the central processing unit, immediately stopping all movements of the robotic arm. In addition, an absolute safety depth threshold (e.g., 30mm) is set. This threshold is fixed at the software level, and no adjustment command can make the massage depth exceed this limit, forming an independent software safety lock.

[0033] It is worth noting that for a specific area of ​​the user (accurately matched by the region code), if there is an "optimal massage depth" in the history, the value is directly used as the initial depth. If the area is new, but a personalized massage strategy model has been trained for the user, the depth predicted by the model is used as the initial value. If neither of the above is met, the conservative massage depth set globally by the system (such as 5mm) is used.

[0034] An adaptive massage method based on visual recognition and 3D modeling includes the following steps: S1: Obtain a preliminary 3D model of the user's body parts through the 3D scanning module; S2: Identify visually marked areas on the user's body parts and / or acquire visual images of the body parts through the visual recognition module; S3: Merge the visually marked area with the preliminary three-dimensional model to determine the massage target area; and / or, analyze the appearance image through an AI recognition model to automatically identify the suspected strain area as the massage target area; S4: Generate a massage trajectory based on the massage target area; S5: Control the massage execution module to perform massage operations according to the massage trajectory and based on an initial massage depth; S6: During the massage, it receives real-time feedback signals from the user and dynamically adjusts the massage depth based on these signals.

[0035] After step S4 and before step S5, step S4a is also included: based on the position of the massage target area, output posture adjustment guidance to the user and confirm that the user's posture is adjusted in place.

[0036] In step S4a, after outputting the posture guidance, the system initiates a confirmation process: a "I have finished adjusting" button is displayed on the UI interface, and the user clicks to confirm. To achieve higher automation, a lightweight posture verification can be introduced: the new posture of the user is quickly scanned by the 3D scanning module and compared with the standard 3D model of the posture stored in the posture library (such as calculating the similarity of key joint angles). When the similarity exceeds the threshold (such as 85%), it is automatically determined that the adjustment is in place. Otherwise, the system will provide more detailed guidance (such as "Please bend forward 10 degrees").

[0037] In step S3, when the target area is obtained simultaneously through two channels, the system defaults to giving the manually marked area the highest priority, as it represents the doctor's professional judgment. Simultaneously, the system can provide an intelligent fusion mode: performing a union or intersection operation on the AI-marked area and the manually marked area. For example, in assisted diagnosis mode, the AI-marked area can serve as a prompt, assisting the doctor in confirming or expanding the marking range, with the doctor's final confirmation of the marking taking precedence. This clarifies the responsibilities of human-machine collaboration and avoids potential decision-making conflicts.

[0038] After multiple massages, the central processing unit generates a personalized massage strategy model for the user based on recorded user feedback data and corresponding massage parameters using a machine learning model. This personalized massage strategy model is used to predict the user's preferred comfort depth for unmasked areas. Specifically, after each massage, the system automatically records three sets of key data: "user physical characteristics" (such as height, weight, BMI), "massage context" (such as the coding of the massaged area), and "feedback result" (the user's final confirmed comfort depth). When the data accumulates to a certain scale, the system uses this historical data to train a personalized regression prediction model (such as gradient boosting decision tree or support vector regression). This model learns the hidden mapping relationship between the user's specific physical characteristics and the comfort depth of different areas. The final effect is that when the user uses the system again and prepares to massage a new area that has never been massaged before, the system can proactively predict the most likely comfortable massage depth as an initial value based on the user's physical characteristics and the coding of the new area. This greatly reduces the tedious steps of repeated adjustments and realizes personalized intelligent service.

[0039] It should be noted that all user data is stored locally and encrypted, or uploaded to the cloud platform after obtaining explicit authorization from the user. Personalized models are isolated using the user ID as the primary key to ensure privacy. Model updates can be triggered on a scheduled basis (e.g., weekly) or by events (e.g., when the number of records increases by 50). During updates, the model is retrained with full data or uses a more efficient online learning algorithm to combine the new data with the old model for incremental updates, so as to continuously optimize prediction accuracy.

[0040] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. An adaptive massage robot system based on visual recognition and 3D modeling, characterized in that, include The 3D scanning module is used to acquire 3D models of the user's body parts; The visual recognition module is used to identify visually marked areas on the user's body that are preset by the user or doctor. The central processing unit, connected to the 3D scanning module and the visual recognition module, is used to fuse the visually marked area with the 3D model to determine the massage target area and generate a massage trajectory based on the massage target area. A massage execution module, connected to the central processing unit, is used to perform massage operations according to the massage trajectory; The user feedback module is used to receive real-time feedback signals on the user's massage depth. The central processing unit adjusts the massage depth of the massage execution module in real time based on the feedback signal received by the user feedback module.

2. The adaptive massage robot system based on visual recognition and 3D modeling according to claim 1, characterized in that, The central processing unit also includes a posture guidance module, which outputs corresponding posture adjustment instructions to the user based on the position information of the massage target area on the body part.

3. The adaptive massage robot system based on visual recognition and 3D modeling according to claim 1, characterized in that, The visual recognition module is also used to acquire the appearance image of the body part; the central processing unit has a built-in trained AI recognition model, which analyzes the appearance image to identify physiological features related to muscle strain, and automatically marks the suspected strain area on the three-dimensional model based on the physiological features.

4. The adaptive massage robot system based on visual recognition and 3D modeling according to claim 3, characterized in that, The physiological characteristics associated with muscle strain include at least one of the following: localized changes in skin color, localized capillary distribution, abnormal muscle contour morphology, and differences in body surface temperature distribution.

5. The adaptive massage robot system based on visual recognition and 3D modeling according to claim 1, characterized in that, The user feedback module can be a handheld physical button, a virtual control on a touchscreen, or a voice recognition unit.

6. The adaptive massage robot system based on visual recognition and 3D modeling according to claim 1, characterized in that, The specific method by which the central processing unit adjusts the massage depth based on feedback signals is as follows: Initialize a preset conservative massage depth; upon receiving an incrementing signal triggered by the user, control the massage execution module to gradually increase the massage depth in preset steps.

7. The adaptive massage robot system based on visual recognition and 3D modeling according to claim 6, characterized in that, The central processing unit also records and stores the optimal massage depth for a specific user and / or a specific massage target area, and calls this depth as the initial value in subsequent massages.

8. An adaptive massage method based on any of the systems described in claims 1-7, characterized in that, Includes the following steps: S1: Obtain a preliminary 3D model of the user's body parts through the 3D scanning module; S2: Identify visually marked areas on the user's body parts and / or acquire visual images of the body parts through the visual recognition module; S3: Merge the visually marked area with the preliminary three-dimensional model to determine the massage target area; and / or, analyze the appearance image through an AI recognition model to automatically identify the suspected strain area as the massage target area; S4: Generate a massage trajectory based on the massage target area; S5: Control the massage execution module to perform massage operations according to the massage trajectory and based on an initial massage depth; S6: During the massage, it receives real-time feedback signals from the user and dynamically adjusts the massage depth based on these signals.

9. The adaptive massage method based on visual recognition and 3D modeling according to claim 8, characterized in that, After step S4 and before step S5, step S4a is also included: based on the position of the massage target area, output posture adjustment guidance to the user and confirm that the user's posture is adjusted in place.

10. The adaptive massage method based on visual recognition and 3D modeling according to claim 8, characterized in that, After multiple massages, the central processing unit generates a personalized massage strategy model for the user based on the recorded user feedback data and corresponding massage parameters through a machine learning model. The personalized massage strategy model is used to predict the user's preference for comfort depth in unmasked areas.

Images