Massage robot massage scene visualization method and device based on virtual reality

Abstract

The invention provides a massage robot massage scene visualization method and device based on virtual reality, and the method comprises the steps: obtaining first images of a real massage scene at multiple angles, and generating a virtual massage scene image based on the first images of the real massage scene at multiple angles; obtaining a second image of a massage user at multiple angles in the real massage scene, and obtaining a human body three-dimensional model image of the massage user through a trained human body generation model based on the second image of the massage user at multiple angles; and sending the virtual massage scene image and the human body three-dimensional model image to virtual reality equipment, so that the virtual reality equipment displays the virtual massage scene image and the human body three-dimensional model image to an operation user. According to the method, remote visualization of the massage scene can be realized in the remote massage process, and the massage effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote massage, and in particular to a method and device for visualizing a massage scene of a massage robot based on virtual reality. Background Art

[0002] With the progress of modern civilization, robots have become well known and recognized by people. Robots have moved from narrow special fields such as industry to service fields closely related to humans. In recent years, service robots, as a member of the robot family, have developed rapidly. Various types of service robots such as medical, rehabilitation, cleaning, construction, and disaster relief have entered our lives.

[0003] At present, massage has become the only choice for many people to maintain health, relax and have fun. However, with the development of science and technology, many traditional industries have been replaced by robots, and the massage industry is no exception. Massage robots that integrate high technology have also entered people's lives. However, when massaging, existing massage robots use pre-set massage programs to achieve massage. It is impossible for operators to remotely operate the massage robots to perform targeted massages on massage users, and it is even more impossible to achieve a visual display of the massage scene during the remote massage process, resulting in poor massage effects. Therefore, during the remote massage process, how to achieve remote visualization of the massage scene and how to improve the massage effect are technical problems that need to be solved urgently. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for visualizing a massage scene of a massage robot based on virtual reality to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present invention provides a method for visualizing a massage scene of a massage robot based on virtual reality, the method comprising:

[0006] Acquire first images of a real massage scene at multiple angles, and generate a virtual massage scene image based on the first images of the real massage scene at multiple angles;

[0007] Acquire second images of the massage user at multiple angles in the real massage scene, and obtain a three-dimensional human body model image of the massage user based on the second images of the massage user at multiple angles by using a trained human body generation model;

[0008] The virtual massage scene image and the human body three-dimensional model image are sent to a virtual reality device, so that the virtual reality device displays the virtual massage scene image and the human body three-dimensional model image to an operating user.

[0009] In some embodiments of the present invention, obtaining second images of the massaging user at multiple angles in the real massage scene includes:

[0010] Based on the image capturing device at the end of the mechanical arm of the massage robot in the real massage scene, second images of the massage user at multiple angles are acquired.

[0011] In some embodiments of the present invention, obtaining second images of the massage user at multiple angles based on an image capture device at the end of the mechanical arm of the massage robot in the real massage scene includes:

[0012] Determining a shooting motion path of a mechanical arm of the massage robot;

[0013] Determine the height data and posture data of the image capturing device;

[0014] Based on the shooting motion path, height data and posture data, shooting a posture video of the massage user by the image shooting device;

[0015] Second images of the massaging user at multiple angles are determined based on the gesture video.

[0016] In some embodiments of the present invention, determining the shooting motion path of the mechanical arm of the massage robot includes:

[0017] Determine multiple motion path points of the robotic arm;

[0018] Based on the multiple motion path points, the shooting motion path of the mechanical arm of the massage robot is obtained by an interpolation polynomial calculation method.

[0019] In some embodiments of the present invention, obtaining a human body three-dimensional model image of the massage user from a trained human body generation model based on the second images of the massage user at multiple angles includes:

[0020] The human body generation model determines shooting parameters corresponding to each second image, wherein the shooting parameters include internal parameters and external parameters;

[0021] Calculate the distance information between each pixel in the second image and the image capturing device based on the capturing parameters;

[0022] A three-dimensional human body model image of the massage user is generated based on the shooting parameters and the distance information.

[0023] In some embodiments of the invention, the method comprises:

[0024] The thermal imaging data of the massage user is obtained, and the thermal imaging data is sent to a virtual reality device, so that the virtual reality device displays the thermal imaging data to the operating user.

[0025] In some embodiments of the present invention, the posture video includes a plurality of videos captured by the image capturing device at different height data.

[0026] In some embodiments of the present invention, the method further comprises:

[0027] Determine a sample data set and an initial human body generation model, wherein the sample data in the sample data set includes multi-angle 2D sample images and human body three-dimensional model sample images;

[0028] The human body generation model is pre-trained based on the sample data set to obtain a trained human body generation model.

[0029] According to another aspect of the present invention, a virtual reality-based massage robot massage scene visualization system is also disclosed, the system comprising a processor, a memory and a computer program stored in the memory, the processor is used to execute the computer program, and when the computer program is executed, the system implements the steps of the method described in any of the above embodiments.

[0030] According to yet another aspect of the present invention, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0031] The virtual reality-based massage scene visualization method of a massage robot of the present application generates a virtual massage scene image based on first images of multiple angles of a real massage scene, and obtains a three-dimensional human body model image of the massage user through a trained human body generation model based on second images of multiple angles of the massage user, and finally sends the virtual massage scene image and the three-dimensional human body model image to a virtual reality device to be displayed to an operating user through the virtual reality device, so that the operating user can remotely control the massage robot based on the content displayed in the virtual reality device to achieve remote massage; this method enables the operator to remotely operate the massage robot to perform targeted massage on the massage user, and can also achieve remote visualization of the massage scene during the remote massage process, thereby improving the massage effect.

[0032] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0033] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. The components in the drawings are not drawn to scale, but are only for illustrating the principles of the present invention. In order to facilitate the illustration and description of some parts of the present invention, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present invention. In the drawings:

[0035] Figure 1 This is a flow chart of a method for visualizing a massage scene of a massage robot based on virtual reality according to an embodiment of the present application.

[0036] Figure 2 Schematic diagram of the process of generating a human body three-dimensional model image according to an embodiment of the present application.

[0037] Figure 3 A schematic diagram of coordinate relationships of a massage scene visualization method according to an embodiment of the present application.

[0038] Figure 4 This is a flow chart of a method for visualizing a massage scene of a massage robot based on virtual reality according to another embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0040] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0041] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0042] It should also be noted here that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection with an intermediary, not only to a wired connection, but also to a wireless connection, and the specific connection can be changed based on the actual application scenario.

[0043] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0044] Figure 1 FIG. 1 is a flow chart of a method for visualizing a massage scene of a massage robot based on virtual reality according to an embodiment of the present application. Figure 1 As shown, the massage robot massage scene visualization method at least includes steps S10 to S30.

[0045] Step S10: Acquire first images of a real massage scene at multiple angles, and generate a virtual massage scene image based on the first images of the real massage scene at multiple angles.

[0046] In this step, pictures of the massage scene can be collected based on multiple image acquisition devices of the real massage scene. The image acquisition device can be exemplarily a camera, and the camera can be specifically fixed on a camera bracket at the scene, and the camera bracket can be further fixed on the ground or on the massage bed. In order to enable the operating user to better observe the state of the massage user remotely, the first images of the real massage scene at multiple angles can be taken by multiple cameras; it can be understood that the first images of the real massage scene at multiple angles in this step refer to the image of the entire massage scene, and the massage scene includes the massage bed, the massage user, the scene environment, and the massage robot.

[0047] Step S20: obtaining second images of the massage user at multiple angles in the real massage scene, and obtaining a three-dimensional human body model image of the massage user based on the second images of the massage user at multiple angles by using a trained human body generation model.

[0048] When the massage user is performing a massage, the massage user can lie on the massage bed. Although the first images of the massage scene at multiple angles obtained based on the above steps can observe the status of the robotic arm and the patient, the status of the massage user displayed in the first images at multiple angles in step S10 is not intuitive. Therefore, this step further obtains the second images of the massage user at multiple angles, and further generates a three-dimensional human body model image of the massage user. The second images of the massage user at multiple angles obtained in this step are different from the images in step S10 in that the second images obtained in this step mainly express the massage user content, and the second images do not include content such as the robotic arm. Exemplarily, the second images of the massage user at multiple angles may specifically include images of the massage user taken from various angles, such as images directly above the massage user, images on the left, and images on the right, etc.

[0049] Similarly, the second images of the massage user at multiple angles in the real massage scene can be obtained by capturing the second images at multiple angles by an image capturing device. In one embodiment, obtaining the second images of the massage user at multiple angles in the real massage scene may specifically include: obtaining the second images of the massage user at multiple angles by an image capturing device at the end of the mechanical arm of the massage robot in the real massage scene. In this embodiment, an image capturing device is fixed at the end of the mechanical arm, and the image capturing device may be exemplarily a camera. In this embodiment, by moving the mechanical arm, the second images of the massage user at different angles may be captured by the camera at the end of the mechanical arm.

[0050] In one embodiment, the image capture device at the end of the mechanical arm of the massage robot in the real massage scene obtains the second image of the massage user at multiple angles, which may specifically include: determining the shooting motion path of the mechanical arm of the massage robot; determining the height data and posture data of the image capture device; shooting the posture video of the massage user through the image capture device based on the shooting motion path, height data and posture data; and determining the second image of the massage user at multiple angles based on the posture video. In this embodiment, the posture video of the massage user is shot by moving the mechanical arm, and then the second image of the massage user at multiple angles can be generated based on the posture video. Exemplarily, when the massage user lies on the massage bed, the height data of the image capture device can be understood as the height distance between the image capture device and the massage bed or the massage user, and the posture data of the image capture device can be understood as the shooting angle of the camera during the shooting process, such as the camera's viewing angle change range during the shooting process can be set to no more than 30 degrees. In addition, in order to obtain a complete posture video of the massage user, this embodiment can set the initial position and the end position of the image capture device to be the same, that is, the image capture device at the end of the robotic arm starts from the initial position and moves along the shooting motion path for one circle until the image capture device moves to its initial position, thus completing the shooting of the posture video.

[0051] Exemplarily, the shooting motion path of the robotic arm can be set based on the contour curve of the human body. At this time, the image capture device moves along the shooting motion path generated based on the contour curve of the human body. When the image capture device moves from the initial position to the end position, the shooting of the massage user's posture video is completed. In order to ensure the integrity of the captured posture video, the end position is the same as the initial position.

[0052] Further, determining the shooting motion path of the mechanical arm of the massage robot includes: determining a plurality of motion path points of the mechanical arm; obtaining the shooting motion path of the mechanical arm of the massage robot by an interpolation polynomial calculation method based on the plurality of motion path points. In this embodiment, the plurality of motion path points can be determined in advance based on the contour curve of the massage user, or can be determined in advance based on the size of the massage bed, or can be determined based on the pre-set lying position of the massage user. For example, in order to achieve smooth operation of the mechanical arm, a fifth-order interpolation polynomial method can be used for trajectory planning; when determining the shooting motion path, first determine a series of motion path points distributed along the human body curve, and assign appropriate time points to every two adjacent path points. By using the fifth-order polynomial interpolation formula, the smooth trajectory of the mechanical arm is calculated under the premise of ensuring the continuity of the initial displacement, velocity, acceleration and the target displacement, velocity, and acceleration. This interpolation method not only ensures the continuity of displacement, velocity, and acceleration, but also avoids sudden changes in velocity or acceleration of the mechanical arm, and ensures the smooth transition of the camera between path points.

[0053] Step S30: sending the virtual massage scene image and the human body three-dimensional model image to a virtual reality device, so that the virtual reality device displays the virtual massage scene image and the human body three-dimensional model image to an operating user.

[0054] In this step, the virtual massage scene image determined in step S10 and the human body three-dimensional model image determined in step S20 are displayed to the operating user through the virtual reality device, so that the operating user can intuitively observe the state of the massage user when performing remote massage on the massage user. The operating user can specifically be a masseur, a doctor, etc., and the massage user is correspondingly a person receiving massage, such as a patient, etc.

[0055] In some embodiments of the present invention, the human body three-dimensional model image of the massage user is obtained through a trained human body generation model based on the second images of the massage user at multiple angles, including: the human body generation model determines the shooting parameters corresponding to each of the second images, and the shooting parameters include internal parameters and external parameters; based on the shooting parameters, the distance information between each pixel point in the second image and the image shooting device is calculated; based on the shooting parameters, the human body three-dimensional model image of the massage user is generated. In this embodiment, the second images of the massage user at multiple angles are input into the trained human body generation model to obtain the human body three-dimensional model image of the massage user. In order to obtain the trained human body generation model, in one embodiment, the massage scene visualization method of the massage robot based on virtual reality also includes the following steps: determining a sample data set and an initial human body generation model, the sample data in the sample data set includes multi-angle 2D sample images and human body three-dimensional model sample images; pre-training the human body generation model based on the sample data set to obtain a trained human body generation model.

[0056] Exemplarily, the human body generation model can be a NeRF model combined with the instant-ngp algorithm, that is, this embodiment realizes real-time ray tracing and high-fidelity three-dimensional scene reconstruction based on the trained NeRF model. NeRF, the full name of which is Neural Radiance Field, can estimate the camera imaging parameters corresponding to each picture by inputting a series of 2D pictures of the same scene taken from different angles and positions as input data, wherein the camera imaging parameters include the intrinsic parameters and extrinsic parameters of the camera, the intrinsic parameters (Intrinsic Parameters) describe the internal characteristics of the camera, including focal length, optical center (center point on the image plane), distortion parameters, etc.; the extrinsic parameters (Extrinsic Parameters) describe the position and posture of the camera in the world coordinate system, and the extrinsic parameters include rotation matrices and translation vectors, which are used to convert the coordinates in the camera coordinate system into the coordinates in the world coordinate system.

[0057] Based on multiple 2D pictures of massage users at different angles, the 3D model of the human body is obtained. The main purpose is to complete the conversion between the 2D plane coordinate system and the 3D world coordinate system. After receiving a series of 2D photos, the NeRF model further calculates each depth value based on the camera imaging parameters corresponding to each photo. The depth value is the distance data between the pixel point and the image capture device. The NeRF model further converts the 2D pixel coordinates in the picture into world coordinates in 3D space according to the depth value and camera imaging parameters, thereby obtaining the 3D model of the human body. Based on the NeRF model, instant-ngp takes advantage of the modern GPU architecture to achieve fast training and reasoning of the NeRF model. In addition, the real-time and efficient characteristics of instant-ngp well meet the massage robot's needs for fast, efficient and high-fidelity 3D reconstruction of the human body model, and can better achieve the reconstruction of the 3D model image of the human body.

[0058] Figure 3 Schematic diagram of coordinate relationship of a massage scene visualization method according to an embodiment of the present application. Figure 3 As shown, Ow is the world coordinate system, Oc is the camera coordinate system, O is the image coordinate system, O1 is the pixel coordinate system, p(u1, v1) is the coordinate of the object projected to the pixel coordinate system after the projective transformation, and P(Xw, Yw, Zw) is the coordinate of the object in the world coordinate system; in this application, 2D photos of the massage user at multiple angles can be taken based on the depth camera. For the depth camera, the depth value Z corresponding to the pixel coordinate can be directly obtained. Based on the pinhole imaging technology of the camera, the camera coordinates corresponding to the pixel coordinates are calculated by the principle of triangle similarity. The calculation formula is as follows: The camera coordinates of point P can be further obtained based on the following formula: Among them, f x 、f y , u0, v0 are the camera internal parameters, z c is the depth value, f x 、f y They represent the effective focal lengths of the x-axis and y-axis respectively. u0 and v0 are the coordinates of the origin of the image coordinate system in the pixel coordinate system. (u1, v1) represents the image coordinates of point p. (x c ,y c， z c ) represents the camera coordinates of point p. It can be understood that the coordinate conversion formula listed in this embodiment is only an example, and in other embodiments, other calculation methods can also be used to achieve the conversion between two-dimensional coordinates and three-dimensional space coordinates.

[0059] Figure 2 A schematic diagram of a flow chart of a method for generating a human body three-dimensional model image according to an embodiment of the present application. Figure 2As shown, in order to generate a three-dimensional model image of the human body, the robot arm is first initialized and moved to the initial position. When it moves to the initial position, it enters the photo taking position, and then starts to execute the set human body shooting trajectory; after the execution is completed, the shot video is first processed by colmap to become a 2D photo, and then imported into the instant-ngp model, and then the model is trained to obtain a 3D model of the human body; further, the trained 3D model of the human body is transmitted to the VR device, and the reconstruction of the three-dimensional model of the human body is completed. The "training model" mentioned in this embodiment is specifically a three-dimensional model of the human body of a massage user generated based on the trained NeRF model; in this embodiment, the camera used to shoot the posture video of the massage user is fixed at the end of the robot arm, and the camera angle of view does not change by more than 30 degrees during the shooting process.

[0060] In addition, in order to further improve the effect of the human body three-dimensional model image, the posture video includes multiple videos shot by the image capture device at different height data, that is, in this embodiment, the robotic arm moves along the shooting motion path for multiple cycles, and the height of the image capture device corresponding to each cycle is different. Exemplarily, the image capture device can shoot three complete videos, and the shooting of the three complete videos is completed by the robotic arm moving along the shooting motion path. The shooting motion paths corresponding to the three videos are the same, while the heights of the image capture devices are different, for example, the heights are 450mm, 500mm and 550mm respectively. It can be understood that the number of posture videos and the height of the image capture device listed in this embodiment are only some examples. In other embodiments, the number of posture videos shot and the height of the image capture device can be set according to actual needs.

[0061] In addition, the virtual reality device can be a VR device, and the human body 3D model image is specifically sent to the VR device; exemplarily, a script can be used to automatically store the generated new human body 3D model file into the Addressables resource group, and then package and publish the resource to the CDN server. The VR end updates the human body 3D model at any time by detecting the update of the server, so that the operating user can see the 3D model of the massage user through VR. Addressables is a resource management system launched by Unity. It is currently mainly used in hot updates in games. The system allows players to receive the latest content without downloading the complete game update package; this application uses the characteristics of Addressables to enable the operating user to see the 3D model of the massage user in VR.

[0062] In some embodiments of the present application, the method further includes: obtaining thermal imaging data of the massage user, and sending the thermal imaging data to a virtual reality device, so that the virtual reality device displays the thermal imaging data to the operating user. In this embodiment, the thermal imaging data of the massage user can be obtained based on a sensor, and the sensor is specifically fixed to the end of the robotic arm, that is, when the robotic arm moves along the shooting motion path, the sensor at its end detects and monitors the temperature distribution on the human body surface, and the human body surface temperature is visually displayed in the virtual reality device, so that the operating user can intuitively observe the muscle tension area or the blood flow sluggish area of ​​the massage user, so as to massage the massage user more targeted.

[0063] The virtual reality-based massage robot massage scene visualization method of the present application can intuitively display virtual massage scene images, human body three-dimensional model images and thermal imaging data of massage users in virtual reality devices, so that the operating user can better control the massage robot to perform remote massage on the massage user.

[0064] Figure 4 FIG. 1 is a flow chart of a method for visualizing a massage scene of a massage robot based on virtual reality according to another embodiment of the present application. Figure 4 As shown, in this embodiment, the mechanical arm is first initialized, that is, the mechanical arm moves to the initial position (which can also be understood as a preset photo position); the mechanical arm further shoots along the preset shooting motion path, and while executing the shooting trajectory, the thermal imaging system detects and monitors the temperature distribution on the human body surface; the video obtained by executing the shooting trajectory is processed into a 2D photo by colmap and then imported into the instant-ngp model in real time, and then a human body three-dimensional model is generated based on the instant-ngp model training, and the human body three-dimensional model image is transmitted to the VR device for display; in addition, the thermal imaging image obtained by the thermal imaging system is processed to obtain the body state data of the massage user that can be directly observed by the operating user, so that when the operating user performs remote massage on the massage user, the massage user can refer to the thermal imaging data of the massage user to formulate a targeted massage plan. In this embodiment, the massage user is a patient and the operating user is a physician. After processing the thermal imaging image obtained by the thermal imaging system, the body state data of the patient that can be directly observed by the physician is further obtained, so that when the physician performs remote massage on the patient, the patient can refer to the thermal imaging data of the patient to formulate a targeted massage plan.

[0065] In addition, in some other embodiments, when the operating user performs remote massage on the massage user, remote video communication can also be realized based on VR equipment; for example, video communication can be realized based on WebRTC. WebRTC has the advantage of low latency, so real-time communication between the operating user and the massage user can be realized. The existing WebRTC is difficult to communicate across platforms, so a browser can be built in VR and the website display interface can be logged in to realize cross-platform video communication. In this embodiment, a robotic arm is bound to a video communication channel, so that after the massage user turns on the switch of the robotic arm and the industrial computer, the video communication is automatically turned on and the camera image is sent on the corresponding channel. After the operating user logs in to the account on the VR side, it automatically enters the corresponding video communication channel to receive the image of the massage user, so that the operating user can more accurately control the massage robot to perform a better massage on the massage user.

[0066] Correspondingly, the present invention also provides a massage scene visualization system for a massage robot based on virtual reality, the system comprising a processor, a memory and a computer program stored in the memory, the processor being used to execute the computer program, and when the computer program is executed, the system implements the steps of the method described in any of the above embodiments.

[0067] Through the above embodiments, it can be found that the virtual reality-based massage robot massage scene visualization method and device disclosed in the present application generates a virtual massage scene image based on first images of multiple angles of a real massage scene, and obtains a three-dimensional human body model image of the massage user through a trained human body generation model based on second images of multiple angles of the massage user, and finally sends the virtual massage scene image and the three-dimensional human body model image to the virtual reality device to be displayed to the operating user through the virtual reality device, so that the operating user can remotely control the massage robot based on the content displayed in the virtual reality device to achieve remote massage of the massage user; this method allows the operator to remotely operate the massage robot to perform targeted massage on the massage user, and realize remote visualization of the massage scene during the remote massage process, thereby improving the massage effect.

[0068] The embodiments of the present invention also provide a computer-readable storage medium and a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0069] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0070] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0071] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A massage scene visualization method for a massage robot based on virtual reality, characterized in that: The method comprises: Acquire first images of a real massage scene at multiple angles, and generate a virtual massage scene image based on the first images of the real massage scene at multiple angles; Acquire second images of the massage user at multiple angles in the real massage scene, and obtain a three-dimensional human body model image of the massage user based on the second images of the massage user at multiple angles by using a trained human body generation model; The virtual massage scene image and the three-dimensional human body model image are sent to a virtual reality device, so that the virtual reality device displays the virtual massage scene image and the three-dimensional human body model image to an operating user.

2. The method for visualizing a massage scene of a massage robot based on virtual reality according to claim 1, characterized in that: Acquiring second images of the massaging user at multiple angles in the real massage scene, including: Based on the image capturing device at the end of the mechanical arm of the massage robot in the real massage scene, second images of the massage user at multiple angles are acquired.

3. The method for visualizing a massage scene of a massage robot based on virtual reality according to claim 2, characterized in that: Acquiring second images of the massage user at multiple angles based on an image capturing device at the end of the mechanical arm of the massage robot in the real massage scene includes: Determining a shooting motion path of a mechanical arm of the massage robot; Determine the height data and posture data of the image capturing device; Based on the shooting motion path, height data and posture data, shooting a posture video of the massage user by the image shooting device; Second images of the massaging user at multiple angles are determined based on the gesture video.

4. The method for visualizing a massage scene of a massage robot based on virtual reality according to claim 3, characterized in that: Determining a shooting motion path of a mechanical arm of the massage robot includes: Determine multiple motion path points of the robotic arm; Based on the multiple motion path points, the shooting motion path of the mechanical arm of the massage robot is obtained by an interpolation polynomial calculation method.

5. The method for visualizing a massage scene of a massage robot based on virtual reality according to claim 3, characterized in that: Obtaining a three-dimensional human body model image of the massage user from a trained human body generation model based on the second images of the massage user at multiple angles, including: The human body generation model determines shooting parameters corresponding to each second image, wherein the shooting parameters include internal parameters and external parameters; Calculate the distance information between each pixel in the second image and the image capturing device based on the capturing parameters; A three-dimensional human body model image of the massage user is generated based on the shooting parameters and the distance information.

6. The method for visualizing a massage scene of a massage robot based on virtual reality according to claim 1, characterized in that: The method comprises: The thermal imaging data of the massage user is obtained, and the thermal imaging data is sent to a virtual reality device, so that the virtual reality device displays the thermal imaging data to the operating user.

7. The method for visualizing a massage scene of a massage robot based on virtual reality according to claim 3, characterized in that: The posture video includes a plurality of videos captured by the image capturing device at different height data.

8. The method for visualizing a massage scene of a massage robot based on virtual reality according to any one of claims 1 to 7, characterized in that: The method further comprises: Determine a sample data set and an initial human body generation model, wherein the sample data in the sample data set includes multi-angle 2D sample images and human body three-dimensional model sample images; The human body generation model is pre-trained based on the sample data set to obtain a trained human body generation model.

9. A massage scene visualization system for a massage robot based on virtual reality, the system comprising a processor, a memory and a computer program stored in the memory, characterized in that: The processor is used to execute the computer program. When the computer program is executed, the system implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images