A virtual-real fusion method and system for digital human and entity

Abstract

The present invention discloses a virtual-real fusion method and system for digital humans and entities, belonging to the field of augmented reality technology. The scheme realizes the virtual-real fusion of digital humans and entity mechanical structures. First, the entity mechanical structure is skeletally modeled, the properties of each joint of the entity mechanical structure are determined, and each joint part is modeled to form a three-dimensional skeletal model of the entity mechanical structure. The virtual digital human image is skeletally redirected, and the skeletal redirection process is specifically as follows: the size and direction of the bones of the existing virtual digital human image are adjusted so that the redirected virtual digital human image matches the three-dimensional skeletal model of the entity mechanical structure. The entity mechanical structure is superimposed on the virtual digital human image, and the corresponding positions of each joint of the virtual digital human image and the entity mechanical structure are controlled to change simultaneously and synchronously, and it is judged in real time whether the virtual digital human image matches the three-dimensional skeletal model of the entity mechanical structure. If they do not match, they are adjusted in real time.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a virtual-reality fusion method and system for digital humans and entities. Background Art

[0002] Digital humans provide visual interactive properties, while mechanical systems provide kinesthetic interactive properties. By effectively integrating vision and kinesthetics, it is possible to render more realistic virtual images while retaining the kinesthetic advantages of physical mechanical structures. In terms of vision, digital humans can render human images more realistically, and the interactive images are variable, which can provide interactive emotional properties and save the tedious process of designing and making humanoid robots. In terms of kinesthetics, the physical mechanical structure is still retained, which can provide tactile interaction and collaborate with users to complete tasks.

[0003] The patent with application number CN202010302983.X discloses a virtual-real fusion human-machine collaboration simulation method and system, and its technical solution is: according to the three-dimensional posture information of the human body and the return results of the data glove, the three-dimensional posture sequence and corresponding force information of the human body, arm and hand are calculated; combined with the three-dimensional posture sequence and corresponding force information of the human body, arm and hand, and the virtual robot model motion simulation results, based on the collision detection algorithm and the physical simulation algorithm, the position interaction information and force interaction information between the virtual robot and the three-dimensional posture sequence of the human body are detected, and the human-machine collaboration process is simulated and analyzed. The invention can verify and experiment various control algorithms in human-machine collaboration research, and experiment the interaction, collision, force, and coordination of movements between humans and robots; it is closer to the real scene, can truly reflect the movement of the human body, and avoids the establishment of complex models. It also has the advantages of ensuring human safety, people can truly feel the collaboration process, and more accurate simulation. The system is not based on entity-based virtual-real fusion technology, cannot generate any force feedback, and does not have the ability to be used for interaction. What users see are all virtual images. Although they provide visual attributes during the interaction process, they do not have any kinesthetic attributes. Users cannot touch the virtual images, nor can they cooperate with the virtual images to complete tasks. The technology of virtual-reality fusion relies purely on dynamic modeling, not on entity-based virtual-reality fusion technology.

[0004] The patent with application number 202110109325.3 discloses a system and method for rapid and accurate teaching of industrial robots based on augmented reality technology. The system includes a physical industrial robot, a QR code and augmented reality glasses; the physical industrial robot includes an industrial robot body and a robot controller; the augmented reality glasses recognize the QR code on the industrial robot body, and use the recognized QR code posture to generate a virtual robot model that coincides with the real robot; the augmented reality glasses continuously capture the terminal posture of the virtual robot model, and solve the rotation angle of each joint axis of the robot in real time, and synchronously update the robot virtual model; each joint angle is transmitted to the robot controller, and the robot controller controls the joint angle movement of the industrial robot body according to each joint angle. The invention helps to improve the teaching speed of the robot, adopts a three-dimensional registration and tracking module based on markers, improves the teaching accuracy, and has a strong reference significance for the field of industrial manufacturing and processing. The invention is a virtual-real fusion technology based on entities, but the virtual image is not superimposed on the mechanical structure. Although kinesthetic attributes are provided during the interaction process, it does not have any visual attributes. What the user observes through the display device is still the same industrial physical robot, which is not mapped to another different image in the virtual environment.

[0005] Therefore, there is currently a lack of an interaction solution that can integrate humans with digital-physical fusion robots. Summary of the invention

[0006] In view of this, the present invention provides a method and system for virtual-reality fusion of digital humans and entities, which combines the advantages of digital humans and humanoid entity robots to achieve an efficient and intelligent human-computer interaction mode.

[0007] To achieve the above object, the technical solution of the present invention comprises the following steps:

[0008] Step 1: Perform skeletal modeling on the physical mechanical structure to determine the properties of each joint of the physical mechanical structure; the properties include length, size and orientation; the physical mechanical structure includes at least one joint, wherein the joint includes at least one axis and at least one connecting rod, and the connecting rod is connected to the axis to achieve rotation; after knowing the spatial relationship between the joints of the physical mechanical structure, each joint part can be modeled to form a three-dimensional skeletal model of the physical mechanical structure.

[0009] Step 2: Redirect the skeleton of the virtual digital human image. The skeleton redirection process is as follows: adjust the size and direction of the skeleton of the existing virtual digital human image so that the redirected virtual digital human image matches the three-dimensional skeleton model of the physical mechanical structure.

[0010] Step 3: Superimpose the physical mechanical structure with the virtual digital human image, control the corresponding positions of the joints of the virtual digital human image and the physical mechanical structure, and change them simultaneously and synchronously, so as to judge in real time whether the three-dimensional skeletal model of the virtual digital human image and the physical mechanical structure matches, and make real-time adjustments if they do not match.

[0011] Furthermore, skeleton modeling is performed on the physical mechanical structure, specifically using one of the following modeling methods: Modeling method one: manually measure the length, thickness, direction of the connecting rods in each joint of the physical mechanical structure in advance, as well as the rotation angle limit of the axis, so as to determine the spatial relationship of each joint and achieve modeling; Modeling method two: use a calibrated camera to shoot the physical mechanical structure, and then calculate the length, thickness and direction of the connecting rods in the mechanical structure, determine the spatial relationship between the joints, and achieve modeling.

[0012] Furthermore, step 2 specifically includes the following steps:

[0013] First, the bone structure of the virtual digital human image is adjusted so that the bone structure of the virtual digital human image is the same as the three-dimensional bone structure of the physical mechanical structure.

[0014] Then, using manual or Unreal Engine algorithms, adjust the spacing between each joint of the virtual digital human image so that the bone length and orientation are the same.

[0015] Finally, the thickness of the bones of the virtual digital human image is readjusted to match the volume of the three-dimensional bone model of the physical mechanical structure.

[0016] Preferably, it is determined whether the virtual digital human image matches the three-dimensional skeleton model of the physical mechanical structure, and if they do not match, they are adjusted in real time; specifically: when the production accuracy of the virtual digital human image is not less than 1 mm, when the virtual digital human image and the physical mechanical structure are superimposed, the offsets of the axes in the joints in the three directions in the spatial coordinate system are all less than 5 mm, and the color difference value is not greater than 4.0, then the virtual digital human image matches the three-dimensional skeleton model of the physical mechanical structure.

[0017] Preferably, the physical mechanical structure is superimposed on the virtual digital human image, and the corresponding positions of the joints of the virtual digital human image and the physical mechanical structure are controlled to change simultaneously and synchronously. Specifically, the joints in the virtual digital human image and the physical mechanical structure move simultaneously, and the deviation of the spatial position moved to is less than 1 cm, and the delay time is less than 0.5 seconds.

[0018] Another embodiment of the present invention further provides a virtual-real fusion system for digital humans and entities, comprising a physical mechanical structure, a positioning device, a computing terminal, a display device, and a virtual digital human image in a virtual environment;

[0019] The physical mechanical structure comprises at least one joint, wherein the joint comprises at least one shaft and at least one connecting rod, and the connecting rod is connected to the shaft to realize rotation.

[0020] The positioning device is installed on the physical mechanical structure. The positioning device obtains the real-time spatial coordinates and orientation of the physical mechanical structure and transmits its six-degree-of-freedom posture to the computing end.

[0021] The computing end obtains the six-degree-of-freedom posture, provides multiple virtual digital human images, and receives the user's selection instructions. It determines the virtual digital human image to be displayed according to the selection instructions, and renders the virtual digital human image in the virtual environment. The position of the virtual digital human image is the same as the position of the physical mechanical structure, and the physical mechanical structure is superimposed on the virtual digital human image. The computing end controls the corresponding positions of each joint of the virtual digital human image and the physical mechanical structure, and changes them simultaneously and synchronously, and judges in real time whether the three-dimensional skeletal model of the virtual digital human image and the physical mechanical structure matches. If they do not match, they are adjusted in real time.

[0022] The display device is a head-mounted display device, and the virtual environment is presented to the user through the head-mounted display; the head-mounted display device can be a video transmission type or an optical transmission type display device.

[0023] Beneficial effects:

[0024] 1. The present invention provides a method and system for the virtual-real fusion of digital humans and entities. In view of the lack of interactive functional attributes of existing virtual digital humans and the difficulty of humanoid physical robots to achieve interactive emotional attributes, a virtual-real fusion solution for the fusion of digital humans and entities is proposed. The solution realizes the virtual-real fusion of digital humans and physical machine mechanical structures. While retaining the movement and tactile interaction of the physical mechanical structure, part or the whole body of the digital human is superimposed on the physical mechanical structure to achieve the interaction between humans and digital-physical fusion robots in a more effective way. The present invention can simplify the design of humanoid physical robots in an augmented reality environment and provide diverse and variable virtual images. The solution provides a new interaction mode between humans and digital-physical fusion robots, which comprehensively improves the interactive experience of users in a mixed reality environment. The present invention can be applied to a series of virtual-real fusion scenarios such as digital tourism and digital shopping, and has broad application prospects.

[0025] 2. Compared with designing a physical humanoid robot, the present invention is simpler and faster. It simplifies the cumbersome steps of designing a humanoid robot, and eliminates the process of designing facial expressions, skin, facial features, etc. It also has higher mechanical stability.

[0026] 3. The present invention forms a system that combines optional and variable virtual digital humans and entities, which can increase the realism of the character image in augmented reality, and can freely switch the appearance of the interactive object according to the user's preferences, providing a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A framework diagram of a virtual-reality fusion system for digital humans and entities provided by the present invention;

[0028] Figure 2 A flowchart of a virtual-reality fusion system for digital humans and entities provided by the present invention;

[0029] Figure 3 An exemplary schematic diagram of virtual-real fusion provided by one of the embodiments of the present invention;

[0030] Figure 4 A schematic diagram of an example of interaction of virtual-reality fusion provided for one of the embodiments of the present invention. DETAILED DESCRIPTION

[0031] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0032] The present invention provides a virtual-reality fusion technology for digital humans and entities, including partial virtual-reality fusion and full virtual-reality fusion technology. Virtual-reality fusion technology is to superimpose a virtual digital human on a physical mechanical device, so that users can not only see more realistic and vivid character images, but also cooperate and interact with the physical mechanical device. The fusion of digital humans and physical mechanical structures can be divided into two types, one is based on augmented reality, integrating part of the digital human and physical mechanical structure, and the other is to superimpose the entire digital human on a humanoid physical robot.

[0033] This virtual-real fusion method includes four steps:

[0034] Step 1: The physical mechanical structure includes at least one joint, wherein the joint includes at least one axis and at least one connecting rod; the physical mechanical structure is skeletally modeled to determine the length, size, orientation and other properties of each part of the physical mechanical structure. The skeletal modeling method of the physical mechanical structure can be achieved by manually measuring the length, thickness and orientation of the connecting rods of each part of the physical mechanical structure in advance, the rotation angle limit of the axis, or by using a calibrated camera to shoot the physical mechanical structure, and then calculating the spatial relationship from each connection of the mechanical structure to the next connection. After knowing the spatial relationship of the physical structure, the computing end can model each joint part to form a three-dimensional skeletal model of the physical mechanical structure.

[0035] Step 2: Next, the virtual digital image is reoriented. The reorientation process adjusts the size and orientation attributes of the existing virtual digital human image so that the reoriented virtual model matches the model of the physical mechanical structure. First, adjust the skeletal structure of the virtual digital human so that it is the same as the skeletal structure modeled in step 1. Then, use the algorithm in the artificial or Unreal Engine to adjust the spacing between each joint so that the length and orientation of the bones are the same. Finally, re-adjust the size (thickness) of the local position of the virtual digital human model to match the volume of the model.

[0036] Step 3: Determine whether the virtual digital model matches the physical mechanical structure, and make real-time adjustments. When the production accuracy of the virtual digital human image is not less than 1 mm; when the virtual digital human image and the physical mechanical structure are superimposed, the three-direction offset of the axis in the joint is less than 5 mm, and the color difference value is not greater than 4.0, the virtual image can be considered to match the physical mechanical structure (the three directions here refer to the spatial coordinate system constructed according to the physical mechanical structure, the x-axis, y-axis and z-axis directions of the coordinate system). Alternatively, the user can specify the matching accuracy according to actual needs.

[0037] Step 4: During the virtual-real fusion process, the corresponding positions of the virtual digital human image and the physical mechanical structure change simultaneously and synchronously. For example, if interacting with the hand of a digital-real fusion robot, the movement of the digital human's virtual arm should be simultaneous with the movement of the physical mechanical arm, with a spatial position deviation of less than 1 cm and a delay of less than 0.5 seconds.

[0038] The present invention can realize partial virtual-real fusion and full-body virtual-real fusion:

[0039] Partial fusion of reality and virtuality

[0040] In partial virtual-real fusion, the physical mechanical structure or digital virtual image only contains partial components. Partial virtual-real fusion often does not require the full-body virtual digital human to be combined with the physical mechanical structure. Only by combining key parts can the needs of interaction be met efficiently.

[0041] The physical mechanical structure uses a movable humanoid robot, and the virtual image uses a highly realistic virtual digital human. Using the model and texture above the virtual digital human's chest, the bones of the virtual digital human's head, shoulders, and chest are redirected so that the outline of the virtual image and the physical mechanical structure have roughly the same outline. Users can watch the changes in the face and expression of the realistic digital human through the display device. Compared with a pure physical robot, this method can save the cumbersome robot facial mechanical design and material production process, and is more efficient and quick.

[0042] Full body virtual and real fusion

[0043] Figure 3 The figure shows an example of partial virtual-real fusion. Full-body virtual-real fusion refers to combining the entire digital virtual human with a physical mechanical structure or a humanoid physical robot, and integrating the digital image of the entire body into the physical mechanical structure. In the full-body virtual-real fusion system, the full-body digital human model can be directly superimposed on the physical mechanical structure, or the full-body digital human model can be split at each joint and then re-superimposed on the physical mechanical structure.

[0044] Figure 4 The demonstration showed a full-body virtual-real fusion interaction example. Users wearing head-mounted display devices can observe virtual images. When users interact with the digital-physical hybrid robot, such as reaching out to shake hands with it, the mechanical arm on the physical robot performs the handshake action, and at the same time, the virtual hand of the digital person also performs the handshake action. While feeling the touch and force feedback of the hand, the user can see the hand gestures and facial smile of the virtual person, which greatly improves the authenticity of the interaction.

[0045] The hardware and software equipment based on the virtual-reality fusion system: The hardware and software equipment based on the virtual-reality fusion technology includes virtual digital human images, physical mechanical structures, positioning equipment, computing terminals and display devices. The overall framework of the virtual-reality fusion system is as follows: Figure 1 , the functions of each part of the system are as follows Figure 2 shown.

[0046] Virtual digital human image and physical mechanical structure

[0047] The digital human virtual image and the physical mechanical structure are two basic components of the integration of virtual and real. The digital human is a virtual character image, which can be the whole body, half body or head. The digital human is an image agent in the virtual space, an interactive object for the user, and provides visual interactive properties. The physical mechanical structure can be a robotic arm, a manipulator or a full-body robot, etc., as a proxy object for touch or kinesthetics.

[0048] The image of a digital human should have a partial or full body model, skeleton, textures, and animations. The physical mechanical system should have the ability to move and be able to match the movement and body movements of the virtual human to a certain extent. Figure 3 In the example of virtual-real fusion shown, the physical mechanical structure is a movable and interactive robot, but it is difficult for the physical structure to form highly realistic facial features. By superimposing a highly realistic digital human image on the robot through an augmented reality display device, it is possible to render a high-precision, real-time virtual human image while ensuring tactile interaction with a physical structure.

[0049] Positioning equipment

[0050] The positioning device is installed on the physical mechanical structure. The positioning device obtains the real-time spatial coordinates and orientation of the physical mechanical structure and transmits its six-degree-of-freedom posture to the computing end. Through the positioning device, the computing end can obtain the current state of the physical mechanical device, which is convenient for superimposing the virtual image upward. If the mechanical structure is complex, there can be several positioning devices.

[0051] Computing and head-mounted display devices

[0052] The computing end is responsible for the calculation of the virtual-real fusion process. Figure 2 As shown, first, the positioning device obtains the position of the physical mechanical structure and transmits the six-degree-of-freedom posture to the computing end. The six degrees of freedom are the x-axis, y-axis, and z-axis coordinates of the spatial coordinate system and the rotation angles around each axis. The computing end provides one or more virtual images for the user to choose. When the user selects a virtual image, the computing end adjusts the size and position of the virtual image in real time, and a partial or full-body virtual-real fusion method can be used. Use the method in Section 4.2.2 to perform virtual-real fusion, where the method for adjusting the position and size can be but is not limited to translation, rotation, proportional scaling, local scaling, etc. At the same time, the computing end calculates and determines in real time whether the connection between the virtual digital image and the mechanical entity coincides. If an offset occurs, the computing end recalculates and positions. After the virtual digital image is rendered by the computing end, it is presented to the user through a head-mounted display device. The head-mounted display device can use a video transmission type or an optical transmission type display device.

[0053] After the user interacts with the digital-physical fusion robot, if the virtual image or the physical mechanical structure undergoes a significant change that exceeds the range threshold limited by the user-defined virtual-real matching, the positioning device will reposition and the computing end will repeat the above steps to readjust and render the virtual image.

[0054] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A virtual-real fusion method for digital humans and entities, characterized in that: The steps include: Step 1: Perform skeletal modeling on the physical mechanical structure to determine the properties of each joint of the physical mechanical structure; the properties include length, size and orientation; the physical mechanical structure includes at least one joint, wherein the joint includes at least one axis and at least one connecting rod, and the connecting rod is connected to the axis to achieve rotation; after knowing the spatial relationship of each joint of the physical mechanical structure, each joint part can be modeled to form a three-dimensional skeletal model of the physical mechanical structure; Step 2: Redirect the skeleton of the virtual digital human image. The skeleton redirection process specifically includes: adjusting the size and direction of the skeleton of the existing virtual digital human image so that the redirected virtual digital human image matches the three-dimensional skeleton model of the physical mechanical structure; Step 3: superimpose the physical mechanical structure with the virtual digital human image, control the corresponding positions of the joints of the virtual digital human image and the physical mechanical structure to change simultaneously and synchronously, and judge in real time whether the three-dimensional skeletal model of the virtual digital human image and the physical mechanical structure matches. If not, adjust in real time.

2. The method according to claim 1, characterized in that The skeleton modeling of the entity mechanical structure is specifically performed by one of the following modeling methods: Modeling method 1: By manually measuring the length, thickness, direction of the connecting rods in each joint of the physical mechanical structure in advance, as well as the rotation angle limit of the axis, the spatial relationship of each joint is determined to achieve modeling; Modeling method 2: Use a calibrated camera to shoot the physical mechanical structure, then calculate the length, thickness and direction of the connecting rod in the mechanical structure, determine the spatial relationship between the joints, and achieve modeling.

3. The method according to claim 1, characterized in that The step 2 specifically includes the following steps: First, the bone structure of the virtual digital human image is adjusted so that the bone structure of the virtual digital human image is the same as the three-dimensional bone structure of the physical mechanical structure; Then, using manual or Unreal Engine algorithms, adjust the spacing between each joint of the virtual digital human image so that the bone length and orientation are the same; Finally, the thickness of the bones of the virtual digital human image is readjusted to match the volume of the three-dimensional bone model of the physical mechanical structure.

4. The method according to claim 1, 2 or 3, characterized in that: The method of determining whether the virtual digital human image matches the three-dimensional skeleton model of the physical mechanical structure, and adjusting in real time if they do not match, is specifically: When the production accuracy of the virtual digital human image is not less than 1 mm, when the virtual digital human image and the physical mechanical structure are superimposed, the offset of the axis in the joint in the three directions in the spatial coordinate system is less than 5 mm, and the color difference value is not greater than 4.0, then the virtual digital human image matches the three-dimensional skeletal model of the physical mechanical structure.

5. The method according to claim 4, characterized in that The physical mechanical structure is superimposed on the virtual digital human image, and the corresponding positions of the joints of the virtual digital human image and the physical mechanical structure are controlled to change simultaneously and synchronously, specifically: The movement of the joints in the virtual digital human image and the physical mechanical structure is carried out simultaneously, with the deviation of the spatial position moved to less than 1 cm and the delay time less than 0.5 seconds.

6. A virtual-reality fusion system for digital humans and entities, characterized in that: It includes a physical mechanical structure, a positioning device, a computing terminal, a display device, and a virtual digital human image in a virtual environment; The physical mechanical structure comprises at least one joint, wherein the joint comprises at least one shaft and at least one connecting rod, and the connecting rod is connected to the shaft to realize rotation; The positioning device is installed on the physical mechanical structure, and the positioning device obtains the real-time spatial coordinates and orientation of the physical mechanical structure, and transmits its six-degree-of-freedom posture to the computing end; The computing end obtains a six-degree-of-freedom posture, provides multiple virtual digital human images, and receives a user's selection instruction, determines the virtual digital human image to be displayed according to the selection instruction, and renders and displays the virtual digital human image in a virtual environment. The position of the virtual digital human image is the same as the position of the physical mechanical structure, and the physical mechanical structure is superimposed on the virtual digital human image. The computing end controls the corresponding positions of the joints of the virtual digital human image and the physical mechanical structure to change simultaneously and synchronously, and judges in real time whether the three-dimensional skeletal model of the virtual digital human image matches the three-dimensional skeletal model of the physical mechanical structure, and adjusts in real time if they do not match; The display device is a head-mounted display device, and the virtual environment is presented to the user through the head-mounted display device; the head-mounted display device adopts a video transmission type or an optical transmission type display device.

Images