Human-machine cooperation digital twinning simulation method based on virtual reality interaction

Abstract

The invention discloses a man-machine cooperation digital twinning simulation method based on virtual reality interaction, and relates to the technical field of man-machine collaboration.The method comprises the steps that a digital twinning three-dimensional scene, a digital twinning human body model and a digital twinning mechanical arm model are constructed, after human body posture data of a real tester and joint movement data of a real mechanical arm are obtained, a digital twin human body model is driven to move based on the human body posture data, a digital twin mechanical arm model is driven to move based on the joint movement data, virtual man-machine cooperation is achieved, and in the virtual man-machine cooperation process, the human body posture data of the real tester and the joint movement data of the real mechanical arm are obtained. And recording a human body collision part and a scene collision area in real time when the collision occurs, and subsequently testing and optimizing the collision prediction model. According to the method, a highly-simulated and immersive digital twin system is constructed, a real working scene can be safely and reliably reproduced, man-machine cooperation behaviors can be optimized, and the collision prediction model can be repeatedly tested and optimized under the condition that no real collision risk exists.

Description

Technical Field

[0001] The present application relates to the technical field of human-machine collaboration, and in particular to a human-machine collaborative digital twin simulation method based on virtual reality interaction. Background Art

[0002] As a cutting-edge industrial production paradigm, human-machine collaboration technology enables collaborative work between workers and intelligent robotic arms. However, in human-machine collaborative work scenarios, the problem of collision monitoring between workers and robotic arms has always been an important safety challenge. At present, there are two strategies for dealing with human-machine collision problems, namely collision compliance control and collision prediction model. Collision compliance control is based on the use of compliant admittance control after the force sensor detects the collision signal to prevent the robotic arm from remaining in an extruded state after the collision, thereby reducing secondary damage to the human body, but there is still a risk of collision. The collision prediction model can learn the sensor signals of the robotic arm and predict the potential collision hazards of the scene, so as to provide real-time warnings and early avoidance, but this method requires continuous high-risk scene testing and model optimization.

[0003] If the collision prediction model is tested and optimized directly in the actual scene, the tester will inevitably be exposed to potential collision risks. Once the robot arm fails to accurately identify and accidentally collides with the human body, it may not only cause damage to the equipment and interrupt the production process, but also cause harm to the tester, resulting in incalculable losses. The digital twin technology can not only reproduce the human-machine collaborative working conditions by establishing a virtual digital environment that is highly similar to the physical system, and use virtual reality technology to provide real-time immersive first-person action interaction and scene feedback, but also repeatedly test the collision prediction model without the risk of real collision, and further optimize the human-machine collaborative behavior. This method can not only significantly improve the performance of the algorithm, but also greatly reduce the safety risks in the testing and optimization process, providing a safer and more efficient solution for human-machine collaboration in the field of intelligent manufacturing. Summary of the invention

[0004] The purpose of this application is to provide a human-machine collaborative digital twin simulation method based on virtual reality interaction, to construct a highly realistic and immersive digital twin system that can not only safely and reliably reproduce real working scenes, but also optimize human-machine collaborative behavior, and can repeatedly test and optimize collision prediction models without real collision risks.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a human-machine collaborative digital twin simulation method based on virtual reality interaction, and the human-machine collaborative digital twin simulation method based on virtual reality interaction includes:

[0007] Construct a digital twin 3D scene corresponding to the real working environment, as well as a digital twin human model and digital twin robotic arm model for virtual human-machine collaboration in the digital twin 3D scene;

[0008] Acquire the human body posture data of the real test person and the joint motion data of the real robot arm; the human body posture data includes position and posture; the joint motion data includes the speed and angle of each joint;

[0009] The digital twin human body model is driven to move based on the human body posture data, and the digital twin mechanical arm model is driven to move based on the joint motion data, so that the digital twin human body model is synchronized with the movements of the real test person, and the digital twin mechanical arm model is synchronized with the movements of the real mechanical arm;

[0010] Determine whether a physical collision occurs between the digital twin human body model and the physical assets within the digital twin three-dimensional scene. If so, determine and record the human body collision part and the scene collision area, and return to the step of "obtaining the human body posture data of the real test person and the joint motion data of the real robotic arm"; if not, return to the step of "obtaining the human body posture data of the real test person and the joint motion data of the real robotic arm".

[0011] Optionally, a digital twin 3D scene corresponding to the real working environment is constructed, specifically including:

[0012] The real working environment of human-machine collaboration is modeled to obtain a digital twin 3D scene, and the physical terrain boundaries of the digital twin 3D scene are set for the digital twin 3D scene to make the digital twin 3D scene consistent with the real working environment.

[0013] Optionally, the digital twin human body model includes a digital twin human body skeleton model and a digital twin human body physical model, and the digital twin robotic arm model includes a digital twin robotic arm skeleton model and a digital twin robotic arm physical model; the digital twin human body skeleton model can bind different digital twin human body physical models through a skeleton redirector, and the digital twin robotic arm skeleton model can bind different digital twin robotic arm physical models through a skeleton redirector, which is suitable for different human-machine collaborative work scenarios, and different human-machine collaborative work scenarios include handling, welding and disassembly and assembly.

[0014] Optionally, the human body posture data is data collected using a virtual reality device, and the joint motion data is data collected using a sensor;

[0015] The virtual reality device includes a head-mounted display device worn on the head of a real tester, a handle worn on the hand of the real tester, and multiple positioning base stations installed in the test area where the real tester is located; the positioning base station is used to emit a laser signal; the head-mounted display device and the handle are both provided with a gyroscope and a laser receiver, the gyroscope is used to measure the posture of the real tester, and the laser receiver is used to measure the position of the real tester based on the received laser signal; the head-mounted display device is used to view the virtual human-machine collaboration between the digital twin human model and the digital twin mechanical arm model from a first-person perspective; the handle has buttons, and the buttons are used to control the movement and hand movements of the digital twin human model;

[0016] The sensors include a speed sensor and an angle sensor installed at each joint of the real robot arm.

[0017] Optionally, driving the digital twin human body model to move based on the human body posture data specifically includes: taking the human body posture data and the digital twin human body skeleton model as input, and using a skeleton inverse kinematics solver to obtain joint data of the digital twin human body skeleton model; and driving the digital twin human body model to move based on the joint data.

[0018] Optionally, each joint of the digital twin robotic arm skeleton model is set with a rotational pair constraint and a joint angle variable, and the digital twin robotic arm model is driven to move based on the joint motion data, specifically including: taking the joint motion data as input, using a state updater to update the joint angle variable of each joint of the digital twin robotic arm skeleton model, and driving the digital twin robotic arm model to move.

[0019] Optionally, before determining whether a physical collision occurs between the digital twin human body model and a physical asset within the digital twin three-dimensional scene, the human-computer collaborative digital twin simulation method based on virtual reality interaction further includes: determining whether the digital twin human body model moves to the physical terrain boundary of the digital twin three-dimensional scene, and if so, triggering an interface pop-up window and a virtual air wall to block it.

[0020] Optionally, determine and record the human body collision part and scene collision area, specifically including:

[0021] Using a collision event distributor, the human joint posture data of the digital twin human model in a frame before the collision and the human joint posture data of the digital twin human model in a current frame of the collision are respectively recorded to obtain posture recording data;

[0022] Based on the posture recording data, a ray collision detection function is used to determine the human body collision part and the scene collision area;

[0023] Record the collision parts of the human body and the collision areas of the scene.

[0024] Optionally, after determining and recording the human body collision part and the scene collision area, the human-machine collaborative digital twin simulation method based on virtual reality interaction further includes: visualizing the human body collision part and the scene collision area;

[0025] Among them, the human body collision parts and scene collision areas are visualized, including:

[0026] Visualize the collision parts of the human body in the form of a virtual reality pop-up interface;

[0027] Visualize the scene's collision area as a capsule.

[0028] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0029] The present application provides a human-machine collaborative digital twin simulation method based on virtual reality interaction. The method first constructs a digital twin three-dimensional scene corresponding to a real working environment and a digital twin human model and a digital twin robotic arm model for virtual human-machine collaboration in the digital twin three-dimensional scene. After obtaining the human posture data of a real tester and the joint motion data of a real robotic arm, the digital twin human model is driven to move based on the human posture data, and the digital twin robotic arm model is driven to move based on the joint motion data to achieve virtual human-machine collaboration. During the virtual human-machine collaboration process, it is determined in real time whether a physical collision occurs between the digital twin human model and the physical assets in the digital twin three-dimensional scene. When a collision occurs, the human body collision part and the scene collision area are determined and recorded. The recorded human body collision part and scene collision area can provide input data for the testing and optimization of the collision prediction model. Therefore, the present application does not require actual human-machine collaboration, but can safely and reliably reproduce the real working scene and optimize the human-machine collaborative behavior. The collision prediction model can also be repeatedly tested and optimized without the risk of real collision. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 This is an application environment diagram of a human-machine collaborative digital twin simulation method based on virtual reality interaction provided in Example 1 of the present application.

[0032] Figure 2 A flowchart of a human-machine collaborative digital twin simulation method based on virtual reality interaction provided in Example 1 of the present application.

[0033] Figure 3 A schematic diagram of the principle of a human-computer collaborative digital twin simulation method based on virtual reality interaction provided in Example 1 of the present application.

[0034] Figure 4 A schematic diagram of a digital twin three-dimensional scene of a factory processing line provided in Example 1 of the present application.

[0035] Figure 5 A schematic diagram of a digital twin human skeleton model and a digital twin human physical model provided in Example 1 of the present application; wherein: Figure 5 (a) in the figure is a digital twin human skeleton model. Figure 5 (b) in the figure is the digital twin human physical model.

[0036] Figure 6 A schematic diagram of real-time posture mapping of a digital twin human model provided in Example 1 of the present application.

[0037] Figure 7 Schematic diagram of hand motion mapping of the digital twin human model provided in Example 1 of the present application.

[0038] Figure 8 A schematic diagram of an example robotic arm provided in Example 1 of the present application.

[0039] Fig. 9 A flow chart of the digital twin human body status monitoring algorithm provided in Example 1 of the present application.

[0040] Fig.10 A schematic diagram of ray collision detection provided in Example 1 of the present application.

[0041] Fig.11 A schematic diagram of the human body collision parts and scene collision areas provided in Example 1 of the present application.

[0042] Fig.12 A schematic diagram of the structure of a computer device provided in Example 2 of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] Example 1

[0045] The human-machine collaborative digital twin simulation method based on virtual reality interaction provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send the human posture data and joint motion data to be processed to the server. After the server receives the human posture data and joint motion data to be processed, the server constructs a digital twin three-dimensional scene corresponding to the real working environment and a digital twin human model and a digital twin mechanical arm model for virtual human-machine collaboration in the digital twin three-dimensional scene for the human posture data and joint motion data to be processed; the digital twin human model is driven to move based on the human posture data, and the digital twin mechanical arm model is driven to move based on the joint motion data, so that the digital twin human model is synchronized with the real tester's action, and the digital twin mechanical arm model is synchronized with the real mechanical arm action; it is determined whether the digital twin human model and the physical assets in the digital twin three-dimensional scene have a physical collision, and if so, the human body collision part and the scene collision area are determined and recorded.

[0046] In addition, in some embodiments, the human-computer collaborative digital twin simulation method based on virtual reality interaction can also be implemented independently by a server or a terminal. For example, the terminal can directly process the human body posture data and joint motion data to be processed, or the server can obtain the human body posture data and joint motion data to be processed from the data storage system and process the human body posture data and joint motion data to be processed.

[0047] In the current field of human-machine collaboration research, there is still a lack of a virtual simulation operation platform that can highly restore the real scene of human-machine collaboration and accurately simulate human-machine interaction. Testers need to optimize and standardize human-machine collaboration behavior in a safe virtual simulation operation platform, and conduct comprehensive pre-testing and repeated debugging of the collision prediction model, so as to avoid the collision risk brought by direct human-machine collaboration testing in complex working environments, and adopt a safer and more efficient digital solution. Based on this, this embodiment proposes a human-machine collaboration digital twin simulation method based on virtual reality interaction, which provides testers with a highly realistic and immersive digital twin system, which can not only safely and reliably reproduce real working scenes, but also optimize human-machine collaboration behavior, and provide strong technical support for safety testing and task optimization in intelligent production and complex working environments.

[0048] Specifically, Figure 2 and Figure 3As shown, this embodiment provides a human-machine collaborative digital twin simulation method based on virtual reality interaction. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0049] Step S1, constructing a digital twin three-dimensional scene corresponding to the real working environment and a digital twin human body model and a digital twin robotic arm model for virtual human-machine collaboration in the digital twin three-dimensional scene.

[0050] Step S2, obtaining the human body posture data of the real test person and the joint motion data of the real robot arm; the human body posture data includes position and posture; the joint motion data includes the speed and angle of each joint.

[0051] Step S3: driving the digital twin human body model to move based on the human body posture data, and driving the digital twin robotic arm model to move based on the joint motion data, so that the digital twin human body model is synchronized with the movements of the actual tester, and the digital twin robotic arm model is synchronized with the movements of the actual robotic arm.

[0052] Step S4, determine whether a physical collision occurs between the digital twin human body model and the physical assets in the digital twin three-dimensional scene. If so, determine and record the human body collision part and the scene collision area, and return to the step of "obtaining the human body posture data of the real test person and the joint motion data of the real robotic arm"; if not, return to the step of "obtaining the human body posture data of the real test person and the joint motion data of the real robotic arm".

[0053] By implementing the above steps S1 to S4, this embodiment constructs a digital twin three-dimensional scene, a digital twin human model and a digital twin mechanical arm model, drives the movement of the digital twin human model based on the human posture data of the real tester, and drives the movement of the digital twin mechanical arm model based on the joint motion data of the real mechanical arm, so as to simulate the human-machine collaborative behavior through virtual reality, and record the human body collision part and scene collision area when the collision occurs in the process of simulating the human-machine collaborative behavior, and provide input data for the testing and optimization of the collision prediction model. Digital twin technology can not only reproduce the human-machine collaborative working conditions by establishing a virtual digital environment that is highly similar to the physical system, and use virtual reality technology to provide real-time immersive first-person action interaction and scene feedback, but also repeatedly test and optimize the collision prediction model without the risk of real collision, and the tester will know which human-machine collaborative actions are prone to collision, consciously avoid using human-machine collaborative actions that are prone to collision, and further optimize the human-machine collaborative behavior. This method can not only significantly improve the performance of the algorithm, but also greatly reduce the safety risks in the testing and optimization process, and provide a safer and more efficient solution for human-machine collaboration in the field of intelligent manufacturing.

[0054] In step S1, a digital twin three-dimensional scene is first constructed to simulate the actual working environment (which is the specific place for actual human-machine collaboration), and then a digital twin human body model and a digital twin robotic arm model are constructed in the digital twin three-dimensional scene, which have real-time perception and can perform virtual human-machine collaboration in the digital twin three-dimensional scene.

[0055] The digital twin 3D scene is a simulation model based on the real working environment, such as Figure 4 As shown, it is a digital twin 3D scene of a factory processing line. In step S1, a digital twin 3D scene corresponding to the real working environment is constructed, specifically including: scene modeling of the real working environment of human-machine collaboration to obtain a digital twin 3D scene, and setting a scene physical terrain boundary for the digital twin 3D scene so that the digital twin 3D scene is consistent with the real working environment. By setting the scene physical terrain boundary, the working area consistency of the digital twin 3D scene and the real working environment is maintained.

[0056] In step S1, Figure 5As shown, the digital twin human body model includes a digital twin human body skeleton model and a digital twin human body physical model, and the digital twin robotic arm model includes a digital twin robotic arm skeleton model and a digital twin robotic arm physical model. Among them, the skeleton models of the digital twin human body model and the digital twin robotic arm model can be designed to be bindable, and the physical models of different human bodies and robotic arms can be bound through the skeleton redirector, that is, through the skeleton redirector, when the skeletons are similar, different human body physical models and robotic arm physical models can be bound. With the help of the transferable characteristics of digital skeleton assets, it can adapt to different types of workers, different models of robotic arms and different operation scenes, etc., to adapt to different human-machine collaborative operation scenes. Specifically, the digital twin human body skeleton model can be bound to different digital twin human body physical models through the skeleton redirector, and the digital twin robotic arm skeleton model can be bound to different digital twin robotic arm physical models through the skeleton redirector, which is suitable for different human-machine collaborative operation scenes, and different human-machine collaborative operation scenes include handling, welding and disassembly.

[0057] In this embodiment, the human posture data of the real test person and the joint motion data of the real robotic arm are collected by the virtual reality device and the sensor respectively. In step S2, the human posture data is the data collected by the virtual reality device, and the joint motion data is the data collected by the sensor.

[0058] The virtual reality device includes a head-mounted display device worn on the head of the real tester, a handle worn on the hand of the real tester, and multiple positioning base stations installed in the test area where the real tester is located. The positioning base station is used to emit laser signals. The head-mounted display device and the handle are both equipped with a gyroscope and a laser receiver. The gyroscope is used to measure the posture of the real tester, and the laser receiver is used to measure the position of the real tester based on the received laser signal. The head-mounted display device is used to view the virtual human-machine collaboration between the digital twin human model and the digital twin mechanical arm model from a first-person perspective. The handle has buttons, which are used to control the movement and hand movements of the digital twin human model. The handle can control the digital twin human model to move forward, backward, left, and right, and can also control the digital twin human model to turn. It can also select and map hand movements for the digital twin human model. The hand movements can physically interact with the digital twin three-dimensional scene, such as dragging the mechanical arm for teaching, grasping objects, and sliding doors.

[0059] The virtual reality device of this embodiment may also include devices worn on other parts of the human body, such as the waist, arms, legs, and knee joints.

[0060] The specific process of human posture data collection is as follows: the real test personnel use the head-mounted display device and handle that come with the virtual reality device to measure the actual human posture data and input the control signal. The human posture data is based on the high-frequency laser signal emitted by multiple positioning base stations arranged in the test area. The laser receiver in the head-mounted display device and the handle is responsible for receiving this high-frequency laser signal, and by analyzing the core parameters of the high-frequency laser signal such as the time difference, distance and angle, it measures the position coordinates and orientation angle (i.e., position) of the real test personnel's body in the real three-dimensional space. At the same time, the gyroscope can measure the posture of the real test personnel's body in the real three-dimensional space. The control signal is sent by executing different buttons on the handle during human-computer interaction.

[0061] The sensors include speed sensors and angle sensors installed at each joint of the real robot arm.

[0062] The specific process of joint motion data collection is as follows: the real robotic arm is driven by a motion trajectory planning program, and the actual joint motion data of the real robotic arm is collected through the speed sensors and angle sensors installed on each joint (including the end) of the real robotic arm. The joints can be driven by motors.

[0063] The posture mapping of the digital twin human model is to input the sensor data of the virtual reality device, the input action mapping set and the digital twin human skeleton model into the skeleton inverse kinematics solver, solve the basic joint data of the digital twin human skeleton model, and trigger the corresponding digital twin human body model motion part animation to complete the real-time mapping of the real motion posture and the real-time mapping of the real hand movement, such as Figure 6 and Figure 7 As shown, the input action mapping set is defined by different buttons of the handle, among which the touchpad, trigger button, side grip button and other buttons correspond to different hand operation animations (i.e. hand movements), and can also control the forward, backward, left, right movement and steering of the human body.

[0064] Then in step S3, the digital twin human model is driven to move based on the human posture data, specifically including: taking the human posture data and the digital twin human skeleton model as input, using the skeleton inverse kinematics solver to solve and obtain the joint data of the digital twin human skeleton model, and driving the digital twin human model to move based on the joint data. When the handle has the function of selecting hand movements, the human posture data at this time also includes hand movements.

[0065] The digital twin robotic arm model is based on the three-dimensional model of the robotic arm. By setting the base as the root skeleton and connecting each joint in turn towards the end, a motion chain of the digital twin robotic arm model is formed. Revolute constraints and angle variables are set between each joint. After the sensors on each joint of the real robotic arm collect the joint motion data of the robotic arm, the joint motion data is packaged into data packets, and the real-time joint motion data is sent to the state updater of the digital twin robotic arm skeleton model through the TCP socket communication service. After the state updater of the digital twin robotic arm skeleton model receives the joint motion data collected by the sensor in real time, it updates the joint angle variables of the digital twin robotic arm skeleton model to synchronize the virtual and real motion of the robotic arm. For demonstration purposes, this embodiment provides the following example. Figure 8 The 7-axis robotic arm model of the Franka EmikaPanda model is shown, and the actual implementation method is not limited to the digital twin mapping of this robotic arm.

[0066] In step S3, each joint of the digital twin robotic arm skeleton model is set with a rotational sub constraint and a joint angle variable, and the digital twin robotic arm model is driven to move based on the joint motion data. Specifically, it includes: taking the joint motion data as input, using the state updater to update the joint angle variable of each joint of the digital twin robotic arm skeleton model, and driving the digital twin robotic arm model to move.

[0067] This embodiment designs a digital twin human body state monitoring algorithm to detect and record collision data between the digital twin human body model and the physical assets of the digital twin three-dimensional scene, and visualize the human body collision parts and scene collision areas.

[0068] like Fig. 9As shown in the figure, the process of the digital twin human body monitoring algorithm is: first, based on the human body posture data obtained by the virtual reality device, it is synchronized to the digital twin human body model in the digital twin 3D scene to achieve accurate positioning of the digital twin human body model in the digital twin 3D scene; then, it is determined whether the digital twin human body model exceeds the scene physical terrain boundary of the digital twin 3D scene. Once it is detected that the digital twin human body model touches the scene physical terrain boundary of the digital twin 3D scene, an interface pop-up window and a virtual air wall will be triggered. The interface pop-up window refers to the prompt displayed on the head-mounted display device that the physical terrain boundary of the scene has been reached, and the virtual air wall will be blocked. Air wall blocking means preventing the digital twin human body model from actually going beyond the digital twin 3D scene. If it does not go beyond the physical terrain boundary of the digital twin 3D scene, it will enter the subsequent human body physical collision detection process; the human body physical collision detection process is: when a volume overlap collision occurs between the digital twin human body model and the physical assets in the digital twin 3D scene (i.e., all objects in the digital twin 3D scene, including but not limited to the digital twin robotic arm model) in the digital twin 3D scene, the collision event distributor is immediately triggered, and the collision event distributor responds quickly and records the human body joint posture data of the previous frame and the current frame of the collision; Fig.10 As shown, the ray collision detection function is called to combine the geometric characteristics of the digital twin human model with the grid boundaries of the physical assets of the digital twin 3D scene to calculate and determine the human body collision parts and scene collision areas; the scene collision area in the digital twin 3D scene is rendered and displayed in the form of a capsule, and the human body collision parts are updated in the human body collision interface, that is, the human body collision parts are visualized in the form of a virtual reality pop-up window interface, such as Fig.11 As shown, the collision data is recorded, the collision data includes the collision part of the human body and the collision area of ​​the scene, and the human-machine status is updated.

[0069] Among them, when calculating the human body collision part and scene collision area, the intersection of the ray and the physical asset is calculated according to the mesh boundaries of the digital twin human body model and the physical asset of the digital twin three-dimensional scene. The starting point of the ray is the wearing point of the virtual reality device, and the direction is to point to the volume overlapping grid, so as to determine the specific location where the collision occurs, that is, to determine the human body collision part and scene collision area. In this process, the digital twin human body model is usually simplified into a geometric shape (such as a capsule, a sphere, etc.) based on the human body joint posture data of the previous frame and the current frame of the collision.

[0070] Before determining whether a physical collision occurs between the digital twin human body model and the physical assets within the digital twin three-dimensional scene, the human-computer collaborative digital twin simulation method based on virtual reality interaction of this embodiment also includes: determining whether the digital twin human body model moves to the physical terrain boundary of the digital twin three-dimensional scene, and if so, triggering an interface pop-up window and a virtual air wall to block it.

[0071] Determine whether there is a physical collision between the digital twin human body model and the physical assets in the digital twin three-dimensional scene. If so, determine and record the human body collision part and the scene collision area, return to the step of "obtaining the human body posture data of the real test person and the joint motion data of the real robotic arm" to update the human-machine status; if not, return to the step of "obtaining the human body posture data of the real test person and the joint motion data of the real robotic arm" to update the human-machine status.

[0072] Among them, determining and recording the human body collision part and the scene collision area specifically includes: using the collision event distributor to respectively record the human body joint posture data of the digital twin human body model in the frame before the collision and the human body joint posture data of the digital twin human body model in the current frame of the collision to obtain posture recording data; based on the posture recording data, using the ray collision detection function to determine the human body collision part and the scene collision area; recording the human body collision part and the scene collision area.

[0073] After determining and recording the collision parts of the human body and the collision areas of the scene, the human-machine collaborative digital twin simulation method based on virtual reality interaction of this embodiment further includes: visualizing the collision parts of the human body and the collision areas of the scene.

[0074] Among them, the human body collision parts and scene collision areas are visualized, specifically including: visualizing the human body collision parts in the form of a virtual reality pop-up window interface; visualizing the scene collision area in the form of a capsule.

[0075] The entire digital twin system of this embodiment can be built in the Unreal Engine UE5, and each function used therein can be implemented by calling the encapsulated functions in the Unreal Engine.

[0076] This embodiment discloses a human-machine collaborative digital twin simulation method based on virtual reality interaction, which belongs to the field of human-machine interaction and virtual reality technology. The method includes: first, collecting the human posture data of the real test personnel and the joint motion data of the real mechanical arm through virtual reality equipment and sensors respectively; then, in the digital twin three-dimensional scene simulating the real working environment, constructing a real-time perceived digital twin human model and a digital twin mechanical arm model, wherein the skeleton models of the digital twin human model and the digital twin mechanical arm model can be bound to different physical models through the skeleton redirector to adapt to different human-machine collaborative working scenes; finally, based on the digital twin human state monitoring algorithm, the collision between the digital twin human model and the physical assets of the digital twin three-dimensional scene is detected in real time and the collision data is recorded, and the collision part of the human body and the collision area of ​​the scene are visualized. This embodiment constructs a highly realistic and immersive digital twin system, which can not only safely and reliably reproduce the real working scene, but also optimize the human-machine collaborative behavior, and provide strong technical support for safety testing and task optimization in intelligent production and complex working environments.

[0077] The method of this embodiment has the following advantages:

[0078] (1) The digital twin system accurately simulates the real working conditions of human-machine collaboration and reproduces human-machine interaction behaviors, allowing testers to optimize and standardize human-machine collaborative behaviors in a safe, reliable, and immersive environment, and can verify the collision prediction model, thereby improving test efficiency and reducing test costs. Compared with actual scene testing, more human-machine collaborative actions can be tested.

[0079] (2) After detecting a collision, the digital twin human body status monitoring algorithm will trigger the collision event distributor, record the human body collision data, and display the visual interface of the scene collision area and the human body collision part, providing collision visualization feedback and related data support for the research on human-machine collaborative algorithms.

[0080] (3) The digital skeleton model can be bound to different human body physical models or robotic arm physical models according to specific practical requirements, without the need for tedious and complicated modeling work. It can also be migrated to different virtual scenes for plug-and-play use, and has universal applicability in different working environments.

[0081] The robotic arm of this embodiment can be replaced by other types of robots.

[0082] The embodiment of the present application also provides an application scenario, which applies the above-mentioned human-machine collaborative digital twin simulation method based on virtual reality interaction. Specifically, the human-machine collaborative digital twin simulation method based on virtual reality interaction provided in this embodiment can be applied in a model test scenario. The model test scenario includes a data acquisition link, a simulation link and a test link. The data acquisition link is used to obtain human posture data and joint motion data. The simulation link is used to simulate virtual human-machine collaboration based on human posture data and joint motion data to obtain collision data. The test link is used to test and optimize the collision prediction model based on collision data. The human-machine collaborative digital twin simulation method based on virtual reality interaction provided in this embodiment belongs to the simulation link.

[0083] Example 2

[0084] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.12 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a human-computer collaborative digital twin simulation method based on virtual reality interaction is implemented.

[0085] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0086] In an exemplary embodiment, a computer device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the human-computer collaborative digital twin simulation method based on virtual reality interaction described in Example 1.

[0087] Example 3

[0088] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the human-computer collaborative digital twin simulation method based on virtual reality interaction described in Example 1 is implemented.

[0089] Example 4

[0090] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the human-computer collaborative digital twin simulation method based on virtual reality interaction described in Example 1.

[0091] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A human-machine collaborative digital twin simulation method based on virtual reality interaction, characterized in that: The human-machine collaborative digital twin simulation method based on virtual reality interaction includes: Construct a digital twin 3D scene corresponding to the real working environment, as well as a digital twin human model and digital twin robotic arm model for virtual human-machine collaboration in the digital twin 3D scene; Acquire the human body posture data of the real test person and the joint motion data of the real robot arm; the human body posture data includes position and posture; the joint motion data includes the speed and angle of each joint; The digital twin human body model is driven to move based on the human body posture data, and the digital twin mechanical arm model is driven to move based on the joint motion data, so that the digital twin human body model is synchronized with the movements of the real test person, and the digital twin mechanical arm model is synchronized with the movements of the real mechanical arm; Determine whether a physical collision occurs between the digital twin human body model and the physical assets within the digital twin three-dimensional scene. If so, determine and record the human body collision part and the scene collision area, and return to the step of "obtaining the human body posture data of the real test person and the joint motion data of the real robotic arm"; if not, return to the step of "obtaining the human body posture data of the real test person and the joint motion data of the real robotic arm".

2. The human-machine collaborative digital twin simulation method based on virtual reality interaction according to claim 1 is characterized in that: Construct a digital twin 3D scene corresponding to the real working environment, including: The real working environment of human-machine collaboration is modeled to obtain a digital twin 3D scene, and the physical terrain boundaries of the digital twin 3D scene are set for the digital twin 3D scene to make the digital twin 3D scene consistent with the real working environment.

3. The human-machine collaborative digital twin simulation method based on virtual reality interaction according to claim 1 is characterized in that: The digital twin human body model includes a digital twin human body skeleton model and a digital twin human body physical model, and the digital twin robotic arm model includes a digital twin robotic arm skeleton model and a digital twin robotic arm physical model; the digital twin human body skeleton model can be bound to different digital twin human body physical models through a skeleton redirector, and the digital twin robotic arm skeleton model can be bound to different digital twin robotic arm physical models through a skeleton redirector, so as to be suitable for different human-machine collaborative work scenarios, and different human-machine collaborative work scenarios include handling, welding and disassembly and assembly.

4. The human-machine collaborative digital twin simulation method based on virtual reality interaction according to claim 1 is characterized in that: The human body posture data is data collected by using virtual reality equipment, and the joint movement data is data collected by using sensors; The virtual reality device includes a head-mounted display device worn on the head of a real tester, a handle worn on the hand of the real tester, and multiple positioning base stations installed in the test area where the real tester is located; the positioning base station is used to emit a laser signal; the head-mounted display device and the handle are both provided with a gyroscope and a laser receiver, the gyroscope is used to measure the posture of the real tester, and the laser receiver is used to measure the position of the real tester based on the received laser signal; the head-mounted display device is used to view the virtual human-machine collaboration between the digital twin human model and the digital twin mechanical arm model from a first-person perspective; the handle has buttons, and the buttons are used to control the movement and hand movements of the digital twin human model; The sensors include a speed sensor and an angle sensor installed at each joint of the real robot arm.

5. The human-machine collaborative digital twin simulation method based on virtual reality interaction according to claim 3 is characterized in that: Driving the digital twin human body model to move based on the human body posture data specifically includes: taking the human body posture data and the digital twin human body skeleton model as input, and using a skeleton inverse kinematics solver to obtain joint data of the digital twin human body skeleton model; and driving the digital twin human body model to move based on the joint data.

6. The human-machine collaborative digital twin simulation method based on virtual reality interaction according to claim 3 is characterized in that: A rotational pair constraint and a joint angle variable are set for each joint of the digital twin robotic arm skeleton model, and the digital twin robotic arm model is driven to move based on the joint motion data. Specifically, the process includes: taking the joint motion data as input, using a state updater to update the joint angle variable of each joint of the digital twin robotic arm skeleton model, and driving the digital twin robotic arm model to move.

7. The human-machine collaborative digital twin simulation method based on virtual reality interaction according to claim 1 is characterized in that: Before determining whether a physical collision occurs between the digital twin human body model and the physical assets within the digital twin three-dimensional scene, the human-computer collaborative digital twin simulation method based on virtual reality interaction also includes: determining whether the digital twin human body model moves to the physical terrain boundary of the digital twin three-dimensional scene, and if so, triggering an interface pop-up window and a virtual air wall to block it.

8. The human-machine collaborative digital twin simulation method based on virtual reality interaction according to claim 1 is characterized in that: Determine and record the human body collision parts and scene collision areas, including: Using a collision event distributor, the human joint posture data of the digital twin human model in a frame before the collision and the human joint posture data of the digital twin human model in a current frame of the collision are respectively recorded to obtain posture recording data; Based on the posture recording data, a ray collision detection function is used to determine the human body collision part and the scene collision area; Record the collision parts of the human body and the collision areas of the scene.

9. The human-machine collaborative digital twin simulation method based on virtual reality interaction according to claim 1 is characterized in that: After determining and recording the human body collision part and the scene collision area, the human-machine collaborative digital twin simulation method based on virtual reality interaction further includes: visualizing the human body collision part and the scene collision area; Among them, the human body collision parts and scene collision areas are visualized, including: Visualize the collision parts of the human body in the form of a virtual reality pop-up interface; Visualize the scene's collision area as a capsule.

Images