Human-Computer Collaborative Digital Twin Simulation Method Based on Virtual Reality Interaction

Abstract

This application discloses a human-computer collaboration digital twin simulation method based on virtual reality interaction, belonging to the field of human-computer collaboration technology. The method includes: constructing a digital twin 3D scene, a digital twin human body model, and a digital twin robotic arm model; after acquiring the human posture data of a real test subject and the joint motion data of a real robotic arm, driving the movement of the digital twin human body model based on the human posture data, and driving the movement of the digital twin robotic arm model based on the joint motion data, thereby realizing virtual human-computer collaboration. During the virtual human-computer collaboration process, the collision points of the human body and the scene collision area are recorded in real time when a collision occurs, and the collision prediction model is subsequently tested and optimized. This application constructs a highly realistic and immersive digital twin system that can not only safely and reliably reproduce real work scenarios but also optimize human-computer collaboration behavior, allowing for repeated testing and optimization of the collision prediction model without real-world collision risks.

Description

Technical Field

[0001] This application relates to the field of human-computer collaboration technology, and in particular to a human-computer collaborative digital twin simulation method based on virtual reality interaction. Background Technology

[0002] Human-robot collaboration technology, as a cutting-edge industrial production paradigm, enables collaborative work between workers and intelligent robotic arms. However, collision detection between workers and robotic arms remains a significant safety challenge in human-robot collaborative work scenarios. Currently, collision handling strategies can be divided into two types: collision compliance control and collision prediction models. Collision compliance control, based on force sensors detecting collision signals, employs compliant admittance control to prevent the robotic arm from remaining in a compressed state after a collision, reducing secondary injuries to the worker, but the risk of collision still exists. Collision prediction models can learn the robotic arm's sensor signals and predict potential collision hazards in the scene, thus providing real-time warnings and early avoidance. However, this method requires continuous testing in high-risk scenarios and model optimization.

[0003] If collision prediction models are tested and optimized directly in real-world scenarios, testers will inevitably be exposed to potential collision risks. If the robotic arm fails to accurately identify and accidentally collides with a human, it could not only damage the equipment and disrupt the production process, but also injure the testers, resulting in incalculable losses. Digital twin technology, however, can not only replicate human-machine collaboration by creating a virtual digital environment highly similar to the physical system, and provide real-time immersive first-person action interaction and scene feedback using virtual reality technology, but it can also repeatedly test collision prediction models without real-world collision risks, further optimizing human-machine collaboration. This approach can significantly improve algorithm performance and greatly reduce safety risks during testing and optimization, providing a safer and more efficient solution for human-machine collaboration in the field of intelligent manufacturing. Summary of the Invention

[0004] The purpose of this application is to provide a human-computer collaborative digital twin simulation method based on virtual reality interaction, which constructs a highly realistic and immersive digital twin system that can not only safely and reliably reproduce real work scenarios, but also optimize human-computer collaborative behavior. It can repeatedly test and optimize the collision prediction model without the risk of real-world collisions.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] Firstly, this application provides a human-computer collaborative digital twin simulation method based on virtual reality interaction, the human-computer collaborative digital twin simulation method based on virtual reality interaction comprising:

[0007] Constructing a digital twin 3D scene corresponding to the real working environment, as well as a digital twin human body model and a digital twin robotic arm model for virtual human-machine collaboration in the digital twin 3D scene;

[0008] Acquire human posture data of a real test subject and joint motion data of a real robotic arm; the human posture data includes position and posture; the joint motion data includes the velocity and angle of each joint;

[0009] The digital twin human body model is driven to move based on the human posture data, and the digital twin robotic arm model is driven to move based on the joint motion data, so that the digital twin human body model is synchronized with the actual test personnel's movements, and the digital twin robotic arm model is synchronized with the actual robotic arm's movements.

[0010] Determine whether a physical collision occurs between the digital twin human body model and the physical assets within the digital twin 3D scene. If so, determine and record the collision location of the human body and the collision area of ​​the scene, and return to the step of "obtaining the human posture data of the real test personnel and the joint motion data of the real robotic arm". If not, return to the step of "obtaining the human posture data of the real test personnel and the joint motion data of the real robotic arm".

[0011] Optionally, a digital twin 3D scene corresponding to the real-world working environment is constructed, specifically including:

[0012] A scene model is created for the real-world working environment of human-machine collaboration to obtain a digital twin 3D scene. The physical terrain boundary of the digital twin 3D scene is then set to ensure that the digital twin 3D scene is consistent with the real-world working environment.

[0013] Optionally, the digital twin human body model includes a digital twin human skeletal model and a digital twin human physical model, and the digital twin robotic arm model includes a digital twin robotic arm skeletal model and a digital twin robotic arm physical model; the digital twin human skeletal model can be bound to different digital twin human physical models through a skeletal redirector, and the digital twin robotic arm skeletal model can be bound to different digital twin robotic arm physical models through a skeletal redirector, adapting to different human-machine collaborative operation scenarios, including handling, welding, and disassembly / assembly.

[0014] Optionally, the human posture data is data collected using virtual reality equipment, and the joint motion data is data collected using sensors;

[0015] The virtual reality device includes a head-mounted display worn on the head of a real-world tester, a controller worn on the hands of the real-world tester, and multiple positioning base stations installed within the test area where the real-world tester is located. The positioning base stations are used to emit laser signals. Both the head-mounted display and the controllers are equipped with gyroscopes and laser receivers. The gyroscopes are used to measure the posture of the real-world tester, and the laser receivers are used to measure the position of the real-world tester based on the received laser signals. The head-mounted display is used to view virtual human-machine collaboration between the digital twin human body model and the digital twin robotic arm model from a first-person perspective. The controllers have buttons for controlling the movement and hand actions of the digital twin human body model.

[0016] The sensors include speed sensors and angle sensors installed at each joint of the actual robotic arm.

[0017] Optionally, driving the movement of the digital twin human model based on the human posture data specifically includes: using the human posture data and the digital twin human skeleton model as input, using a skeletal inverse kinematics solver to obtain the joint data of the digital twin human skeleton model; and driving the movement of the digital twin human model based on the joint data.

[0018] Optionally, each joint of the digital twin robotic arm skeleton model is provided with rotational joint constraints and joint angle variables. The movement of the digital twin robotic arm model is driven based on the joint motion data. Specifically, this includes: using the joint motion data as input, updating the joint angle variables of each joint of the digital twin robotic arm skeleton model using a state updater, thereby driving the movement of the digital twin robotic arm model.

[0019] Optionally, before determining whether a physical collision occurs between the digital twin human body model and the physical assets within the digital twin 3D scene, the human-computer collaborative digital twin simulation method based on virtual reality interaction further includes: determining whether the digital twin human body model has moved to the scene physical terrain boundary of the digital twin 3D scene; if so, triggering an interface pop-up and a virtual air wall to block it.

[0020] Optionally, the collision points of the human body and the collision area of ​​the scene are identified and recorded, specifically including:

[0021] The pose recording data is obtained by using a collision event dispatcher to record the human joint pose data of the digital twin human model in the frame before the collision and the human joint pose data of the digital twin human model in the current frame of the collision.

[0022] Based on the posture recording data, the collision points of the human body and the collision areas of the scene are determined using the ray collision detection function.

[0023] Record the points of human collision and the areas of scene collision.

[0024] Optionally, after determining and recording the human collision site and the scene collision area, the human-computer collaborative digital twin simulation method based on virtual reality interaction further includes: visualizing the human collision site and the scene collision area;

[0025] This includes visualizing the collision points of the human body and the collision areas of the scene, specifically including:

[0026] Visualize the points of human collision in the form of a virtual reality pop-up interface;

[0027] The collision area of ​​the scene is visualized in the form of a capsule.

[0028] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0029] This application provides a human-computer collaboration digital twin simulation method based on virtual reality interaction. First, a digital twin 3D scene corresponding to the real-world work environment is constructed, along with a digital twin human body model and a digital twin robotic arm model for virtual human-computer collaboration within the 3D scene. After acquiring the human posture data of the real test personnel and the joint motion data of the real robotic arm, the movement of the digital twin human body model is driven based on the human posture data, and the movement of the digital twin robotic arm model is driven based on the joint motion data, thus achieving virtual human-computer collaboration. During the virtual human-computer collaboration process, it is determined in real time whether physical collisions occur between the digital twin human body model and the physical assets within the digital twin 3D scene. When a collision occurs, the collision point of the human body and the scene collision area are identified and recorded. The recorded collision point of the human body and the scene collision area can provide input data for testing and optimizing the collision prediction model. Therefore, this application can safely and reliably reproduce real-world work scenarios and optimize human-computer collaboration behavior without actual human-computer collaboration, and can repeatedly test and optimize the collision prediction model without the risk of real-world collisions. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is an application environment diagram of a human-computer collaborative digital twin simulation method based on virtual reality interaction, provided in Embodiment 1 of this application.

[0032] Figure 2 This is a flowchart illustrating a human-computer collaborative digital twin simulation method based on virtual reality interaction, as provided in Embodiment 1 of this application.

[0033] Figure 3 This is a schematic diagram illustrating the principle of a human-computer collaborative digital twin simulation method based on virtual reality interaction, as provided in Embodiment 1 of this application.

[0034] Figure 4 This is a schematic diagram of a digital twin three-dimensional scene of a factory processing line provided in Embodiment 1 of this application.

[0035] Figure 5 This is a schematic diagram of the digital twin human skeletal model and the digital twin human physical model provided in Embodiment 1 of this application; wherein, Figure 5 (a) in the image is a digital twin human skeletal model. Figure 5 (b) in the figure is a digital twin human physical model.

[0036] Figure 6 This is a schematic diagram of real-time pose mapping of a digital twin human body model provided in Embodiment 1 of this application.

[0037] Figure 7 This is a schematic diagram of the hand movement mapping of the digital twin human body model provided in Embodiment 1 of this application.

[0038] Figure 8 This is a schematic diagram of an example robotic arm provided in Embodiment 1 of this application.

[0039] Figure 9 This is a flowchart illustrating the digital twin human state monitoring algorithm provided in Embodiment 1 of this application.

[0040] Figure 10 This is a schematic diagram of the X-ray collision detection provided in Embodiment 1 of this application.

[0041] Figure 11 This is a schematic diagram of the human body collision site and scene collision area provided in Embodiment 1 of this application.

[0042] Figure 12 This is a schematic diagram of the structure of a computer device provided in Embodiment 2 of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] Example 1

[0045] The human-computer collaborative digital twin simulation method based on virtual reality interaction provided in this application can be applied to, for example... Figure 1 The application environment shown is as follows. The terminal communicates with the server via a network. The data storage system stores the data the server needs to process. The data storage system can be set up independently, integrated into the server, or placed in the cloud or on another server. The terminal can send the human posture data and joint motion data to be processed to the server. After receiving the human posture data and joint motion data, the server constructs a digital twin 3D scene corresponding to the real-world working environment, as well as a digital twin human body model and a digital twin robotic arm model for virtual human-machine collaboration within the digital twin 3D scene. Based on the human posture data, the server drives the movement of the digital twin human body model; based on the joint motion data, it drives the movement of the digital twin robotic arm model, ensuring that the digital twin human body model synchronizes with the movements of the real test personnel, and the digital twin robotic arm model synchronizes with the movements of the real robotic arm. The server then determines whether physical collisions occur between the digital twin human body model and the physical assets within the digital twin 3D scene. If so, it identifies and records the collision points of the human body and the scene.

[0046] Furthermore, in some embodiments, the human-computer collaborative digital twin simulation method based on virtual reality interaction can also be implemented by a server or a terminal alone. For example, the terminal can directly process the human posture data and joint motion data to be processed, or the server can obtain the human posture data and joint motion data to be processed from the data storage system and process the human posture data and joint motion data to be processed.

[0047] In the current research field of human-computer collaboration, there is a lack of a virtual simulation platform that can highly reproduce real-world human-computer collaboration scenarios and accurately simulate human-computer interaction. Testers need a safe virtual simulation platform to optimize and standardize human-computer collaboration behavior and conduct comprehensive pre-testing and repeated debugging of collision prediction models. This avoids the collision risks associated with directly conducting human-computer collaboration tests in complex work environments, and allows for a safer and more efficient digital solution. Based on this, this embodiment proposes a human-computer collaboration digital twin simulation method based on virtual reality interaction. It provides testers with a highly realistic and immersive digital twin system that not only safely and reliably reproduces real-world work scenarios but also optimizes human-computer collaboration behavior, providing strong technical support for safety testing and task optimization in intelligent production and complex work environments.

[0048] Specifically, such as Figure 2 and Figure 3As shown, this embodiment provides a human-computer collaborative digital twin simulation method based on virtual reality interaction. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking the server in the example, the following steps are included:

[0049] Step S1: Construct a digital twin 3D scene corresponding to the real working environment, as well as a digital twin human body model and a digital twin robotic arm model for virtual human-machine collaboration in the digital twin 3D scene.

[0050] Step S2: Obtain the human posture data of the actual test personnel and the joint motion data of the actual robotic arm; the human posture data includes position and posture; the joint motion data includes the speed and angle of each joint.

[0051] Step S3: Drive the movement of the digital twin human body model based on the human posture data, and drive the movement of the digital twin robotic arm model based on the joint motion data, so that the digital twin human body model is synchronized with the movement of the real test person, and the digital twin robotic arm model is synchronized with the movement of the real robotic arm.

[0052] Step S4: Determine whether a physical collision occurs between the digital twin human body model and the physical assets in the digital twin 3D scene. If yes, determine and record the collision location of the human body and the collision area of ​​the scene, and return to the step of "obtaining the human body posture data of the real test personnel and the joint motion data of the real robotic arm". If no, return to the step of "obtaining the human body posture data of the real test personnel and the joint motion data of the real robotic arm".

[0053] By implementing steps S1 to S4 above, this embodiment constructs a digital twin 3D scene, a digital twin human body model, and a digital twin robotic arm model. The digital twin human body model is driven to move based on the human posture data of a real tester, and the digital twin robotic arm model is driven to move based on the joint motion data of the real robotic arm. This allows for the simulation of human-machine collaboration through virtual reality, recording the human body collision points and scene collision areas during the simulated human-machine collaboration process. This provides input data for testing and optimizing the collision prediction model. Digital twin technology not only replicates human-machine collaboration scenarios by establishing a virtual digital environment highly similar to the physical system, and provides real-time immersive first-person action interaction and scene feedback using virtual reality technology, but it also allows for repeated testing and optimization of the collision prediction model without real-world collision risks. Testers are aware of which human-machine collaboration actions are prone to collisions and consciously avoid using these actions, further optimizing human-machine collaboration. This method not only significantly improves algorithm performance but also greatly reduces safety risks during testing and optimization, providing a safer and more efficient solution for human-machine collaboration in the field of intelligent manufacturing.

[0054] In step S1, a digital twin 3D scene is first constructed to simulate the real working environment (which is the specific place for real human-machine collaboration). Then, in the digital twin 3D scene, a digital twin human body model and a digital twin robotic arm model are constructed that can be perceived in real time and can perform virtual human-machine collaboration in the digital twin 3D scene.

[0055] Digital twin 3D scenes are simulation models based on real-world working environments, such as... Figure 4 As shown, this is a digital twin 3D scene of a factory production line. Step S1, constructing the digital twin 3D scene corresponding to the real-world work environment, specifically includes: modeling the real-world work environment of human-machine collaboration to obtain the digital twin 3D scene, and setting physical terrain boundaries for the digital twin 3D scene to ensure consistency between the digital twin 3D scene and the real-world work environment. By setting the physical terrain boundaries, the consistency of the work area between the digital twin 3D scene and the real-world work environment is maintained.

[0056] In step S1, such as Figure 5As shown, the digital twin human body model includes a digital twin human skeletal model and a digital twin human body physical model, and the digital twin robotic arm model includes a digital twin robotic arm skeletal model and a digital twin robotic arm physical model. The skeletal models of both the digital twin human body model and the digital twin robotic arm model can be designed to be bound. Different human body and robotic arm physical models can be bound using a skeletal redirector. That is, with similar skeletons, different human body physical models and robotic arm physical models can be bound using the skeletal redirector. Leveraging the transferability of digital skeletal assets, it can adapt to different types of workers, different models of robotic arms, and different work scenarios, thus adapting to various human-machine collaborative work scenarios. Specifically, the digital twin human skeletal model can be bound to different digital twin human body physical models through the skeletal redirector, and the digital twin robotic arm skeletal model can be bound to different digital twin robotic arm physical models through the skeletal redirector, adapting to different human-machine collaborative work scenarios, including handling, welding, and assembly / disassembly.

[0057] In this embodiment, the human posture data of the real test personnel and the joint motion data of the real robotic arm are collected by virtual reality equipment and sensors respectively. In step S2, the human posture data is the data collected by virtual reality equipment and the joint motion data is the data collected by sensors.

[0058] The virtual reality equipment includes a head-mounted display worn by a real-world tester, controllers worn by the tester, and multiple positioning base stations installed within the test area where the tester is located. The positioning base stations emit laser signals. Both the head-mounted display and the controllers contain gyroscopes and laser receivers. The gyroscopes measure the tester's posture, and the laser receivers measure the tester's position based on the received laser signals. The head-mounted display provides a first-person perspective of virtual human-machine collaboration between a digital twin human body model and a digital twin robotic arm model. The controllers have buttons for controlling the movement and hand gestures of the digital twin human body model. The controllers can move the digital twin human body model forward, backward, left, and right, as well as turn it. They can also select and map hand gestures to the digital twin human body model, allowing for physical interaction with the digital twin 3D scene, such as dragging the robotic arm for teaching, grasping objects, and pushing or pulling doors.

[0059] The virtual reality device in this embodiment may also include devices worn on other parts of the human body, such as the waist, arms, legs, and knees.

[0060] The human posture data acquisition process is as follows: The test personnel use the head-mounted display and controllers integrated into the virtual reality device to measure actual human posture data and input control signals. The human posture data is based on high-frequency laser signals emitted by multiple positioning base stations deployed within the test area. The laser receivers in the head-mounted display and controllers receive these high-frequency laser signals and analyze core parameters such as the time difference, distance, and angle of the high-frequency laser signals to measure the position coordinates and orientation angle (i.e., position) of the test personnel in real three-dimensional space. Simultaneously, the gyroscope measures the posture of the test personnel in real three-dimensional space. Control signals are emitted by executing different buttons during human-computer interaction using the controllers.

[0061] The sensors include speed sensors and angle sensors installed at each joint of the actual robotic arm.

[0062] The joint motion data acquisition process is as follows: The real robotic arm is driven by a motion trajectory planning program. The actual joint motion data of the real robotic arm is collected by speed sensors and angle sensors installed on each joint (including the end effector). The joints can be driven by motors.

[0063] The pose mapping of a digital twin human body model involves inputting sensor data from virtual reality devices, input motion mapping sets, and the digital twin human skeletal model into a skeletal inverse kinematics solver. This solver obtains the basic joint data of the digital twin human skeletal model and triggers corresponding animations of moving parts of the digital twin human body model, achieving real-time mapping of realistic motion postures and real-time mapping of realistic hand movements. Figure 6 and Figure 7 As shown, the input action mapping set is defined by different buttons on the gamepad. The touchpad, trigger button, side grip button, and other buttons correspond to different hand operation animations (i.e., hand movements), and can also control the forward, backward, left, right movement and turning of the human body.

[0064] In step S3, driving the motion of the digital twin human body model based on human posture data specifically includes: using human posture data and the digital twin human skeleton model as input, using a skeletal inverse kinematics solver to obtain the joint data of the digital twin human skeleton model, and driving the motion of the digital twin human body model based on the joint data. When the handle has the function of selecting hand movements, the human posture data at this time also includes hand movements.

[0065] The digital twin robotic arm model is based on a 3D model of a robotic arm. By setting the base as the root skeleton and connecting each joint sequentially towards the end effector, a kinematic chain is formed. Rotational joint constraints and angular variables are set between each joint. Sensors on each joint of the real robotic arm collect joint motion data, package this data into data packets, and send the real-time joint motion data to the state updater of the digital twin robotic arm skeleton model via a TCP socket communication service. Upon receiving the real-time joint motion data from the sensors, the state updater of the digital twin robotic arm skeleton model updates the joint angle variables, ensuring synchronization between the virtual and real robotic arm movements. For demonstration purposes, this embodiment provides the following... Figure 8 The 7-axis robotic arm model shown in the Franka EmikaPanda series can be implemented in ways that go beyond the digital twin mapping of this type of robotic arm.

[0066] In step S3, each joint of the digital twin robotic arm skeleton model is set with rotational joint constraints and joint angle variables. The movement of the digital twin robotic arm model is driven based on the joint motion data. Specifically, the joint motion data is used as input, and the joint angle variables of each joint of the digital twin robotic arm skeleton model are updated using a state updater to drive the movement of the digital twin robotic arm model.

[0067] This embodiment designs a digital twin human body state monitoring algorithm to detect and record collisions between the digital twin human body model and the physical assets of the digital twin 3D scene, and visualizes the collision parts of the human body and the collision area of ​​the scene.

[0068] like Figure 9As shown, the process of the digital twin human body detection algorithm is as follows: First, based on the human posture data obtained from the virtual reality device, it is synchronized to the digital twin human body model in the digital twin 3D scene to achieve accurate positioning of the digital twin human body model in the digital twin 3D scene; then, it is determined whether the digital twin human body model exceeds the scene physical terrain boundary of the digital twin 3D scene. Once it is detected that the digital twin human body model touches the scene physical terrain boundary of the digital twin 3D scene, an interface pop-up window and a virtual air wall will be triggered. The interface pop-up window refers to the prompt displayed on the head-mounted display device that the scene physical terrain boundary has been reached, and the virtual air wall will be triggered. Air wall blocking refers to preventing the digital twin human model from actually exceeding the boundaries of the digital twin 3D scene. If it does not exceed the physical terrain boundaries of the digital twin 3D scene, it proceeds to the subsequent human body physical collision detection process. The human body physical collision detection process is as follows: when a volume overlap collision occurs between the digital twin human model and a physical asset (i.e., all objects in the digital twin 3D scene, including but not limited to the digital twin robotic arm model) within the digital twin 3D scene, the collision event dispatcher is immediately triggered. This collision event dispatcher quickly responds and records the human joint pose data of the frame before and the current frame of the collision; for example... Figure 10 As shown, the ray collision detection function is called, combining the geometric characteristics of the digital twin human model with the mesh boundaries of the physical assets of the digital twin 3D scene to calculate and determine the human collision points and scene collision areas. The scene collision areas within the digital twin 3D scene are rendered and displayed in capsule form, while the human collision points are updated in the human collision interface, that is, the human collision points are visualized in the form of a virtual reality pop-up interface, as shown. Figure 11 As shown, it records collision data, including the collision points of the human body and the collision areas of the scene, and updates the human-machine status.

[0069] In calculating the collision points of the human body and the scene, the intersection points of rays with the physical assets are calculated based on the mesh boundaries of the physical assets of the digital twin human body model and the digital twin 3D scene. The starting point of the ray is the wearing point of the virtual reality device, and the direction is towards the overlapping mesh of the volume, thereby determining the specific location where the collision occurs, i.e., determining the human body collision point and the scene collision area. In this process, the digital twin human body model is usually simplified into a geometric shape (such as a capsule, sphere, etc.) based on the human joint pose data of the frame before and the frame before the collision.

[0070] Before determining whether a physical collision occurs between the digital twin human body model and the physical assets within the digital twin 3D scene, the human-computer collaborative digital twin simulation method based on virtual reality interaction in this embodiment further includes: determining whether the digital twin human body model has moved to the scene physical terrain boundary of the digital twin 3D scene; if so, triggering an interface pop-up and a virtual air wall to block it.

[0071] Determine whether a physical collision occurs between the digital twin human body model and the physical assets within the digital twin 3D scene. If so, identify and record the collision location of the human body and the collision area of ​​the scene, and return to the step of "obtaining the human posture data of the real test personnel and the joint motion data of the real robotic arm" to update the human-machine state. If not, return to the step of "obtaining the human posture data of the real test personnel and the joint motion data of the real robotic arm" to update the human-machine state.

[0072] Specifically, determining and recording the human body collision location and scene collision area includes: using a collision event dispatcher to record the human joint posture data of the digital twin human model in the frame before the collision and the human joint posture data of the digital twin human model in the current frame of the collision, to obtain posture recording data; based on the posture recording data, using a ray collision detection function to determine the human body collision location and scene collision area; and recording the human body collision location and scene collision area.

[0073] After determining and recording the human body collision site and the scene collision area, the human-computer collaborative digital twin simulation method based on virtual reality interaction in this embodiment further includes: visualizing the human body collision site and the scene collision area.

[0074] The visualization of human collision sites and scene collision areas includes: visualizing human collision sites in the form of virtual reality pop-up interfaces; and visualizing scene collision areas in the form of capsules.

[0075] The entire digital twin system in this embodiment can be built in Unreal Engine 5, and all the functions used can be implemented by calling the functions encapsulated in Unreal Engine.

[0076] This embodiment discloses a human-computer collaborative digital twin simulation method based on virtual reality interaction, belonging to the field of human-computer interaction and virtual reality technology. The method includes: first, collecting human posture data of a real-world tester and joint motion data of a real-world robotic arm using virtual reality devices and sensors; then, constructing a real-time perceptible digital twin human body model and a digital twin robotic arm model in a digital twin 3D scene simulating a real-world work environment. The skeletal models of the digital twin human body model and the digital twin robotic arm model can be bound to different physical models via a skeletal redirector to adapt to different human-computer collaborative work scenarios; finally, based on a digital twin human body state monitoring algorithm, real-time detection and collision data recording are performed between the digital twin human body model and the physical assets of the digital twin 3D scene, and the collision points of the human body and the scene are visualized. This embodiment constructs a highly realistic and immersive digital twin system that can not only safely and reliably reproduce real-world work scenarios but also optimize human-computer collaborative behavior, providing strong technical support for safety testing and task optimization in intelligent production and complex work environments.

[0077] The method in this embodiment has the following advantages:

[0078] (1) Digital twin systems accurately simulate real-world human-computer collaboration scenarios and reproduce human-computer interaction behaviors, enabling testers to optimize and standardize human-computer collaboration behaviors in a safe, reliable, and immersive environment. They also allow for the verification of collision prediction models, improving testing efficiency and reducing testing costs. Compared to real-world scenario testing, more human-computer collaboration actions can be tested.

[0079] (2) The digital twin human body status monitoring algorithm will trigger the collision event dispatcher after detecting a collision, record human body collision data, and display the scene collision area and human body collision part visualization interface, providing collision visualization feedback and related data support for human-computer collaborative algorithm research.

[0080] (3) The digital skeleton model can be bound to different human physical models or robotic arm physical models according to specific actual requirements, without the need for tedious and complicated modeling work, and can be transferred to different virtual scenes for plug-and-play use, with universality for different working environments.

[0081] The robotic arm in this embodiment can be replaced with other types of robots.

[0082] This application also provides an application scenario where the aforementioned human-computer collaborative digital twin simulation method based on virtual reality interaction is applied. Specifically, the human-computer collaborative digital twin simulation method based on virtual reality interaction provided in this embodiment can be applied in a model testing scenario. The model testing scenario includes a data acquisition stage, a simulation stage, and a testing stage. The data acquisition stage is used to acquire human posture data and joint motion data. The simulation stage is used to perform virtual human-computer collaboration simulation based on the human posture data and joint motion data to obtain collision data. The testing stage is used to test and optimize the collision prediction model based on the collision data. The human-computer collaborative digital twin simulation method based on virtual reality interaction provided in this embodiment belongs to the simulation stage.

[0083] Example 2

[0084] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 12 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a human-computer collaborative digital twin simulation method based on virtual reality interaction.

[0085] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0086] In one exemplary embodiment, a computer device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the human-computer collaborative digital twin simulation method based on virtual reality interaction described in Embodiment 1.

[0087] Example 3

[0088] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the human-computer collaborative digital twin simulation method based on virtual reality interaction described in Embodiment 1.

[0089] Example 4

[0090] This application provides a computer program product, including a computer program that, when executed by a processor, implements the human-computer collaborative digital twin simulation method based on virtual reality interaction described in Embodiment 1.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A human-in-the-loop digital twin simulation method based on virtual reality interaction, characterized in that, The human-machine cooperation digital twin simulation method based on virtual reality interaction comprises: constructing a digital twin three-dimensional scene corresponding to a real work environment, and a digital twin human body model and a digital twin mechanical arm model for virtual human-machine cooperation in the digital twin three-dimensional scene; acquiring human body posture data of a real test person and joint motion data of a real mechanical arm; the human body posture data comprises position and posture; the joint motion data comprises the speed and angle of each joint; driving the digital twin human body model to move based on the human body posture data, and driving the digital twin mechanical arm model to move based on the joint motion data, so that the digital twin human body model is synchronized with the action of the real test person, and the digital twin mechanical arm model is synchronized with the action of the real mechanical arm; determining whether physical collision occurs between the digital twin human body model and physical assets in the digital twin three-dimensional scene, and if so, determining and recording the human body collision part and the scene collision area, providing input data for testing and optimization of a collision prediction model, updating the human-machine state, and returning to the step of acquiring the human body posture data of the real test person and the joint motion data of the real mechanical arm; if not, updating the human-machine state and returning to the step of acquiring the human body posture data of the real test person and the joint motion data of the real mechanical arm; The digital twin human body model comprises a digital twin human body skeleton model and a digital twin human body physical model, and the digital twin mechanical arm model comprises a digital twin mechanical arm skeleton model and a digital twin mechanical arm physical model; the digital twin human body skeleton model can bind different digital twin human body physical models through a skeleton repositioner, the digital twin mechanical arm skeleton model can bind different digital twin mechanical arm physical models through a skeleton repositioner, and different human-machine cooperation work scenes, including carrying, welding and disassembly, are adapted to. Before determining whether physical collision occurs between the digital twin human body model and physical assets in the digital twin three-dimensional scene, the human-machine cooperation digital twin simulation method based on virtual reality interaction further comprises: determining whether the digital twin human body model moves to the scene physical terrain boundary of the digital twin three-dimensional scene, and if so, triggering an interface pop-up window and a virtual air wall block; the interface pop-up window is used to display a prompt that the scene physical terrain boundary has been reached, and the virtual air wall block is used to prevent the digital twin human body model from exceeding the digital twin three-dimensional scene. Determining and recording the human body collision part and the scene collision area specifically comprises: recording the human body joint posture data of the digital twin human body model in the frame before collision and the human body joint posture data of the digital twin human body model in the current frame of collision respectively by using a collision event distributor to obtain posture record data; determining the human body collision part and the scene collision area based on the posture record data by using a ray collision detection function; and recording the human body collision part and the scene collision area.

2. The human co-operative digital twin simulation method based on virtual reality interaction according to claim 1, characterized in that, Constructing a digital twin three-dimensional scene corresponding to a real work environment specifically comprises: The scene modeling is performed on a real work environment of human-robot collaboration to obtain a digital twin three-dimensional scene, and a scene physical terrain boundary is set for the digital twin three-dimensional scene, so that the digital twin three-dimensional scene is consistent with the real work environment.

3. The human co-operative digital twin simulation method based on virtual reality interaction according to claim 1, characterized in that, The human body posture data is collected by a virtual reality device, and the joint motion data is collected by a sensor. The virtual reality device includes a head-mounted display device worn on the head of a real tester, a handle worn on the hand of the real tester, and a plurality of positioning base stations installed in a test area where the real tester is located; the positioning base stations are used to emit laser signals; the head-mounted display device and the handle are both provided with a gyroscope and a laser receiver, the gyroscope is used to measure the posture of the real tester, and the laser receiver is used to measure the position of the real tester based on the received laser signals; the head-mounted display device is used to view virtual human-robot collaboration between the digital twin human body model and the digital twin mechanical arm model in a first-person perspective; the handle has a key for controlling the movement and hand action of the digital twin human body model; The sensor includes a speed sensor and an angle sensor installed at each joint of the real mechanical arm.

4. The human co-operative digital twin simulation method based on virtual reality interaction according to claim 1, characterized in that, The digital twin human body model is driven based on the human body posture data, specifically including: taking the human body posture data and the digital twin human body skeleton model as input, using a skeletal inverse kinematics solver to solve joint data of the digital twin human body skeleton model; and driving the digital twin human body model based on the joint data.

5. The human co-operative digital twin simulation method based on virtual reality interaction according to claim 1, characterized in that, Each joint of the digital twin mechanical arm skeleton model is provided with a revolute pair constraint and a joint angle variable, and the digital twin mechanical arm model is driven based on the joint motion data, specifically including: taking the joint motion data as input, using a state updater to update the joint angle variable of each joint of the digital twin mechanical arm skeleton model, and driving the digital twin mechanical arm model.

6. The human co-operative digital twin simulation method based on virtual reality interaction according to claim 1, characterized in that, After determining and recording the human body collision part and the scene collision area, the human-robot collaboration digital twin simulation method based on virtual reality interaction further includes visualizing the human body collision part and the scene collision area. The human body collision part and the scene collision area are visualized, specifically including: The human body collision part is visualized in the form of a virtual reality pop-up interface; The scene collision area is visualized in the form of a capsule.

Images