Cyber-physical fusion man-machine collaborative three-dimensional visual interaction deduction method and system

Abstract

The present application belongs to the field of agent motion deduction, and particularly relates to a virtual-real fusion man-machine collaborative three-dimensional visual interaction deduction method and system. The method comprises: mapping a virtual and real environment state space through an augmented reality device to obtain a virtual-real fusion deduction space for display to a user; acquiring in real time situation information of an entity model of a deduction object in a real environment region, including ontology information and motion information; and superimposing the entity model situation information to a corresponding entity model in the deduction space for display through the augmented reality device; the entity model moves in the real environment according to a corresponding trajectory through a XYZ three-dimensional direction movable displacement device; and the situation information of each virtual deduction object and a preset trajectory in the XYZ three-dimensional direction are calculated in real time through the situation information of the entity model and other virtual deduction objects to control the virtual deduction object to move according to the preset trajectory, and the situation information of the virtual deduction object is superimposed to the virtual deduction object for display through the augmented reality device.

Description

Technical Field

[0001] This invention belongs to the field of intelligent agent motion simulation, specifically involving a virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation method and system. Background Technology

[0002] Intelligent agent motion simulation refers to the prediction, planning, and reproduction of information such as the position and trajectory of an intelligent agent over time, given an environment, constraints, and objectives, through model calculation, simulation iteration, or algorithmic simulation, using an intelligent agent with autonomous perception, decision-making, and execution capabilities. For complex task simulations, especially in scenarios involving the fusion of virtual and real elements, human intervention is often required to form human-machine collaboration. Current technologies for traditional human-machine collaborative simulations typically involve twinning the actual scene into a small-scale experimental space and combining physical entities for virtual-real fusion simulations. Although a 3D scene can be constructed through mapping, this is usually limited to a 3D display of the simulation scene; the interaction between physical and virtual entities remains confined to a 2D plane, and human-machine collaboration is primarily achieved through data detached from the actual scene.

[0003] Chinese invention patent application CN121053835A discloses a virtual-real fusion traffic twin simulation testing system and method for air-ground collaboration, belonging to the field of intelligent transportation design optimization, verification, or simulation. It aims to provide a highly flexible multi-scenario experimental platform for teaching and research through the concept of "physical twin-dynamic mapping." Based on dynamic display technology, the system twins real traffic scenarios such as urban roads, emergency evacuation, and multi-level parking into a small-scale experimental space, supporting rapid switching of scenarios and spatial layouts to achieve seamless mapping between virtual and real environments. The central control system, through virtual-real fusion technology, synchronizes the motion trajectories of physical devices such as drones and intelligent vehicles with virtual traffic flow in real time, forming a two-way interactive closed loop of "small-space entity-central control virtual." With the core idea of ​​"multi-scenario adaptive integration," combined with air-ground collaboration capabilities, it supports dynamic reconstruction and nonlinear migration of tasks such as logistics delivery and emergency response, enabling cross-scenario experiments without hardware modifications. However, the interactive simulation of vehicles in this solution is mainly implemented in a two-dimensional plane, and the motion trajectory is collected in real time through a motion capture system, processed by the central control system, and synchronized to the cloud for data monitoring. Summary of the Invention

[0004] The purpose of this invention is to provide a human-computer collaborative 3D visual interactive simulation method and system that integrates virtual and real elements, in order to solve the problem that the interaction simulation of physical and virtual objects in the prior art has spatial limitations and mainly achieves human-computer collaboration in the form of data detached from the scene.

[0005] To achieve the above objectives, the first aspect of the present invention provides a human-computer collaborative 3D visual interactive inference method that integrates virtual and real elements, comprising: By mapping virtual and real environment state spaces using augmented reality devices, a virtual-real fusion simulation space is obtained and displayed to the user; The system acquires real-time situational information of the simulated object entity model in the real environment area, including ontology information and motion information; and displays the situational information of the entity model on the corresponding entity model in the simulation space through augmented reality equipment; the entity model moves in the real environment according to the corresponding trajectory through a displacement device that can move in the XYZ three-dimensional direction. By using the situational information of the physical model and other virtual simulation objects, the situational information and preset trajectory of each virtual simulation object in the XYZ three-dimensional direction are calculated in real time to control the virtual simulation object to move according to the preset trajectory. The situational information of the virtual simulation object is then superimposed onto the corresponding virtual simulation object in the simulation space through augmented reality devices for display.

[0006] The first aspect of the present invention provides a virtual-real fusion human-computer collaborative three-dimensional visual interactive inference method. In one possible implementation, the motion information of the entity model is obtained based on the position and shape of the motion marker corresponding to the entity model in the image, as well as the positional and posture relationships between the motion marker and the entity model body. The identity information of an entity model is obtained by scanning the ontology marker corresponding to that entity model. The motion marker and the body marker are the same identifier code. This identifier code contains the identity information of the physical model and is fixed on the surface of the physical model that can be photographed. The motion pattern of the identifier code is consistent with that of the physical model.

[0007] The first aspect of the present invention provides a virtual-real fusion human-computer collaborative three-dimensional visual interactive inference method. In one possible implementation, the method of obtaining the virtual-real fusion inference space displayed to the user includes: using scene registration technology to complete the mapping of virtual and real environment state spaces, and performing consistency matching of virtual and real scenes. Based on the functions and components displayed to users in augmented reality devices, including gesture recognition and UI interfaces, users are provided with various perspectives and detailed zoom displays in the simulation space.

[0008] The first aspect of the present invention provides a human-computer collaborative three-dimensional visual interactive simulation method that integrates virtual and real elements. In one possible implementation, it further includes: acquiring real-time state information of the entity model and the virtual simulation object in real time, and displaying the virtual state markers corresponding to the real-time state information on the corresponding entity model and the virtual simulation object in the simulation space through an augmented reality device.

[0009] The first aspect of this invention provides a human-computer collaborative three-dimensional visual interactive simulation method that integrates virtual and real elements. In one possible implementation, situational information, including basic attribute information, performance parameters, energy information, preset trajectory, and tracking trajectory, is superimposed onto the physical model and the virtual simulation object by an augmented reality device. The tracking trajectory is the actual motion trajectory of the simulation object. The method for determining the preset trajectory of the entity model includes: triggering the corresponding path planning algorithm based on the entity model preset trajectory planning function selected by the user when operating on the corresponding interface in the simulation space, and obtaining a new preset trajectory of the entity model.

[0010] The technical solution described above provides a novel human-computer collaborative 3D visual interactive simulation method that integrates virtual and real elements. Its beneficial effects include: breaking the traditional spatial limitations of simulation by using augmented reality devices for human-computer collaborative participation; and integrating the real-time position, motion state, and other situational information data of the physical simulation object with virtual situational elements in the virtual-real fusion dimension, superimposing the corresponding situational information onto the virtual and physical simulation objects. This intuitive design allows operators to intuitively understand the spatial correspondence between the virtual and real environments and their corresponding simulation objects, reducing the operator's cognitive load. Both the physical and virtual models can achieve XYZ three-dimensional motion. Combined with the construction of a virtual-real fusion 3D spatial simulation environment and the interactive linkage between physical and virtual simulation objects, virtual-real interactive simulation is no longer limited to a two-dimensional plane, facilitating interactive human-computer collaborative visual simulation correction. Under this simulation system, operators and machines can cooperate efficiently. The machine uses a simulation system and a specific path planning algorithm to simulate the movement of complex groups, thereby realizing the path simulation of virtual and real intelligent agents in a complex simulation environment. The results can assist operators in making decisions. This mechanism for sharing and updating situational awareness information in real time can effectively deal with complex environmental information and improve the efficiency of simulation decision-making.

[0011] The second aspect of the present invention provides a virtual-real integrated human-computer collaborative three-dimensional visual interactive simulation system, including an augmented reality device, an image acquisition device, a back-end computing module, a simulation environment of physical objects and a simulation object entity model; the entity model can move in the real environment according to a corresponding trajectory through a displacement device movable in the XYZ three-dimensional direction; Image acquisition devices are used to acquire images of the simulation environment and the entity model of the simulation object to construct a real environment state space; The backend computing module is used to acquire the situation information of the deduced object entity model in the real environment area in real time based on the image acquired by the image acquisition device, including the body information and motion information; it is also used to calculate the situation information and preset trajectory of each virtual deduced object in the XYZ three-dimensional direction in real time through the situation information of the entity model and other virtual deduced objects, so as to control the virtual deduced objects to move according to the preset trajectory. Augmented reality devices are used to map virtual and real environment state spaces to obtain a virtual-real fusion simulation space displayed to the user; they are also used to overlay the situation information of entity models onto the corresponding entity models in the simulation space for display, and to overlay the situation information of virtual simulation objects onto the corresponding virtual simulation objects in the simulation space for display via augmented reality devices.

[0012] The second aspect of the present invention provides a virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation system. In one possible implementation, the motion information of the entity model is obtained based on the position and shape of the motion marker corresponding to the entity model in the image acquired by the image acquisition device, as well as the positional and posture relationships between the motion marker and the entity model body. The identity information of the entity model is obtained by scanning the corresponding ontological markers of the entity model using an image acquisition device. The motion marker and the body marker are the same identifier code. This identifier code contains the identity information of the physical model and is fixed on the surface of the physical model that can be photographed. The motion pattern of the identifier code is consistent with that of the physical model.

[0013] The second aspect of the present invention provides a virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation system. In one possible implementation, the method of obtaining the virtual-real fusion simulation space displayed to the user includes: using scene registration technology to complete the mapping of virtual and real environment state spaces, and performing consistency matching of virtual and real scenes. Based on the functions and components displayed to users in augmented reality devices, including gesture recognition and UI interfaces, users are provided with various perspectives and detailed zoom displays in the simulation space.

[0014] The second aspect of the present invention provides a virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation system. In one possible implementation, the augmented reality device is also used to acquire real-time status information of the entity model and the virtual simulation object, and to overlay the virtual status markers corresponding to the real-time status information onto the corresponding entity model and the virtual simulation object in the simulation space for display.

[0015] The second aspect of the present invention provides a virtual-real integrated human-computer collaborative three-dimensional visual interactive simulation system. In one possible implementation, situational information displayed on the physical model and the virtual simulation object is superimposed by augmented reality devices, including basic attribute information, performance parameters, energy information, preset trajectory and tracking trajectory; the tracking trajectory is the actual motion trajectory of the simulation object. Augmented reality devices are also used to trigger corresponding path planning algorithms to obtain new preset trajectories for entity models when users operate on the corresponding interface in the simulation space and select the entity model's preset trajectory planning function.

[0016] The technical solution of the virtual-real fusion human-computer collaborative three-dimensional visual interactive inference system of the present invention can achieve the same beneficial effects as the virtual-real fusion human-computer collaborative three-dimensional visual interactive inference method. Attached Figure Description

[0017] Figure 1 This is an example block diagram of the virtual-real fusion human-computer collaborative 3D visual interactive simulation system in the embodiment of the present invention; Figure 2a This is an example of the first effect of a physical aircraft model moving in the air in an embodiment of the virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation system of the present invention; Figure 2b This is an example diagram illustrating the second effect of a physical aircraft model moving in the air in an embodiment of the virtual-real fusion human-computer collaborative 3D visual interactive simulation system of the present invention; Figure 3 This is an example block diagram illustrating the virtual-real fusion human-computer collaborative 3D visual interactive inference method in the implementation of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0019] Implementation of a Human-Computer Collaborative 3D Visual Interactive Simulation System that Integrates Virtual and Real Realities This embodiment presents a technical solution for a virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation system. By constructing a virtual-real fusion three-dimensional simulation space with full-domain perception capabilities, virtual situations are superimposed on real intelligent agents. Operators can use the information presented by mixed reality devices to simulate the complex movements of intelligent agents in the virtual-real fusion scene, so as to complete the simulation of complex tasks in the three-dimensional virtual-real fusion scene.

[0020] Reference Figure 1The system includes augmented reality equipment, image acquisition devices, back-end computing modules, a physical simulation environment, and a physical model of the simulation object; the physical model can move along a corresponding trajectory in the real environment through a three-dimensional displacement device that can move in the XYZ directions. Image acquisition devices are used to acquire images of the simulation environment and the entity model of the simulation object to construct a real environment state space; The backend computing module is used to acquire the situation information of the deduced object entity model in the real environment area in real time based on the image acquired by the image acquisition device, including the body information and motion information; it is also used to calculate the situation information and preset trajectory of each virtual deduced object in the XYZ three-dimensional direction in real time through the situation information of the entity model and other virtual deduced objects, so as to control the virtual deduced objects to move according to the preset trajectory. Augmented reality devices are used to map virtual and real environment state spaces to obtain a virtual-real fusion simulation space displayed to the user; they are also used to overlay the situation information of entity models onto the corresponding entity models in the simulation space for display, and to overlay the situation information of virtual simulation objects onto the corresponding virtual simulation objects in the simulation space for display via augmented reality devices.

[0021] Therefore, by using augmented reality devices for human-machine collaborative simulation, the traditional spatial limitations of simulation are broken. By fusing the real-time position and motion status of physical simulation objects with virtual situational elements in a virtual-real fusion dimension, the corresponding situational information is superimposed onto both virtual and physical simulation objects. This intuitive design allows operators to directly understand the spatial correspondence between the virtual and real environments and their respective simulation objects, reducing the operator's cognitive load. Both physical and virtual models can achieve XYZ three-dimensional motion. Combined with the construction of a virtual-real fusion three-dimensional spatial simulation environment and the interactive linkage between physical and virtual simulation objects, virtual-real interactive simulation is no longer limited to a two-dimensional plane, facilitating interactive human-machine collaborative visual simulation correction. Under this simulation system, operators and machines can cooperate efficiently. The machine, through the simulation system and specific path planning algorithms, simulates the movement of complex groups, thereby realizing path simulation of virtual and real intelligent agents in complex simulation environments. The results can assist operators in decision-making. This mechanism of sharing and real-time updating of situational awareness information can effectively cope with complex environmental information and improve the efficiency of simulation decision-making.

[0022] In this embodiment, the method of obtaining the virtual-real fusion simulation space displayed to the user includes: using scene registration technology to complete the mapping of virtual and real environment state spaces, and performing consistency matching of virtual and real scenes; Based on the functions and components displayed to users in augmented reality devices, including gesture recognition and UI interfaces, users are provided with various perspectives and detailed zoom displays in the simulation space.

[0023] Specifically, augmented reality devices are used to construct a virtual-real fusion 3D simulation space with omnidirectional perception capabilities, namely a 3D sandbox system, which includes a virtual environment state space and a real environment state space. The virtual environment state space presents virtual simulation objects, their situational information, and future predictions; the real environment state space presents the simulation environment and simulation objects based on physical objects. Considering the geometric consistency, lighting consistency, and compositional consistency of the virtual and real scenes, scene registration technology is used to complete the mapping between the virtual and real spaces and to perform consistency matching between the virtual and real scenes. This system enables interaction between virtual and physical simulation objects within the 3D sandbox system, completing a visual simulation of the virtual-real fusion 3D space.

[0024] In terms of human-computer interaction, the system supports free switching between multiple dimensions, such as a global overview and a single-user perspective. Users can interact with the virtual simulation environment using augmented reality devices according to the simulation requirements. In this system, HoloLens 2 MR glasses can transmit shared virtual information to the operator. The operator can achieve immersive interaction based on gesture recognition, UI interface, and other functions and components, enabling free switching between multiple perspectives and zooming in and out of details.

[0025] Through extensive testing of different cameras' coverage of the physical sand table surface and their performance in recognizing and tracking QR codes of the physical simulation object model, this embodiment uses a Jierui Microcom USB industrial-grade camera (with main performance parameters of ultra-wide-angle 140 degrees and distortion-free). Considering the advantages of the OpenCV function library being open-source and cross-platform, the OpenCV function library is used to complete the relevant video reading and writing operations. When using dynamic camera calibration, the chessboard calibration board is fixed on a plane. The camera takes pictures of the chessboard calibration board from different angles and positions, reads the directory where the chessboard calibration board image is located, and uses the functions provided by OpenCV to pass the defined chessboard type and grayscale image to the function cv2.findChessboardCorners() to find the chessboard corner points. The corner points returned by this function are then passed to the function cv2.cornerSubPix() to find sub-pixel points based on the original corner points. The function cv2.calibrateCamera() is used to obtain the intrinsic and extrinsic parameters of the camera matrix and the distortion coefficients. This achieves the mapping of the virtual and real environment state space, resulting in a simulation space that blends the virtual and real worlds and is displayed to the user.

[0026] In one specific embodiment, a suspension frame is used as a movable displacement device in the XYZ three-dimensional direction to suspend the physical simulation object (i.e., a physical model, such as an airplane model). Images acquired by an image acquisition device can be used to calculate the trajectory, speed, and orientation information of the simulation object (airplane) in real time. This allows for real-time acquisition of the situational information of the simulation object model in the real environment, enabling path planning as needed and achieving motion control in the XYZ three-dimensional direction. This accurately presents the flight path and movements of the physical model in the air. Combined with a virtual model (e.g., a virtual airplane model) presented in an augmented reality device, its position and attitude information are calculated, enabling interaction and linkage between the virtual and physical airplanes in the air. The situational information of this virtual and physical airplanes is also overlaid and presented in the AR device. In other embodiments, the movable displacement device in the XYZ three-dimensional direction can also be a movable support device installed on the ground, which lifts the model upwards to achieve movement in the Z-axis direction, or other displacement devices that can assist the physical model in achieving movable movement in the XYZ three-dimensional direction. For example, an example diagram of the physical model's movement in the air is shown below. Figure 2a , 2b As shown, the white trajectory is the preset trajectory of the simulation object in the XYZ three-dimensional direction, and the red trajectory is the tracking trajectory of the simulation object, that is, the actual motion trajectory of the simulation object.

[0027] In this embodiment, the motion information of the entity model is obtained based on the position and shape of the motion marker corresponding to the entity model in the image acquired by the image acquisition device, as well as the positional and posture relationships between the motion marker and the entity model body; the identity information of the entity model is obtained by scanning the body marker corresponding to the entity model through the image acquisition device.

[0028] For example, the motion markers corresponding to the physical model can be specific patterns attached to the physical model. Based on images of the physical model captured by an image acquisition device installed at a known location, and according to the position and shape of the motion markers in the images, as well as the positional and orientational relationships between the motion markers and the physical model itself, the position and shape of the motion markers can be converted into the position and orientation of the physical model itself. The identity information of the physical model can be in the form of barcodes, QR codes, or other information codes, or it can be in the form of identity information content directly printed on the surface of the physical model.

[0029] In this embodiment, the motion marker and the body marker are the same identification code. This identification code contains the identity information of the physical model, and it is fixed to the surface of the physical model that can be photographed. The motion pattern of the identification code is consistent with that of the physical model. In other embodiments, the motion marker and the body marker may also be specific patterns or characteristic components attached to the physical model using other existing technologies, and there are no specific limitations.

[0030] In a preferred embodiment, the motion marker and the body marker are the same QR code; the QR code has high recognition efficiency, and its shape in the image can accurately correspond to the posture of the physical model. Using an image acquisition device, such as a camera, the identity information of the physical model can be obtained by scanning the QR code, realizing the function of identifying the physical model; furthermore, the motion information of the physical model can be obtained by recognizing the position and shape of the QR code, realizing the function of tracking the physical model. In this embodiment, for the physical simulation object, the spatial information of the physical 3D sand table is calibrated and integrated using a camera, and the position, angle, and other motion data of the physical simulation object in the sand table are analyzed and determined by recognizing the QR code using the camera. Simultaneously, an AR interactive device (i.e., an augmented reality device) drives the presentation of the virtual simulation object in the 3D sand table, showing the motion trajectory and state changes of the virtual simulation object.

[0031] Based on this, the real-time situational information transmitted by the images acquired by physical devices (i.e., image acquisition devices) can be combined to calculate the situational information, and the corresponding situational information can be superimposed on the physical sand table to realize the interaction and linkage between the physical simulation object and the virtual simulation object.

[0032] Furthermore, in this embodiment, the situational information displayed by the augmented reality device overlaid on the physical model and the virtual simulation object includes basic attribute information, performance parameters, energy information, preset trajectory, and tracking trajectory; wherein, the tracking trajectory is the actual movement trajectory of the simulation object. In a preferred embodiment, the situational information displayed on the physical model and the virtual simulation object is specifically displayed by overlaying it on top of the physical model and the virtual simulation object; the user can use the augmented reality device to click different buttons displayed in the simulation space to display different types of detailed situational information.

[0033] In a preferred embodiment, using an aircraft as the simulation object, the basic attribute information includes relevant information describing the aircraft's working principle, such as approach attitude and landing method, which provides a reference for the operator when making decisions about the aircraft's departure and return.

[0034] Performance parameters include relevant parameters describing the aircraft's structure and performance, such as structural information like length, wingspan, altitude, and empty weight, as well as performance information like maximum speed, maximum range, and thrust-to-weight ratio. The purpose is to characterize the aircraft's physical model, aiding in tasks such as airport layout and mission execution.

[0035] Energy information includes details describing the aircraft's energy usage, such as remaining fuel, electricity, or gas. Energy usage data helps operators assess the aircraft's range during missions, ensuring the safety and efficiency of mission execution.

[0036] The preset trajectory and the tracking trajectory describe the execution effect of aircraft path planning. The preset trajectory, based on the start point, end point, and obstacle information, uses a path planning algorithm to plan a reasonable preset path for the aircraft. The tracking trajectory displays the actual trajectory of the aircraft when moving according to the preset trajectory. The preset trajectory provides action guidance for the aircraft through algorithms, saving manpower, while the tracking trajectory shows the operator the deviation between the actual trajectory and the preset trajectory, providing visual guidance for operator intervention and correction.

[0037] In this embodiment, after the operator wears VR glasses, they can see three information buttons on the top of the aircraft, which indicate basic attributes, performance parameters, and energy information, respectively. The operator can click the Basic Attributes button to view the aircraft's operating principles, such as approach attitude and landing method. The operator needs to comprehensively consider this information to rationally arrange the aircraft's departure and return. Clicking the Performance Parameters button allows the operator to view the aircraft's structure and performance. Based on the aircraft's size, weight, maximum speed, and other information, the operator can adjust the aircraft's deployment position at the airport and the order of mission execution. Clicking the Energy Information button allows the operator to view the aircraft's energy usage. By viewing, analyzing, and evaluating the consumption of fuel, electricity, gas, and other energy sources, the operator can decide whether the aircraft should continue its mission, its priority during mission execution, or to perform actions such as energy replenishment. Clicking the Path Planning panel allows the operator to select the corresponding aircraft and destination. The path planning algorithm will then plan a white path to guide the aircraft's navigation. The operator can also click the Tracking Trajectory button above the aircraft to activate the aircraft's tracking trajectory. At this time, a red real-time tracking trajectory will be displayed at the tail of the aircraft as it moves. The operator can also intervene to adjust the aircraft's movement and analyze the deviation between the two trajectories, providing a case study for the next path planning.

[0038] Augmented reality devices are also used to acquire real-time status information of physical models and virtual simulation objects, and to overlay virtual markers corresponding to the real-time status information onto the corresponding physical models and virtual simulation objects in the simulation space.

[0039] To accurately represent the current state of the simulation object (e.g., an aircraft), in this embodiment, the augmented reality device is also used to acquire real-time state information of the physical model and the virtual simulation object, and to overlay virtual status markers corresponding to the real-time state information onto the corresponding physical model and virtual simulation object in the simulation space. Different status markers are designed to provide feedback on the real-time state information of the aircraft. Different markers represent different aircraft states; for example, if the aircraft is currently parked, a red marker representing the parked state can be selected.

[0040] In the dimension of virtual-real integration, the system uses standardized data interface processing to deeply integrate the real-time position and motion status data of physical simulation objects with the situational elements of virtual forms (such as simulation tasks and routes). This integration can achieve not only static situational overlay (such as basic attribute information, performance parameters, energy information, etc. displayed on the physical model and virtual simulation object), but also dynamic situational overlay (such as dynamically calculated preset trajectories and tracking trajectories). A more intuitive human-computer interaction interface design reduces the cognitive load on operators.

[0041] Augmented reality devices are also used to trigger corresponding path planning algorithms to obtain new preset trajectories for entity models when users operate on the corresponding interface in the simulation space and select the entity model's preset trajectory planning function.

[0042] In a preferred embodiment, interactive visualization correction is performed based on the aforementioned 3D sand table. This process is divided into two parts: 3D trajectory (preset trajectory) generation and human-machine collaborative correction. The 3D trajectory generation part relies on machine intelligence algorithms for backend calculations and uses augmented reality devices for frontend visualization. The system uses the path planning algorithm of the backend calculation module to calculate the path of the group composed of complex entity models (i.e., physical simulation objects) and virtual simulation objects, i.e., the preset trajectory. The calculated situational information and preset trajectory information are then superimposed onto the virtual and physical simulation objects in the 3D simulation space using augmented reality devices, driving the movement of the virtual and physical simulation objects. The path planning for physical simulation objects can be triggered by the operator as needed (specifically, it can be triggered by the operator's operation on the corresponding interface in the simulation space), while the preset trajectory for virtual simulation objects is planned in real time.

[0043] The human-machine collaborative correction function provides operators with an intuitive interface and a preset trajectory correction entry point. Relying on augmented reality devices, it accurately overlays and presents the 3D trajectory (i.e. the preset trajectory) generated by machine intelligence algorithms in the visual space. Operators can modify the preset trajectory by controlling the physical model based on their experience and cognition to obtain the tracking trajectory of the physical model (i.e. the actual movement trajectory of the physical model).

[0044] The aforementioned system enables human-machine collaborative simulations using augmented reality (AR) devices, breaking the traditional spatial limitations of simulations and allowing operators to immerse themselves in a three-dimensional simulation environment. With AR devices, the system can overlay virtual situational information onto a real sandbox scene. Operators can intuitively understand the spatial correspondence between virtual tracks and physical models, and can trigger preset trajectory correction tasks through interaction with physical simulation objects. It also enables multi-person collaborative simulations, where multiple operators share the simulation scene and information, achieving synchronous interaction and collaborative decision-making. Through multiple iterations, a human-machine collaborative simulation mode with virtual-real interaction is formed.

[0045] Implementation Method of Human-Computer Collaborative 3D Visual Interactive Inference Method that Integrates Virtual and Real Realities This embodiment presents a technical solution for a virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation method. By constructing a virtual-real fusion three-dimensional simulation space with full-domain perception capabilities, the virtual situation is superimposed on the real intelligent agent. The operator can use the information presented by the mixed reality device to simulate the complex intelligent agent movement in the virtual-real fusion scene, so as to complete the simulation of complex tasks in the three-dimensional virtual-real fusion scene.

[0046] Reference Figure 3 The method includes: By mapping virtual and real environment state spaces using augmented reality devices, a virtual-real fusion simulation space is obtained and displayed to the user; The system acquires real-time situational information of the simulated object entity model in the real environment area, including ontology information and motion information; and displays the situational information of the entity model on the corresponding entity model in the simulation space through augmented reality equipment; the entity model moves in the real environment according to the corresponding trajectory through a displacement device that can move in the XYZ three-dimensional direction. By using the situational information of the physical model and other virtual simulation objects, the situational information and preset trajectory of each virtual simulation object in the XYZ three-dimensional direction are calculated in real time to control the virtual simulation object to move according to the preset trajectory. The situational information of the virtual simulation object is then superimposed onto the corresponding virtual simulation object in the simulation space through augmented reality devices for display.

[0047] This method breaks through the traditional spatial limitations of simulation by using augmented reality devices for human-machine collaborative simulation. By fusing real-world data such as the real-time position and motion status of physical simulation objects with virtual situational elements, the corresponding situational information is overlaid onto both virtual and physical simulation objects. This intuitive design allows operators to directly understand the spatial correspondence between the virtual and real environments and their respective simulation objects, reducing cognitive load. Both physical and virtual models can achieve XYZ three-dimensional motion. Combined with the construction of a virtual-real fusion three-dimensional spatial simulation environment and the interactive linkage between physical and virtual simulation objects, virtual-real interactive simulation is no longer limited to a two-dimensional plane, facilitating interactive human-machine collaborative visual simulation correction. In this simulation system, operators and machines can collaborate efficiently. The machine, through the simulation system and specific path planning algorithms, simulates the movement of complex groups, enabling path simulation of virtual and real intelligent agents in complex simulation environments. The results can assist operators in decision-making. This sharing and real-time updating mechanism of situational awareness information effectively addresses complex environmental information and improves the efficiency of simulation decision-making.

[0048] In this embodiment, the method of obtaining the virtual-real fusion simulation space displayed to the user includes: using scene registration technology to complete the mapping of virtual and real environment state spaces, and performing consistency matching of virtual and real scenes; Based on the functions and components displayed to users in augmented reality devices, including gesture recognition and UI interfaces, users are provided with various perspectives and detailed zoom displays in the simulation space.

[0049] Specifically, augmented reality devices are used to construct a virtual-real fusion 3D simulation space with omnidirectional perception capabilities, namely a 3D sandbox system, which includes a virtual environment state space and a real environment state space. The virtual environment state space presents virtual simulation objects, their situational information, and future predictions; the real environment state space presents the simulation environment and simulation objects based on physical objects. Considering the geometric consistency, lighting consistency, and compositional consistency of the virtual and real scenes, scene registration technology is used to complete the mapping between the virtual and real spaces and to perform consistency matching between the virtual and real scenes. This system enables interaction between virtual and physical simulation objects within the 3D sandbox system, completing a visual simulation of the virtual-real fusion 3D space.

[0050] In terms of human-computer interaction, the constructed 3D simulation space supports free switching between multiple dimensions, such as a global overview and a single-user perspective. Users can interact with the virtual simulation environment using augmented reality devices according to their simulation needs. In this system, HoloLens 2 MR glasses can transmit shared virtual information to the operator. The operator can achieve immersive interaction based on gesture recognition, UI interface, and other functions and components, enabling free switching between multiple perspectives and zooming in and out on details.

[0051] Through extensive testing of different cameras' coverage of the physical sand table surface and their performance in recognizing and tracking QR codes of the physical simulation object model, this embodiment uses a Jierui Microcom USB industrial-grade camera (with main performance parameters of ultra-wide-angle 140 degrees and distortion-free). Considering the advantages of the OpenCV function library being open-source and cross-platform, the OpenCV function library is used to complete the relevant video reading and writing operations. When using dynamic camera calibration, the chessboard calibration board is fixed on a plane. The camera takes pictures of the chessboard calibration board from different angles and positions, reads the directory where the chessboard calibration board image is located, and uses the functions provided by OpenCV to pass the defined chessboard type and grayscale image to the function cv2.findChessboardCorners() to find the chessboard corner points. The corner points returned by this function are then passed to the function cv2.cornerSubPix() to find sub-pixel points based on the original corner points. The function cv2.calibrateCamera() is used to obtain the intrinsic and extrinsic parameters of the camera matrix and the distortion coefficients. This achieves the mapping of the virtual and real environment state space, resulting in a simulation space that blends the virtual and real worlds and is displayed to the user.

[0052] In one specific embodiment, a suspension frame is used as a movable displacement device in the XYZ three-dimensional direction to suspend the physical simulation object (i.e., a physical model, such as an airplane model). Images acquired by an image acquisition device can be used to calculate the trajectory, speed, and orientation information of the simulation object (airplane) in real time. This allows for real-time acquisition of the situational information of the simulation object model in the real environment, enabling path planning as needed and achieving motion control in the XYZ three-dimensional direction. This accurately presents the flight path and actions of the physical model in the air. Combined with a virtual model (e.g., a virtual airplane model) presented in an augmented reality device, its position and attitude information are calculated, enabling interaction and linkage between the virtual and physical airplanes in the air. The situational information of this virtual and physical airplanes is also overlaid and presented in the AR device. In other embodiments, the movable displacement device in the XYZ three-dimensional direction can also be a movable support device installed on the ground, which lifts the model upwards to achieve movement in the Z-axis direction, or other displacement devices that can assist the physical model in achieving movable movement in the XYZ three-dimensional direction.

[0053] In this embodiment, the motion information of the entity model is obtained based on the position and shape of the motion marker corresponding to the entity model in the image acquired by the image acquisition device, as well as the positional and posture relationships between the motion marker and the entity model body; the identity information of the entity model is obtained by scanning the body marker corresponding to the entity model through the image acquisition device.

[0054] For example, the motion markers corresponding to the physical model can be specific patterns attached to the physical model. Based on images of the physical model captured by an image acquisition device installed at a known location, and according to the position and shape of the motion markers in the images, and the positional and orientational relationships between the motion markers and the physical model itself, the position and shape of the motion markers can be converted into the position and orientation of the physical model itself. The identity information of the physical model can be in the form of barcodes, QR codes, or other information codes, or it can be identity information content directly printed on the surface of the physical model. In other embodiments, the motion markers and the body markers can also respectively adopt other existing specific patterns or characteristic components attached to the physical model, without any specific limitations.

[0055] In this embodiment, the motion marker and the body marker are the same identifier code. The identifier code contains the identity information of the physical model and is fixed on the surface of the physical model that can be photographed. The motion pattern of the identifier code is consistent with that of the physical model.

[0056] In a preferred embodiment, the motion marker and the body marker are the same QR code; the QR code has high recognition efficiency, and its shape in the image can accurately correspond to the posture of the physical model. Using an image acquisition device, such as a camera, the identity information of the physical model can be obtained by scanning the QR code, realizing the function of identifying the physical model; furthermore, the motion information of the physical model can be obtained by recognizing the position and shape of the QR code, realizing the function of tracking the physical model. In this embodiment, for the physical simulation object, the spatial information of the physical 3D sand table is calibrated and integrated using a camera, and the position, angle, and other motion data of the physical simulation object in the sand table are analyzed and determined by recognizing the QR code using the camera. Simultaneously, an AR interactive device (i.e., an augmented reality device) drives the presentation of the virtual simulation object in the 3D sand table, showing the motion trajectory and state changes of the virtual simulation object.

[0057] Based on this, the real-time situational information transmitted by the images acquired by physical devices (i.e., image acquisition devices) can be combined to calculate the situational information, and the corresponding situational information can be superimposed on the physical sand table to realize the interaction and linkage between the physical simulation object and the virtual simulation object.

[0058] Furthermore, in this embodiment, the situational information displayed by the augmented reality device overlaid on the physical model and the virtual simulation object includes basic attribute information, performance parameters, energy information, preset trajectory, and tracking trajectory; wherein, the tracking trajectory is the actual movement trajectory of the simulation object. In a preferred embodiment, the situational information displayed on the physical model and the virtual simulation object is specifically displayed by overlaying it on top of the physical model and the virtual simulation object; the user can use the augmented reality device to click different buttons displayed in the simulation space to display different types of detailed situational information.

[0059] In a preferred embodiment, using an aircraft as the simulation object, the basic attribute information includes relevant information describing the aircraft's working principle, such as approach attitude and landing method, which provides a reference for the operator when making decisions about the aircraft's departure and return.

[0060] Performance parameters include relevant parameters describing the aircraft's structure and performance, such as structural information like length, wingspan, altitude, and empty weight, as well as performance information like maximum speed, maximum range, and thrust-to-weight ratio. The purpose is to characterize the aircraft's physical model, aiding in tasks such as airport layout and mission execution.

[0061] Energy information includes details describing the aircraft's energy usage, such as remaining fuel, electricity, or gas. Energy usage data helps operators assess the aircraft's range during missions, ensuring the safety and efficiency of mission execution.

[0062] The preset trajectory and the tracking trajectory describe the execution effect of aircraft path planning. The preset trajectory, based on the start point, end point, and obstacle information, uses a path planning algorithm to plan a reasonable preset path for the aircraft. The tracking trajectory displays the actual trajectory of the aircraft when moving according to the preset trajectory. The preset trajectory provides action guidance for the aircraft through algorithms, saving manpower, while the tracking trajectory shows the operator the deviation between the actual trajectory and the preset trajectory, providing visual guidance for operator intervention and correction.

[0063] In this embodiment, after the operator wears VR glasses, they can see three information buttons on the top of the aircraft, which indicate basic attributes, performance parameters, and energy information, respectively. The operator can click the Basic Attributes button to see the aircraft's operating principles, such as approach attitude and landing method. The operator needs to consider this information comprehensively to rationally plan the aircraft's departure and return. Clicking the Performance Parameters button allows the operator to view the aircraft's structure and performance. Based on the aircraft's size, weight, maximum speed, and other information, the operator can adjust the aircraft's deployment position at the airport and the order of mission execution. Clicking the Energy Information button allows the operator to view the aircraft's energy usage. By viewing, analyzing, and evaluating the consumption of fuel, electricity, and gas, the operator can decide whether the aircraft should continue its mission, its priority during mission execution, or to perform actions such as refueling. Clicking the Path Planning panel allows the operator to select the corresponding aircraft and destination. The path planning algorithm will then plan a white path to guide the aircraft's navigation. The operator can also click the Tracking Trajectory button above the aircraft to activate the aircraft's tracking trajectory. At this time, a red real-time tracking trajectory will be displayed at the tail of the aircraft as it moves. The operator can also intervene to adjust the aircraft's movement and analyze the deviation between the two trajectories, providing a case study for the next path planning.

[0064] In this embodiment, augmented reality devices are used to acquire real-time status information of the physical model and the virtual simulation object, and the virtual status markers corresponding to the real-time status information are superimposed onto the corresponding physical model and the virtual simulation object in the simulation space for display.

[0065] To accurately represent the current state of the simulation object (e.g., an aircraft), in this embodiment, the augmented reality device is also used to acquire real-time state information of the physical model and the virtual simulation object, and to overlay virtual status markers corresponding to the real-time state information onto the corresponding physical model and virtual simulation object in the simulation space. Different status markers are designed to provide feedback on the real-time state information of the aircraft. Different markers represent different aircraft states; for example, if the aircraft is currently parked, a red marker representing the parked state can be selected.

[0066] In the dimension of virtual-real integration, through standardized data interface processing, the real-time position and motion status data of physical simulation objects are deeply integrated with the situational elements of virtual forms (such as simulation tasks and simulation routes). This integration can not only achieve static situational overlay (such as basic attribute information, performance parameters, energy information, and other static situational information overlaid on physical models and virtual simulation objects), but also achieve the effect of dynamic situational overlay (such as dynamically calculated preset trajectories and tracking trajectories). A more intuitive human-computer interaction interface design reduces the cognitive load on operators.

[0067] Augmented reality devices are also used to trigger corresponding path planning algorithms to obtain new preset trajectories for entity models when users operate on the corresponding interface in the simulation space and select the entity model's preset trajectory planning function.

[0068] In a preferred embodiment, interactive visualization correction is performed based on the aforementioned 3D sand table. This process is divided into two parts: 3D trajectory (preset trajectory) generation and human-machine collaborative correction. The 3D trajectory generation part relies on machine intelligence algorithms for backend calculations and uses augmented reality devices for frontend visualization. The system uses the path planning algorithm of the backend calculation module to calculate the path of the group composed of complex entity models (i.e., physical simulation objects) and virtual simulation objects, i.e., the preset trajectory. The calculated situational information and preset trajectory information are then superimposed onto the virtual and physical simulation objects in the 3D simulation space using augmented reality devices, driving the movement of the virtual and physical simulation objects. The path planning for physical simulation objects can be triggered by the operator as needed (specifically, it can be triggered by the operator's operation on the corresponding interface in the simulation space), while the preset trajectory for virtual simulation objects is planned in real time.

[0069] The human-machine collaborative correction function provides operators with an intuitive interface and a preset trajectory correction entry point. Relying on augmented reality devices, it accurately overlays and presents the 3D trajectory (i.e. the preset trajectory) generated by machine intelligence algorithms in the visual space. Operators can modify the preset trajectory by controlling the physical model based on their experience and cognition to obtain the tracking trajectory of the physical model (i.e. the actual movement trajectory of the physical model).

[0070] The aforementioned method enables human-machine collaborative simulation using augmented reality (AR) devices, breaking through traditional simulation space limitations and allowing operators to immerse themselves in a three-dimensional simulation environment. With AR devices, the system can overlay virtual situational information onto a real sandbox scene. Operators can intuitively understand the spatial correspondence between virtual tracks and physical models, and can trigger preset trajectory correction tasks through interaction with physical simulation objects. It also enables multi-person collaborative simulation, where multiple operators share the simulation scene and information, achieving synchronous interaction and collaborative decision-making. Through multiple iterations, a human-machine collaborative simulation mode with virtual-real interaction is formed.

[0071] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or explanatory of the principles of the present invention, and do not constitute a limitation thereof.

Claims

1. A human-computer collaborative 3D visual interactive simulation method that integrates virtual and real elements, characterized in that, include: By mapping virtual and real environment state spaces using augmented reality devices, a virtual-real fusion simulation space is obtained and displayed to the user; The system acquires real-time situational information of the simulated object entity model in the real environment area, including ontology information and motion information; and displays the situational information of the entity model on the corresponding entity model in the simulation space through augmented reality equipment; the entity model moves in the real environment according to the corresponding trajectory through a displacement device that can move in the XYZ three-dimensional direction. By using the situational information of the physical model and other virtual simulation objects, the situational information and preset trajectory of each virtual simulation object in the XYZ three-dimensional direction are calculated in real time to control the virtual simulation object to move according to the preset trajectory. The situational information of the virtual simulation object is then superimposed onto the corresponding virtual simulation object in the simulation space through augmented reality devices for display.

2. The virtual-real fusion human-computer collaborative three-dimensional visual interactive inference method according to claim 1, characterized in that, The motion information of the entity model is obtained based on the position and shape of the corresponding motion marker in the image, as well as the positional and orientational relationships between the motion marker and the entity model itself. The identity information of an entity model is obtained by scanning the ontology marker corresponding to that entity model. The motion marker and the body marker are the same identifier code. This identifier code contains the identity information of the physical model and is fixed on the surface of the physical model that can be photographed. The motion pattern of the identifier code is consistent with that of the physical model.

3. The virtual-real fusion human-computer collaborative three-dimensional visual interactive inference method according to claim 1 or 2, characterized in that, The methods for obtaining the virtual-real fusion simulation space to be displayed to the user include: using scene registration technology to complete the mapping of virtual and real environment state spaces, and performing consistency matching between virtual and real scenes; Based on the functions and components displayed to users in augmented reality devices, including gesture recognition and UI interfaces, users are provided with various perspectives and detailed zoom displays in the simulation space.

4. The virtual-real fusion human-computer collaborative three-dimensional visual interactive inference method according to claim 1 or 2, characterized in that, Also includes: Real-time status information of physical models and virtual simulation objects is acquired, and virtual status markers corresponding to the real-time status information are superimposed onto the corresponding physical models and virtual simulation objects in the simulation space using augmented reality devices.

5. The virtual-real fusion human-computer collaborative three-dimensional visual interactive inference method according to claim 1 or 2, characterized in that, The situational information displayed by superimposing augmented reality devices onto the physical model and the virtual simulation object includes basic attribute information, performance parameters, energy information, preset trajectory and tracking trajectory. The tracking trajectory is the actual motion trajectory of the simulated object; The method for determining the preset trajectory of the entity model includes: triggering the corresponding path planning algorithm based on the entity model preset trajectory planning function selected by the user when operating on the corresponding interface in the simulation space, and obtaining a new preset trajectory of the entity model.

6. A human-computer collaborative 3D visual interactive simulation system that integrates virtual and real elements, characterized in that, It includes augmented reality devices, image acquisition devices, back-end computing modules, physical simulation environment and simulation object entity models; the entity models can move in the real environment according to corresponding trajectories through three-dimensional displacement devices in the XYZ directions; Image acquisition devices are used to acquire images of the simulation environment and the entity model of the simulation object to construct a real environment state space; The backend computing module is used to acquire the situation information of the deduced object entity model in the real environment area in real time based on the image acquired by the image acquisition device, including the body information and motion information; it is also used to calculate the situation information and preset trajectory of each virtual deduced object in the XYZ three-dimensional direction in real time through the situation information of the entity model and other virtual deduced objects, so as to control the virtual deduced objects to move according to the preset trajectory. Augmented reality devices are used to map virtual and real environment state spaces to obtain a virtual-real fusion simulation space displayed to the user; they are also used to overlay the situation information of entity models onto the corresponding entity models in the simulation space for display, and to overlay the situation information of virtual simulation objects onto the corresponding virtual simulation objects in the simulation space for display via augmented reality devices.

7. The virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation system according to claim 6, characterized in that, The motion information of the entity model is obtained based on the position and shape of the motion marker corresponding to the entity model in the image acquired by the image acquisition device, as well as the positional and orientational relationships between the motion marker and the entity model body. The identity information of the entity model is obtained by scanning the corresponding ontological markers of the entity model using an image acquisition device. The motion marker and the body marker are the same identifier code. This identifier code contains the identity information of the physical model and is fixed on the surface of the physical model that can be photographed. The motion pattern of the identifier code is consistent with that of the physical model.

8. The virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation system according to claim 6 or 7, characterized in that, The methods for obtaining the virtual-real fusion simulation space to be displayed to the user include: using scene registration technology to complete the mapping of virtual and real environment state spaces, and performing consistency matching between virtual and real scenes; Based on the functions and components displayed to users in augmented reality devices, including gesture recognition and UI interfaces, users are provided with various perspectives and detailed zoom displays in the simulation space.

9. The virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation system according to claim 6 or 7, characterized in that, Augmented reality devices are also used to acquire real-time status information of physical models and virtual simulation objects, and to overlay virtual markers corresponding to the real-time status information onto the corresponding physical models and virtual simulation objects in the simulation space.

10. The virtual-real fusion human-computer collaborative three-dimensional visual interactive simulation system according to claim 6 or 7, characterized in that, The situational information displayed by superimposing augmented reality devices onto the physical model and the virtual simulation object includes basic attribute information, performance parameters, energy information, preset trajectory and tracking trajectory; the tracking trajectory is the actual movement trajectory of the simulation object. Augmented reality devices are also used to trigger corresponding path planning algorithms to obtain new preset trajectories for entity models when users operate on the corresponding interface in the simulation space and select the entity model's preset trajectory planning function.

Images