Electronic sand table system and multi-man interactive visual deduction method oriented to electronic sand table

Abstract

The invention relates to an electronic sand table system and an electronic sand table-oriented multi-man interactive visualization deduction method, and belongs to a virtual-real fusion man-machine interaction technology. According to the invention, the constructed simulation deduction environment is a virtual-real fusion environment, and the three-dimensional virtual model of the constructed simulation deduction object on the sand table deduction platform and the motion of the trolley entity are displayed through the global situation map and the man-machine interaction device. A plurality of operators can use the global situation map and the information presented by the man-machine interaction equipment to deduce the complex cluster motion of the virtual-real mixed environment; and meanwhile, the trolley entity can be controlled to run, and a control result is fed back to the global situation map. Through the method, simulation deduction of multi-man-machine cooperative complex group movement is realized, and an entity sand table is added, so that an operator can visually observe and intuitively understand the deduction process in real time, and a reference basis is provided for manual control decision making of a complex group movement scene.

Description

Technical Field

[0001] The present invention relates to an electronic sand table system and a multi - human - machine interactive visualization deduction method for an electronic sand table, belonging to the human - machine interaction technology of virtual - real fusion. Background Art

[0002] The evolution of various movement plans of a complex group consumes a large amount of actual costs, and real - life drills require a large amount of manpower and material resources. The electronic deduction sand table can simulate and deduce the drill scenarios and movement plans through multi - party cooperation. Although the traditional screen - based electronic sand table can also display three - dimensional scenes, the display means is a two - dimensional computer screen, which does not conform to human visual habits. And restricted by physical display devices, all operators can only observe the situation from the same perspective, and the information display is insufficient. For this reason, an electronic sand table system capable of displaying three - dimensional scenes has been proposed, such as the Chinese patent application document with the application publication number CN107479705A. This document discloses a command post collaborative operation electronic sand table system based on HoloLens. This system displays the real - time situation of the electronic sand table in a holographic manner, vividly reproducing the three - dimensional scenes of all elements, various types, and multiple levels of the scene, providing an "environment for virtual scene discussion" for operators, supporting multiple operators to collaborate, and realizing real - time sharing of situation data. Although the electronic sand table of this solution can provide a three - dimensional scene, this electronic sand table is all virtualized and lacks a physical sand table, resulting in difficulties for operators to visually observe and intuitively understand the deduction process in real time, and it is also impossible to apply human decision - making experience for correction in a timely manner. Summary of the Invention

[0003] The purpose of the present invention is to provide an electronic sand table system and a multi - human - machine interactive visualization deduction method for an electronic sand table to solve the problem that in the current multi - human - machine interactive visualization deduction process, it is impossible to apply human decision - making experience for correction in a timely manner.

[0004] The present invention provides a multi - human - machine interactive visualization deduction method for an electronic sand table to solve the above - mentioned technical problems. This method includes:

[0005] 1) Construct a cluster movement simulation deduction environment for the complex group where the simulation deduction object is located, including a virtual environment state space and a real environment state space. The virtual environment state space includes a three - dimensional virtual model of the simulation deduction object and the virtual situation information of the simulation deduction object. The real environment state space includes a physical electronic sand table, and the physical electronic sand table includes a sand table deduction platform and intelligent vehicle entities running on the sand table deduction platform representing the simulation deduction objects.

[0006] 2) Use the global situation map to display the 3D virtual models of the virtualized simulation deduction objects and the movement process of the intelligent vehicle entities, and present the virtual information in the simulation deduction environment to the operator based on the human-computer interaction device worn by the operator, so that each operator can deduce the cluster movement of the simulation deduction objects in the virtual-real hybrid environment by using the information presented by the global situation map and the human-computer interaction device;

[0007] 3) The human-machine fusion controls the operation of the intelligent vehicle entities and feeds back the control results to the global situation map.

[0008] Further, the human-machine fusion control of the operation of the intelligent vehicle entities includes that the operator does not intervene to correct the operation of the vehicle entities and intervenes to correct the operation of the physical vehicle. The non-intervention to correct the operation of the vehicle entities means that the vehicle entities run according to the preset trajectory given by the machine; the intervention to correct the operation of the physical vehicle means that the operation of the vehicle entities is controlled according to the preset trajectory.

[0009] Further, the deduction process includes: The operator takes the preset trajectory given by the machine as a reference, determines the dynamic change of the global situation of the scene in combination with the global situation map, and determines the actual running trajectory of the intelligent vehicle entities through distributed deduction and correction using human experience.

[0010] Further, controlling the operation of the vehicle entities according to the preset trajectory is that the operator controls the actions of the intelligent vehicle entities by remote control.

[0011] Further, the method also includes superimposing the actual running trajectory of the intelligent vehicle entities on the information presented by the human-computer interaction device.

[0012] Further, the method includes intuitively displaying the virtual-real fusion scene presented by the human-computer interaction device through a visualization large screen for the multi-perspective situation deduction process.

[0013] Further, a UI design of virtual situation information related to the simulation deduction object corresponding to the intelligent vehicle entity is loaded on the intelligent vehicle entity model presented by the human-computer interaction device, and the superimposed virtual situation information UI interface moves with the movement of the intelligent vehicle entity.

[0014] Further, the human-computer interaction device is Hololens2.

[0015] The present invention also provides an electronic sand table system, and this electronic sand table system performs deduction by using the above-mentioned multi-human-machine interactive visualization deduction method for the electronic sand table.

[0016] The beneficial effects of the present invention are as follows: As an improved invention and creation type, the simulation and deduction environment constructed by the present invention is a virtual-real hybrid environment, including a three-dimensional virtual model of the simulation and deduction object, the virtual situation information of the simulation and deduction object, and an intelligent vehicle entity representing the simulation and deduction object running on the sand table deduction platform. The three-dimensional virtual model of the constructed sand table deduction platform and the motion plan of the intelligent vehicle entity are displayed through the global situation map, and the virtual information in the simulation and deduction environment is presented to the operator based on the human-computer interaction device worn by the operator, facilitating multiple operators to deduce the complex group movement in the virtual-real hybrid environment using the information presented by the global situation map and the human-computer interaction device; at the same time, the operation results of the vehicle entity can also be fed back into the global situation map. Through this method, multi-operator collaborative simulation and deduction of complex groups are realized, and a physical sand table is added, enabling the operator to visually observe and intuitively understand the deduction process in real time, and being able to correct in a timely manner using the operator's own experience to achieve a more realistic simulation and deduction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of the multi-operator interactive visualization deduction method for the electronic sand table of the present invention;

[0018] Figure 2 is a schematic diagram of the human-machine virtual fusion interaction and collaboration mode adopted in the embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of the three-dimensional visualization model based on the electronic deduction sand table constructed by the present invention;

[0020] Figure 4 is according to the present invention Figure 3 Schematic diagram of the superposition of the virtual-real fusion situation information obtained from the virtual scene;

[0021] Figure 5 is a schematic diagram of multi-operator collaborative interaction correction adopted in the embodiment of the present invention;

[0022] Figure 6 Schematic diagram of the movement of the AGV vehicle entity adopted in the embodiment of the present invention;

[0023] Figure 7 is a schematic diagram of the corresponding virtual entity movement state of the virtual-real mapping of the global situation map adopted in the embodiment of the present invention;

[0024] Figure 8 is a schematic diagram for comparing the planned routes of different movement plans of carrier-based aircraft in the embodiment of the present invention;

[0025] Figure 9 Schematic diagram of multi-operator scenario sharing and interactive deduction from different perspectives in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The following further describes the specific embodiments of the present invention in conjunction with the accompanying drawings.

[0027] The simulation and deduction environment constructed by the present invention is a virtual-real fusion environment, including a three-dimensional virtual model of the simulation and deduction object, the virtual situation information of the simulation and deduction object, and an intelligent vehicle entity representing the simulation and deduction object running on the sand table deduction platform. The movement postures of the three-dimensional virtual model and the vehicle entity on the constructed sand table deduction platform are displayed through the global situation map and the human-computer interaction device, facilitating multiple operators to deduce the complex group movement of the virtual-real hybrid environment using the information presented by the global situation map and the human-computer interaction device; at the same time, the final movement result of the vehicle entity can also be fed back into the global situation map.

[0028] Method Embodiment

[0029] The deduction method of the present invention first uses the virtual situation information related to the scene to construct a complex group movement simulation and deduction environment and displays it through the global situation map. Based on the human-computer interaction device to share virtual information, the joint deduction personnel are in the same virtual-real fusion scene environment. Based on the global situation map and the shared virtual information, multiple participants can simultaneously perform distributed parallel deductions on the complex group movement of the virtual-real hybrid environment; based on the deduction results, the machine intelligence calculation recommendation scheme is corrected to achieve the complementary advantages of human-machine collaboration and realize the optimization and correction of the complex group movement scheme. The implementation process of this method is as Figure 1 shown. The following takes the aircraft cluster in the support operation scene as a complex group to elaborate in detail the specific implementation process of the invention.

[0030] 1. Construct a complex group movement simulation and deduction environment.

[0031] The present invention uses relevant situation data to construct a complex group movement simulation and deduction environment (referred to as the simulation scene), including a virtual environment state space and a real environment state space. The virtual environment state space includes a three-dimensional virtual model of the simulation and deduction object and the virtual situation information of the simulation and deduction object. The real environment state space includes a real electronic sand table scene and related human-computer interaction devices. The real electronic sand table scene is virtual-real fusion, including a real sand table deduction platform and an intelligent vehicle entity representing the simulation and deduction object running on the sand table deduction platform. Moreover, the vehicle entity corresponds one-to-one with the three-dimensional virtual model and maps the virtual situation information of the corresponding simulation and deduction object.

[0032] In this embodiment, the aircraft cluster in the support operation is selected as a complex group for research. Among them, the simulation and deduction object is the aircraft. The constructed virtual environment state space includes a three-dimensional virtual model of the aircraft and the virtual situation information of the aircraft. The real sand table deduction platform simulates the aircraft support operation scene, and the intelligent vehicle entity represents the aircraft.

[0033] When constructing a simulation and deduction environment for complex group movement, the present invention also defines a virtual-real fusion state space Θ = V ο ∷∞R ο ∷;V ο represents the virtual environment state space; R ο represents the real environment state space; the virtual-real hybrid space is naturally connected by the ∞ symbol. The virtual-real mapping function M: where f v represents the interaction that causes the state transition change in the virtual space; f R represents the interaction that causes the state transition change in the real space. The real environment state space in this embodiment refers to the physical electronic sand table, and it is set that the physical entity of the intelligent vehicle corresponds to A i ={A1, A2, A 3, ...}; the virtual entity of the simulation and deduction object in the virtual environment (the aircraft virtual entity in this embodiment) corresponds to It means that the physical entity of the intelligent vehicle and the virtual entity in the virtual-real hybrid space correspond one by one. Then, according to the virtual-real mapping, the state change of the intelligent vehicle entity in the real environment will be mapped to the virtual environment, that is

[0034] The global situation map is used to display the virtual environment state space and the real environment state space, including the three-dimensional virtual model of the simulation and deduction object on the sand table deduction platform (the three-dimensional virtual model of the aircraft in this embodiment to ensure the operation scenario) and the movement trend of the vehicle entity.

[0035] 2. Through virtual-real fusion multi-person machine scenario sharing, based on the sharing of virtual information by human-machine interaction devices, multiple participants simultaneously perform distributed parallel deduction on the complex group movement of the virtual-real hybrid environment.

[0036] The essence of the deduction method is an iterative optimization process of "human-machine-environment" virtual-real fusion collaborative interaction operation. The decision-making scheme is cooperated by humans and machines to achieve the purpose of hybrid enhanced intelligent optimization. As Figure 2 shown, this is a continuous evolution process from abstraction to concretization, relying on multi-party collaboration, rather than being completed unilaterally by humans or machine algorithms. The present invention uses Hololens2 as the human-machine interaction device, and presents the shared virtual information to the wearing operator through the MR glasses of Hololens2. In order for all operators to see the shared virtual information, the present invention also displays the shared virtual information presented by the MR glasses through a visualization large screen. Model the operation state space of the jointly participating operators: H i ∷ = bHSet∞rHSet, H i represents the state space for multiple people to jointly and actively intervene in interactive correction operations. The elements of the model set respectively represent the operator interaction behavior set bHSet and the rule constraint model set rHSet.

[0037] In the present invention, the interaction between humans, machines, and the environment mainly relies on the MR mixed reality helmet glasses device of Hololens2. Through immersive interaction (interacting based on gesture recognition and UI by dragging and dropping), human-machine interaction interface layout design (promoting intuitive human understanding of information and reducing cognitive load), virtual-real fusion situation superposition (the virtual situation information moves synchronously with the movement of the intelligent vehicle), etc., the human-machine intelligence fusion is enhanced.

[0038] 3. Based on the electronic deduction sand table for interactive visualization deduction and correction, use the deduction results to correct the machine intelligence calculation recommendation, and realize the optimization and correction of the complex group cluster movement plan through the complementary advantages of human-machine collaboration.

[0039] The movement route of the complex group is planned through the path planning algorithm provided by the machine. The planned route is used as the preset trajectory and presented through the MR glasses situation superposition. Based on the path planning algorithm, the machine calculates the movement plan route of each virtual entity of the simulation deduction object in the scene, denoted as S mi ={S m1 , S m2 , S m3 ,...}, and at the same time, the preset trajectory is sent to the corresponding intelligent vehicle entity, and the intelligent vehicle entity is controlled to run according to the preset trajectory, and the preset trajectory is presented to the operator through the MR glasses, as Figure 6 shown. Without human active intervention to correct the movement of the entity vehicle, the vehicle entity can run according to the machine recommended plan (i.e., the preset trajectory). When it is necessary to intervene and correct the movement of the vehicle entity, the vehicle entity runs according to the corrected trajectory (determined according to the preset trajectory).

[0040] Specifically, the operator can actively intervene or passively access the correction process of the path of the object as needed. The specific process is as follows: Based on the preset trajectory recommended by the machine as a reference, and then combined with the auxiliary support of multi-device source information and the monitoring of the global situation map, the operator can control the dynamic changes of the global situation of the scene, fully integrate human characteristic capabilities to efficiently correct the preset trajectory, and obtain the actual movement route of the intelligent vehicle entity after the operator's interactive operation intervention and correction, denoted as S hi ={S h1 , S h2 , S h3 ,...}, during the correction, multiple operators can also be combined. The formal representation of the combined correction is The actual movement trajectory of the intelligent vehicle entity, that is, the tracking trajectory, will be visually presented in the MR glasses and the global situation map. The actual movement trajectory route plan presented through the MR human-machine interaction device based on the MR glasses superposition and corrected by humans is denoted as S' hi ={S'h1 , S' h2 , S' h3 ,...}; As Figure 7 shown, map the movement route presented by the virtual-real mapping of S' hi = {S' h1 , S' h2 , S' h3 ,...} in the global situation map as S'' hi = {S'' h1 , S'' h2 , S'' h3 ,...}; It means that the actual route obtained after the intelligent vehicle entity is corrected by humans is consistent with the three types of movement trajectory route forms presented by the MR glasses situation superposition and the global situation map visual presentation. Among them, the operator can control the movement of the corresponding intelligent vehicle entity through a remote control device (such as a car remote control), and finally obtain the movement trajectory of the entity as S hi = {S h1 , S h2 , S h3 ,...}, as Figure 8 shown.

[0041] Among them, the global situation map depicts the overall state of the entire scene, and the information is the most comprehensive. The human-computer interaction device is equivalent to the first perspective, and the information is not as comprehensive as the global situation map. What is presented in the human-computer interaction device is the virtual UI situation information (preset trajectory, tracking trajectory, status information, etc.) superimposed on the simulation and deduction object; as part of the visual analysis module, the global situation map is presented on the visual display screen, which can display the top view of the entire movement scene and present the complex group movement situation in the entire scene. The virtual objects in the global situation map correspond one by one to the movement of the physical cars on the sand table. The information presented by the global situation map and the human-computer interaction device during the deduction process are both used.

[0042] The following is illustrated with a specific example. The initial formal design method for multi-person joint collaborative operation is the combination of human and machine, that is, the integration of rationality and sensibility. In this collaborative mode, the interaction surface between humans and machines is wider, deepening the breadth and depth of the perceived situation information and assisting in supporting efficient decision-making. The three-dimensional visualization model based on the electronic deduction sand table constructed by the present invention is as Figure 3 shown. For the immersive environment provided by mixed reality, the rule constraint conditions of space-time resources represent the true scientific nature of the interaction. Specific states or abnormal information, etc. can also be corrected in a timely manner under the supervision of humans. The movement plans of intelligent agents, including situation information such as preset trajectories and tracking trajectories, will be presented through the virtual-real fusion visualization of the mixed reality helmet glasses. Realize multi-human-machine collaborative deduction through the electronic deduction sand table, and use the human-computer interaction device to connect the virtual environment and the real world, asFigure 4 As shown, the visual display of the superposition of virtual and real fusion situation information. By perceiving through multiple channels, the sources of situation information acquisition are increased, and human command experience is fused in real time. The perception and cognitive abilities of humans are combined with the computing power of machines. For example, Figure 5 As shown, through the man-machine interaction device, the operator can not only see the physical intelligent vehicle, but also see the man-machine interaction interface. The commander can control the corresponding instructions through gestures. Multiple people and machines jointly conduct interactive visual deduction and correction of the path planning scheme recommended by the machine algorithm. Finally, "what is modified is what is seen" is realized, and the situation information such as the preset trajectory and the tracking trajectory is visually presented in a virtual-real fusion manner through the mixed reality helmet glasses.

[0043] The interactive deduction of different perspectives in the multi-person and multi-machine scenario sharing is as Figure 9 As shown, a total of 3 operators share through the virtual-real fusion multi-person and multi-machine scenario. The perspectives of different operators are different. Based on the man-machine interaction device, virtual information is shared, and multiple participants simultaneously conduct distributed parallel deduction of the situation information in the virtual-real hybrid environment. The perception of people is combined with the accuracy and consistency of the machine system, and the situation awareness ability of the system is improved through human-machine cooperation, forming a consistent understanding of the scene situation. As other implementation manners, more operators can also perform cooperative control.

[0044] System embodiment

[0045] The electronic sand table system of the present invention conducts deduction by using the above-mentioned multi-person and multi-machine interactive visual deduction method for the electronic sand table. The specific implementation process has been described in detail in the method embodiment and will not be elaborated here.

[0046] The present invention can realize the efficient and intelligent cooperation between the operator and the machine, complete the deduction of complex group movements based on the electronic deduction sand table, make up for the deficiencies of the intelligent algorithm by using the experience of the operator, and use the machine intelligence to assist the operator in making human decisions. The human-machine fusion collaborative interaction can flexibly respond to the fuzzy and uncertain environmental information under various complex situations, realize the complementation and sharing of the global situation perception and cognition, and the added physical sand table enables the operator to visually observe and intuitively understand the deduction process in real time, and can correct it in time by using his own experience. At the same time, based on the mixed reality technology, an immersive interactive virtual-real fusion deduction simulation environment can be given, combined with visual situation analysis, to improve the human's control of the overall situation of complex group movements.

Claims

1. A multi-person computer interactive visualization deduction method for an electronic sandbox, characterized in that: The method includes: 1) Constructing a cluster motion simulation environment of a complex group of simulation objects, including a virtual environment state space and a real environment state space. The virtual environment state space includes a three-dimensional virtual model of the simulation object and virtual situation information of the simulation object. The real environment state space includes a physical electronic sand table. The physical electronic sand table includes a sand table simulation platform and an intelligent car entity running on the sand table simulation platform. 2) Using the global situation map to display the three-dimensional virtual model of the virtualized simulation deduction object and the movement process of the intelligent car entity, and presenting the virtual information in the simulation deduction environment to the operator based on the human-computer interaction device worn by the operator, so that each operator can use the global situation map and the information presented by the human-computer interaction device to deduce the complex group movement in the virtual-real mixed environment; 3) Human-machine fusion controls the physical operation of the intelligent car and feeds back the control results to the global situation map.

2. The multi-person computer interactive visualization deduction method for electronic sandbox according to claim 1 is characterized in that: The human-machine fusion control of the intelligent car's physical operation includes the operator not intervening in the correction of the car's physical operation and the operator intervening in the correction of the physical car's operation. Not intervening in the correction of the physical car's operation means that the physical car runs according to the preset trajectory given by the machine; intervening in the correction of the physical car's operation means controlling the physical car's operation according to the preset trajectory.

3. The multi-person computer interactive visualization deduction method for electronic sandbox according to claim 2 is characterized in that: The deduction process includes: the operator uses the preset trajectory given by the machine as a reference, combines the global situation map to determine the dynamic changes of the scene's global situation, and uses human experience to correct through distributed deduction to determine the actual running trajectory of the smart car entity.

4. The multi-person computer interactive visualization deduction method for electronic sandbox according to claim 1 is characterized in that: Controlling the physical operation of the smart car according to a preset trajectory means that the operator controls the physical actions of the smart car through remote control.

5. The multi-person computer interactive visualization deduction method for electronic sandbox according to claim 3 is characterized in that: The method also includes superimposing the actual running trajectory of the intelligent car entity onto the information presented by the human-computer interaction device.

6. The multi-person computer interactive visualization deduction method for electronic sandbox according to claim 1 is characterized in that: The method includes intuitively displaying the multi-perspective situation deduction process of the virtual-reality fusion scene presented by the human-computer interaction device through a large visualization screen.

7. The multi-person computer interactive visualization deduction method for electronic sandbox according to claim 2 is characterized in that: The smart car entity model presented by the human-computer interaction device is loaded with a UI design of virtual situation information related to the simulation deduction object corresponding to the smart car entity, and the superimposed virtual situation information UI interface moves with the movement of the smart car entity.

8. The multi-person computer interactive visualization deduction method for electronic sandbox according to claim 1 is characterized in that: The human-computer interaction device is Hololens2.

9. An electronic sand table system, characterized in that: The electronic sandbox system uses the multi-person computer interactive visualization deduction method for the electronic sandbox as described in any one of claims 1-8 to perform deduction.

Images