Human-in-loop virtual-real fusion digital twinning system

Abstract

The invention relates to a human-in-loop virtual-real fusion digital twinning system, and belongs to the technical field of digital twinning. According to the method, the entity sand table system is used for simulating the entity environment and the entity agent of the complex operation system where the person is located in the loop, virtual simulation is carried out on the complex operation system where the person is located in the loop through the virtual agent and virtual personnel of the twin deduction system, and twin simulation can also be carried out based on state information of the entity sand table system; or the two are combined to realize virtual-real fusion simulation. According to the virtual-real fusion digital twinning system, an entity agent-virtual human agent virtual-real collaborative operation mode is achieved, a human agent performs virtual-real collaborative operation with other agents in a simulation deduction platform in a virtual-real fusion mode, and credible simulation of a person in a complex operation system where a loop is located is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to a virtual-real fusion digital twin system of man-in-the-loop, and belongs to the technical field of digital twin. BACKGROUND

[0002] Large-scale heterogeneous cluster collaborative operation refers to a heterogeneous cluster composed of a large number of operation agents in a complex scene to serve various operation tasks. Operation efficiency is the core index for evaluating the operation scheme. With the help of computers and other tools, each type of operation scheme is evaluated and verified, which is the only way to deploy the above-mentioned scheme verification algorithm to practical application and transform it into productive forces. However, due to the high cost and risk of field verification, most operation scheme evaluations and verifications still rely on virtual simulation technology, which is highly questionable in terms of authenticity.

[0003] In recent years, with the development of artificial intelligence and Internet of Things technologies, digital twin-based scheme verification has been widely concerned and has made rapid development in the fields of workshop production, rail transportation, medical care, power systems, and urban management. The main comparison with cluster operation is traffic digital twin. Existing traffic digital twin is mainly focused on real-time data analysis and prediction based on real physical world, and the virtual entity is generally a digital reproduction of the actual environment. On the one hand, it relies too much on the actual environment and has potential safety risks when connecting to the actual environment for testing. On the other hand, the potential of virtual space has not been fully explored. In view of these problems, Dong et al. designed a vehicle motion virtual-real fusion digital twin platform based on sand table deduction, which provides great convenience for verifying vehicle motion in road networks. However, this platform only considers simple lane-following vehicle motion and cannot be used to simulate complex operations of heterogeneous clusters composed of multiple types of agents, especially for man-in-the-loop systems where personnel behavior is also important. Existing physical robots cannot achieve the freedom of human beings, so personnel agents are generally not set in physical sand tables. Without personnel agents, it is impossible to comprehensively simulate and test complex systems with man-in-the-loop, and the effect is poor.

[0004] In summary, current complex systems with man-in-the-loop either use purely virtual digital models for simulation and demonstration, which rely on real environment data and have potential safety risks when connecting to the actual environment for testing, or use physical sand tables for deduction, but physical sand tables cannot fully represent the behavior of personnel agents in complex systems with man-in-the-loop, and the simulation credibility is low. SUMMARY

[0005] The purpose of the present application is to provide a virtual-real fusion digital twin system of man-in-the-loop to solve the problem of relying on real environment data and low credibility when testing complex systems with man-in-the-loop.

[0006] In order to solve the above technical problems, the present invention provides a digital twin system of virtual-reality fusion of people in the loop. The digital twin system includes a cloud control system, a physical sandbox system and a twin deduction system. The cloud control system is used to create an operation task and send the operation task to the physical sandbox system, and collect the status information of the physical sandbox system in real time; the physical sandbox system includes a physical sandbox body, which is used to simulate the environment of people in the loop. The physical sandbox body is equipped with a non-personnel physical intelligent body, and the physical sandbox system is used to simulate the environment and control the operation of the physical intelligent body thereon according to the received operation task; the twin deduction system is used to perform virtual object motion simulation according to the simulation task generated by itself, perform twin simulation according to the status information of the physical sandbox system sent by the cloud control system, or perform virtual-reality fusion simulation based on the simulation task generated by itself and the status information of the physical sandbox system sent by the cloud control system; the virtual objects include virtual non-personnel intelligent bodies and virtual human intelligent bodies.

[0007] Furthermore, the system also includes an interactive interface, which is used to enable interaction between the user and the physical sandbox system and / or the twin deduction system.

[0008] Furthermore, the interactive interface is used to enable the user to enter the real world of the physical sandbox system and the virtual world of the twin simulation system from a first-person perspective, and to issue instructions to the physical sandbox system and the twin simulation system to achieve human-computer collaborative control.

[0009] Furthermore, the twin deduction system adopts the HLA architecture and integrates algorithm modules for path planning, non-human intelligent agent motion simulation, and human intelligent agent motion simulation.

[0010] Furthermore, the HLA architecture adopted by the twin deduction system includes four layers, namely, the data resource layer, the system component layer, the application service layer and the user interface layer from bottom to top; the data resource layer is used to provide various data storage and management functions for twin deduction; the system component layer is used to provide the model algorithm components, object management components and scenario setting components required for twin deduction; the application service layer is used to orchestrate the various components provided by the system component layer according to different application requirements, thereby forming different types of application services; the user interface layer is used to provide three-dimensional visualization display, a general management interface and UI design according to different users and different application requirements.

[0011] Furthermore, the model algorithm component provided by the system component layer includes relevant model algorithms. The object management component is used to manage the various virtual objects in the twin deduction system, perform object-oriented modeling and storage on all virtual objects in the system, and provide a unified interface to facilitate various operations on the entity objects in the twin deduction system; the scene setting component provides the twin deduction system with twin scene construction, scene rendering, and scene import functions.

[0012] Furthermore, when creating a work task, the cloud control system needs to integrate the complex work system where the human is in the loop into a model, and create the work task based on the modeling results. The created work task includes the task type, execution conditions, execution time and target parameters.

[0013] Furthermore, the cloud control system is also used to obtain and store the status information of each virtual object in the twin deduction system.

[0014] Furthermore, the cloud control system integrates motion control, path planning, target detection and tracking, and obstacle avoidance algorithms for controlling the operation of intelligent agents. The cloud control system generates action instructions for each intelligent agent based on the work task, and sends the generated action instructions to the intelligent agents mounted on the physical sandbox body to control the operation of each intelligent agent.

[0015] Furthermore, the physical intelligent body carried on the physical sandbox body is embedded with motion control, path planning, target detection and tracking, and obstacle avoidance algorithms. The intelligent body performs motion control independently based on the work tasks issued by the cloud control system.

[0016] The beneficial effects of the present invention are as follows: as an improved invention, the present invention uses a physical sandbox system to simulate the physical environment and physical intelligent body of the complex operating system where the person is in the loop, and uses the virtual intelligent body and virtual personnel of the twin deduction system to perform virtual simulation of the complex operating system where the person is in the loop. Twin simulation can also be performed based on the state information of the physical sandbox system, or the two can be combined to achieve virtual-real fusion simulation. The virtual-real fusion digital twin system of the present invention realizes the virtual-real collaborative operation mode of "physical intelligent body-virtual human intelligent body", allowing the human intelligent body to perform virtual-real collaborative operation with other intelligent bodies in the simulation deduction platform through virtual-real fusion, ensuring the credible simulation of the complex operating system where the person is in the loop. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of the overall design framework of the virtual-reality fusion digital twin system in the loop by the inventor;

[0018] Figure 2a It is a schematic diagram of the physical sandbox environment state detected by the cloud control system in a specific embodiment of the present invention;

[0019] Figure 2b It is a schematic diagram of the state of the intelligent agent in the physical sandbox detected by the cloud control system in a specific embodiment of the present invention;

[0020] Figure 3a This is a schematic diagram of a job task sequence arranged by a cloud control system in a specific embodiment of the present invention;

[0021] Figure 3b This is a schematic diagram of an intelligent agent operation task set by the cloud control system in a specific embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of unified data management performed by a cloud control system in a specific embodiment of the present invention;

[0023] Figure 5 This is a functional architecture diagram of a twin deduction system in a specific embodiment of the present invention;

[0024] Figure 6a This is a business flow chart of cluster motion simulation and deduction performed by the twin deduction system in a specific embodiment of the present invention;

[0025] Figure 6b This is a business flow chart of cluster motion simulation playback performed by the twin deduction system in a specific embodiment of the present invention;

[0026] Figure 7a It is a user UI interface rendering displayed by the user interface layer in a specific embodiment of the present invention;

[0027] Figure 7b It is a three-dimensional rendering of a cluster motion simulation displayed on the user interface layer in a specific embodiment of the present invention;

[0028] Figure 8a Schematic diagram of a user interface layer using a driving simulator to take over a virtual agent in a specific embodiment of the present invention;

[0029] Figure 8b Schematic diagram of a user interface layer using a driving simulator to take over an entity agent in a specific embodiment of the present invention;

[0030] Figure 9 It is a functional framework diagram of the inventor's digital twin system integrating virtual and real elements in the loop. DETAILED DESCRIPTION

[0031] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0032] The present invention utilizes a physical sandbox system and a twin deduction system to realize the virtual-real collaborative operation mode of "physical intelligent agent-virtual human intelligent agent", so that the human intelligent agent can perform virtual-real collaborative operations with other intelligent agents through virtual-real fusion in the simulation deduction platform.

[0033] Virtual-real fusion digital twin system implementation of man-in-the-loop

[0034] The virtual-real fusion digital twin system (also known as a test bed) of man-in-the-loop, as shown in Figure 1 The cloud control system is responsible for information collection, analysis, and the issuance of cluster individual scheduling results / global path planning results. The entity sand table system and the digital twin simulation system represent the real world state and the virtual world state, respectively. The process of virtual-real fusion of the system is to map the real world state to the virtual world, depicting the virtual world as originating from the real world and having properties higher than the real world. The digital twin simulation system simulates and deduces objects with higher properties than the real world based on the global path planning results issued by the cloud control system. The interactive interface system realizes virtual-real fusion and interaction through displays, driving simulators, headsets, and handles, and plays the role of intelligent enhancement. The cloud control system is used to create a job task and issue it to the entity sand table system, and to collect real-time state information of the entity sand table system. The entity sand table system includes an entity sand table body, which is used to simulate the environment of man-in-the-loop. The entity sand table body carries non-personnel entity agents. The entity sand table system is used to simulate the environment and control the running of entity agents on it based on the received job task. The digital twin simulation system is used to simulate the motion of virtual objects based on the simulation task generated by itself, to perform digital twin simulation based on the state information of the entity sand table system sent by the cloud control system, or to perform virtual-real fusion simulation based on the simulation task generated by itself and the state information of the entity sand table system sent by the cloud control system. The design and implementation process of the above four parts are described in detail below.

[0035] Cloud control system

[0036] The cloud control system has a global view of the virtual-real fusion digital twin system of man-in-the-loop, and is the central brain of the virtual-real fusion digital twin system of man-in-the-loop, which is used for real-time monitoring, remote control, data analysis, etc. of the entire virtual-real world. Before introducing the functions of the cloud control system in detail, the virtual-real fusion digital twin system of man-in-the-loop is defined and modeled for easy understanding.

[0037] The virtual-real fusion digital twin system of man-in-the-loop can be abstractly represented as a virtual-real fusion system Θ in which heterogeneous agents perform collaborative tasks under the constraints of space, resources, etc. The fusion system Θ can be expressed as:

[0038] Θ=(W∪V,R,P,C,H)

[0039] W represents the real world state:

[0040] W=W(t)={E(t),P(t),C(t)}

[0041] where E(t), P(t), and C(t) represent the state of the environment and heterogeneous agents in the real world at time t, respectively. Taking the complex human-in-the-loop operation system as an example, the intelligent support system is represented by |P(t)| = N, where |P(t)| = N represents the number of supported agents in Θ, and |C(t)| = M represents the total number of vehicle agents performing the support operation.

[0042] V represents the state of the virtual and real world:

[0043]

[0044] in H(t) represents the state of the virtual world environment, all protected agents, all vehicle agents performing support operations, and all human agents performing support operations at time t. |H(t)|=O represents the total number of human agents performing security operations.

[0045] R represents the task sequence in Θ:

[0046] R={R1,R2,…,R D}

[0047] Where D is the number of subtasks, R i (i∈[1,D]) represents the attribute state set of the i-th subtask, RC i RH i ∈R i They represent the support vehicle and support personnel information required for the i-th subtask. i ={RC ij ,j∈[1,CM]} represents the number of support vehicles of type j required by subtask i, CM represents the total number of support vehicle types, RH i ={RH ij ,j∈[1,HO]} represents the number of support personnel of type j required by subtask i, and HO represents the total number of support personnel types.

[0048] P represents the attribute information of the task and target position of the protected agent in Θ:

[0049] P={<PT1,PS1>,<PT2,PS2>,…,<PT N ,PS N >}

[0050] in represents the subtask sequence of the protected agent i, n 1irepresents the total number of subtasks of the i-th supported agent, i∈[1,N], PT ij ={PTT ij ,PTP ij ,PTB ij ,PTC ij ,PTH ij} respectively represent the type, priority, start time, actual assigned support vehicle, and support personnel information of the supported agent i. ij ={PTC ijk ,k∈[1,CM]} represents the number of vehicles of type k actually assigned to the jth subtask of the i-th protected agent, PTH ij ={PTH ijk ,k∈[1,HO]} represents the number of support personnel of type k actually assigned to the j-th subtask of the supported agent i, Represents the destination information set of the protected agent i during the task process, where n 2i represents the total number of destinations of protected agent i, PS ij =<pts ij ,ptg ij >, pts ij represents the time when the protected agent i goes to the jth destination, ptg ij represents the j-th destination of the protected agent i.

[0051] C represents the attribute information of the vehicle mission and target location in Θ:

[0052] C={<CT1,CS1>,<CT2,CS2>,…,<CT M ,CS M >}

[0053] CT i Indicates the type of protected vehicle i, Represents the destination information set during the mission of vehicle i, where m 2i represents the total number of destinations for vehicle i, CS ij =<cts ij ,ctg ij >, cts ij Indicates the time when vehicle i is guaranteed to go to the jth destination, ctg ij represents the j-th destination of the guarantee vehicle i.

[0054] H represents the attribute information of the support personnel tasks and target locations in Θ:

[0055] H={<HT1,HS1>,<HT2,HS2>,…,<HTO ,HS O >}

[0056] Among them HT i Indicates the type of support personnel i, represents the destination information set during the mission of support personnel i, o 2i represents the total number of destinations of person i, HS ij =<hts ij ,htg ij >, its hts ij represents the time when support personnel i goes to the jth destination, htg ij represents the j-th destination of support personnel i.

[0057] Therefore, the complex operation problem of the intelligent support system in this embodiment is transformed into a heterogeneous cluster motion integrated modeling problem for the operation tasks of the supported intelligent bodies, that is, the known environmental information and the supported intelligent body operation task list information R, Calculate the system status W∪V and the execution status of the job tasks and target location information Target location information of all support vehicles Target location information for all support personnel The information required for calculation can be categorized into three types: task information for the protected agent, target location information for clustered individuals (e.g., protected agents, support vehicles, and support personnel), and system status. The first two types can be derived using a scheduling algorithm based on known system information. A global path planning algorithm is then used to determine the global path for each individual based on their target location information. Furthermore, cluster simulation methods can be used to determine the system's state at any given moment.

[0058] Through the above modeling, the digital twin of the present invention provides an integrated mathematical description of the relationship between the heterogeneous cluster operation tasks, spatiotemporal motion behaviors and other elements involved. This allows for fine-grained modeling of complex operation processes by simply inputting the operation tasks of each main intelligent agent during application, without having to specify tasks for each object within the cluster. Based on this, the specific functions of the cloud control system of the present invention include:

[0059] Virtual and real world monitoring and control: The virtual world is constructed by the twin deduction system, and the real world is constructed by the physical sandbox system. Therefore, the virtual and real world monitoring here refers to the communication and interaction between the cloud control system, the twin deduction system and the physical sandbox system. The cloud control system uses the capture module in the physical sandbox system to monitor the status of the physical sandbox (online status, operating status, detailed parameters, etc.). The cloud control system also remotely controls the environmental status and the status of intelligent objects (intelligent bodies) on the physical sandbox through communication (system effects such as Figure 2a and Figure 2b As shown). Through communication and interaction with the twin deduction system, users can set the attribute status of each smart object in the virtual world in real time according to the status of the detected physical sandbox.

[0060] Job task arrangement and issuance: Users can create and configure job tasks in the cloud control system, including task type, execution conditions, execution time, target device and other parameters (for example, job task sequence information R, number of subtasks for each agent, type, priority, etc.). Once the task is created and confirmed, the user can use the task issuing function of the cloud control platform to send the task instructions to each intelligent object that performs the task in the physical sandbox system through network communication. It can also be sent to the twin deduction system (system effect is as follows Figure 3a and Figure 3b shown).

[0061] Data integration analysis and prediction: Taking the complex operating system where people are in the loop as an intelligent security system as an example, the status information of the physical entity of the intelligent security system at the time of operation will be transmitted to the cloud control system through the data capture modules such as the optical motion capture system, radio frequency identification positioning system, RGB image data acquisition system, and depth data acquisition system deployed in the physical sandbox. The status information of various pure virtual objects in the twin deduction system can also be transmitted to the cloud control system. Taking into account the diverse and large-scale characteristics of operating data, the cloud control system uses various types of data storage systems to store the above data, such as relational databases for storing structured data, memory databases for storing real-time data, text databases for storing text data, object databases for storing discrete data, and commonly used file systems. In order to uniformly manage these databases, the cloud control system abstracts the operations of various databases (such as Figure 4 ), providing a unified access interface for all types of data. In addition to storing the aforementioned data, the cloud control system can also perform data cleaning, alignment and conversion, model lightweighting, statistical analysis, and prediction. When potential risks are predicted, the cloud control system can also issue timely warnings and provide appropriate response recommendations.

[0062] Physical sandbox system

[0063] The physical sandbox is a composite sandbox obtained by scaling and folding the actual system. It is equipped with physical intelligent bodies and can perform sandbox simulations of heterogeneous cluster movements. In other words, the physical sandbox system of the present invention is used to display the real working scenes of complex working systems where people are in the loop. Taking the intelligent security system as an example of a specific complex working system, the physical intelligent bodies carried in the physical sandbox in this embodiment include all the secured intelligent bodies and all the intelligent bodies of vehicles performing security operations, and can simulate various environmental information, including obstacles, etc. In order to facilitate the cloud control system to obtain the environmental information of the physical sandbox system and the status information of the intelligent bodies in real time, the present invention also deploys data capture modules such as optical motion capture system, radio frequency identification and positioning system, RGB image data acquisition system, and depth data acquisition system in the physical sandbox.

[0064] Twin deduction system

[0065] The twin deduction system uses HLA as the architecture to integrate algorithm modules such as path planning, various intelligent body motion simulation, and crowd motion simulation, thereby realizing diversified and efficient collaborative modeling of large-scale heterogeneous clusters. The twin deduction system using HLA architecture has strong generalization capabilities and can be migrated to scenarios such as logistics scheduling and operation simulation. The twin deduction system of the present invention includes four layers from bottom to top: data resource layer, system component layer, application service layer, and user interface layer. Figure 5 shown.

[0066] Data resource layer: This layer serves as the basic data support and provides various data storage and management functions for twin deduction, including algorithm model library, simulation database, case database, three-dimensional model library, business database, etc. In addition to managing the input data required by the system, it also stores and manages the simulation result data, algorithm output data, business data, etc. during and after the operation of the twin deduction system.

[0067] System component layer: This layer is the core function of the twin deduction system, including model algorithm components, object management components, scene setting components, etc. Among them, the model algorithm component includes path planning, cluster simulation and other related model algorithms. Users can choose the algorithms they need to use and integrate them into the twin deduction system. For each model algorithm that needs to be integrated into the twin deduction system, it is connected to the twin deduction system by building a dynamic library or a static library. The object management component mainly manages the various virtual objects in the twin deduction system (for the intelligent support system, the virtual objects include the protected intelligent body, the support vehicle intelligent body, the support personnel intelligent body, obstacles, etc.), and performs object-oriented modeling on all virtual objects in the twin deduction system and stores them in the global data manager. Then, a unified interface is provided to the outside world to facilitate other modules such as the cloud control platform to perform various operations on the virtual objects in the system. The scene setting component provides the twin simulation system with functions such as twin scene construction, scene rendering, and scene import. Users can import three-dimensional models into the system for scene construction according to their own needs. Based on the powerful rendering function of Unreal Engine, users can render realistic lighting, weather and other special effects elements to enhance the richness and fidelity of scene simulation.

[0068] Application service layer: This layer is based on the different components provided in the system component layer. It flexibly arranges system components according to different application requirements and forms different types of application services, including cluster simulation, scenario guidance, parameter optimization, algorithm testing and other functions. Figure 6a For the business process of cluster motion simulation and deduction, after the twin scene is imported and rendered, the data required for cluster motion simulation and deduction is input and processed, the corresponding intelligent agent is created according to the data, and then the calculation of each algorithm model in the cluster simulation is started. Figure 6b For the business process of cluster motion simulation playback function, after the twin scene is imported and rendered, there is no need to perform algorithm calculations, and the existing simulation data can be directly used for visual rendering.

[0069] User interface layer: This layer provides three-dimensional visualization, general management interfaces, and UI interfaces for different users and application needs. To use an application service in the twinning system, users need to interact with the application service through the visual UI interface provided by this layer. To migrate the twinning system to a different application scenario, in addition to rearranging the twinning system components according to business needs, only targeted modifications to the UI in the user interface layer are required. Figure 7a The following is a customized user UI interface rendering. Figure 7b Shown is a three-dimensional effect display of cluster motion simulation.

[0070] It can be seen that the twin inference system of the application adopts the HLA distributed simulation architecture to design the federal members of each part of the system based on the data provided by the data resource layer and according to the needs of the users of the application service layer, each federal member represents a certain system component in the system component layer, the completed federal members collectively form a simulation federation, and each federal member can be distributed in different platform systems, on the one hand, the system bottom logic and related algorithms are decoupled, which greatly improves the expandability of the system, on the other hand, the data barriers between algorithms are broken, data sharing is realized, and the data communication between algorithms can be more flexible. The twin inference system of the application can perform virtual object motion simulation based on the simulation tasks generated by itself, and can also access the state information (environment and job agents, etc.) of the entity sand table, fuse the real world simulated by the entity sand table and the virtual world simulated by the twin inference, and then generate a hybrid space of virtual and real to perform three-dimensional visualization display in the twin inference. In the display effect, part of the objects are digital copies of physical entities, and the other part are virtual objects calculated by cluster motion simulation algorithms in the virtual space.

[0071] Interactive interface system

[0072] The interactive interface system is a middle component connecting the user, the entity sand table and the twin inference, and the interactive interface system can make the user enter the real world of the entity sand table and the virtual world of the twin inference in the first perspective and issue control instructions by means of the driving simulator, the AR augmented reality helmet and other devices, the user can take over the control of the protected agents, vehicles and personnel in the real world and the virtual world, and realize the man-machine collaborative control. The interactive interface module can realize various user intervention virtual-real fusion modes such as man-virtual simulation, man-entity sand table, man-entity sand table-virtual simulation, etc. Figure 8a and Figure 8b respectively are effect schematic diagrams of the driving simulator taking over the twin inference virtual world vehicle (virtual agent) and the entity sand table vehicle agent (entity agent).

[0073] In summary, the virtual-real fusion digital twin system of the man-in-the-loop of the application integrates the cloud control system, the entity sand table system, the twin inference system and the interactive interface system, and provides the user with scene generation, task allocation, situation awareness, layout optimization and other application services, and the functional module relationship of the above four system components is as shown in Figure 9 .

[0074] Among them, the cloud control system is responsible for functions such as monitoring and control of the virtual and real world, scheduling and issuing work tasks, and data integration analysis and prediction. Therefore, the system integrates algorithms such as data fusion, cluster control, task scheduling, path planning, evaluation and verification; the physical sandbox system is used to simulate the real world and is equipped with a variety of heterogeneous intelligent work objects. These intelligent objects serve as edge intelligence and are embedded with motion control, path planning, target detection and tracking, obstacle avoidance algorithms, etc.; the twin deduction system is responsible for realizing diversified and efficient collaborative modeling of large-scale heterogeneous clusters. The system integrates algorithms such as scene modeling, path planning, intelligent body motion simulation, three-dimensional modeling and rendering; the interactive interface system serves as the middleware connecting users, physical sandbox and twin deduction, and is embedded with algorithms such as tracking and positioning, virtual and real fusion, and multi-channel interaction.

[0075] The cloud control system can control objects in both the physical sandbox and the twinning system, so there is some functional overlap between the cloud control system and the physical sandbox, as well as between the cloud control system and the twinning system. For example, path planning algorithms can be integrated into the cloud control system or into the physical smart objects on the physical sandbox. The difference is that after being placed in the cloud control system, they need to be transmitted to the physical smart objects via network communication. Of course, motion control algorithms can also be integrated into the cloud control system or the physical smart objects. The advantage of integrating into the cloud control system is that it can easily implement cluster operation control, but the disadvantage is the high communication cost and severe communication dependency. The advantage of integrating into the physical smart objects is that it greatly reduces communication dependency, but the disadvantage is that cluster collaborative operations are more difficult.

[0076] The physical sandbox system and twin deduction system adopted in the present invention jointly realize the digital twin of virtual-reality fusion that originates from reality and is higher than reality. Therefore, the physical sandbox and the twin deduction system also have overlapping functions. For example, the physical intelligent objects on the physical sandbox require path planning algorithms, and the virtual protected targets / protection vehicles / protection personnel in the twin deduction system also require path planning algorithms. Different algorithms can be embedded here to achieve this, which also facilitates the comparison of the effects of different algorithms.

[0077] In the present invention, when deploying applications, in order to alleviate the pressure of data transmission and storage, the database can be deployed separately on a server. In addition, in order to enhance the data transmission capability between the systems and reduce interference factors in the network, the four systems can be deployed together in a local area network.

Claims

1. A virtual-reality fusion digital twin system with humans in the loop, characterized by: The digital twin system includes a cloud control system, a physical sandbox system and a twin deduction system. The cloud control system is used to create an operation task and send the operation task to the physical sandbox system, and collect the status information of the physical sandbox system in real time; the physical sandbox system includes a physical sandbox body, which is used to simulate the environment of people in the loop. The physical sandbox body is equipped with non-personnel physical intelligent bodies. The physical sandbox system is used to simulate the environment and control the operation of the physical intelligent bodies thereon according to the received operation tasks; the twin deduction system is used to perform virtual object motion simulation according to the simulation tasks generated by itself, perform twin simulation according to the status information of the physical sandbox system sent by the cloud control system, or perform virtual-reality fusion simulation based on the simulation tasks generated by itself and the status information of the physical sandbox system sent by the cloud control system. The virtual objects include virtual non-personnel intelligent bodies and virtual human intelligent bodies.

2. The virtual-reality fusion digital twin system of human-in-the-loop according to claim 1 is characterized in that: The system also includes an interactive interface, which is used to enable interaction between the user and the physical sandbox system and / or the twin deduction system.

3. The virtual-reality fusion digital twin system of human-in-the-loop according to claim 2 is characterized in that: The interactive interface is used to enable users to enter the real world of the physical sandbox system and the virtual world of the twin deduction system from a first-person perspective, and to issue instructions to the physical sandbox system and the twin deduction system to achieve human-computer collaborative control.

4. The virtual-reality fusion digital twin system of human-in-the-loop according to claim 3 is characterized in that: The twin deduction system adopts the HLA architecture and integrates algorithm modules for path planning, non-human intelligent agent motion simulation, and human intelligent agent motion simulation.

5. The human-in-the-loop virtual-reality fusion digital twin system according to claim 4 is characterized in that: The HLA architecture adopted by the twin deduction system includes four layers: data resource layer, system component layer, application service layer and user interface layer from bottom to top; the data resource layer is used to provide various data storage and management functions for twin deduction; the system component layer is used to provide the model algorithm components, object management components and scenario setting components required for twin deduction; The application service layer is used to orchestrate the various components provided by the system component layer according to different application requirements, thereby forming different types of application services; The user interface layer is used to provide three-dimensional visualization, general management interface and UI design according to different users and different application requirements.

6. The human-in-the-loop virtual-reality fusion digital twin system according to claim 5, characterized in that: The model algorithm components provided by the system component layer include relevant model algorithms. The object management component is used to manage the various virtual objects in the twin deduction system, perform object-oriented modeling and storage on all virtual objects in the system, and provide a unified interface to facilitate various operations on the entity objects in the twin deduction system; the scene setting component provides the twin deduction system with twin scene construction, scene rendering, and scene import functions.

7. The human-in-the-loop virtual-reality fusion digital twin system according to claim 3 is characterized in that: When creating a work task, the cloud control system needs to integrate the complex work system where the human is in the loop into a model, and create the work task based on the modeling results. The created work task includes the task type, execution conditions, execution time and target parameters.

8. The human-in-the-loop virtual-reality fusion digital twin system according to claim 3 is characterized in that: The cloud control system is also used to obtain and store the status information of each virtual object in the twin deduction system.

9. The human-in-the-loop virtual-reality fusion digital twin system according to claim 3 is characterized in that: The cloud control system integrates motion control, path planning, target detection and tracking, and obstacle avoidance algorithms for controlling the operation of intelligent agents. The cloud control system generates action instructions for each intelligent agent based on the work task, and sends the generated action instructions to the intelligent agents carried on the physical sandbox body to control the operation of each intelligent agent.

10. The human-in-the-loop virtual-reality fusion digital twin system according to claim 3, characterized in that: The intelligent body carried on the physical sandbox body is embedded with motion control, path planning, target detection and tracking, and obstacle avoidance algorithms. The intelligent body performs motion control independently based on the work tasks issued by the cloud control system.

Images