Virtual-reality fusion simulation test system and method for ocean vehicles

Abstract

The present invention provides a virtual-real fusion simulation test system and method for ocean vehicles, including: monitoring end: monitoring the operation of the physical end, controlling the operation mode and state of the physical end and the virtual end; the physical end: receiving the task planning instructions and operation instructions issued by the monitoring end and the motion state quantity of the virtual ocean vehicle output by the virtual end, controlling the motion of the physical ocean vehicle, collecting the internal data information of the physical ocean vehicle and uploading it to the monitoring end and the virtual end; the virtual end: providing a high-fidelity digital environment and a mirror image of the vehicle model for the virtual-real fusion simulation, promoting the operation of the physical end system in a data-driven manner, verifying the operation logic of the physical end system, and debugging the algorithm; including a digital test environment system and an Ethernet module; a multimodal data transmission link: standardizing the data format between the systems of the platform, and providing a standard communication interface and protocol to the outside. The simulation results of the present invention are closer to reality and have high agility.

Description

Technical Field

[0001] The present invention relates to the technical field of ocean vehicle simulation test, and in particular to a virtual-reality fusion simulation test system and method for ocean vehicles. Background Art

[0002] Ocean vehicles are a general term for a type of equipment, including unmanned ships, autonomous underwater vehicles, and various new vehicles. The development cost of ocean vehicles is high, and the development process of a vehicle from design drawings to prototypes takes a lot of effort from researchers. In the process of developing ocean vehicles, due to the special environment of the ocean, the cost and risk of directly conducting physical tests are too high or even unacceptable, and some test scenarios are difficult to simulate in reality. Therefore, it is an essential basic research work to build an ocean vehicle simulation system with high automated testing efficiency, realistic visual effects, convenient operation, and high confidence in results.

[0003] At present, system simulation is the main way to study, implement and debug the autonomous navigation control system of marine vehicles and their embedded algorithms. System simulation can be divided into three types according to the simulation model used: pure digital simulation, virtual-real fusion simulation and full physical simulation. In actual application, full physical simulation is difficult and cost-effective, while pure digital simulation has low credibility and incomplete verification, and both cannot meet the application needs of marine vehicle simulation. Virtual-real fusion simulation refers to a simulation system that combines physical hardware devices with computer software and digital models to build a virtual reality environment and interact with hardware devices in real time. It is characterized by "hardware in the loop"; in the marine vehicle simulation scene, virtual-real fusion simulation can be used to simulate the marine environment, marine vehicle motion behavior and sensor output. Compared with the pure digital simulation system, the virtual-real fusion simulation system connects the hardware to the simulation system, reduces the impact of the corresponding model modeling error and uncertainty, and the results are more accurate and reliable. Compared with full physical simulation, the virtual-real fusion simulation system can verify the system function in a virtual operating environment with low risk and cost, and has the advantages of high efficiency and cost-effectiveness. After investigation and analysis, it was found that in the field of ocean vehicles, most simulation systems are pure digital simulations. The existing ocean vehicle virtual-reality fusion simulation systems have an overly centralized and closed architecture, which cannot achieve agile access to ocean vehicle systems without modification and cannot meet diverse and distributed simulation needs. In addition, in the construction of the digital simulation environment on the virtual end, the visual fidelity is low, which is not conducive to intuitive inspection by test personnel. Summary of the invention

[0004] In view of the defects in the prior art, the purpose of the present invention is to provide a virtual-reality fusion simulation test system and method for ocean vehicles.

[0005] A virtual-reality fusion simulation test system for ocean vehicles provided by the present invention comprises:

[0006] Monitoring end: monitors the operation of the physical end, has the ability of human-computer interaction and the ability to issue task planning instructions and operation instructions to the physical end and virtual end, and controls the operation mode and status of the physical end and virtual end;

[0007] Physical end: receives the mission planning instructions and operation instructions issued by the monitoring end and the motion state of the virtual ocean vehicle output by the virtual end, controls the motion of the physical ocean vehicle, collects the internal data information of the physical ocean vehicle and uploads it to the monitoring end and the virtual end;

[0008] Virtual end: Provides high-fidelity digital environment and aircraft model mirror for virtual-reality fusion simulation, promotes the operation of physical end system in a data-driven manner, verifies the operation logic of physical end system, and debugs algorithms; includes digital test environment system and Ethernet module;

[0009] Multimodal data transmission link: standardizes the data format between various systems on the platform and provides standard communication interfaces and protocols to the outside world.

[0010] Preferably, in the monitoring terminal:

[0011] It includes a universal monitoring system and a movable ground station; the universal monitoring system runs on the movable ground station or runs on other computers;

[0012] The universal monitoring system has multi-channel communication capability, and is used for physical communication with the entity end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment, and for physical communication with the virtual end through the Ethernet module;

[0013] At the software level, the channel object based on the Serial and Socket modules is encapsulated. After the channel object is instantiated, it is used to communicate with the physical end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment at the physical level, and to communicate with the virtual end through the Ethernet module. The channel object can work in real time and in parallel, and receive user instructions to configure the channel parameters online.

[0014] The universal monitoring system has universal monitoring capabilities and can monitor and control various types of aircraft that exchange information based on modal data transmission links. It internally encapsulates data transmission protocol modules in S4DTP-Hex and S4DTP-Str formats to achieve decoding and encoding of the protocol, and can monitor and control various types of aircraft that exchange information based on the protocol.

[0015] Preferably, in the entity end:

[0016] Including autonomous flight control system; the autonomous flight control system is based on the MOOS framework, adopts a multi-node parallel architecture, and has a simulation debugging node built inside to collect information from other nodes inside the autonomous flight control system and synchronize it to the virtual end, and accept virtual end data injection; the autonomous flight control system has a handle remote control mode, an autonomous navigation mode and a simulation operation mode. In the simulation operation mode, a data closed-loop interactive channel between the system and the virtual end is established through the simulation debugging node to perform hardware-in-the-loop virtual-real fusion simulation. The handle remote control mode and the autonomous navigation mode accept monitoring end data, but not virtual end data. This is the actual working mode of the ocean vehicle;

[0017] The monitoring end opens the virtual end and configures the physical end to simulation mode. In this mode, the virtual end establishes a data connection with the virtual end through its simulation node via a multimodal data link; the monitoring end issues task-level instructions to the physical end, and the physical end starts running and interacts with the virtual end in real-time data. The physical end sends driving instruction data to the virtual end, and the virtual end returns the current status information of the virtual aircraft to the physical end, including position, attitude, speed and other information. The physical end receives the data sent by the virtual end through the simulation node and injects the data into the autonomous flight control system to drive the system to make the next decision and control, thereby realizing virtual-real fusion simulation;

[0018] In the remote control mode, the user uses the panoramic camera, radar images, and nautical charts as visual input, and remotely generates commands using the handle to send drive control commands to the bottom layer of the autonomous flight control system through the universal monitoring system for direct control;

[0019] In autonomous navigation mode, after the universal monitoring system plans the task, it sends the task type and task parameters to the autonomous navigation control system, which autonomously completes the path planning, controls the movement of the aircraft and maintains the attitude stability in accordance with the preset standards, autonomously senses and detects the target and completes obstacle avoidance. After completing the task, it will stand by autonomously and wait for the next instruction;

[0020] In the simulation operation mode, the autonomous flight control system establishes a closed-loop data interaction channel with the virtual end through its internal simulation debugging node, and performs virtual-real fusion simulation of hardware in the loop. In this mode, the physical end uses the interactive data generated by the virtual end to drive the operation, and conducts online non-destructive control system function and algorithm tests on the spacecraft, replacing some lake and sea function tests, so as to achieve the purpose of pre-risk control, cost reduction and efficiency improvement.

[0021] The physical end establishes physical communication with the monitoring end through the wired radio, Ethernet module, 4G module, and Beidou module, and establishes physical communication with the virtual end through the Ethernet module.

[0022] Preferably, in the virtual terminal:

[0023] The digital test environment system architecture is divided into a data acquisition layer, a model calculation layer, a twin function layer and an immersive interaction layer. Under the UE5 physical engine, the ECS design pattern is used to organize and schedule the operation of each layer of functional modules. Each layer of functional modules is implemented in a mixed way of C++ and Python. The C++ implementation module and the Python implementation module exchange information through a shared memory mechanism. It has a scene rendering module for three-dimensional high-fidelity rendering of environmental scenes and virtual ocean vehicle motion scenes, an ocean vehicle dynamics model library and an environmental dynamics model library for multi-scale physical effect simulation calculations, and a network communication module for establishing data connections with the entity end and the monitoring end.

[0024] The digital test environment system is deployed in a mobile ground station or other computers; it is physically connected to the monitoring end and the entity end through an Ethernet module.

[0025] Preferably, in the multimodal data transmission link:

[0026] The multimodal data transmission link includes S4DTP-String and S4DTP-Hex message formats, including single frame transmission and data packet transmission modes; the S4DTP-String message format is used for data encoding, decoding and transmission processes between the monitoring end and the virtual end, the physical end and the virtual end; the S4DTP-Hex message format is used for data encoding, decoding and transmission processes between the monitoring end and the physical end;

[0027] The single-frame transmission mode is used for the transmission of operation instructions, data instructions and status instructions between the monitoring end and the virtual end, the physical end and the virtual end; the data packet transmission mode is used for the transmission of task planning instructions between the monitoring end and the physical end.

[0028] A virtual-reality fusion simulation test method for ocean vehicles provided by the present invention comprises:

[0029] Step S1: The monitoring end is instructed to monitor the operation of the physical end, have the ability of human-computer interaction and the ability to issue task planning instructions and operation instructions to the physical end and the virtual end, and control the operation mode and status of the physical end and the virtual end;

[0030] Step S2: The physical end receives the mission planning instructions and operation instructions issued by the monitoring end and the motion state quantity of the virtual ocean vehicle output by the virtual end, controls the motion of the physical ocean vehicle, collects the internal data information of the physical ocean vehicle and uploads it to the monitoring end and the virtual end;

[0031] Step S3: The virtual end provides a high-fidelity digital environment and a vehicle model image for the virtual-reality fusion simulation, promotes the operation of the physical end system in a data-driven manner, verifies the operation logic of the physical end system, and debugs the algorithm; including a digital test environment system and an Ethernet module.

[0032] Preferably, in the monitoring terminal:

[0033] It includes a universal monitoring system and a movable ground station; the universal monitoring system runs on the movable ground station or runs on other computers;

[0034] The universal monitoring system has multi-channel communication capability, and is used for physical communication with the entity end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment, and for physical communication with the virtual end through the Ethernet module;

[0035] At the software level, the channel object based on the Serial and Socket modules is encapsulated. After the channel object is instantiated, it is used to communicate with the physical end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment at the physical level, and to communicate with the virtual end through the Ethernet module. The channel object can work in real time and in parallel, and receive user instructions to configure the channel parameters online.

[0036] The universal monitoring system has universal monitoring capabilities and can monitor and control various types of aircraft that exchange information based on modal data transmission links. It internally encapsulates data transmission protocol modules in S4DTP-Hex and S4DTP-Str formats to achieve decoding and encoding of the protocol, and can monitor and control various types of aircraft that exchange information based on the protocol.

[0037] Preferably, in the entity end:

[0038] Including autonomous flight control system; the autonomous flight control system is based on the MOOS framework, adopts a multi-node parallel architecture, and has a simulation debugging node built inside to collect information from other nodes inside the autonomous flight control system and synchronize it to the virtual end, and accept virtual end data injection; the autonomous flight control system has a handle remote control mode, an autonomous navigation mode and a simulation operation mode. In the simulation operation mode, a data closed-loop interactive channel between the system and the virtual end is established through the simulation debugging node to perform hardware-in-the-loop virtual-real fusion simulation. The handle remote control mode and the autonomous navigation mode accept monitoring end data, but not virtual end data. This is the actual working mode of the ocean vehicle;

[0039] The monitoring end opens the virtual end and configures the physical end to simulation mode. In this mode, the virtual end establishes a data connection with the virtual end through its simulation node via a multimodal data link; the monitoring end issues task-level instructions to the physical end, and the physical end starts running and interacts with the virtual end in real-time data. The physical end sends driving instruction data to the virtual end, and the virtual end returns the current status information of the virtual aircraft to the physical end, including position, attitude, speed and other information. The physical end receives the data sent by the virtual end through the simulation node and injects the data into the autonomous flight control system to drive the system to make the next decision and control, thereby realizing virtual-real fusion simulation;

[0040] In the remote control mode, the user uses the panoramic camera, radar images, and nautical charts as visual input, and remotely generates commands using the handle to send drive control commands to the bottom layer of the autonomous flight control system through the universal monitoring system for direct control;

[0041] In autonomous navigation mode, after the universal monitoring system plans the task, it sends the task type and task parameters to the autonomous navigation control system, which autonomously completes the path planning, controls the movement of the aircraft and maintains the attitude stability in accordance with the preset standards, autonomously senses and detects the target and completes obstacle avoidance. After completing the task, it will stand by autonomously and wait for the next instruction;

[0042] In the simulation operation mode, the autonomous flight control system establishes a closed-loop data interaction channel with the virtual end through its internal simulation debugging node, and performs virtual-real fusion simulation of hardware in the loop. In this mode, the physical end uses the interactive data generated by the virtual end to drive the operation, and conducts online non-destructive control system function and algorithm tests on the spacecraft, replacing some lake and sea function tests, so as to achieve the purpose of pre-risk control, cost reduction and efficiency improvement.

[0043] The physical end establishes physical communication with the monitoring end through the wired radio, Ethernet module, 4G module, and Beidou module, and establishes physical communication with the virtual end through the Ethernet module.

[0044] Preferably, in the virtual terminal:

[0045] The digital test environment system architecture is divided into a data acquisition layer, a model calculation layer, a twin function layer and an immersive interaction layer. Under the UE5 physical engine, the ECS design pattern is used to organize and schedule the operation of each layer of functional modules. Each layer of functional modules is implemented in a mixed way of C++ and Python. The C++ implementation module and the Python implementation module exchange information through a shared memory mechanism. It has a scene rendering module for three-dimensional high-fidelity rendering of environmental scenes and virtual ocean vehicle motion scenes, an ocean vehicle dynamics model library and an environmental dynamics model library for multi-scale physical effect simulation calculations, and a network communication module for establishing data connections with the entity end and the monitoring end.

[0046] The digital test environment system is deployed in a mobile ground station or other computers; it is physically connected to the monitoring end and the entity end through an Ethernet module.

[0047] Preferably, the multimodal data transmission link includes S4DTP-String and S4DTP-Hex message formats, including single frame transmission and data packet transmission modes; the S4DTP-String message format is used for data encoding, decoding and transmission processes between the monitoring end and the virtual end, the entity end and the virtual end; the S4DTP-Hex message format is used for data encoding, decoding and transmission processes between the monitoring end and the entity end;

[0048] The single-frame transmission mode is used for the transmission of operation instructions, data instructions and status instructions between the monitoring end and the virtual end, the physical end and the virtual end; the data packet transmission mode is used for the transmission of task planning instructions between the monitoring end and the physical end.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. The present invention can perform virtual-reality fusion simulation without destroying the original ocean vehicle system architecture, and the simulation result is closer to reality and has high agility. After the simulation is completed, the ocean vehicle system can be switched back to the actual working mode by changing the mode, and the test optimization iteration can be carried out quickly, which greatly reduces the test cycle and the investment in manpower and material resources.

[0051] 2. The virtual-reality fusion simulation method provided by the present invention fully mobilizes the digital resources and physical resources in the system, so that the development process that was originally carried out linearly under the design-modeling-simulation-verification mode can be carried out in parallel in the simulation and verification stages, effectively improving development efficiency and reducing potential risks.

[0052] 3. By using the virtual-reality fusion simulation platform and method proposed in the present invention, it is only necessary to construct simulation models of different types of ocean vehicles or adjust the parameters of the original models to match different types of ocean vehicles for simulation verification. It has strong versatility and high scalability.

[0053] 4. The virtual terminal provided by the present invention can provide high-fidelity real-time three-dimensional virtual vision, enhancing the overall control and intuitive understanding of the entire mission cycle of the ocean vehicle simulation process.

[0054] 5. Through the virtual-reality fusion simulation platform of the present invention, simulation training for physical operators of ocean vehicles can be further carried out, which greatly reduces training costs and ensures safety; it can also provide a training environment for the artificial intelligence algorithm embedded in the aviation system of ocean vehicles and obtain a large amount of generalized simulation data, which is conducive to the optimization learning of the model and improves efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0056] Figure 1 A schematic diagram of the structure of a virtual-reality fusion simulation platform for ocean vehicles provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of a multi-modal data transmission link message format provided for an embodiment of the invention;

[0058] Figure 3 A schematic diagram of a flow chart of a virtual-reality fusion simulation method for ocean vehicles provided by an embodiment of the present invention;

[0059] Figure 4 An information processing circuit of a virtual-reality fusion simulation platform for ocean vehicles provided by an embodiment of the present invention;

[0060] Figure 5 The digital test environment system architecture provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0062] Embodiment 1:

[0063] A virtual-reality fusion simulation test method for ocean vehicles provided by the present invention comprises:

[0064] Step S1: The monitoring end is instructed to monitor the operation of the physical end, have the ability of human-computer interaction and the ability to issue task planning instructions and operation instructions to the physical end and the virtual end, and control the operation mode and status of the physical end and the virtual end;

[0065] Specifically, in the monitoring terminal:

[0066] It includes a universal monitoring system and a movable ground station; the universal monitoring system runs on the movable ground station or runs on other computers;

[0067] The universal monitoring system has multi-channel communication capability, and is used for physical communication with the entity end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment, and for physical communication with the virtual end through the Ethernet module;

[0068] At the software level, the channel object based on the Serial and Socket modules is encapsulated. After the channel object is instantiated, it is used to communicate with the physical end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment at the physical level, and to communicate with the virtual end through the Ethernet module. The channel object can work in real time and in parallel, and receive user instructions to configure the channel parameters online.

[0069] The universal monitoring system has universal monitoring capabilities and can monitor and control various types of aircraft that exchange information based on modal data transmission links. It internally encapsulates data transmission protocol modules in S4DTP-Hex and S4DTP-Str formats to achieve decoding and encoding of the protocol, and can monitor and control various types of aircraft that exchange information based on the protocol.

[0070] Step S2: The physical end receives the mission planning instructions and operation instructions issued by the monitoring end and the motion state quantity of the virtual ocean vehicle output by the virtual end, controls the motion of the physical ocean vehicle, collects the internal data information of the physical ocean vehicle and uploads it to the monitoring end and the virtual end;

[0071] Specifically, in the entity side:

[0072] Including autonomous flight control system; the autonomous flight control system is based on the MOOS framework, adopts a multi-node parallel architecture, and has a simulation debugging node built inside to collect information from other nodes inside the autonomous flight control system and synchronize it to the virtual end, and accept virtual end data injection; the autonomous flight control system has a handle remote control mode, an autonomous navigation mode and a simulation operation mode. In the simulation operation mode, a data closed-loop interactive channel between the system and the virtual end is established through the simulation debugging node to perform hardware-in-the-loop virtual-real fusion simulation. The handle remote control mode and the autonomous navigation mode accept monitoring end data, but not virtual end data. This is the actual working mode of the ocean vehicle;

[0073] The monitoring end opens the virtual end and configures the physical end to simulation mode. In this mode, the virtual end establishes a data connection with the virtual end through its simulation node via a multimodal data link; the monitoring end issues task-level instructions to the physical end, and the physical end starts running and interacts with the virtual end in real-time data. The physical end sends driving instruction data to the virtual end, and the virtual end returns the current status information of the virtual aircraft to the physical end, including position, attitude, speed and other information. The physical end receives the data sent by the virtual end through the simulation node and injects the data into the autonomous flight control system to drive the system to make the next decision and control, thereby realizing virtual-real fusion simulation;

[0074] In the remote control mode, the user uses the panoramic camera, radar images, and nautical charts as visual input, and remotely generates commands using the handle to send drive control commands to the bottom layer of the autonomous flight control system through the universal monitoring system for direct control;

[0075] In autonomous navigation mode, after the universal monitoring system plans the task, it sends the task type and task parameters to the autonomous navigation control system, which autonomously completes the path planning, controls the movement of the aircraft and maintains the attitude stability in accordance with the preset standards, autonomously senses and detects the target and completes obstacle avoidance. After completing the task, it will stand by autonomously and wait for the next instruction;

[0076] In the simulation operation mode, the autonomous flight control system establishes a closed-loop data interaction channel with the virtual end through its internal simulation debugging node, and performs virtual-real fusion simulation of hardware in the loop. In this mode, the physical end uses the interactive data generated by the virtual end to drive the operation, and conducts online non-destructive control system function and algorithm tests on the spacecraft, replacing some lake and sea function tests, so as to achieve the purpose of pre-risk control, cost reduction and efficiency improvement.

[0077] The physical end establishes physical communication with the monitoring end through the wired radio, Ethernet module, 4G module, and Beidou module, and establishes physical communication with the virtual end through the Ethernet module.

[0078] Step S3: The virtual end provides a high-fidelity digital environment and a vehicle model image for the virtual-reality fusion simulation, promotes the operation of the physical end system in a data-driven manner, verifies the operation logic of the physical end system, and debugs the algorithm; including a digital test environment system and an Ethernet module.

[0079] Specifically, in the virtual terminal:

[0080] The digital test environment system architecture is divided into a data acquisition layer, a model calculation layer, a twin function layer and an immersive interaction layer. Under the UE5 physical engine, the ECS design pattern is used to organize and schedule the operation of each layer of functional modules. Each layer of functional modules is implemented in a mixed way of C++ and Python. The C++ implementation module and the Python implementation module exchange information through a shared memory mechanism. It has a scene rendering module for three-dimensional high-fidelity rendering of environmental scenes and virtual ocean vehicle motion scenes, an ocean vehicle dynamics model library and an environmental dynamics model library for multi-scale physical effect simulation calculations, and a network communication module for establishing data connections with the entity end and the monitoring end.

[0081] The digital test environment system is deployed in a mobile ground station or other computers; it is physically connected to the monitoring end and the entity end through an Ethernet module.

[0082] Specifically, the multimodal data transmission link includes S4DTP-String and S4DTP-Hex message formats, including single-frame transmission and data packet transmission modes; the S4DTP-String message format is used for data encoding, decoding and transmission processes between the monitoring end and the virtual end, the physical end and the virtual end; the S4DTP-Hex message format is used for data encoding, decoding and transmission processes between the monitoring end and the physical end;

[0083] The single-frame transmission mode is used for the transmission of operation instructions, data instructions and status instructions between the monitoring end and the virtual end, the physical end and the virtual end; the data packet transmission mode is used for the transmission of task planning instructions between the monitoring end and the physical end.

[0084] Embodiment 2:

[0085] Embodiment 2 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0086] The present invention also provides a virtual-reality fusion simulation test system for ocean vehicles. The virtual-reality fusion simulation test system for ocean vehicles can be realized by executing the process steps of the virtual-reality fusion simulation test method for ocean vehicles, that is, technical personnel in this field can understand the virtual-reality fusion simulation test method for ocean vehicles as a preferred implementation of the virtual-reality fusion simulation test system for ocean vehicles.

[0087] According to the present invention, a virtual-reality fusion simulation test system for ocean vehicles is provided. Figure 1-Figure 5 As shown, including:

[0088] Monitoring end: monitors the operation of the physical end, has the ability of human-computer interaction and the ability to issue task planning instructions and operation instructions to the physical end and virtual end, and controls the operation mode and status of the physical end and virtual end;

[0089] Specifically, in the monitoring terminal:

[0090] It includes a universal monitoring system and a movable ground station; the universal monitoring system runs on the movable ground station or runs on other computers;

[0091] The universal monitoring system has multi-channel communication capability, and is used for physical communication with the entity end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment, and for physical communication with the virtual end through the Ethernet module;

[0092] At the software level, the channel object based on the Serial and Socket modules is encapsulated. After the channel object is instantiated, it is used to communicate with the physical end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment at the physical level, and to communicate with the virtual end through the Ethernet module. The channel object can work in real time and in parallel, and receive user instructions to configure the channel parameters online.

[0093] The universal monitoring system has universal monitoring capabilities and can monitor and control various types of aircraft that exchange information based on modal data transmission links. It internally encapsulates data transmission protocol modules in S4DTP-Hex and S4DTP-Str formats to achieve decoding and encoding of the protocol, and can monitor and control various types of aircraft that exchange information based on the protocol.

[0094] Physical end: receives the mission planning instructions and operation instructions issued by the monitoring end and the motion state of the virtual ocean vehicle output by the virtual end, controls the motion of the physical ocean vehicle, collects the internal data information of the physical ocean vehicle and uploads it to the monitoring end and the virtual end;

[0095] Specifically, in the entity side:

[0096] Including autonomous flight control system; the autonomous flight control system is based on the MOOS framework, adopts a multi-node parallel architecture, and has a simulation debugging node built inside to collect information from other nodes inside the autonomous flight control system and synchronize it to the virtual end, and accept virtual end data injection; the autonomous flight control system has a handle remote control mode, an autonomous navigation mode and a simulation operation mode. In the simulation operation mode, a data closed-loop interactive channel between the system and the virtual end is established through the simulation debugging node to perform hardware-in-the-loop virtual-real fusion simulation. The handle remote control mode and the autonomous navigation mode accept monitoring end data, but not virtual end data. This is the actual working mode of the ocean vehicle;

[0097] The monitoring end opens the virtual end and configures the physical end to simulation mode. In this mode, the virtual end establishes a data connection with the virtual end through its simulation node via a multimodal data link; the monitoring end issues task-level instructions to the physical end, and the physical end starts running and interacts with the virtual end in real-time data. The physical end sends driving instruction data to the virtual end, and the virtual end returns the current status information of the virtual aircraft to the physical end, including position, attitude, speed and other information. The physical end receives the data sent by the virtual end through the simulation node and injects the data into the autonomous flight control system to drive the system to make the next decision and control, thereby realizing virtual-real fusion simulation;

[0098] In the remote control mode, the user uses the panoramic camera, radar images, and nautical charts as visual input, and remotely generates commands using the handle to send drive control commands to the bottom layer of the autonomous flight control system through the universal monitoring system for direct control;

[0099] In autonomous navigation mode, after the universal monitoring system plans the task, it sends the task type and task parameters to the autonomous navigation control system, which autonomously completes the path planning, controls the movement of the aircraft and maintains the attitude stability in accordance with the preset standards, autonomously senses and detects the target and completes obstacle avoidance. After completing the task, it will stand by autonomously and wait for the next instruction;

[0100] In the simulation operation mode, the autonomous flight control system establishes a closed-loop data interaction channel with the virtual end through its internal simulation debugging node, and performs virtual-real fusion simulation of hardware in the loop. In this mode, the physical end uses the interactive data generated by the virtual end to drive the operation, and conducts online non-destructive control system function and algorithm tests on the spacecraft, replacing some lake and sea function tests, so as to achieve the purpose of pre-risk control, cost reduction and efficiency improvement.

[0101] The physical end establishes physical communication with the monitoring end through the wired radio, Ethernet module, 4G module, and Beidou module, and establishes physical communication with the virtual end through the Ethernet module.

[0102] Virtual end: Provides high-fidelity digital environment and aircraft model mirror for virtual-reality fusion simulation, promotes the operation of physical end system in a data-driven manner, verifies the operation logic of physical end system, and debugs algorithms; includes digital test environment system and Ethernet module;

[0103] Specifically, in the virtual terminal:

[0104] The digital test environment system architecture is divided into a data acquisition layer, a model calculation layer, a twin function layer and an immersive interaction layer. Under the UE5 physical engine, the ECS design pattern is used to organize and schedule the operation of each layer of functional modules. Each layer of functional modules is implemented in a mixed way of C++ and Python. The C++ implementation module and the Python implementation module exchange information through a shared memory mechanism. It has a scene rendering module for three-dimensional high-fidelity rendering of environmental scenes and virtual ocean vehicle motion scenes, an ocean vehicle dynamics model library and an environmental dynamics model library for multi-scale physical effect simulation calculations, and a network communication module for establishing data connections with the entity end and the monitoring end.

[0105] The digital test environment system is deployed in a mobile ground station or other computers; it is physically connected to the monitoring end and the entity end through an Ethernet module.

[0106] Multimodal data transmission link: standardizes the data format between various systems on the platform and provides standard communication interfaces and protocols to the outside world.

[0107] Specifically, in the multimodal data transmission link:

[0108] The multimodal data transmission link includes S4DTP-String and S4DTP-Hex message formats, including single frame transmission and data packet transmission modes; the S4DTP-String message format is used for data encoding, decoding and transmission processes between the monitoring end and the virtual end, the physical end and the virtual end; the S4DTP-Hex message format is used for data encoding, decoding and transmission processes between the monitoring end and the physical end;

[0109] The single-frame transmission mode is used for the transmission of operation instructions, data instructions and status instructions between the monitoring end and the virtual end, the physical end and the virtual end; the data packet transmission mode is used for the transmission of task planning instructions between the monitoring end and the physical end.

[0110] Embodiment 3:

[0111] Embodiment 3 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0112] The purpose of the present invention is to provide a virtual-real fusion ocean vehicle simulation test platform and method, which can be used to debug and verify the software, hardware and embedded algorithms of the ocean vehicle control system in a low-cost, agile, efficient and universal manner.

[0113] On the one hand, the present invention proposes a virtual-real fusion ocean vehicle simulation test platform, which adopts a universal / open design architecture, decouples and modularizes the platform components, so that each system has the ability to operate completely independently. The platform at least includes a monitoring end, a physical end, a virtual end, and a multimodal data transmission link. The monitoring end, the physical end, and the virtual end perform bidirectional data exchange through the multimodal data transmission link.

[0114] The monitoring end is used to monitor the operation of the physical end, and has the ability of human-computer interaction and the ability to issue task planning instructions and operation instructions to the physical end and the virtual end, and can control the operation mode and status of the physical end and the virtual end. Furthermore, the monitoring end includes a universal monitoring system and a movable ground station. The universal monitoring system can run on a movable ground station or on other computers. The universal monitoring system has multi-channel communication capabilities and can be used for physical communication with the physical end through communication equipment such as radio stations, Ethernet modules, 4G modules, Beidou modules, and physical communication with the virtual end through Ethernet modules. The universal monitoring system has universal monitoring capabilities and can monitor and control various types of aircraft that interact with information based on modal data transmission links.

[0115] The universal monitoring system has multi-channel communication capabilities and encapsulates universal channel objects based on multiple modules such as Serial and Socket at the software level. After instantiation, these channel objects can be used for physical communication with the entity end through the physical layer radio station, Ethernet module, 4G module, and Beidou module communication equipment, and for physical communication with the virtual end through the Ethernet module; in addition, these channel objects can work in real time and in parallel, and can receive user instructions for online configuration of channel parameters.

[0116] The universal monitoring system has universal monitoring capabilities, and internally encapsulates data transmission protocol modules in S4DTP-Hex and S4DTP-Str formats, which can decode and encode the protocol and monitor and control various types of aircraft that exchange information based on the protocol.

[0117] The entity end is used to receive the task planning instructions and operation instructions issued by the monitoring end and the motion state quantity of the virtual ocean vehicle output by the virtual end, control the motion of the entity ocean vehicle, collect the internal data information of the entity ocean vehicle and upload it to the monitoring end and the virtual end. The entity end includes an autonomous navigation control system. The autonomous navigation control system is based on the MOOS framework and adopts a multi-node parallel architecture. A simulation debugging node is established inside to realize the collection of other node information inside the autonomous navigation control system and synchronize it to the virtual end, and accept virtual end data injection; the autonomous navigation control system has a handle remote control mode, an autonomous navigation mode and a simulation operation mode. In the simulation operation mode, a data closed-loop interactive channel between the system and the virtual end can be established through the simulation debugging node to perform a "hardware in the loop" virtual-real fusion simulation. The handle remote control mode and the autonomous navigation mode only accept the monitoring end data, not the virtual end data. This is the actual working mode of the ocean vehicle. The entity end can establish physical communication with the monitoring end through a radio, an Ethernet module, a 4G module, and a Beidou module, and can establish physical communication with the virtual end through an Ethernet module.

[0118] In the handle remote control mode, the user uses panoramic cameras, radar images, nautical charts, etc. as visual input, and remotely generates commands using the handle to send drive control commands to the bottom layer of the autonomous navigation control system through the universal monitoring system for direct control, which is flexible and agile; in the autonomous navigation mode, after the universal monitoring system plans the task, it sends the task type and task parameters to the autonomous navigation control system, which autonomously completes the path planning, controls the movement of the aircraft and maintains a stable attitude, autonomously senses and detects targets and completes obstacle avoidance, and waits for the next instruction after completing the task. In the simulation operation mode, the autonomous navigation control system establishes a data closed-loop interaction channel with the virtual end through its internal simulation debugging node, and performs virtual-real fusion simulation of hardware in the loop. In this mode, the physical end can use the interactive data generated by the virtual end to drive the operation, and can carry out online non-destructive control system function and algorithm testing on the aircraft, thereby replacing some lake and sea function tests, and achieving the purpose of risk pre-control, cost reduction and efficiency improvement.

[0119] The virtual end is used to provide a high-fidelity digital environment and a mirror image of the aircraft model for the virtual-real fusion simulation, and to promote the operation of the physical end system in a data-driven manner, so as to verify the operation logic of the physical end system, debug the algorithm, etc. The virtual end includes a digital test environment system and an Ethernet module. The digital test environment system architecture is divided into a data acquisition layer, a model calculation layer, a twin function layer, and an immersive interaction layer. Under the UE5 physical engine, the ECS (Entity-Component-System) design pattern is used to organize and schedule the specific operation of each layer of functional modules. Each layer of functional modules is implemented in a mixed way of C++ and Python. The C++ implementation module and the Python implementation module interact with each other through a shared memory mechanism; it has a scene rendering module for three-dimensional high-fidelity rendering of environmental scenes and virtual ocean aircraft motion scenes, an ocean aircraft dynamics model library and an environmental dynamics model library for multi-scale physical effect simulation calculations, and a network communication module for establishing data connections with the physical end and the monitoring end. The digital test environment system can be deployed in a mobile ground station or other computer; it is physically connected to the monitoring end and the physical end through an Ethernet module.

[0120] The multimodal data transmission link is used to standardize the data format between the systems of the platform and provide standard communication interfaces and protocols to the outside world. The multimodal data transmission link includes S4DTP-String and S4DTP-Hex message formats, including single-frame transmission and data packet transmission modes. The S4DTP-String message format is used for the data encoding, decoding and transmission process between the monitoring end and the virtual end, and the entity end and the virtual end; the S4DTP-Hex message format is used for the data encoding, decoding and transmission process between the monitoring end and the entity end. The single-frame transmission mode is used for the transmission of operation instructions, data instructions and status instructions between the monitoring end and the virtual end, and the entity end and the virtual end; the data packet transmission mode is used for the transmission of task planning instructions between the monitoring end and the entity end.

[0121] On the other hand, the present invention also proposes a virtual-real fusion ocean vehicle simulation test method, which is based on the ocean vehicle simulation test platform as described above and specifically includes the following steps:

[0122] The monitoring end accepts user operations and sets the entity end to the simulation operation mode through the multimodal data transmission link; the virtual end selects the ocean vehicle dynamics model corresponding to the entity end, sets the environmental dynamics model parameters, and selects to load the environmental scene. The monitoring end issues actual task planning instructions to the entity end through the multimodal data transmission link. After the entity end receives the actual task planning instructions, it autonomously calculates the decision amount, control amount and driving amount, and synchronizes them to the virtual end through the simulation debugging node through the multimodal data transmission link. After the ocean vehicle dynamics model of the virtual end receives the decision amount, control amount and driving amount of the entity end, it simulates and calculates the motion state of the virtual ocean vehicle and transmits it back to the entity end. The scene rendering module of the virtual end renders and presents the three-dimensional motion state and functional behavior of the virtual ocean vehicle in real time. The entity end accepts the motion state of the virtual ocean vehicle from the virtual end, replaces the motion state of the physical ocean vehicle collected by the physical sensor of the entity end, and drives the entity end to perform the next step of calculation. The monitoring end, the entity end and the virtual end form a closed loop of simulation data until the task planning instruction is completed.

[0123] Innovation:

[0124] A distributed ocean vehicle virtual / real fusion simulation system architecture is proposed, which can perform virtual / real fusion simulation without changing the original system architecture of the vehicle entity. The simulation system has functions such as data acquisition, transmission and processing, simulation modeling, data twins, service applications, etc., and supports plug-in replacement of environmental mirrors and digital prototypes, which effectively reduces the overall system delay and coupling degree, and improves the reliability, real-time and compatibility of the system.

[0125] A method for building a digital test environment system based on the UE5 virtual engine and the ECS mode is proposed, which enables the entire simulation system to have the ability to efficiently process and render real-time images of three-dimensional dynamic ocean environments and simulate the physical effects of twin ocean vehicles. It provides good three-dimensional visual display, physical simulation capabilities and dynamic interactive interfaces, and improves the immersive application experience and high real-time dynamic interactive effects of the entire simulation system.

[0126] Embodiment 4:

[0127] Embodiment 4 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0128] Figure 1 The schematic diagram of the structure of a virtual-reality linkage simulation test system for drones provided by the present embodiment is shown, including: a monitoring terminal, a physical terminal, a virtual terminal and a multimodal data transmission link. The monitoring terminal, the physical terminal and the virtual terminal are respectively connected in a bidirectional communication manner through the multimodal data transmission link.

[0129] The monitoring end is used to monitor the operation of the entity end, has the ability of human-computer interaction and the ability to issue task planning instructions and operation instructions to the entity end and the virtual end, and can control the operation mode and status of the entity end and the virtual end. The universal monitoring system of the monitoring end is designed based on the MVC mode, and has aircraft status monitoring, path planning, action setting, parameter configuration, emergency operation, link management, load operation and system management modules. The development tools used are not limited to QT, Python, Folium, etc., and run on the Windows operating system; the universal monitoring system establishes a human-computer interaction interface, which can accept user operations and present dynamic data and pictures to the user. The movable ground station is internally integrated with an industrial computer for running the universal monitoring system, an embedded processor for collecting the internal environmental data of the ground station and controlling the power status of the equipment, and a wireless radio, Ethernet module, 4G module, Beidou module and other communication equipment for physical communication with the entity end and the virtual end, and physical communication with the virtual end through the Ethernet module. The universal monitoring system has multi-channel communication capabilities, and can be used for physical communication with the entity end through the mobile ground station's radio station, Ethernet module, 4G module, Beidou module and other communication equipment, and can be used for physical communication with the virtual end through the mobile ground station's Ethernet module. The universal monitoring system has universal monitoring capabilities, and can monitor and control various types of aircraft that interact with information based on modal data transmission links by modifying the addresses of monitored objects.

[0130] The entity end is used to receive the task planning instructions and operation instructions issued by the monitoring end and the motion state of the virtual ocean vehicle output by the virtual end, control the motion of the physical ocean vehicle, collect the internal data information of the physical ocean vehicle and upload it to the monitoring end and the virtual end. The entity end includes an autonomous flight control system, which may include a marine vehicle drive mechanism, energy equipment, sensor equipment, etc. The autonomous flight control system is designed based on the MOOS framework and adopts a multi-node parallel mode. Each functional node interacts with information through the MOOSDB subscription / publishing mechanism. A simulation debugging node is established inside to realize the collection of other node information inside the autonomous flight control system and synchronize it to the virtual end, and accept virtual end data injection. The autonomous flight control system runs in the Linux operating system and is installed with the MOOS application framework. The autonomous flight control system has handle remote control mode, autonomous navigation mode and simulation operation mode. In the simulation operation mode, a data closed-loop interaction channel between the system and the virtual end can be established through the simulation debugging node to perform "hardware in the loop" virtual-reality fusion simulation. The handle remote control mode and autonomous navigation mode only accept monitoring end data, not virtual end data. This is the actual working mode of the ocean vehicle.

[0131] The virtual end is used to provide a high-fidelity digital environment and a mirror image of the aircraft model for virtual-real fusion simulation, and to promote the operation of the physical end system in a data-driven manner, thereby verifying the operation logic of the physical end system, debugging algorithms, etc. The virtual end includes a digital test environment system and an Ethernet module. The digital test environment system has a scene rendering module for three-dimensional high-fidelity rendering of environmental scenes and virtual ocean aircraft motion scenes, an ocean aircraft dynamics model library and an environmental dynamics model library for multi-scale physical effect simulation calculations, and a network communication module for establishing data connections with the physical end and the monitoring end. The Ethernet module is used to establish a communication physical connection with the physical end and the monitoring end. The digital test environment system can be deployed in a mobile ground station or other computer; it is physically connected to the monitoring end and the physical end through the Ethernet module. The digital test environment system is based on the ECS design mode, developed using the Unreal Engine, and runs in the Windows operating system. In addition to building an ocean environment, a large number of high-precision three-dimensional models must be built in the digital test environment system. The use of traditional rendering methods requires a trade-off between performance and picture detail performance; in order to pursue a high-quality user experience, this embodiment uses the latest Nanite technology for rendering.

[0132] The multimodal data transmission link is used to standardize the interactive data format between the universal monitoring system, autonomous flight control system, and digital test environment system in the platform, and provides standard communication interfaces and protocols for the platform's external systems so that they can be connected to the platform. The multimodal data transmission link includes S4DTP-String and S4DTP-Hex message formats, including single-frame transmission and data packet transmission modes. The S4DTP-String message format is used for the data encoding and decoding and transmission process between the monitoring end and the virtual end, and the entity end and the virtual end; the S4DTP-Hex message format is used for the data encoding and decoding and transmission process between the monitoring end and the entity end. The single-frame transmission mode is used for the transmission of operation instructions, data instructions, and status instructions between the monitoring end and the virtual end, and the entity end and the virtual end; the data packet transmission mode is used for the transmission of task planning instructions between the monitoring end and the entity end.

[0133] Various information (instruction stream, state stream, data stream) generated by the physical end and the virtual end forms a closed loop through the standard data link in the entire distributed virtual / real fusion simulation system and performs real-time, efficient and reliable closed-loop processing. The closed-loop information processing process is as follows: Figure 4 As shown in the figure, each subsystem exchanges data and information between services based on network sockets, and the end-to-end uses a high-level interface in the form of command / response for system access. The universal ocean vehicle monitoring system obtains user input through event interaction as the simulation task entry of the virtual / real fusion simulation system; during the operation of the simulation system, the monitoring system, as the overall data integration and management center, monitors the operation of the entire simulation system while assuming the functions of background data collection, storage, management, processing and analysis, so that users can grasp the simulation and twin data in a timely manner.

[0134] Figure 2 A schematic diagram of the multimodal data transmission link message format provided in this embodiment is shown. The S4DTP-Hex message format consists of nine parts: frame header, frame length, frame address, transmission mode, frame category, data block, timestamp, check bit and frame tail. The shortest message length under the S4DTP-Hex message format is 16 bytes, and the maximum length does not exceed 255 bytes; when the amount of data to be transmitted exceeds the capacity limit of a single message, it can be cut into multiple messages and sent in sequence according to the frame number, which greatly reduces the pressure on the data transmission link. The S4DTP-String message format consists of six parts: frame header, function code, data area size, data area, timestamp, and frame tail. Comma separators are used to separate the functional area identifiers.

[0135] In the embodiment of the present invention, in the virtual-reality fusion simulation test platform for ocean vehicles, virtual refers to the virtual end, and a virtual ocean vehicle is constructed through an ocean vehicle dynamics model and a scene rendering module; real refers to the physical end and the monitoring end. The physical end can be deployed in a variety of ocean vehicles including unmanned ships, and the monitoring end can monitor and control a variety of ocean vehicles including unmanned ships.

[0136] The present invention divides the architecture of the virtual end digital test environment system into four functional layers, from the bottom to the top: data acquisition layer, model calculation layer, twin function layer and immersion interaction layer. The implementation of each layer is based on the previous layers, which is a further enrichment and expansion of the functions of the previous layers. The specific system architecture is as follows: Figure 5 shown.

[0137] The data acquisition layer is the foundation of the entire digital test environment architecture and supports the operation of the entire upper system. It is mainly composed of three parts: data transmission service, data cleaning and management, and data storage. The data transmission service module of this layer is directly connected to the distributed network communication module to obtain accurate, real-time, and sufficient data source support from the ocean vehicle entity; the data storage module adopts a high-efficiency storage structure and data retrieval structure to provide important guarantees for the storage and rapid extraction of massive historical operation data; the data cleaning and management module is responsible for real-time detection of access data to prevent expired data from causing system operation jitter. The model calculation layer is the background core part of the entire architecture, providing multi-perspective, multi-scale system models and evaluation results for the twin function layer. After obtaining the data provided by the data acquisition layer, this layer uses data-driven methods and methods based on mathematical analytical models to model the vehicle, ocean environment, actuators, and sensors at multi-physics and multi-scale levels, so as to obtain the current behavior based on the physical effect calculation and coordinate mapping that conforms to the real rules on the basis of the state of the simulation object and the ocean environment at the previous moment. The twin function layer in the data twin model mainly provides corresponding services for the actual system design, production, use and maintenance. This layer can be customized and expanded according to actual needs. Based on the powerful information interface provided by the model calculation layer, the twin function layer can meet multiple performance indicators such as high reliability, high accuracy, high real-time performance and intelligent decision-making assistance. In this project, the twin function layer provides corresponding twin services based on the data generated by the model calculation layer, mainly including 3D animation demonstration, virtual / real fusion simulation, state prediction and intelligent algorithm training. The main purpose of the immersive interaction layer is to provide users with a good human-computer interaction environment, so that users can get an immersive technical experience, quickly understand and master the characteristics and functions of complex systems, and can easily access the information provided by the twin function layer to obtain information support for analysis and decision-making. In this system, this layer can provide users with UI interaction functions, support real-time adjustment of 3D perspective, and provide visual effects such as virtual ocean vision, vehicle motion vision, and meteorological and lighting simulation vision.

[0138] Figure 3A flow chart of a virtual-reality fusion simulation method for ocean vehicles provided in this embodiment is shown. The monitoring end sets the entity end to a simulation operation mode: the monitoring end accepts user operations and sets the entity end to a simulation operation mode through the multimodal data transmission link. The virtual end selects a model and a view and sets parameters: the virtual end selects the ocean vehicle dynamics model corresponding to the entity end, sets the environmental dynamics model parameters, and selects to load the environmental view. The monitoring end issues instructions to the entity end: the monitoring end accepts user task setting operations and issues actual task planning instructions to the entity end through a multimodal data transmission link. The entity end runs autonomously and interacts with the virtual end data: after the entity end receives the actual task planning instructions, it autonomously runs and calculates the decision amount, control amount and drive amount, and synchronizes them to the virtual end through the simulation debugging node via the multimodal data transmission link. The virtual end simulates and runs, renders the environment, and transmits data back: after the ocean vehicle dynamics model of the virtual end receives the decision-making quantity, control quantity, and driving quantity of the physical end, it simulates and calculates the motion state quantity of the virtual ocean vehicle and transmits it back to the physical end. The virtual end scene rendering module renders and presents the three-dimensional motion state and functional behavior of the virtual ocean vehicle in real time. The physical end injects the virtual end data: the physical end receives the motion state quantity of the virtual ocean vehicle from the virtual end, replaces the motion state quantity of the physical ocean vehicle collected by the physical sensor of the physical end, and injects it into each functional node, driving the physical end to perform the next step of calculation. The data flow is closed until the task is completed: the monitoring end, the physical end, and the virtual end form a simulation data closed loop until the task planning instruction is completed.

[0139] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0140] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A virtual-reality fusion simulation test system for ocean vehicles, characterized in that: include: Monitoring end: monitors the operation of the physical end, has the ability of human-computer interaction and the ability to issue task planning instructions and operation instructions to the physical end and virtual end, and controls the operation mode and status of the physical end and virtual end; Physical end: receives the mission planning instructions and operation instructions issued by the monitoring end and the motion state of the virtual ocean vehicle output by the virtual end, controls the motion of the physical ocean vehicle, collects the internal data information of the physical ocean vehicle and uploads it to the monitoring end and the virtual end; Virtual end: Provides high-fidelity digital environment and aircraft model mirror for virtual-reality fusion simulation, promotes the operation of physical end system in a data-driven manner, verifies the operation logic of physical end system, and debugs algorithms; includes digital test environment system and Ethernet module; Multimodal data transmission link: standardizes the data format between various systems on the platform and provides standard communication interfaces and protocols to the outside world.

2. The virtual-reality fusion simulation test system for ocean-going vehicles according to claim 1 is characterized in that: In the monitoring terminal: It includes a universal monitoring system and a movable ground station; the universal monitoring system runs on the movable ground station or runs on other computers; The universal monitoring system has multi-channel communication capability, and is used for physical communication with the entity end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment, and for physical communication with the virtual end through the Ethernet module; At the software level, the channel object based on the Serial and Socket modules is encapsulated. After the channel object is instantiated, it is used to communicate with the entity end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment at the physical level, and to communicate with the virtual end through the Ethernet module. Channel objects can work in real time and in parallel, and receive user instructions to configure channel parameters online; The universal monitoring system has universal monitoring capabilities and can monitor and control various types of aircraft that interact with each other based on modal data transmission links; It encapsulates the data transmission protocol modules in S4DTP-Hex and S4DTP-Str formats, implements decoding and encoding of the protocol, and monitors and controls various types of vehicles that exchange information based on the protocol.

3. The virtual-reality fusion simulation test system for ocean-going vehicles according to claim 1 is characterized in that: In the entity side: Including autonomous flight control system; the autonomous flight control system is based on the MOOS framework, adopts a multi-node parallel architecture, and has a simulation debugging node built inside to collect information from other nodes inside the autonomous flight control system and synchronize it to the virtual end, and accept virtual end data injection; the autonomous flight control system has a handle remote control mode, an autonomous navigation mode and a simulation operation mode. In the simulation operation mode, a data closed-loop interactive channel between the system and the virtual end is established through the simulation debugging node to perform hardware-in-the-loop virtual-real fusion simulation. The handle remote control mode and the autonomous navigation mode accept monitoring end data, but not virtual end data. This is the actual working mode of the ocean vehicle; The monitoring end opens the virtual end and configures the physical end to simulation mode. In this mode, the virtual end establishes a data connection with the virtual end through its simulation node via a multimodal data link; the monitoring end issues task-level instructions to the physical end, and the physical end starts running and interacts with the virtual end in real-time data. The physical end sends driving instruction data to the virtual end, and the virtual end returns the current status information of the virtual aircraft to the physical end, including position, attitude, speed and other information. The physical end receives the data sent by the virtual end through the simulation node and injects the data into the autonomous flight control system to drive the system to make the next decision and control, thereby realizing virtual-real fusion simulation; In the remote control mode, the user uses the panoramic camera, radar images, and nautical charts as visual input, and remotely generates commands using the handle to send drive control commands to the bottom layer of the autonomous flight control system through the universal monitoring system for direct control; In autonomous navigation mode, after the universal monitoring system plans the task, it sends the task type and task parameters to the autonomous navigation control system, which autonomously completes the path planning, controls the movement of the aircraft and maintains the attitude stability in accordance with the preset standards, autonomously senses and detects the target and completes obstacle avoidance. After completing the task, it will stand by autonomously and wait for the next instruction; In the simulation operation mode, the autonomous flight control system establishes a closed-loop data interaction channel with the virtual end through its internal simulation debugging node, and performs virtual-real fusion simulation of hardware in the loop. In this mode, the physical end uses the interactive data generated by the virtual end to drive the operation, and conducts online non-destructive control system function and algorithm tests on the spacecraft, replacing some lake and sea function tests, so as to achieve the purpose of pre-risk control, cost reduction and efficiency improvement. The physical end establishes physical communication with the monitoring end through the wired radio, Ethernet module, 4G module, and Beidou module, and establishes physical communication with the virtual end through the Ethernet module.

4. The virtual-reality fusion simulation test system for ocean-going vehicles according to claim 1 is characterized in that: In the virtual terminal: The digital test environment system architecture is divided into a data acquisition layer, a model calculation layer, a twin function layer and an immersive interaction layer. Under the UE5 physical engine, the ECS design pattern is used to organize and schedule the operation of each layer of functional modules. Each layer of functional modules is implemented in a mixed way of C++ and Python. The C++ implementation module and the Python implementation module exchange information through a shared memory mechanism. It has a scene rendering module for three-dimensional high-fidelity rendering of environmental scenes and virtual ocean vehicle motion scenes, an ocean vehicle dynamics model library and an environmental dynamics model library for multi-scale physical effect simulation calculations, and a network communication module for establishing data connections with the entity end and the monitoring end. The digital test environment system is deployed in a mobile ground station or other computers; it is physically connected to the monitoring end and the entity end through an Ethernet module.

5. The virtual-reality fusion simulation test system for ocean-going vehicles according to claim 1 is characterized in that: In the multimodal data transmission link: The multimodal data transmission link includes S4DTP-String and S4DTP-Hex message formats, including single frame transmission and data packet transmission modes; the S4DTP-String message format is used for data encoding, decoding and transmission processes between the monitoring end and the virtual end, the physical end and the virtual end; the S4DTP-Hex message format is used for data encoding, decoding and transmission processes between the monitoring end and the physical end; The single-frame transmission mode is used for the transmission of operation instructions, data instructions and status instructions between the monitoring end and the virtual end, the physical end and the virtual end; the data packet transmission mode is used for the transmission of task planning instructions between the monitoring end and the physical end.

6. A virtual-reality fusion simulation test method for ocean vehicles, characterized in that: include: Step S1: The monitoring end is instructed to monitor the operation of the physical end, have the ability of human-computer interaction and the ability to issue task planning instructions and operation instructions to the physical end and the virtual end, and control the operation mode and status of the physical end and the virtual end; Step S2: The physical end receives the mission planning instructions and operation instructions issued by the monitoring end and the motion state quantity of the virtual ocean vehicle output by the virtual end, controls the motion of the physical ocean vehicle, collects the internal data information of the physical ocean vehicle and uploads it to the monitoring end and the virtual end; Step S3: The virtual end provides a high-fidelity digital environment and a vehicle model image for the virtual-reality fusion simulation, promotes the operation of the physical end system in a data-driven manner, verifies the operation logic of the physical end system, and debugs the algorithm; including a digital test environment system and an Ethernet module.

7. The virtual-reality fusion simulation test method for ocean-oriented vehicles according to claim 6 is characterized by: In the monitoring terminal: It includes a universal monitoring system and a movable ground station; the universal monitoring system runs on the movable ground station or runs on other computers; The universal monitoring system has multi-channel communication capability, and is used for physical communication with the entity end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment, and for physical communication with the virtual end through the Ethernet module; At the software level, the channel object based on the Serial and Socket modules is encapsulated. After the channel object is instantiated, it is used to communicate with the entity end through the radio station, Ethernet module, 4G module, and Beidou module communication equipment at the physical level, and to communicate with the virtual end through the Ethernet module. Channel objects can work in real time and in parallel, and receive user instructions to configure channel parameters online; The universal monitoring system has universal monitoring capabilities and can monitor and control various types of aircraft that interact with each other based on modal data transmission links; It encapsulates the data transmission protocol modules in S4DTP-Hex and S4DTP-Str formats, implements decoding and encoding of the protocol, and monitors and controls various types of vehicles that exchange information based on the protocol.

8. The virtual-reality fusion simulation test method for ocean-oriented vehicles according to claim 6 is characterized by: In the entity side: Including autonomous flight control system; the autonomous flight control system is based on the MOOS framework, adopts a multi-node parallel architecture, and has a simulation debugging node built inside to collect information from other nodes inside the autonomous flight control system and synchronize it to the virtual end, and accept virtual end data injection; the autonomous flight control system has a handle remote control mode, an autonomous navigation mode and a simulation operation mode. In the simulation operation mode, a data closed-loop interactive channel between the system and the virtual end is established through the simulation debugging node to perform hardware-in-the-loop virtual-real fusion simulation. The handle remote control mode and the autonomous navigation mode accept monitoring end data, but not virtual end data. This is the actual working mode of the ocean vehicle; The monitoring end opens the virtual end and configures the physical end to simulation mode. In this mode, the virtual end establishes a data connection with the virtual end through its simulation node via a multimodal data link; the monitoring end issues task-level instructions to the physical end, and the physical end starts running and interacts with the virtual end in real-time data. The physical end sends driving instruction data to the virtual end, and the virtual end returns the current status information of the virtual aircraft to the physical end, including position, attitude, speed and other information. The physical end receives the data sent by the virtual end through the simulation node and injects the data into the autonomous flight control system to drive the system to make the next decision and control, thereby realizing virtual-real fusion simulation; In the remote control mode, the user uses the panoramic camera, radar images, and nautical charts as visual input, and remotely generates commands using the handle to send drive control commands to the bottom layer of the autonomous flight control system through the universal monitoring system for direct control; In autonomous navigation mode, after the universal monitoring system plans the task, it sends the task type and task parameters to the autonomous navigation control system, which autonomously completes the path planning, controls the movement of the aircraft and maintains the attitude stability in accordance with the preset standards, autonomously senses and detects the target and completes obstacle avoidance. After completing the task, it will stand by autonomously and wait for the next instruction; In the simulation operation mode, the autonomous flight control system establishes a closed-loop data interaction channel with the virtual end through its internal simulation debugging node, and performs virtual-real fusion simulation of hardware in the loop. In this mode, the physical end uses the interactive data generated by the virtual end to drive the operation, and conducts online non-destructive control system function and algorithm tests on the spacecraft, replacing some lake and sea function tests, so as to achieve the purpose of pre-risk control, cost reduction and efficiency improvement. The physical end establishes physical communication with the monitoring end through the wired radio, Ethernet module, 4G module, and Beidou module, and establishes physical communication with the virtual end through the Ethernet module.

9. The virtual-reality fusion simulation test method for ocean-oriented vehicles according to claim 6, characterized in that: In the virtual terminal: The digital test environment system architecture is divided into a data acquisition layer, a model calculation layer, a twin function layer and an immersive interaction layer. Under the UE5 physical engine, the ECS design pattern is used to organize and schedule the operation of each layer of functional modules. Each layer of functional modules is implemented in a mixed way of C++ and Python. The C++ implementation module and the Python implementation module exchange information through a shared memory mechanism. It has a scene rendering module for three-dimensional high-fidelity rendering of environmental scenes and virtual ocean vehicle motion scenes, an ocean vehicle dynamics model library and an environmental dynamics model library for multi-scale physical effect simulation calculations, and a network communication module for establishing data connections with the entity end and the monitoring end. The digital test environment system is deployed in a mobile ground station or other computers; it is physically connected to the monitoring end and the entity end through an Ethernet module.

10. The virtual-reality fusion simulation test method for ocean-oriented vehicles according to claim 6, characterized in that: The multimodal data transmission link includes S4DTP-String and S4DTP-Hex message formats, including single frame transmission and data packet transmission modes; the S4DTP-String message format is used for data encoding, decoding and transmission processes between the monitoring end and the virtual end, the physical end and the virtual end; the S4DTP-Hex message format is used for data encoding, decoding and transmission processes between the monitoring end and the physical end; The single-frame transmission mode is used for the transmission of operation instructions, data instructions and status instructions between the monitoring end and the virtual end, the physical end and the virtual end; the data packet transmission mode is used for the transmission of task planning instructions between the monitoring end and the physical end.

Images