Underwater virtual simulation interaction system, method and equipment
By introducing the Fish ID mechanism and asynchronous task dispatch module into the underwater virtual simulation system, the bandwidth contention and thread safety issues of control and media interaction in the prior art are solved, and stable bidirectional closed-loop interaction is achieved, supporting efficient communication in multi-entity concurrency and complex network environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-08
- Publication Date
- 2026-03-13
AI Technical Summary
Existing underwater virtual simulation systems based on Unreal Engine 5 suffer from problems in control and media interaction, such as bandwidth contention, complex packet fragmentation/reassembly, poor thread safety, difficulty in multi-entity addressing, and lack of multi-view backhaul links, making it difficult to achieve efficient and stable bidirectional closed-loop interaction.
The system employs a control server to assign a unique entity identifier, Fish ID, to each connection. Control commands are encapsulated into asynchronous tasks through an asynchronous task dispatch module. Compressed image transmission is achieved using media uplink and downlink channels. By combining the asynchronous task dispatch module with the two-level thread model of the main game thread, thread-safe bidirectional closed-loop interaction is realized.
It achieves thread-safe, loosely coupled, and easily scalable bidirectional closed-loop interaction, supports concurrent access from multiple external control terminals, has high robustness, and can stably establish closed-loop data flow in complex network environments, eliminating the risk of bandwidth contention between control commands and media data.
Smart Images
Figure CN121664863A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of underwater virtual simulation technology, and in particular to an underwater virtual simulation interactive system, method and device. Background Technology
[0002] Underwater robots, including autonomous underwater vehicles (AUVs), remotely operated underwater vehicles (ROVs), and biomimetic robotic fish, are playing an increasingly important role in marine resource exploration, underwater facility maintenance, search and rescue, and environmental monitoring. Using virtual simulation platforms for underwater robot operation training, algorithm verification, and system testing has become a key means to reduce the cost of real-world experiments and improve R&D efficiency. Among these, Unreal Engine 5 (UE5), with its powerful real-time rendering capabilities and physics engine, provides an ideal platform for building high-fidelity underwater simulation environments.
[0003] However, existing simulation systems based on UE5 or similar engines still have significant architectural flaws and technical bottlenecks in real-time interaction and collaboration with external control and perception algorithms, mainly in the following aspects: 1. Control and media multiplexing use a single TCP / UDP channel, which is prone to bandwidth contention, and media peaks can squeeze control latency; 2. Image transmission based on streaming byte streams lacks unified framing rules, making packet splicing / splitting processing complex; 3. The UE engine's thread model is complex, and direct cross-thread manipulation of the world state can easily cause crashes or undefined behavior; 4. Multi-entity addressing lacks a unified mechanism, making concurrent control from multiple external control terminals difficult; 5. The absence of downlink superposition links in multi-view media feedback makes it difficult for human-machine co-driving and interpretable interaction to form a closed loop.
[0004] Therefore, there is an urgent need for an underwater virtual simulation communication architecture that is thread-safe, loosely coupled, highly robust, and supports bidirectional closed-loop interaction. Summary of the Invention
[0005] The purpose of this application is to provide an underwater virtual simulation interaction system, method, and device that can achieve thread-safe, loosely coupled, easily expandable, and highly robust bidirectional closed-loop interactive underwater virtual simulation communication.
[0006] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides an underwater virtual simulation interaction system, the underwater virtual simulation interaction system comprising: The simulation terminal includes multiple robot actors and a control server; each robot actor is equipped with a camera component, an IMU simulation component, and a frame stream component; the control server is used to wait for connection requests from external algorithm terminals on a specified TCP port, assign a unique entity identifier Fish ID to each newly connected connection, and internally establish a mapping relationship between Fish ID and the corresponding robot actor; The media uplink channel is actively connected to a designated address of an external algorithm by the frame stream component, which is used to encode the rendered images captured by the camera component into compressed images and send them periodically in a preset frame format. The media downlink channel is passively monitored by the frame stream component at a port paired with the media downlink channel to receive the processed superimposed image frames returned by the external algorithm end. The superimposed image frames adopt the same format as the preset frame format in the media uplink channel. The asynchronous task dispatch module is used to encapsulate the control instructions parsed by the control server's working thread into asynchronous tasks and deliver them to the main game thread of UE5. The robot Actor with the corresponding Fish ID performs pose updates in the main game thread. The IMU simulation component obtains the coordinates and attitude angles of the robot Actor after the pose update, transforms the coordinates and attitude angles from the UE world coordinate system to the local coordinate system, calculates acceleration, attitude angles, and angular velocities based on the coordinates and attitude angles in the local coordinate system, adds zero-mean Gaussian noise to the acceleration, attitude angles, and angular velocities, and outputs sensor data aligned with the image frame timestamp.
[0007] Optionally, the underwater virtual simulation interactive system further includes a picture-in-picture interface module, used to display the superimposed image frames and sensor data with the image frame timestamps aligned in a picture-in-picture format.
[0008] Optionally, the preset frame format is specifically a "length prefix + compressed image" format.
[0009] Optionally, the control instructions are in text line format, and the control instructions specifically include the pose parameters of the robot Actor corresponding to the Fish ID.
[0010] Secondly, this application provides an underwater virtual simulation interaction method, including: S1: Start the control server on the UE5 emulation terminal and listen to the preset control port; S2: Respond to connection requests from external algorithms, assign a unique Fish ID to each new connection, and establish a mapping relationship between the Fish ID and the robot Actor in the scene; S3: Receives control commands in text line format in the worker thread of the control server, parses out the pose parameters, and encapsulates the control commands into asynchronous tasks according to the Fish ID; S4: The asynchronous task is delivered to the main game thread, and the robot Actor with the corresponding Fish ID performs first-order interpolation to smoothly update its coordinates and attitude angles; S5: In the frame stream component of the robot Actor, the rendered image is read from the camera component at a preset frame rate, the rendered image is encoded into a compressed image, and the encoded rendered image is periodically sent to the external algorithm terminal through the media uplink channel in a preset frame format. S6: Listens for and receives overlay image frames returned by the external algorithm terminal at the paired port of the media downlink channel. After reading the complete frame according to the preset frame format, it decodes and updates the UI texture in the main game thread through an asynchronous task to achieve real-time picture-in-picture display. S7: Obtain the updated coordinates and attitude angles from step 4 through the IMU simulation component, transform the updated coordinates and attitude angles from the UE world coordinate system to the local coordinate system, calculate the acceleration, attitude angle and angular velocity based on the coordinates and attitude angles in the local coordinate system, add zero-mean Gaussian noise to the acceleration, attitude angle and angular velocity, output the sensor data frame aligned with the image frame timestamp in step 5, and display the sensor data frame in picture-in-picture in real time.
[0011] Optionally, the control instructions are in text line format, and the control instructions specifically include the pose parameters of the robot Actor corresponding to the Fish ID.
[0012] Optionally, the preset frame format is specifically a "length prefix + compressed image" format.
[0013] Optionally, the IMU simulation component acquires the updated robot Actor coordinates and attitude angles, and transforms the updated coordinates and attitude angles from the UE world coordinate system to the local coordinate system, specifically including: The updated coordinates and attitude angles are transformed from the UE world coordinate system to the local coordinate system using a transformation matrix, as shown in the formula: ,in, The coordinates of the robot's Actor in the local coordinate system; The coordinates of the robot actor in the UE world coordinate system; The transformation matrix; The attitude angles of the robot actor in the UE world coordinate system are offset and corrected to obtain the attitude angles of the robot actor in the local coordinate system, as shown in the formula: ,in, Let be the roll angle of the robot Actor in the local coordinate system; Let $\r$ be the roll angle of the robot Actor in the UE world coordinate system. Let yaw angle be the robot Actor's yaw angle in the local coordinate system; Let yaw angle be the robot Actor's yaw angle in the UE world coordinate system; Let be the pitch angle of the robot Actor in the local coordinate system; Let be the pitch angle of the robot Actor in the UE world coordinate system.
[0014] Optionally, the step of calculating acceleration, attitude angle, and angular velocity based on the coordinates and attitude angles in the local coordinate system specifically includes: The formula for calculating the linear velocity of the robot Actor is as follows: ; in, This represents the position of the robot Actor at time t; express The location of the time-lapse robot Actor; This represents the time interval between the position and attitude acquisition times of adjacent robot Actors; The formula for calculating the acceleration of the robot Actor at time t is: ; in, Let t represent the linear velocity of the robot Actor at time t; express The linear velocity of the robot Actor at any given moment; The formula for the attitude angle of the robot Actor at time t is: ,in, Let be the roll angle of the robot Actor at time t in the local coordinate system; Let be the yaw angle of the robot Actor at time t in the local coordinate system; Let be the pitch angle of the robot Actor at time t in the local coordinate system; The formula for calculating the angular velocity of the robot Actor at time t is: ; in, This represents the attitude angle of the robot Actor at time t; express The pose angle of the robot Actor at any given moment.
[0015] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the underwater virtual simulation interaction method described in any one of the above.
[0016] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides an underwater virtual simulation interaction system, method, and device. The system includes a simulation terminal, a media uplink channel, a media downlink channel, and an asynchronous task dispatch module. The simulation terminal includes multiple robot actors and a control server; each robot actor is equipped with a camera component, an IMU simulation component, and a frame stream component; the control server assigns a unique entity identifier (Fish ID) to each newly connected connection and internally establishes a mapping relationship between Fish IDs and corresponding robot actors. This application integrates all robot actors into a Fish ID mapping table, achieving efficient routing and management of external control commands. The media uplink channel in this application encodes the rendered images captured by the camera component into compressed images and sends them periodically in a preset frame format; the media downlink channel receives superimposed image frames returned by the external algorithm; the asynchronous task dispatch module encapsulates the control instructions parsed by the control server's worker thread into asynchronous tasks and delivers them to the main game thread of UE5, where the robot Actor with the corresponding Fish ID performs pose updates in the main game thread, eliminating the risk of bandwidth contention and blocking between control instructions and media data on the same TCP channel. Through the dual media channels initiated in opposite directions, a closed-loop data flow can be stably established in complex network environments (such as NAT / firewall policy restrictions), realizing a real-time feedback mechanism of "simulation end sending original images → algorithm end analysis → downlink injection results"; at the same time, this application adopts a two-level thread model of asynchronous dispatch of worker thread + main game thread (Game Thread); the worker thread only receives and parses, and does not directly modify the world state; all entity updates are uniformly executed by the main game thread, completely avoiding common thread safety risks in UE engines, thereby supporting concurrent access of multiple external control ends and stable cross-platform operation. This application enables bidirectional closed-loop interactive underwater virtual simulation communication, and features thread safety, low coupling, easy expansion, and high robustness. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 A functional module diagram of an underwater virtual simulation interactive system provided in one embodiment of this application; Figure 2 A flowchart illustrating an underwater virtual simulation interaction method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0021] In one exemplary embodiment, such as Figure 1 As shown, an underwater virtual simulation interactive system is provided, including: a simulation terminal, a media uplink channel, a media downlink channel, an asynchronous task dispatch module, and a picture-in-picture interface module.
[0022] The simulation terminal includes multiple robot actors and a control server; each robot actor is equipped with a camera component, an IMU simulation component, and a frame stream component; the control server is used to wait for connection requests from external algorithm terminals on a specified TCP port, assign a unique entity identifier Fish ID to each newly connected connection, and internally establish a mapping relationship between Fish ID and the corresponding robot actor.
[0023] In this embodiment, efficient addressing is achieved by utilizing the mapping relationship between Fish ID and corresponding robot Actor. A unique connection relationship is established between the external algorithm and the internal robot through connection-level mapping. The rules for maintaining the mapping relationship are maintained through scene-level lifecycle management. Through the Fish ID that runs throughout the process, unified and accurate routing of all control commands and media data is achieved, making multi-robot parallel simulation well-organized and highly scalable.
[0024] The frame stream component actively connects to a designated address of an external algorithm terminal through the media uplink channel to encode the rendered images captured by the camera component into compressed images and send them periodically in a preset frame format.
[0025] The frame stream component passively listens at the port paired with the media downlink channel to receive the processed superimposed image frames returned by the external algorithm end. The superimposed image frames adopt the same format as the preset frame format in the media uplink channel.
[0026] Specifically, the media uplink channel port can be 8989 + 2. Index; the media downlink channel port can be 8990 + 2 Index.
[0027] In this embodiment, the preset frame format is specifically a "length prefix + compressed image" format.
[0028] The asynchronous task dispatch module encapsulates the control instructions parsed by the control server's working thread into asynchronous tasks and delivers them to the main game thread of UE5. The robot Actor with the corresponding Fish ID performs pose updates in the main game thread. The IMU simulation component obtains the robot Actor's coordinates and attitude angles after the pose update, transforms the coordinates and attitude angles from the UE world coordinate system to the local coordinate system, calculates acceleration, attitude angles, and angular velocities based on the coordinates and attitude angles in the local coordinate system, adds zero-mean Gaussian noise to the acceleration, attitude angles, and angular velocities, and outputs sensor data aligned with the image frame timestamp.
[0029] Specifically, the control instructions are in text line format, and the control instructions specifically include the pose parameters of the robot Actor corresponding to the Fish ID.
[0030] In this embodiment, the transmission of control commands is independent of the media uplink / downlink channels, and the media uplink / downlink channels are symmetrically designed so that they do not block each other.
[0031] The picture-in-picture interface module is used to display the superimposed image frames and the sensor data with the image frame timestamps aligned in a picture-in-picture format.
[0032] Specifically, the picture-in-picture interface module displays the original image and algorithm results in the HUD / UMG. Operators can see the original image and algorithm results simultaneously on the same interface. The picture-in-picture windows in the picture-in-picture interface module are arranged in the side column of the screen according to the index (size / spacing can be configured).
[0033] Specifically, the system described in this application, when deployed: 1. Employs automatic port addressing, generating paired ports (even for sending / odd for receiving or fixed difference mapping) based on stream indexes. This allows for linear scaling to N robots and N media streams, and supports dynamic adjustment of frame rate and compression quality, achieving a flexible trade-off between bandwidth and latency. This mechanism not only facilitates reuse in multi-robot formations and multi-sensor scenarios but also possesses excellent scalability and engineering deployment value; 2. Applies to pluggable encoders, defaults to JPEG format, and supports PNG / WEBP and H.264 / H.265 formats. Deployment is prioritized with minimal dependencies, followed by gradual upgrades.
[0034] Based on the same inventive concept, this application also provides an underwater virtual simulation interaction method. The overall process begins with the input of control commands from an external algorithm, proceeds through entity responses within the UE scene, media data generation and feedback, ultimately forming a closed-loop "control-perception-feedback" link. This underwater virtual simulation interaction method is applied to the aforementioned underwater virtual simulation interaction system, such as... Figure 2 As shown, the method includes the following steps: Step 1: Start the control server on the UE5 emulation client and listen to the preset control port.
[0035] Step 2: Respond to connection requests from the external algorithm, assign a unique Fish ID to each new connection, and establish a mapping relationship between the Fish ID and the robot Actor in the scene.
[0036] Specifically, the external algorithm establishes a connection with the virtual terminal through a separate TCP transmission control protocol and a preset listening port (such as listening port 12345). Each connection request represents a robot actor to be controlled. The control server (TCP server) in the UE simulation terminal accepts new connections from the external algorithm in its listening thread and assigns a unique Fish ID to each new connection according to the access order. At the same time, it establishes an internal mapping table of Fish IDs to robot actors in the scene. For example, Fish ID=0 indicates that it is bound to robot actor number 0 in the simulation scene.
[0037] Step 3: Receive control commands in text line format in the worker thread of the control server, parse out the pose parameters, and encapsulate the control commands into asynchronous tasks according to the Fish ID.
[0038] Specifically, the control instructions are in text line format, and the control instructions specifically include the pose parameters of the robot Actor corresponding to the Fish ID.
[0039] Specifically, in this embodiment, the control instruction in text line format is Set Pose:x,y,z,pitch,yaw,roll\n, where (x,y,z) represents the position coordinates of the target robot Actor in the virtual environment, and (pitch,yaw,roll) represents the pitch angle, yaw angle, and roll angle of the target robot Actor.
[0040] Step 4: The asynchronous task is delivered to the main game thread, where the robot Actor with the corresponding Fish ID performs first-order interpolation to smoothly update its coordinates and attitude angles, thereby completing the real-time mapping from external control to simulation execution.
[0041] Specifically, after receiving the instruction, the worker thread of the control server will continuously read and parse the position and posture data (in centimeters and angles) in a blocking manner, and then deliver the control instruction to the main thread through an asynchronous task to call the robot Actor with the corresponding Fish ID to update the pose.
[0042] This embodiment reads and parses a line of commands in a blocking manner. After successful parsing, the world state is not changed directly in the network worker thread of the controller. Instead, it is delivered to the main game thread of the UE as an asynchronous task. In the main game thread, the target robot Actor uses first-order interpolation to smoothly approximate the current position and orientation to the target pose set by the external command. This can maintain thread safety and eliminate the hidden dangers of cross-thread rendering / physics calls.
[0043] Step 5: In the frame stream component of the robot Actor, the rendered image is read from the camera component at a preset frame rate, the rendered image is encoded into a compressed image, and the encoded rendered image is periodically sent to the external algorithm terminal through the media uplink channel in a preset frame format.
[0044] Specifically, in this embodiment, the rendered image is a first-person or multi-view image of the robot Actor captured by the camera component; the image is a 1920×1080 resolution image; in the frame stream component of the robot Actor, the rendered image read from the camera component at a preset frame rate is initially in BGRA format, and the rendered image is encoded from BGRA format into a frame format of "4-byte big-endian length prefix + JPEG compressed image", and periodically sent to the external algorithm end through the media uplink channel to avoid packet fragmentation / splitting.
[0045] Specifically, the frame rate in the encoding parameters of this embodiment The default value is 15 FPS, which can be dynamically adjusted according to network bandwidth and the real-time requirements of external algorithms. The frame rate determines the frequency of image updates per unit time, thus affecting the temporal sampling rate of downstream algorithms (such as path tracking algorithms for tracking figures of eight) (such as target tracking and SLAM algorithms, which typically require 10-30Hz visual input). In addition, the encoding parameters in this embodiment also include JPEG compression quality. This parameter controls the balance between bitrate and image detail; the default value is 75, and it can be configured within the range of (0, 100). At lower speeds, image size is smaller and transmission latency is reduced, but high-frequency details are lost, which may affect the accuracy of feature point detection, semantic segmentation, etc.; when At higher speeds, image clarity improves, making detection and segmentation easier, but network bandwidth consumption and latency increase. Therefore, and Jointly adjustable: Reduces bandwidth limitations or To ensure real-time performance, improve accuracy when algorithm precision requirements are high. Or maintain a higher This is to ensure the continuity of time.
[0046] Step 5 yields the media frame units available for consumption by the external algorithm: ;in, The length of the frame data is represented by a 4-byte big-endian format. For timestamp t The acquired images are subject to frame rate control and compression quality adjustment. The encoded JPEG data stream; In order to be in t The media frame unit at a given moment serves as the basic unit for media uplink transmission. After being decoded at the external algorithm end, it can be directly input into the visual perception module for visual perception (such as target detection, tracking, segmentation, annotation processing, or visual positioning).
[0047] Specifically, in this embodiment, t It represents a specific sampling moment in a discrete time series.
[0048] Step 6: Listen for and receive the overlay image frames returned by the external algorithm on the paired port of the media downlink channel. After reading the complete frame according to the preset frame format, decode and update the UI texture in the main game thread through an asynchronous task to achieve real-time picture-in-picture display.
[0049] Specifically, in this embodiment, after the external algorithm performs visual perception (such as target detection, tracking, segmentation, annotation processing or visual positioning) processing, it is re-transmitted to the simulation end through the media downlink channel in the same "length prefix + compressed image" format.
[0050] In addition to overlaying images, the embodiments in this application also include JSON / binary metadata (detection boxes, poses, semantic labels).
[0051] Specifically, after listening to and receiving the overlay image frames returned by the external algorithm on the paired port of the media downlink channel, the complete frame is strictly determined according to the framing mechanism of "4-byte big-endian length prefix + JPEG compressed image": The background thread first blocks to read the 4-byte length prefix and parses it into frame length L, and then receives exactly L bytes of data in a loop; if it is not read within the timeout, the number of bytes exceeds the limit, or the JPEG start and end markers are missing, it is considered incomplete and discarded; if necessary, CRC check or sequence number increment check can be added. Only when the conditions of "prefix legal + byte number matching + format verification" are met, the buffer is determined to be a complete frame and handed over to JPEG decoding to generate a pixel buffer. Then, the UTexture2D texture is updated in the UE main thread through an asynchronous task and mapped to the HUD / UMG small window to achieve stable picture-in-picture (PIP) multi-channel real-time presentation, ensuring the objective judgment standard of "complete frame", ensuring the reliability of the visual feedback closed loop, and providing the Wait All reading criterion and maximum frame protection.
[0052] In this embodiment, a framing mechanism is employed throughout the entire uplink and downlink media transmission process (decoding only after receiving a complete frame), completely avoiding the packet fragmentation problem. The frame rate and JPEG quality can be flexibly adjusted according to bandwidth / latency constraints to achieve flow control.
[0053] This embodiment forms a complete serial link consisting of "control commands → simulation sensor data acquisition → media uplink encoding → external algorithm processing → media downlink back injection → HUD visual feedback". The input and output relationships between each link are clear, which not only ensures the thread safety of the network and engine, but also realizes multi-entity addressing, bandwidth adaptive control and sensor noise simulation, and can support complex real-time interactive simulation and algorithm verification scenarios.
[0054] Step 7: Obtain the updated coordinates and attitude angles from Step 4 using the IMU simulation component, transform the updated coordinates and attitude angles from the UE world coordinate system to the local coordinate system, calculate the acceleration, attitude angle, and angular velocity based on the coordinates and attitude angles in the local coordinate system, add zero-mean Gaussian noise to the acceleration, attitude angle, and angular velocity, output a sensor data frame aligned with the image frame timestamp in Step 5, and display the sensor data frame in picture-in-picture mode in real time.
[0055] Specifically, the IMU simulation component acquires the updated robot Actor coordinates and attitude angles, and transforms the updated coordinates and attitude angles from the UE world coordinate system to the local coordinate system, specifically including: The updated coordinates and attitude angles are transformed from the UE world coordinate system to the local coordinate system using a transformation matrix, as shown in the formula: ,in, The coordinates of the robot's Actor in the local coordinate system; The coordinates of the robot actor in the UE world coordinate system; The transformation matrix; This indicates a transformation from the UE world coordinate system to the local coordinate system.
[0056] The attitude angles of the robot actor in the UE world coordinate system are offset and corrected to obtain the attitude angles of the robot actor in the local coordinate system, as shown in the formula: ,in, Let be the roll angle of the robot Actor in the local coordinate system; Let $\r$ be the roll angle of the robot Actor in the UE world coordinate system. Let yaw angle be the robot Actor's yaw angle in the local coordinate system; Let yaw angle be the robot Actor's yaw angle in the UE world coordinate system; Let be the pitch angle of the robot Actor in the local coordinate system; Let be the pitch angle of the robot Actor in the UE world coordinate system.
[0057] Specifically, the acceleration, attitude angle, and angular velocity are calculated based on the coordinates and attitude angles in the local coordinate system, including: The formula for calculating the linear velocity of the robot Actor is as follows: ; in, express t The location of the time-lapse robot Actor; express The location of the time-lapse robot Actor; This represents the time interval between the position and attitude acquisition times of adjacent robot Actors.
[0058] The t The formula for calculating the acceleration of the time-lapse robot Actor is: ; in, express t The linear velocity of the robot Actor at any given moment; express The linear velocity of the robot Actor at any given moment.
[0059] The tThe formula for the pose angle of the time-lapse robot Actor is: ,in, In the local coordinate system t The roll angle of the time-lapse robot Actor; In the local coordinate system t The yaw angle of the robot Actor at all times; In the local coordinate system t The pitch angle of the robot Actor at all times.
[0060] The t The formula for calculating the angular velocity of the robot Actor at any given time is: ; in, express t The pose angle of the time-lapse robot Actor; express The pose angle of the robot Actor at any given moment.
[0061] Specifically, zero-mean Gaussian noise is superimposed on the acceleration, attitude angle, and angular velocity channels to simulate the measurement error of a real IMU sensor, using the following formula: ; in, express t Acceleration after adding zero-mean Gaussian noise at any time; The standard deviation of acceleration; Zero-mean Gaussian noise is a unit representing acceleration. express t Acceleration after adding zero-mean Gaussian noise at any time; The standard deviation of angular velocity; Zero-mean Gaussian noise is used to represent angular velocity. express t Attitude angle after adding zero-mean Gaussian noise at any time; This represents the standard deviation of the attitude angles; Zero-mean Gaussian noise represents the attitude angle; where, .
[0062] Specifically, the sensor data frame aligned with the image frame timestamp in step 5 is: .
[0063] In this embodiment, when the external algorithm continuously sends pose commands, the robot in the scene can smoothly track the target path in the UE simulation without thread conflicts or crashes; the detection results from the algorithm can be stably transmitted back and displayed in the HUD interface at a rate of about 15 FPS, verifying the feasibility of this application in terms of real-time performance and interactivity; even when multiple robots (such as 3 to 5 robotic fish) are running simultaneously, the system can still effectively distinguish entities through Fish ID, and the media streams do not interfere with each other, verifying the scalability of the solution; furthermore, in a network environment with limited bandwidth (such as <10 Mbps), smooth interaction can still be maintained by lowering the JPEG quality to 50, indicating that this application has bandwidth adaptive capability.
[0064] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to underwater virtual simulation interaction. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an underwater virtual simulation interaction method.
[0065] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0066] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
[0067] The underwater virtual simulation interaction system, method, and device disclosed in the above embodiments of this application adopt a three-channel communication architecture with separate control command transmission channels and media channels. The control command transmission channel uses a text-based protocol to perform Fish ID addressing and pose control for multiple robots (Actors) in the scene. The media channel uses paired TCP links for one-to-one transmission and receives, and compressed images encapsulated with a length prefix are used for bidirectional streaming transmission. The simulation end completes entity state updates and returns image UI presentation on the main game thread. The communication architecture in this application is thread-safe, loosely coupled, easily scalable, and highly robust, supporting IMU sensor simulation and noise modeling, and can be widely used in virtual simulation, real-time streaming communication, and human-computer interaction fields.
[0068] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0069] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. An underwater virtual simulation interactive system, characterized in that, The underwater virtual simulation interaction system includes: The simulation terminal includes multiple robot actors and a control server; each robot actor is equipped with a camera component, an IMU simulation component, and a frame stream component; the control server is used to wait for connection requests from external algorithm terminals on a specified TCP port, assign a unique entity identifier Fish ID to each newly connected connection, and internally establish a mapping relationship between Fish ID and the corresponding robot actor; The media uplink channel is actively connected to a designated address of an external algorithm by the frame stream component, which is used to encode the rendered images captured by the camera component into compressed images and send them periodically in a preset frame format. The media downlink channel is passively monitored by the frame stream component at a port paired with the media downlink channel to receive the processed superimposed image frames returned by the external algorithm end. The superimposed image frames adopt the same format as the preset frame format in the media uplink channel. The asynchronous task dispatch module is used to encapsulate the control instructions parsed by the control server's working thread into asynchronous tasks and deliver them to the main game thread of UE5. The robot Actor with the corresponding Fish ID performs pose updates in the main game thread. The IMU simulation component obtains the coordinates and attitude angles of the robot Actor after the pose update, transforms the coordinates and attitude angles from the UE world coordinate system to the local coordinate system, calculates acceleration, attitude angles, and angular velocities based on the coordinates and attitude angles in the local coordinate system, adds zero-mean Gaussian noise to the acceleration, attitude angles, and angular velocities, and outputs sensor data aligned with the image frame timestamp.
2. The underwater virtual simulation interactive system according to claim 1, characterized in that, The underwater virtual simulation interaction system also includes: The picture-in-picture interface module is used to display the overlaid image frames and the sensor data with the timestamps of the image frames in a picture-in-picture format.
3. The underwater virtual simulation interactive system according to claim 1, characterized in that, The preset frame format is specifically a "length prefix + compressed image" format.
4. The underwater virtual simulation interactive system according to claim 1, characterized in that, The control instructions are in text line format, and the control instructions specifically include the pose parameters of the robot Actor corresponding to the Fish ID.
5. An underwater virtual simulation interaction method, characterized in that, The underwater virtual simulation interaction method is applied to the underwater virtual simulation interaction system according to any one of claims 1-4, and the underwater virtual simulation interaction method includes: S1: Start the control server on the UE5 emulation terminal and listen to the preset control port; S2: Respond to connection requests from external algorithms, assign a unique Fish ID to each new connection, and establish a mapping relationship between the Fish ID and the robot Actor in the scene; S3: Receives control commands in text line format in the worker thread of the control server, parses out the pose parameters, and encapsulates the control commands into asynchronous tasks according to the Fish ID; S4: The asynchronous task is delivered to the main game thread, and the robot Actor with the corresponding Fish ID performs first-order interpolation to smoothly update its coordinates and attitude angles; S5: In the frame stream component of the robot Actor, the rendered image is read from the camera component at a preset frame rate, the rendered image is encoded into a compressed image, and the encoded rendered image is periodically sent to the external algorithm terminal through the media uplink channel in a preset frame format. S6: Listens for and receives overlay image frames returned by the external algorithm terminal at the paired port of the media downlink channel. After reading the complete frame according to the preset frame format, it decodes and updates the UI texture in the main game thread through an asynchronous task to achieve real-time picture-in-picture display. S7: Obtain the updated coordinates and attitude angles from step 4 through the IMU simulation component, transform the updated coordinates and attitude angles from the UE world coordinate system to the local coordinate system, calculate the acceleration, attitude angle and angular velocity based on the coordinates and attitude angles in the local coordinate system, add zero-mean Gaussian noise to the acceleration, attitude angle and angular velocity, output the sensor data frame aligned with the image frame timestamp in step 5, and display the sensor data frame in picture-in-picture in real time.
6. The underwater simulation interaction method according to claim 5, characterized in that, The control instructions are in text line format, and the control instructions specifically include the pose parameters of the robot Actor corresponding to the Fish ID.
7. The underwater simulation interaction method according to claim 5, characterized in that, The preset frame format is specifically a "length prefix + compressed image" format.
8. The underwater simulation interaction method according to claim 5, characterized in that, The IMU simulation component acquires the updated robot Actor coordinates and attitude angles, and transforms the updated coordinates and attitude angles from the UE world coordinate system to the local coordinate system, specifically including: The updated coordinates and attitude angles are transformed from the UE world coordinate system to the local coordinate system using a transformation matrix, as shown in the formula: ,in, The coordinates of the robot Actor in the local coordinate system; The coordinates of the robot actor in the UE world coordinate system; The transformation matrix; The attitude angles of the robot actor in the UE world coordinate system are offset and corrected to obtain the attitude angles of the robot actor in the local coordinate system, as shown in the formula: ,in, Let be the roll angle of the robot Actor in the local coordinate system; Let $\r$ be the roll angle of the robot Actor in the UE world coordinate system. Let yaw angle be the robot Actor's yaw angle in the local coordinate system; Let yaw angle be the robot Actor's yaw angle in the UE world coordinate system; Let be the pitch angle of the robot Actor in the local coordinate system; Let be the pitch angle of the robot Actor in the UE world coordinate system.
9. The underwater simulation interaction method according to claim 5, characterized in that, The calculation of acceleration, attitude angle, and angular velocity based on the coordinates and attitude angles in the local coordinate system specifically includes: The formula for calculating the linear velocity of the robot Actor is as follows: ； in, express t The location of the time-lapse robot Actor; express The location of the time-lapse robot Actor; This represents the time interval between the position and attitude acquisition times of adjacent robot Actors; The t The formula for calculating the acceleration of the time-lapse robot Actor is: ； in, express t The linear velocity of the robot Actor at any given moment; express The linear velocity of the robot Actor at any given moment; The t The formula for the pose angle of the time-lapse robot Actor is: ,in, In the local coordinate system t The roll angle of the time-lapse robot Actor; In the local coordinate system t The yaw angle of the robot Actor at all times; In the local coordinate system t The pitch angle of the robot Actor; The t The formula for calculating the angular velocity of the robot Actor at any given time is: ； in, express t The pose angle of the robot Actor at any given moment; express The pose angle of the robot Actor at any given moment.
10. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the underwater virtual simulation interaction method according to any one of claims 5-9.