Seabed scene reconstruction method and device based on multi-modal data processing

Abstract

The embodiment of the invention relates to a seabed scene reconstruction method and device based on multi-modal data processing, and the method comprises the steps: obtaining video data, audio data and spatio-temporal data collected by underwater equipment, embedding metadata containing ENU coordinates and timestamps into the video data and the audio data, and outputting a media file in an FFmpeg format; extracting a video stream, an audio stream and a metadata stream from the media file in the FFmpeg format, generating a scene graph stream, synchronizing and cutting the video stream, the audio stream and the scene graph stream, and outputting window data with aligned time; pre-processing the window data subjected to time alignment, sequentially inputting the pre-processed window data into a multi-modal fusion Transform model and a Transform model based on an energy function, and outputting a scene token vector; and converting the scene token vector into an octree data structure, and inputting the octree data structure into a decoder to generate a 3D scene graph of the seabed scene.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of seabed information management technology, specifically to a method, apparatus, and computer-readable storage medium storing a computer program for seabed scene reconstruction based on multimodal data processing. Background Technology

[0002] With the ever-increasing demand for marine resource development, seabed exploration, disaster prediction, marine monitoring, and security, the Internet of Underwater Things (IoUT) technology is gradually becoming a key force driving marine technological innovation. By deploying various sensors, cameras, sonar, hydrophones, and unmanned underwater vehicles in the marine environment, IoUTs can collect real-time data on the marine environment and underwater objects, including acoustic spectra, images, temperature, pressure, and velocity. However, due to the unique characteristics of the underwater environment, such as high pressure, low temperature, and strong noise, marine data acquisition faces numerous challenges, including signal attenuation, data loss, and noise interference.

[0003] Traditional processing methods, such as statistical analysis and rule-based reasoning, are no longer sufficient to handle the increasingly complex marine data. In this context, machine learning and deep learning technologies have become crucial supporting tools for IoUT systems. In marine big data analysis, many tasks involve the reconstruction of temporal or spatial data. For example, the digital reconstruction of seabed scenes is typically based on temporal acoustic spectra and images acquired by hydrophones and cameras. These images and signals contain information such as the trajectory of targets and their spatial structure. Deep learning models, especially convolutional neural networks and recurrent neural networks, can process these spatiotemporal signals, performing signal reconstruction and sequence modeling. Using trained neural networks, different types of seabed objects, such as shipwrecks, marine life, and pipelines, can be identified, and their trajectories or changes in the seabed environment can be inferred based on these identification results. By analyzing equipment operating status, environmental changes, and sensor data, deep learning models can identify early signs of equipment failure, provide warning information, and infer the causes of failures based on historical system data.

[0004] In the digital reconstruction of seabed scenes, multiple sensors within the seabed area capture acoustic spectra and image data of objects at different locations. As an object moves sequentially from one area to another, adjacent sensors and cameras receive its motion acoustic signals and image information in turn. This data is transmitted to a land-based data center via a marine network, where deep learning models process this temporal data to reconstruct the motion trajectory or spatial structure of seabed objects, thereby generating a digital mirror of the seabed environment. A key challenge is how to efficiently optimize the underlying information of 3D scenes to achieve realistic scene reconstruction, especially in real-time visualization in dynamically changing environments, to facilitate the management and decision-making of seabed operations. Summary of the Invention

[0005] The embodiments described herein provide a method, apparatus, and computer-readable storage medium for reconstructing seabed scenes based on multimodal data processing. The method reconstructs seabed scenes using seabed virtual space location information and time-series data, and designs a multi-module neural network architecture to process and analyze seabed virtual space data, thereby achieving high-precision, real-time 3D reconstruction of seabed scenes.

[0006] According to a first aspect of this disclosure, a method for reconstructing seabed scenes based on multimodal data processing is provided, comprising: acquiring video data, audio data, and spatiotemporal data collected by underwater equipment; embedding metadata including ENU coordinates and timestamps into the video data and audio data; and outputting a media file in FFmpeg format; extracting video stream, audio stream, and metadata stream from the FFmpeg format media file to generate a scene graph stream; synchronizing and segmenting the video stream, audio stream, and scene graph stream; outputting time-aligned window data; preprocessing the time-aligned window data and sequentially inputting it into a multimodal fusion Transformer model and an energy function-based Transformer model to output a scene token vector; converting the scene token vector into an octree data structure and inputting the octree data structure into a decoder to generate a 3D scene graph of the seabed scene.

[0007] In some embodiments of this disclosure, acquiring video data, audio data, and spatiotemporal data collected by an underwater device, embedding metadata including ENU coordinates and timestamps into the video and audio data, and outputting a media file in FFmpeg format includes: acquiring video data, audio data, virtual position coordinates, and timestamps collected by the underwater device; mapping the virtual position coordinates to ENU coordinates; using FFmpeg to embed the ENU coordinates and timestamps as metadata into the video data; generating a video file with metadata through the mapping relationship between pixel coordinates and ENU coordinates; and using FFmpeg to embed the ENU coordinates and timestamps as metadata into the audio data; generating an audio file with metadata through the mapping relationship between the timeline and ENU coordinates.

[0008] In some embodiments of this disclosure, mapping virtual location coordinates to ENU coordinates includes: obtaining the latitude and longitude of the region center, setting the origin of the ENU coordinate system on the water surface, and calculating the height of the water surface origin; subtracting the water depth from the height of the water surface origin to obtain the actual height of the underwater region as the upward coordinate of the underwater region; obtaining the eastward and northward local coordinates of each object in the cluster relative to the region center, calculating the upward coordinate of each object based on the water depth, and obtaining the final ENU coordinates of each object.

[0009] In some embodiments of this disclosure, FFmpeg is used to embed ENU coordinates and timestamps as metadata into video data. Generating a video file with metadata by mapping the pixel coordinates to the ENU coordinates involves: obtaining the field of view and shooting angle of the underwater camera through calibration; establishing a linear scaling model to determine the mapping relationship between pixel coordinates and actual ENU coordinates in each video frame; for each video frame, using the established linear scaling model, calculating the corresponding ENU coordinates from the pixel coordinates of the video frame; encapsulating the corresponding ENU coordinates and timestamp into a metadata file; and using FFmpeg to embed the generated metadata into the original video file for encapsulation, outputting a video file in FFmpeg format.

[0010] In some embodiments of this disclosure, FFmpeg is used to embed ENU coordinates and timestamps as metadata into audio data. The process of generating an audio file with metadata by mapping the timeline and ENU coordinates includes: reading the audio file to obtain the audio duration; generating corresponding ENU coordinates for each time point based on a set time interval; writing the timestamp and the corresponding ENU coordinates into a metadata file; and using FFmpeg to output an audio file in FFmpeg format from the generated metadata and the original audio file.

[0011] In some embodiments of this disclosure, extracting video streams, audio streams, and metadata streams from FFmpeg format media files to generate scene graph streams, and synchronizing and segmenting the video streams, audio streams, and scene graph streams to output time-aligned window data includes: using the ffprobe tool to parse video streams, audio streams, and metadata streams from FFmpeg format media files; traversing the video stream, using a pre-trained computer vision model to identify objects and their relationships in each frame of the video stream, using objects as nodes and relationships between objects as edges to generate a scene graph; marking ENU coordinates and timestamps on the scene graph to obtain a time-stamped scene graph stream; defining a fixed time window and obtaining a common time range from the metadata of all data streams; traversing the entire time range, extracting video, audio, and scene graph data within the current time window, checking whether the data within the window is complete, and if incomplete, completing it by interpolation or finding nearest neighbor data; encapsulating the video stream, audio stream, scene graph stream, its timestamp, and ENU coordinates of each time window to output time-aligned window data.

[0012] In some embodiments of this disclosure, after preprocessing time-aligned window data, it is sequentially input into a multimodal fusion Transformer model and an energy function-based Transformer model to output a scene token vector. This process includes: preprocessing time-aligned window data and inputting it into the multimodal fusion Transformer model for semantic encoding to output a scene state token, which includes a video token, an audio token, a scene graph token, spatiotemporal encoding, and a quality / confidence mask; constructing an energy function to evaluate the rationality of the scene state token, including: averaging the scene state token in the time or spatial dimension to generate a pooled scene state vector; using a multilayer perceptron to score the pooled scene state vector and outputting a scalar energy value, where a lower scalar energy value indicates a more rational scene; inputting the scene state token into the energy function-based Transformer model, iteratively optimizing the scene state token to minimize the energy function, and outputting a scene token vector.

[0013] In some embodiments of this disclosure, converting scene token vectors into octree data structures and inputting the octree data structures into decoders to generate 3D scene graphs of the seabed includes: initializing an empty octree, defining a root bounding box, traversing each scene token vector, inserting the feature vector of each token into the corresponding node of the octree, and returning the constructed octree; loading a pre-trained multimodal octree decoder model, processing the octree from the root node to the leaf node, using the stored feature vector at each node to predict the scene attributes of that node, and if the final scene representation is an implicit field, using the implicit field to generate a visible scene, extracting the implicit field into a mesh and rendering it into the final 3D scene graph.

[0014] According to a second aspect of this disclosure, a device for reconstructing a seabed scene based on multimodal data processing is provided. The device includes at least one processor and at least one memory storing a computer program. When the computer program is executed by the at least one processor, the device causes the following actions: acquiring video data, audio data, and spatiotemporal data collected by an underwater device; embedding metadata including ENU coordinates and timestamps into the video and audio data; outputting a media file in FFmpeg format; extracting the video stream, audio stream, and metadata stream from the FFmpeg format media file; generating a scene graph stream; synchronizing and segmenting the video stream, audio stream, and scene graph stream; outputting time-aligned window data; preprocessing the time-aligned window data and sequentially inputting it into a multimodal fusion Transformer model and an energy function-based Transformer model; outputting a scene token vector; converting the scene token vector into an octree data structure; and inputting the octree data structure into a decoder to generate a 3D scene graph of the seabed scene.

[0015] According to a third aspect of this disclosure, a computer-readable storage medium storing a computer program is provided, wherein the computer program, when executed by a processor, implements the steps of the seabed scene reconstruction method based on multimodal data processing according to the first aspect of this disclosure.

[0016] The seabed scene reconstruction method and apparatus based on multimodal data processing according to embodiments of this disclosure, by combining video, audio spectra, graph structure data, and spatiotemporal information (ENU coordinates and timestamps), can more comprehensively perceive complex seabed environments. Through the fusion of multimodal data and optimization of energy functions, it can generate three-dimensional scenes that conform to physical and geometric laws. In the storage and retrieval of 3D scenes, the use of an octree structure greatly improves the storage efficiency and retrieval speed of large-scale scene data. Furthermore, the modular design of multi-stage AI models enhances the scalability and flexibility of the system. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure, wherein:

[0018] Figure 1 An exemplary flowchart of a method for reconstructing a seabed scene based on multimodal data processing according to an embodiment of the present disclosure is shown;

[0019] Figure 2 This is a schematic diagram of the structure of an AI system for reconstructing underwater scenes according to an embodiment of the present disclosure;

[0020] Figure 3 This is a schematic block diagram of a seabed scene reconstruction apparatus based on multimodal data processing according to an embodiment of the present disclosure.

[0021] It should be noted that the elements in the attached diagram are schematic and not drawn to scale. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are also within the scope of protection of this disclosure.

[0023] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having meanings consistent with their meanings in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. Furthermore, terms such as “first” and “second” are used only to distinguish one component (or part of a component) from another component (or another part of a component).

[0024] Underwater sensor networks comprise a variety of sensors and devices used to capture data about the seabed environment. For example, cameras provide real-time video images of the seabed to acquire visual information, facilitating the reconstruction of seabed scenes. Small hydrophone arrays capture acoustic data, particularly acoustic signals from the seabed environment, which can be used for positioning, environmental monitoring, and more. Unmanned underwater vehicles (AUVs) provide autonomous navigation and environmental exploration, acquiring data from a wide range of underwater areas. Remotely operated vehicles (ROVs) are used for more precise tasks, such as equipment inspection and seabed sampling, potentially providing high-definition video streams and acoustic data. Underwater positioning systems (USBL / LBLs) are used to locate underwater equipment, ensuring the precise position of seabed sensors and AUVs. Sonar devices are used to detect underwater objects and map the environment, particularly useful for exploring unknown or uncharted areas. Inertial measurement units (IMUs) capture dynamic data (such as angular velocity and acceleration) to help determine the attitude and position of equipment. These devices work together to form a multimodal data acquisition system, providing comprehensive information about the seabed environment.

[0025] Data acquisition on the seabed relies on the aforementioned sensors. The collected data includes: video data from cameras and ROVs, providing visual information for subsequent scene reconstruction; acoustic spectrum data from hydrophones and sonar, aiding in acoustic localization, environmental detection, and target identification; temporal data such as the motion trajectory of target objects or equipment; and virtual location markers (depth information, regions, clusters, etc.) from underwater sensors, helping to project data into physical space. Data is relayed via surface gateways and unmanned underwater vehicles (UUVs) to ensure data flow between underwater and surface environments. Data forwarding through surface gateways projects virtual locations to physical locations, and then data is transmitted to a data center via terrestrial or satellite networks. This data transmission architecture ensures real-time transmission and processing of data at different levels.

[0026] The objective of this disclosure is to construct a self-supervised, multimodal AI system for seabed scene reconstruction and real-time information visualization management. In seabed scene reconstruction, the selection of the AI ​​model is crucial. The energy-based Transformer model combines the parallel processing capabilities of the Transformer architecture with the complex structural reasoning capabilities of the energy model, effectively addressing the complex tasks involved in seabed scene reconstruction.

[0027] To address this, this disclosure presents a method for reconstructing seabed scenes based on multimodal data processing and a multi-stage AI system architecture. Through efficient data fusion and modular design, it enables accurate reconstruction of the seabed environment. By decoupling perception and generation, the AI ​​system not only processes multimodal data fusion and generates intermediate scene state representations, but also combines these representations with energy functions to generate more reasonable physical scenes. It possesses high debuggability and scalability, effectively supporting real-time information visualization management and decision-making across air, space, land, and sea.

[0028] To further illustrate the embodiments of this disclosure, Figure 1 An exemplary flowchart of a method for reconstructing seabed scenes based on multimodal data processing according to embodiments of the present disclosure is shown. Figure 1 At box S102, video data, audio data, and spatiotemporal data collected by underwater equipment are acquired. Metadata containing ENU coordinates and timestamps is embedded into the video and audio data, and a media file in FFmpeg format is output.

[0029] Specifically, the process begins by acquiring video data, audio data, virtual location coordinates, and timestamps from underwater equipment, and then mapping the virtual location coordinates to ENU coordinates. The video data, obtained from seabed cameras, contains visual information. The audio data is captured by a hydrophone array and converted into a spectrogram. Spatiotemporal information, including region, water depth, and timestamps, constitutes the metadata for both video and audio data.

[0030] In underwater scene reconstruction, the mapping between virtual coordinate systems (such as water depth, region, cluster, etc.) and physical coordinate systems can achieve more accurate positioning and data integration. Physical coordinate systems, such as the ENU (East-North-Up) coordinate system, are right-handed Cartesian coordinate systems with three axes: East (X-axis): pointing geographically east along a horizontal direction; North (Y-axis): pointing geographically north along a horizontal direction; and Up (Z-axis): perpendicular to the ground, pointing to the water surface in underwater scenes, representing depth. Compared to global coordinate systems (such as WGS-84 or geographic coordinate systems), the ENU coordinate system has locality, making it more computationally efficient and able to ignore the influence of the Earth's curvature on accuracy, making it suitable for precise positioning over short distances.

[0031] To map virtual location coordinates to ENU coordinates, the latitude and longitude of the region are obtained. The origin of the ENU coordinate system is set on the water surface, and the height of the water surface origin is calculated. The height of the water surface is calculated based on the height of the geoid and is used as the height of the water surface origin. The water depth is the depth of the underwater region and is a negative value. By subtracting the water depth from the height of the water surface origin, the actual height of the underwater region is obtained and used as the upward coordinate of the underwater region. For each object in the cluster, the local east and north coordinates of each object relative to the center of the region are obtained, and the upward coordinate of the object is calculated based on the water depth. The ENU coordinates of each object are its position relative to the water surface origin. This position is then converted to position coordinates in the ENU coordinate system after offset from the center of the region. The final ENU coordinates of each object are stored in a list. Finally, a list containing the ENU coordinates of all objects is returned.

[0032] Subsequently, metadata (ENU coordinates and timestamps) is embedded into the video and audio data. For example, FFmpeg is used to embed ENU coordinates and timestamps as metadata into the video data, and a video file with metadata is generated through the mapping relationship between pixel coordinates and ENU coordinates. According to one embodiment of this disclosure, by calibrating to obtain the field of view and shooting angle of the underwater camera, a linear scaling model is established, defining the mapping relationship between pixel coordinates and actual ENU coordinates in each video frame. The coordinate mapping can be performed using the following mathematical relationship:

[0033] E = k E (u-u0)+E0

[0034] N = k N (v-v0)+N0

[0035] Where (u,v) are the pixel coordinates in the video frame, (E,N) are the corresponding ENU coordinates, (u0,v0) are the pixel coordinates of the center point of the camera's projection on the screen, (E0,N0) are the ENU coordinates of the camera's physical location in the real world, and k E and k N This is a scaling factor representing the distance (e.g., meters per pixel) of each pixel in the ENU coordinate system. The calibration process includes: selecting the ENU coordinates of two or more known locations and observing their pixel coordinates in the video frame by calibrating known objects in the scene; calculating k using the positions of known points in the image and their corresponding ENU coordinates. E and k N The scaling factor can be determined using the least squares method or linear regression. The center point (u0, v0) of the video image is determined by measuring the field of view and lens angle of the camera and then mapped to its physical location (E0, N0) in the ENU coordinate system.

[0036] For each video frame, using the established linear scaling model, the corresponding ENU coordinates are calculated from the pixel coordinates of the video frame. These ENU coordinates and a timestamp are then encapsulated into a metadata file. Each video frame corresponds to a precise timestamp (usually in seconds or milliseconds), which can be obtained from the frame rate in the video player or other synchronization devices. FFmpeg is used to embed the generated metadata into the original video file, outputting an FFmpeg format video file, thus achieving effective fusion of video data and metadata.

[0037] The fusion of audio data and metadata is similar to video fusion, except that audio associates spatial coordinates with time points rather than frames. FFmpeg is used to embed ENU coordinates and timestamps as metadata into the audio data. Through the mapping relationship between the timeline and ENU coordinates, an audio file with metadata is generated. First, the audio file is read to obtain the audio duration, and ENU coordinates are generated for each time point based on a set time interval (e.g., every second, every 0.5 seconds, etc.). The timestamps and corresponding ENU coordinates are written to a metadata file. FFmpeg is then used to merge the generated metadata file and the original audio file, outputting an audio file in FFmpeg format.

[0038] Subsequently, in box S104, video stream, audio stream, and metadata stream are extracted from the FFmpeg format media file to generate a scene graph stream. The video stream, audio stream, and scene graph stream are then synchronized and segmented to output time-aligned window data.

[0039] FFmpeg or other tools that support metadata reading can be used to parse video, audio, and metadata streams from FFmpeg format media files. The video stream is traversed, and a pre-trained computer vision model is used to identify objects and their relationships in each frame. Objects are treated as nodes, and the relationships between objects are treated as edges to generate a scene graph. For example, a VGT model based on a graph neural network can be used to process each frame of video, identify objects and their relationships, generate a scene graph, and mark the ENU coordinates and timestamps in the scene graph to obtain a time-stamped scene graph stream.

[0040] Next, the multimodal data streams are aligned and windowed. A fixed time window can be defined, obtaining a common time range from the metadata of all data streams. The entire time range is traversed, extracting video, audio, and scene graph data within the current time window. The completeness of the data within the window is checked; if incomplete, it is completed through interpolation or finding the nearest neighbor data. The video stream, audio stream, scene graph stream, and their timestamps and ENU coordinates for each time window are encapsulated, outputting time-aligned window data. For example, video, audio, scene graph, and metadata can be encapsulated into a structure containing four parts: 'V': Contains the video data of the current window; 'A': Contains the audio data of the current window; 'G': Contains the scene graph data of the current window; 'meta': Contains the metadata of the current window.

[0041] Next, in box S106, the time-aligned window data is preprocessed and then sequentially input into the multimodal fusion Transformer model and the energy function-based Transformer model, outputting a scene token vector.

[0042] According to one embodiment of this disclosure, a bimodal Transformer subsystem can combine information from different modalities to better understand and generate scenes. First, time-aligned window data is preprocessed and then input into a multimodal fusion Transformer model for semantic encoding, outputting a scene state token. The scene state token includes a video token, an audio token, a scene graph token, spatiotemporal encoding, and a quality / confidence mask.

[0043] In the preprocessing stage, feature extraction is performed on each window of input data, processing video, audio, and scene graph data separately. Video processing includes techniques such as denoising, dehazing, and refraction calibration to eliminate environmental interference and optical distortion, thereby enhancing the image signal-to-noise ratio and ensuring high usability and quality even in complex environments such as fog and low light. Visual features are extracted using techniques like ViT (Visual Transformer) or Conv (Convolutional Neural Network). Audio features are extracted using AST or CNN-1D. Node and edge attributes of the scene graph are normalized. Spatiotemporal location coding is constructed based on ENU coordinates, timestamps, and sensor poses.

[0044] The compact scene representation obtained after multimodal fusion typically contains 1-8 tokens, representing the main entities, layout, dynamics, and uncertainties in the scene. An auxiliary head (optional) can include tasks such as object tracking, relationship reasoning, sound source localization, and semantic understanding. The model fuses data from different modalities and generates a unified scene state representation through the Transformer's attention mechanism.

[0045] Construct an energy function to evaluate the rationality of scene state tokens, which includes: taking the average of scene state tokens in the time or space dimension to generate a pooled scene state vector, using a multilayer perceptron to score the pooled scene state vector, and outputting a scalar energy value. The lower the scalar energy value, the more rational the scene is.

[0046] Suppose the model receives a batch of data with shape (B, N, d), where B is the batch size, representing multiple scene samples processed simultaneously. N is the number of tokens in each scene, such as the number of videos, audios, ENUs, timestamps, etc. d is the dimension of each token, representing the length of the feature vector of a single token. Pooling these tokens primarily aims to aggregate all tokens in each scene (i.e., each sample) into a single representation. The pooling method is typically mean pooling of the tokens, where each scene is compressed into a vector of dimension d.

[0047] The pooled vectors are then passed to a multilayer perceptron (MLP) for scoring. An MLP is a simple feedforward neural network consisting of multiple fully connected layers and activation functions. The ReLU activation function can be used to enhance the model's non-linear representation capabilities. ReLU activation sets negative values ​​to 0, making the model's output sparser, meaning most features are zero. This sparsity helps the model generalize better. After pooling and MLP scoring, the final output is a scalar value representing the reasonableness of the scene. A low energy value indicates that the scene is coherent and reasonable. This is because a low energy value means that the relationships and states between tokens are as expected, and the model considers the scene appropriate and conflict-free. A high energy value indicates that the scene is inconsistent or unreasonable. A high energy value means that there is inconsistency between tokens, and the model considers the scene to contain contradictory, illogical, or unnatural elements.

[0048] The scene state tokens are input into an energy-function-based Transformer model. The model iteratively optimizes the scene state tokens to minimize the energy function, outputting a scene token vector. The training process of the energy-function-based Transformer model mainly includes: Initializing scene tokens: Randomly initializing or using noisy tokens. Forward propagation: Updating the token representation through the EBT model and calculating the energy value. Loss calculation: The loss function includes reconstruction loss and energy loss. The reconstruction loss constrains the accuracy of scene reconstruction, while the energy loss constrains the rationality of the scene. Backpropagation and optimization: Optimizing model parameters through backpropagation. After training, the trained model can be used for inference. During inference, a scene state token is input, and the model iteratively adjusts these tokens to minimize the energy function, outputting a scene token vector. Through online optimization and incremental updates during the inference phase, the system can adjust and optimize the 3D scene in real time over time, adapting to changes in the dynamic environment (e.g., moving objects, changing lighting conditions).

[0049] Finally, in box S108, the scene token vector is converted into an octree data structure, and the octree data structure is input into the decoder to generate a 3D scene map of the seabed.

[0050] An octree is a hierarchical data structure that allows for efficient partitioning of 3D space, effectively representing sparse 3D space, and creating nodes only in regions with data, avoiding memory waste. Specifically, an empty octree is initialized, a root bounding box is defined, each scene token vector is traversed, and the feature vector of each token is inserted into the corresponding node of the octree, returning the constructed octree. For each scene token vector, its spatial location and multimodal feature data are extracted. The spatial location of the token is represented by ENU (East, North, Up) coordinates, serving as the (x, y, z) coordinates in the octree. The token's feature embeddings (e.g., video, audio, timestamps, etc.) are extracted and concatenated into a feature vector. This feature data will be used to describe the scene content at that location. In the octree, a node represents a region in 3D space. Based on the spatial location of the token, the space is recursively partitioned until a suitable leaf node is found, and the corresponding feature vector is inserted into that node. After constructing the octrees for all tokens, the constructed octree structure is returned.

[0051] A pre-trained multimodal octree decoder model is loaded, processing the octree from the root node to the leaf nodes. At each node, the stored feature vector is used to predict the scene attributes of that node. If the final scene representation is an implicit field, this implicit field is used to generate the visible scene. The implicit field is extracted as a mesh and rendered into the final 3D scene graph. The decoder is a neural network model, such as a 3D CNN or Transformer-based model. These models are trained to process octree data and learn how to interpret features in 3D space. The decoder processes the octree layer by layer, starting from the root node. The feature vector of each node provides scene information at that node's location. As the tree deepens, the decoder can predict scene details with increasing precision. The decoder's output is the final scene representation, typically an implicit field. If the decoder output is an implicit field, 3D geometry (e.g., mesh, point cloud) can be generated using SDF. The generated geometry is then converted into a visualized 3D scene using rendering techniques, ultimately generating a visualized 3D scene graph.

[0052] Figure 2 This is a schematic diagram of the structure of an AI system for reconstructing underwater scenes according to an embodiment of this disclosure. (Refer to...) Figure 2 As shown, the entire system is divided into data input, preprocessing, fusion, and reconstruction. The main data sources for system input include: Video data: providing visual information, typically seabed video frames captured by underwater cameras. Features are extracted from images in the video for subsequent processing. Audio spectrograms: audio data collected by underwater sonar or sensors is converted into audio spectrograms. These spectrograms provide sound characteristics of interaction with the seabed environment, such as the presence of organisms and objects. Metadata: including virtual location (e.g., ENU coordinates) and timestamps. Virtual location information helps to define the spatiotemporal relationships of the data, and timestamps are used to synchronize video, audio, and other sensor data. Data preprocessing and organization are performed through the following steps: Video frame feature extraction: visual features are extracted from video frames using a convolutional neural network (CNN). Audio spectrogram processing: spectral information is extracted from underwater audio data for analyzing environmental sounds (e.g., submersibles, marine life, etc.).

[0053] In the process of reconstructing the underwater scene, it is necessary to map the virtual location to a physical geographic coordinate system to ensure that the reconstructed scene corresponds to the actual environment. Timestamps are used to synchronize the virtual coordinate data with other sensor data in time and space to ensure data consistency in both space and time.

[0054] The system fuses the input multimodal data (video, audio, graph structure data, ENU coordinates and timestamps) to generate a high-dimensional spatial data representation. The purpose of this part is to better organize the data through a scene graph, facilitating subsequent model training and reconstruction. Based on the scene relationships obtained from the input data, a graph structure is constructed to represent the relationships between elements in the underwater scene (e.g., object adjacency, motion paths, etc.). Information from different modalities (video, audio, graph structure data) is fused to generate a unified high-dimensional spatial representation.

[0055] The Multimodal Fusion Transformer (MM-TR) is the core module of the system, focusing on the perceptual alignment of multimodal data to ensure collaborative work between different modalities (video, audio, and spatiotemporal data). The input layer includes video data: CNN features extracted from video frames; audio spectrograms: audio features transformed through time and frequency domains; graph structure data: graph structure data representing scene relationships or object relationships; and spatiotemporal data: including ENU coordinates and corresponding timestamps, providing spatiotemporal reference. A self-attention mechanism establishes relationships between different modalities, enabling video, audio, and graph data to work together. For example, a dedicated encoder is designed for spatiotemporal data (ENU coordinates + timestamps), allowing the system to adaptively adjust in different times and spaces. Based on the contributions of different modalities, a gating mechanism dynamically adjusts the weights of each modality to ensure optimal data fusion. A 128-256 dimensional scene state annotation vector is generated, carrying a compressed semantic environment representation that summarizes the main features of the entire scene, providing a semantic foundation for subsequent reconstruction and analysis.

[0056] The goal of the Energy-Function-Based Transformer Model (EBT-TR) is to transform scene state annotation vectors into physically plausible seabed scenes. A direct energy head can be designed to handle the scene generation process, ensuring the physical plausibility of the generated scene. Scene tokens are pooled, converted into a vector, and a scalar energy is output through an MLP or linear layer. This energy is used to constrain the consistency between the generated scene and the physical structure. The input to EBT-TR comes from the scene tokens (representing objects, elements, etc. in the scene) of the MM-TR and the designed energy function. Output: The generated scene tokens contain information such as location, attributes, objects, and elements, ultimately generating a scene that conforms to physical constraints and structural consistency.

[0057] The decoder / generator module aims to decode the EBT-TR optimized scene tokens into a complete 3D underwater environment for further display or application. It converts the generated scene tokens into 3D meshes or point cloud models, enabling visualization of the underwater environment in 3D space. It assigns physical properties (such as density, material type, reflectivity, etc.) to each object or element in the scene to ensure that the generated scene not only meets aesthetic requirements but also possesses a certain degree of physical plausibility.

[0058] Figure 3 This is a schematic block diagram of a seabed scene reconstruction apparatus based on multimodal data processing according to embodiments of the present disclosure. Figure 3 As shown, the device 300 may include a processor 310 and a memory 320 storing a computer program. When the computer program is executed by the processor 310, the device 300 is made capable of performing actions such as... Figure 1 The steps of the method are shown. In one example, device 300 can be a computer device or a cloud computing node. Device 300 can acquire video data, audio data, and spatiotemporal data collected by underwater equipment, embed metadata containing ENU coordinates and timestamps into the video and audio data, and output a media file in FFmpeg format; extract the video stream, audio stream, and metadata stream from the FFmpeg format media file to generate a scene graph stream, and synchronize and segment the video stream, audio stream, and scene graph stream to output time-aligned window data; after preprocessing the time-aligned window data, input it sequentially into a multimodal fusion Transformer model and an energy function-based Transformer model to output a scene token vector; convert the scene token vector into an octree data structure, and input the octree data structure into a decoder to generate a 3D scene graph of the seabed.

[0059] In the embodiments of this disclosure, the device 300 can acquire video data, audio data, virtual location coordinates, and timestamps collected by underwater equipment, and map the virtual location coordinates to ENU coordinates; use FFmpeg to embed the ENU coordinates and timestamps as metadata into the video data, and generate a video file with metadata through the mapping relationship between pixel coordinates and ENU coordinates; use FFmpeg to embed the ENU coordinates and timestamps as metadata into the audio data, and generate an audio file with metadata through the mapping relationship between the timeline and ENU coordinates.

[0060] In the embodiments of this disclosure, the device 300 can obtain the latitude and longitude of the region center, set the origin of the ENU coordinate system on the water surface, calculate the height of the water surface origin, subtract the water depth from the height of the water surface origin to obtain the actual height of the underwater region as the upward coordinate of the underwater region, obtain the eastward and northward local coordinates of each object in the cluster relative to the region center, calculate the upward coordinate of each object according to the water depth, and obtain the final ENU coordinates of each object.

[0061] In the embodiments of this disclosure, the device 300 can obtain the field of view and shooting angle of the underwater camera through calibration, establish a linear scaling model, and determine the mapping relationship between the pixel coordinates and the actual ENU coordinates in each video frame; for each video frame, the established linear scaling model is used to calculate the corresponding ENU coordinates through the pixel coordinates of the video frame, and the corresponding ENU coordinates and timestamps are encapsulated into the metadata file; FFmpeg is used to embed the generated metadata into the original video file for encapsulation, and output the FFmpeg format video file.

[0062] In the embodiments of this disclosure, the device 300 can read the audio file to obtain the audio duration, generate corresponding ENU coordinates for each time point based on a set time interval, write the timestamp and the corresponding ENU coordinates into a metadata file, and use FFmpeg to output an audio file in FFmpeg format using the generated metadata and the original audio file.

[0063] In the embodiments of this disclosure, the device 300 can use the ffprobe tool to parse video streams, audio streams, and metadata streams from FFmpeg format media files; traverse the video stream, use a pre-trained computer vision model to identify objects and their relationships in each frame of the video stream, treat objects as nodes, and use the relationships between objects as edges to generate a scene graph; mark the ENU coordinates and timestamps on the scene graph to obtain a scene graph stream with timestamps; define a fixed time window and obtain a common time range from the metadata of all data streams; traverse the entire time range, extract video, audio, and scene graph data within the current time window, check whether the data within the window is complete, and if not, complete it by interpolation or finding the nearest neighbor data; encapsulate the video stream, audio stream, scene graph stream, and their timestamps and ENU coordinates of each time window, and output time-aligned window data.

[0064] In embodiments of this disclosure, device 300 can preprocess time-aligned window data and input it into a multimodal fusion Transformer model for semantic encoding, outputting a scene state token. The scene state token includes a video token, an audio token, a scene graph token, spatiotemporal encoding, and a quality / confidence mask. An energy function is constructed to evaluate the rationality of the scene state token, including: averaging the scene state token over time or space to generate a pooled scene state vector; using a multilayer perceptron to score the pooled scene state vector and outputting a scalar energy value, where a lower scalar energy value indicates a more rational scene; inputting the scene state token into the energy function-based Transformer model, iteratively optimizing the scene state token to minimize the energy function, and outputting a scene token vector.

[0065] In embodiments of this disclosure, device 300 can initialize an empty octree, define a root bounding box, traverse each scene token vector, insert the feature vector of each token into the corresponding node of the octree, and return the constructed octree; load a pre-trained multimodal octree decoder model, process the octree from the root node to the leaf node, at each node, use the stored feature vector to predict the scene attributes of the node, if the final scene representation is an implicit field, use the implicit field to generate a visible scene, extract the implicit field into a mesh and render it into the final 3D scene graph.

[0066] In embodiments of this disclosure, processor 310 may be, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a processor based on a multi-core processor architecture, etc. Memory 320 may be any type of memory implemented using data storage technologies, including but not limited to random access memory, read-only memory, semiconductor-based memory, flash memory, disk storage, etc.

[0067] Furthermore, in embodiments of this disclosure, device 300 may also include input device 330, such as a keyboard, mouse, etc. Additionally, device 300 may also include output device 340, such as a display, etc.

[0068] In other embodiments of this disclosure, a computer-readable storage medium storing a computer program is also provided, wherein the computer program, when executed by a processor, is capable of performing the following functions: Figure 1 The steps of the seabed scene reconstruction method based on multimodal data processing are shown.

[0069] In summary, the seabed scene reconstruction method and apparatus based on multimodal data processing according to embodiments of this disclosure, by combining video, audio spectra, graph structure data, and spatiotemporal information (ENU coordinates and timestamps), can more comprehensively perceive complex seabed environments. Through the fusion of multimodal data and optimization of energy functions, it can generate three-dimensional scenes that conform to physical and geometric laws. In the storage and retrieval of 3D scenes, the use of an octree structure greatly improves the storage efficiency and retrieval speed of large-scale scene data. Furthermore, the modular design of multi-stage AI models enhances the scalability and flexibility of the system.

[0070] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses and methods according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0071] Unless otherwise expressly indicated by the context, the singular form of words used herein and in the appended claims includes the plural form, and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the terms “comprising” and “including” shall be interpreted as including rather than exclusively. Likewise, the terms “including” and “or” shall be interpreted as including unless such interpretation is expressly prohibited herein. Where the term “example” is used herein, particularly when it follows a set of terms, “example” is merely exemplary and illustrative and should not be considered exclusive or extensive.

[0072] Further aspects and scope of adaptation become apparent from the description provided herein. It should be understood that various aspects of this application may be implemented individually or in combination with one or more other aspects. It should also be understood that the descriptions and specific embodiments herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0073] Several embodiments of this disclosure have been described in detail above. However, it is obvious that those skilled in the art can make various modifications and variations to the embodiments of this disclosure without departing from the spirit and scope of this disclosure. The scope of protection of this disclosure is defined by the appended claims.

Claims

1. A method for reconstructing seabed scenes based on multimodal data processing, characterized in that, The method includes: Acquire video data, audio data, and spatiotemporal data collected by underwater equipment, embed metadata containing ENU coordinates and timestamps into the video and audio data, and output media files in FFmpeg format; Extract video stream, audio stream, and metadata stream from the FFmpeg format media file, generate scene graph stream, synchronize and segment the video stream, audio stream, and scene graph stream, and output time-aligned window data; The time-aligned window data is preprocessed and then sequentially input into a multimodal fusion Transformer model and an energy function-based Transformer model, outputting a scene token vector; and The scene token vector is converted into an octree data structure, and the octree data structure is input into the decoder to generate a 3D scene map of the seabed.

2. The method for reconstructing seabed scenes based on multimodal data processing according to claim 1, characterized in that, The process of acquiring video data, audio data, and spatiotemporal data collected by underwater equipment, embedding metadata including ENU coordinates and timestamps into the video and audio data, and outputting FFmpeg format media files includes: Acquire video data, audio data, virtual location coordinates, and timestamps collected by underwater equipment, and map the virtual location coordinates to ENU coordinates; Using FFmpeg, the ENU coordinates and timestamps are embedded as metadata into the video data. By mapping the pixel coordinates and ENU coordinates, a video file with metadata is generated. Using FFmpeg, the ENU coordinates and timestamps are embedded as metadata into the audio data. Through the mapping relationship between the timeline and the ENU coordinates, an audio file with metadata is generated.

3. The method for reconstructing seabed scenes based on multimodal data processing according to claim 2, characterized in that, The step of mapping the virtual location coordinates to ENU coordinates includes: Obtain the latitude and longitude of the region center, set the origin of the ENU coordinate system on the water surface, and calculate the height of the water surface origin. Subtracting the water depth from the height of the origin point on the water surface, we obtain the actual height of the underwater region as the upward coordinate of the underwater region; Obtain the local east and north coordinates of each object in the cluster relative to the center of the region, calculate the upward coordinates of each object based on the water depth, and obtain the final ENU coordinates of each object.

4. The method for reconstructing seabed scenes based on multimodal data processing according to claim 2, characterized in that, The step of using FFmpeg to embed the ENU coordinates and timestamps as metadata into the video data, and generating a video file with metadata through the mapping relationship between pixel coordinates and ENU coordinates, includes: By calibrating and obtaining the field of view and shooting angle of the underwater camera, a linear scaling model is established to determine the mapping relationship between the pixel coordinates in each video frame and the actual ENU coordinates. For each video frame, the established linear scaling model is used to calculate the corresponding ENU coordinates from the pixel coordinates of the video frame, and the corresponding ENU coordinates and timestamps are encapsulated into the metadata file. FFmpeg is used to embed the generated metadata into the original video file for encapsulation, and output a video file in FFmpeg format.

5. The method for reconstructing seabed scenes based on multimodal data processing according to claim 2, characterized in that, The step of using FFmpeg to embed the ENU coordinates and timestamps as metadata into the audio data, and generating an audio file with metadata through the mapping relationship between the timeline and ENU coordinates, includes: Read the audio file to obtain the audio duration, generate corresponding ENU coordinates for each time point based on the set time interval, and write the timestamp and corresponding ENU coordinates into a metadata file; Use FFmpeg to generate metadata and the original audio file, and output an audio file in FFmpeg format.

6. The method for reconstructing seabed scenes based on multimodal data processing according to claim 1, characterized in that, The process of extracting video stream, audio stream, and metadata stream from the FFmpeg format media file, generating a scene graph stream, synchronizing and segmenting the video stream, audio stream, and scene graph stream, and outputting time-aligned window data includes: Use the ffprobe tool to parse video streams, audio streams, and metadata streams from FFmpeg format media files; Traverse the video stream, use a pre-trained computer vision model to identify objects and their relationships in each frame of the video stream, and generate a scene graph by treating objects as nodes and the relationships between objects as edges. The ENU coordinates and timestamps are marked on the scene graph to obtain a scene graph flow with timestamps; Define a fixed time window and obtain a common time range from the metadata of all data streams; Iterate through the entire time range, extract video, audio and scene graph data within the current time window, check if the data within the window is complete, and if not, complete it by interpolation or by finding the nearest neighbor data. Encapsulate the video stream, audio stream, scene graph stream, timestamps, and ENU coordinates for each time window, and output time-aligned window data.

7. The method for reconstructing seabed scenes based on multimodal data processing according to claim 1, characterized in that, The time-aligned window data, after preprocessing, is sequentially input into the multimodal fusion Transformer model and the energy function-based Transformer model, and the output scene token vector includes: The time-aligned window data is preprocessed and then input into a multimodal fusion Transformer model for semantic encoding, outputting a scene state token. The scene state token includes a video token, an audio token, a scene graph token, spatiotemporal encoding, and a quality / confidence mask. Constructing an energy function to evaluate the rationality of the scene state token includes: taking the average of the scene state token in the time or space dimension, generating a pooled scene state vector, using a multilayer perceptron to score the pooled scene state vector, and outputting a scalar energy value, wherein the lower the scalar energy value, the more rational the scene is; The scene state token is input into the Transformer model based on the energy function, and the scene state token is iteratively optimized to minimize the energy function, and the scene token vector is output.

8. The method for reconstructing seabed scenes based on multimodal data processing according to claim 1, characterized in that, The step of converting the scene token vector into an octree data structure and inputting the octree data structure into the decoder to generate a 3D scene map of the seabed includes: Initialize an empty octree, define the root bounding box, traverse each scene token vector, insert the feature vector of each token into the corresponding node of the octree, and return the constructed octree. Load a pre-trained multimodal octree decoder model, process the octree from the root node to the leaf node, at each node, use the stored feature vector to predict the scene attributes of that node, if the final scene representation is an implicit field, use the implicit field to generate the visible scene, extract the implicit field as a mesh and render it into the final 3D scene graph.

9. A device for reconstructing seabed scenes based on multimodal data processing, characterized in that, The device includes: At least one processor; and At least one memory storing a computer program; When the computer program is executed by the at least one processor, the device performs the steps of the seabed scene reconstruction method based on multimodal data processing according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the seabed scene reconstruction method based on multimodal data processing according to any one of claims 1 to 8.

Images