Seabed radiation field reconstruction method based on multi-modal sensing method

Abstract

The invention discloses a seabed radiation field reconstruction method based on a multi-mode sensing method, and the method comprises the steps: inputting the dense point cloud data of a seabed environment obtained through the real-time scanning of a laser radar into a spatial distribution filtering algorithm, and carrying out the outlier removal, and obtaining the denoised dense point cloud data; obtaining enhanced RGB image data; defining each point in the obtained de-noised dense point cloud data as a radiation field primitive, projecting the enhanced RGB image data on the radiation field primitive in a corresponding pose, enabling the radiation field primitive to obtain related parameter expression, and then carrying out an initial representation process of a radiation field; inputting the relevant parameters of the radiation field primitives into a parallel sequence fusion rendering algorithm to quickly render to obtain corresponding rendered images; and comparing the rendered image with a corresponding original RGB image to carry out loss calculation and back propagation, updating radiation field primitive related parameters, and updating de-noised dense point cloud data by adopting an active space compensation algorithm at the same time, thereby realizing real-time rendering and efficient reconstruction for the seabed environment.

Description

Technical Field

[0001] The present invention relates to the field of multimodal fusion perception and three-dimensional reconstruction, and more specifically, to a method for reconstructing seabed radiation fields based on a multimodal perception method. Background Art

[0002] Accurate perception and reconstruction of the seabed environment is fundamental to the systematization and intelligentization of underwater equipment. Currently, underwater perception and reconstruction technology primarily relies on devices such as underwater cameras, sonar, and lidar. Underwater cameras are advantageous in capturing detailed local texture information, but are sensitive to illumination and have a limited detection range. Sonar, with its strong directionality and concentrated energy, can capture coarse-grained, long-range information, but its point cloud data has low resolution and is subject to significant noise. Lidar can provide high-precision three-dimensional point cloud data, but is susceptible to the refractive index and scattering effects of water. Currently, these devices alone are often unable to meet the demand for high-precision reconstruction of the seabed environment.

[0003] To address this situation, the seabed radiation field reconstruction technology based on multimodal sensing units is used to integrate the work of underwater cameras and lidars, and a radiation field-guided data fusion and iterative optimization strategy is established to achieve high-precision, wide-range visual perception of the seabed environment, improve the accuracy, completeness and efficiency of underwater operation equipment in obtaining seabed environmental information, and thus meet the reconstruction needs of the seabed environment. Summary of the Invention

[0004] In response to the technical problems raised above, a method for reconstructing the seabed radiation field based on a multimodal perception method is provided. The present invention mainly utilizes a multimodal integrated perception unit based on an underwater camera and a lidar to realize data integration of RGB images and dense point clouds guided by seabed environment reconstruction, and establishes the fusion and iterative optimization of radiation field guided data to achieve real-time rendering and efficient reconstruction of the seabed environment.

[0005] The technical means adopted in the present invention are as follows:

[0006] A method for reconstructing seabed radiation fields based on a multimodal sensing method, comprising:

[0007] S1. Input the dense point cloud data of the seabed environment obtained by real-time scanning of the lidar into the spatial distribution filtering algorithm to remove outliers and obtain denoised dense point cloud data;

[0008] S2, inputting the RGB image data of the seabed environment acquired by the optical camera in real time into the single-scale retina algorithm for deblurring and enhancement to obtain enhanced RGB image data;

[0009] S3, defining each point in the denoised dense point cloud obtained in S1 as a radiation field primitive, and projecting the enhanced RGB image in S2 onto the radiation field primitive with its corresponding posture, so that the radiation field primitive obtains relevant parameter expression, thereby realizing the initial representation of the radiation field;

[0010] S4, inputting the relevant parameters of the radiation field primitives in S3 into the parallel sequence fusion rendering algorithm for rapid rendering to obtain the corresponding rendered image;

[0011] S5. Compare the rendered image in S4 with the corresponding original RGB image to calculate the loss and perform backpropagation. Update the relevant parameters of the radiation field primitives in S3. At the same time, use the active spatial compensation algorithm to update the denoised dense point cloud data in S1, and then update the radiation field representation to achieve real-time rendering and efficient reconstruction of the seabed environment.

[0012] Furthermore, the spatial distribution filtering algorithm performs field statistics on each point in the point cloud data of the environmental space, calculates the average value of the point's distance from its multiple neighboring points, and assumes that the result obeys a Gaussian distribution defined by the mean and standard deviation. The outliers outside the threshold distance expressed by the mean are removed, thereby achieving denoising of underwater dense point cloud data.

[0013] Furthermore, the single-scale retinal algorithm performs deblurring and edge enhancement processing on the sequentially collected RGB images, mainly enhancing high-frequency components, thereby achieving underwater RGB image enhancement.

[0014] Furthermore, the radiation field primitive is based on a centralized three-dimensional Gaussian, and uses its mean vector and covariance matrix to respectively characterize the coordinate position and morphological direction of the radiation field primitive, and introduces opacity to achieve the superposition of diffusion traces for subsequent rendering.

[0015] Furthermore, the initial representation of the radiation field is initially composed of numerous radiation field primitives, whose spatial arrangement is determined by the de-noised dense point cloud data. The overall reconstruction and update of the radiation field is achieved through iterative optimization of a large number of parameter-definable radiation field primitives. The enhanced RGB image is used to project the radiation field primitives pixel by pixel at their corresponding positions. This projection process combines the opacity information of each radiation field primitive with a superposition of diffusion traces to achieve the initial representation of the radiation field.

[0016] Furthermore, the parallel sequence fusion rendering algorithm adopts a parallel computing strategy based on spatial block division and a sequence rendering execution strategy based on depth echelons, achieving efficient computing power division and execution while ensuring the correct rendering order.

[0017] Furthermore, the active spatial compensation algorithm analyzes the radiation field primitives at regular intervals, defines under-reconstruction and over-reconstruction areas according to relevant parameter thresholds, and clones and eliminates radiation field primitives in a targeted manner, thereby achieving compensation and adaptive correction for spatial deficiencies in the radiation field.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] 1. The present invention provides a method for reconstructing the seabed radiation field based on a multimodal perception method. It realizes data integration of RGB images and dense point clouds guided by seabed environment reconstruction through a multimodal integrated perception unit based on underwater cameras and lidar, and establishes the fusion and iterative optimization of radiation field guidance data to achieve real-time rendering and efficient reconstruction of the seabed environment.

[0020] 2. By introducing corresponding radiation field primitives and their related mean vectors, covariance matrices, RGB values, opacity and other data for the seabed environment, the radiation field space of the seabed scene is jointly managed and represented to achieve the digitization and visualization of the seabed environment.

[0021] 3. De-noising and enhancement of multimodal perception unit data input are achieved through a single-scale retinal algorithm and a spatial distribution filtering algorithm. Adaptive optimization and efficiency enhancement of the system are achieved through a parallel sequence fusion rendering algorithm and an active spatial compensation algorithm, solving problems such as distortion, blurring, and poor consistency in underwater multi-source data reconstruction.

[0022] In summary, the technical solution of the present invention addresses the problems of poor consistency in underwater multimodal data fusion reconstruction, weak visualization capabilities for complex environments, lack of full digital representation of the seafloor, and distortion and blurring in underwater reconstruction in existing underwater multimodal fusion perception and 3D reconstruction technologies. This solution achieves efficient visualization and digital reconstruction of the seafloor environment based on a multimodal perception unit. Therefore, the technical solution of the present invention solves the problems of seafloor reconstruction based on multimodal fusion perception in the existing technology.

[0023] Based on the above reasons, the present invention can be widely promoted in the field of multimodal fusion perception and three-dimensional reconstruction technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1This is a flow chart of a method for reconstructing seabed radiation fields based on a multimodal sensing method according to the present invention.

[0026] Figure 2 This is a schematic diagram of the network structure of a method for reconstructing seabed radiation fields based on a multimodal sensing method of the present invention.

[0027] Figure 3 This is a schematic diagram of initializing rendering of a reconstruction target in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] like Figure 1 As shown, the present invention provides a method for reconstructing seabed radiation fields based on a multimodal sensing method, which is characterized by comprising:

[0031] S1. Input the dense point cloud data of the seabed environment obtained by real-time scanning of the lidar into the spatial distribution filtering algorithm to remove outliers and obtain denoised dense point cloud data;

[0032] In specific implementation, as a preferred embodiment of the present invention, the spatial filtering algorithm performs field statistics on each point in the point cloud data of the environmental space, calculates the average value of the point's distance from its multiple neighboring points, and assumes that the result obeys a Gaussian distribution defined by the mean and standard deviation. The outliers outside the threshold distance expressed by the mean are removed, thereby achieving denoising of underwater dense point cloud data.

[0033] S2, inputting the RGB image data of the seabed environment acquired by the optical camera in real time into the single-scale retina algorithm for deblurring and enhancement to obtain enhanced RGB image data;

[0034] In specific implementation, as a preferred embodiment of the present invention, the optical camera should be integrated with the laser radar in S1 to form an underwater multimodal optical perception unit and integrated into a pod module, which is hoisted into an underwater operating equipment with good maneuverability and optical search capabilities to perform perception and reconstruction tasks;

[0035] In specific implementation, as a preferred embodiment of the present invention, the single-scale retinal algorithm performs deblurring and edge enhancement processing on the RGB images collected in sequence, mainly enhancing the high-frequency components, thereby achieving underwater RGB image enhancement.

[0036] S3, defining each point in the denoised dense point cloud obtained in S1 as a radiation field primitive, and projecting the enhanced RGB image in S2 onto the radiation field primitive with its corresponding posture, so that the radiation field primitive obtains relevant parameter expression, thereby realizing the initial representation of the radiation field;

[0037] In a specific implementation, as a preferred embodiment of the present invention, the radiation field primitive is based on a centralized three-dimensional Gaussian, and its mean vector and covariance matrix are used to respectively characterize the coordinate position and morphological direction of the radiation field primitive, and opacity is introduced to achieve the superposition of diffusion traces for subsequent rendering;

[0038] The mean vector is represented by coordinate data within the denoised dense point cloud data;

[0039] The covariance matrix can be decomposed into a scaling matrix and a rotation matrix. By optimizing the scaling matrix and the scale matrix during the iterative optimization process, geometric parameters such as the shape, volume, and direction of the three-dimensional Gaussian can be defined, thereby achieving the correct characterization of the radiation field primitives.

[0040] The opacity is used in the subsequent rendering stage, and when the enhanced RGB image is projected onto the radiation field primitive, its diffusion trace is superimposed through the opacity;

[0041] The initial representation of the radiation field is initially composed of numerous radiation field primitives, whose spatial arrangement is determined by the denoised dense point cloud data. The overall reconstruction and update of the radiation field is achieved through iterative optimization of a large number of parameter-definable radiation field primitives. The enhanced RGB image is used to project the radiation field primitives pixel by pixel at their corresponding positions. The projection process combines the opacity information of each radiation field primitive with a superposition of diffusion traces to achieve the initial representation of the radiation field.

[0042] The projection of the enhanced RGB image on the radiation field primitives, that is, each radiation field primitive in the space leaves a diffusion trace on the image plane due to the projection and gradually superimposes, thereby realizing the progressive generation of a new perspective image.

[0043] S4, inputting the relevant parameters of the radiation field primitives in S3 into the parallel sequence fusion rendering algorithm for rapid rendering to obtain the corresponding rendered image;

[0044] In specific implementation, as a preferred embodiment of the present invention, the parallel sequence fusion rendering algorithm adopts a parallel computing strategy based on spatial block division and a sequential rendering execution strategy based on depth echelons, achieving efficient computing power division and execution while ensuring the correct rendering order;

[0045] The parallel computing strategy divides the rendered image into several blocks with independent threads, and each block selects the radiation field primitives within the viewing cone range for calculation, thereby achieving efficient computing power division and parallel operation management;

[0046] The depth-tiered sequential rendering execution strategy selects radiation field primitives within the viewing cone with an execution degree greater than 99% and instantiates them into three-dimensional Gaussian objects containing the corresponding viewing depth. The three-dimensional Gaussian objects are sorted in a tiered manner and rendered from near to far, thereby achieving rendering of reconstructed targets in the space in the correct order when there is an occlusion relationship.

[0047] S5: Compare the rendered image in S4 with the corresponding original RGB image to calculate the loss and perform backpropagation, update the relevant parameters of the radiation field primitives in S3, and use the active spatial compensation algorithm to update the denoised dense point cloud data in S1, thereby updating the radiation field representation to achieve real-time rendering and efficient reconstruction of the seabed environment;

[0048] In a specific implementation, as a preferred embodiment of the present invention, the loss calculation is to calculate the photometric error of the rendered image compared with the original RGB image, and add the structural similarity error (SSIM) in proportion;

[0049] The active spatial compensation algorithm analyzes the radiation field primitives at regular intervals, defines under-reconstruction and over-reconstruction areas based on relevant parameter thresholds, and clones and eliminates radiation field primitives in a targeted manner, thereby achieving compensation and adaptive correction for spatial missing radiation fields, and thus completely fitting the seabed scene.

[0050] Example

[0051] like Figure 1 The present invention provides a method for reconstructing seabed radiation fields based on a multimodal sensing method; Figure 2 As shown in FIG, the present invention provides a network structure diagram of a method for reconstructing the seabed radiation field based on a multimodal sensing method. Figure 3As shown, this embodiment provides a schematic diagram of initialization rendering of a reconstruction target. After completing the early denoised dense point cloud and enhanced RGB image generation tasks for the reconstruction target, each point in the denoised dense point cloud is defined as a radiation field primitive, which then constitutes the radiation field of the reconstruction target. The enhanced RGB image with its corresponding information is projected onto the radiation field primitive to achieve initialization rendering of the reconstruction target. The radiation field representation is then updated in sequence by the subsequent optimization algorithm to achieve real-time rendering and efficient reconstruction of the seabed environment.

[0052] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0053] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0054] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0055] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0056] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0057] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reconstructing seabed radiation fields based on a multimodal sensing method, characterized in that: include: The dense point cloud data of the seabed environment obtained by real-time scanning of the lidar is input into the spatial distribution filtering algorithm to remove outliers and obtain denoised dense point cloud data; The RGB image data of the seabed environment acquired by the optical camera in real time is input into the single-scale retina algorithm for deblurring and enhancement to obtain enhanced RGB image data; Each point in the acquired denoised dense point cloud data is defined as a radiation field primitive. At the same time, the enhanced RGB image data is projected onto the radiation field primitive with its corresponding posture, so that the radiation field primitive obtains relevant parameter expressions, and then the initial radiation field representation model is performed. Inputting the relevant parameters of the radiation field primitives into the parallel sequence fusion rendering algorithm for fast rendering to obtain the corresponding rendered image; The rendered image is compared with the corresponding original RGB image to calculate the loss and perform back propagation, and the relevant parameters of the radiation field primitives are updated. At the same time, an active spatial compensation algorithm is used to update the denoised dense point cloud data, and then the radiation field representation is updated to achieve real-time rendering and efficient reconstruction of the seabed environment.

2. The method for reconstructing the seabed radiation field based on a multimodal sensing method according to claim 1 is characterized by: A spatial distribution filtering algorithm is used to perform field statistics on each point in the point cloud data of the environmental space, and the average value of the distance between the point and its multiple neighboring points is calculated. Assuming that the result obeys a Gaussian distribution defined by the mean and standard deviation, outliers outside the threshold distance expressed by the mean are removed, thereby achieving denoising of underwater dense point cloud data.

3. The method for reconstructing seabed radiation fields based on a multimodal sensing method according to claim 1 is characterized by: A single-scale retinal algorithm is used to perform deblurring and edge enhancement processing on the sequentially collected RGB images, mainly enhancing the high-frequency components, thereby achieving underwater RGB image enhancement.

4. The method for reconstructing seabed radiation fields based on a multimodal sensing method according to claim 1 is characterized by: The radiation field primitive is based on a centralized three-dimensional Gaussian, and uses its mean vector and covariance matrix to respectively characterize the coordinate position and morphological direction of the radiation field primitive, and introduces opacity to achieve the superposition of diffusion traces for subsequent rendering.

5. The method for reconstructing seabed radiation fields based on a multimodal sensing method according to claim 1 is characterized by: The initial state of the radiation field is composed of numerous radiation field primitives, whose spatial arrangement is determined by denoised dense point cloud data. At the same time, the overall reconstruction and update of the radiation field are achieved by iterative optimization of radiation field primitives defined by a large number of parameters. The initial representation mode of the radiation field is as follows: the enhanced RGB image is used to project pixels onto the radiation field primitives with their corresponding poses. The projection process is combined with the opacity information of each radiation field primitive to perform diffusion trace superposition, thereby realizing the initial representation of the radiation field.

6. The method for reconstructing seabed radiation fields based on a multimodal sensing method according to claim 1 is characterized by: When obtaining the rendered image: the parallel sequence fusion rendering algorithm adopts a parallel computing strategy divided by spatial blocks and a sequence rendering execution strategy based on depth echelons, and performs efficient computing power division and execution to obtain the rendered image while ensuring the rendering order is correct.

7. The method for reconstructing seabed radiation fields based on a multimodal sensing method according to claim 1, characterized in that: When using the active spatial compensation algorithm to update the denoised dense point cloud data: the radiation field primitives are analyzed at regular intervals, under-reconstruction and over-reconstruction areas are defined according to relevant parameter thresholds, and radiation field primitives are cloned and eliminated in a targeted manner, thereby compensating for and adaptively correcting the spatial loss of the radiation field.

Images