Water-air cross-medium object three-dimensional modeling method and system based on visual distortion correction

Abstract

The application discloses a water-air cross-medium object three-dimensional modeling method and system based on visual distortion correction, and the method comprises the following steps: acquiring color information, depth information and incident angle information of each pixel of a cross-medium image, and constructing a mathematical equation of an air-water refraction plane; based on each pixel of the cross-medium image, performing refraction distortion compensation and multi-scale phase shift analysis, and constructing an underwater ray equation set, a relationship matrix of pixels and decoding projection coding values; the mathematical equation of the air-water refraction plane determines a water surface normal vector, and the underwater ray equation set and the relationship matrix of the pixels and the decoding projection coding values are combined to perform depth calculation and joint optimization, thereby generating an integrated three-dimensional model. The application can improve the reconstruction efficiency and visual quality of a three-dimensional environment model of an unmanned aerial vehicle. The application can be widely applied to the technical field of three-dimensional modeling as a water-air cross-medium object three-dimensional modeling method and system based on visual distortion correction.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, and in particular to a method and system for 3D modeling of water and air transmedia objects based on visual distortion correction. Background Technology

[0002] 3D reconstruction and depth perception are core technologies in computer vision. In air environments, mature systems based on structured light and surface motion imaging (SFM) have been established, enabling high-precision modeling under stable illumination. However, in cross-medium imaging, the difference in refractive index between air and water causes light deflection, rendering traditional single-viewpoint models ineffective. Ignoring refraction results in severe geometric distortion and scale errors, necessitating the introduction of a refractive geometry model for compensation. Furthermore, underwater environments present challenges such as illumination attenuation and enhanced scattering, leading to weak target texture and low reliability of feature matching. Structured light technology can actively generate textures by projecting encoded patterns, but current research is largely limited to controlled environments, lacking adaptability to dynamic water surface fluctuations, and failing to fully consider the non-single-viewpoint characteristics of the projection-imaging birefringence path.

[0003] While integrated air-sea UAVs possess multi-mode operational capabilities, existing systems largely rely on two-dimensional image acquisition and lack stable three-dimensional modeling capabilities. Furthermore, they suffer from low computational efficiency, a lack of a unified refraction model, and the absence of an end-to-end cross-media modeling system. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention aims to provide a method and system for three-dimensional modeling of water and air cross-medium objects based on visual distortion correction, which can improve the reconstruction efficiency and visual quality of UAV three-dimensional environment models.

[0005] The first technical solution adopted in this invention is: a three-dimensional modeling method for water and air trans-medium objects based on visual distortion correction, comprising the following steps: Obtain the color, depth, and incident angle information of each pixel in the cross-medium image, and construct the mathematical equation of the air-water refraction plane; Based on each pixel of the cross-medium image, refraction distortion compensation and multi-scale phase shift analysis are performed to construct an underwater ray equation set and a matrix relating pixels to decoded projection coding values. The mathematical equations of the air-water refraction plane determine the water surface normal vector. The depth calculation and joint optimization are performed by combining the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values ​​to generate an integrated 3D model.

[0006] Furthermore, the step of acquiring the RGB information, light intensity information, and angle information of each pixel in the cross-medium image, and constructing the mathematical equation for the air-water refraction plane, specifically includes: The cross-medium UAV flies to the target operation area and hovers stably at the corresponding position, initializes the Gaussian scene, projects a pre-coded structured light pattern onto the target operation area, and acquires cross-medium images containing the pre-coded structured light pattern; Calculate the RGB information, light intensity information and angle information of each pixel in the cross-medium image containing the preset coded structured light pattern, and construct a set of empty Gaussian primitive data structures. Based on the geometric features of the air-water refraction interface obtained by the visual sensor, the Gaussian primitive data structure set is regionalized to obtain the segmented Gaussian scene. Invalid observation regions are removed from the segmented Gaussian scene, and the mathematical equations for the air-water refraction plane are constructed.

[0007] Furthermore, the step of performing refractive distortion compensation and multi-scale phase shift analysis based on each pixel of the cross-medium image to construct the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values ​​specifically includes: Based on each pixel of the cross-medium image, reverse ray tracing is performed to obtain the direction of the light rays incident on the water surface for each pixel of the cross-medium image; The intersection point of the ray and the water surface is determined based on the direction of the ray incident on the water surface, and the direction of the refracted ray is calculated using Snell's law; By combining the intersection of light rays and the water surface with the direction of refracted light rays, a set of underwater ray equations is constructed; Refractive distortion compensation is performed on the cross-medium image to obtain the corrected cross-medium image; Multi-scale pyramid decoding and multi-scale phase shift analysis are performed on the corrected cross-media image to obtain the decoded projection coding values ​​and construct the relationship matrix between pixels and decoded projection coding values.

[0008] Furthermore, the step of performing refractive distortion compensation on the trans-medium image to obtain the corrected trans-medium image specifically includes: Based on the RGB information, light intensity information and angle information of each pixel in the cross-medium image, and combined with Snell's law, the refractive index of air, the refractive index of water and the corresponding angle relationship are determined. Based on the refractive index of air, the refractive index of water, and the corresponding angular relationship, an expression for calculating the refractive vector is constructed. The cross-medium image is divided into several pixel blocks. The refraction Jacobian matrix of the center pixel is calculated. The refraction offset of the pixels in the block is estimated by bilinear interpolation. Analytical compensation for the light refraction distortion is performed to obtain the compensated light refraction path. Reverse ray tracing is performed based on the compensated light refraction path to establish the correspondence between pixel positions and real geometry; The three-dimensional position and normal of the water surface are determined based on the expression for the refraction vector. Combining the correspondence between pixel position and real geometry, the corrected cross-medium image is output.

[0009] Furthermore, the specific expression for calculating the refraction vector is as follows: ; In the above formula, Represents the unit vector of the refracted ray. Indicates the refractive index of air. Indicates the refractive index of water. Indicates the air-side incident angle. Represents the interface normal vector. This represents the unit vector of the incident ray.

[0010] Furthermore, the step of determining the water surface normal vector based on the mathematical equation of the air-water refraction plane, and performing depth calculation and joint optimization by combining the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values ​​to generate an integrated 3D model specifically includes: The water surface normal vector is determined based on the mathematical equation of the air-water refraction plane, and the ray equation in the air is constructed by combining Snell's law with the relationship matrix between pixels and decoded projection encoding values. By combining the ray equations in air and underwater, depth calculations are performed to determine the three-dimensional coordinates of each pixel in the cross-medium image. The set of Gaussian primitives data structures is updated based on the three-dimensional coordinates of each pixel in the cross-media image to obtain the updated set of Gaussian primitives. Based on the updated set of Gaussian primitives, and taking the 3DGS standard photometric loss and refractive geometric constraint loss as optimization objectives, the UAV pose, water surface parameters, and Gaussian parameters are jointly optimized. A Gaussian-ICP variant is used for multi-frame registration, and a factor map is constructed to uniformly optimize the camera pose and Gaussian parameters, generating an integrated 3D model.

[0011] Furthermore, the expression for the depth calculation is as follows: ; ; ; In the above formula, Indicates the depth value. Indicates the camera's focal length. This indicates the baseline distance between the camera and the projection device. This represents the offset corresponding to the disparity or encoding. Represents pixel coordinates, Indicates the coordinates of the principal point. Indicates the equivalent focal length.

[0012] The second technical solution adopted in this invention is: a three-dimensional modeling system for water and air trans-medium objects based on visual distortion correction, comprising: The first module is used to obtain the color information, depth information and incident angle information of each pixel in the cross-medium image, and to construct the mathematical equation of the air-water refraction plane; The second module is used to perform refraction distortion compensation and multi-scale phase shift analysis based on each pixel of the cross-medium image, and to construct the underwater ray equation set and the relationship matrix between pixels and decoded projection coding values. The third module uses the mathematical equations of the air-water refraction plane to determine the water surface normal vector, and combines the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values ​​to perform depth calculation and joint optimization, generating an integrated 3D model.

[0013] The beneficial effects of the method and system of this invention are as follows: This invention acquires the color, depth, and incident angle information of each pixel in a cross-medium image and constructs the mathematical equation of the air-water refraction plane. It actively generates textures using structured light, solving the problem of weak and difficult-to-match target textures in underwater and cross-medium environments. Furthermore, based on each pixel of the cross-medium image, it performs refraction distortion compensation and multi-scale phase shift analysis, constructing an underwater ray equation set and a relationship matrix between pixels and decoded projection encoding values. Through refraction physical modeling and inverse correction, it achieves geometrically accurate restoration of cross-medium imaging. Finally, the mathematical equation of the air-water refraction plane determines the water surface normal vector. Combined with the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values, it performs depth calculation and joint optimization to generate an integrated 3D model. This ensures centimeter-level modeling accuracy while maximizing computational efficiency, improving the visual quality of the UAV 3D environment model. Attached Figure Description

[0014] Figure 1 This is a flowchart of the steps of the three-dimensional modeling method for water and air trans-medium objects based on visual distortion correction according to the present invention; Figure 2 This is a structural block diagram of the three-dimensional modeling system for water and air trans-medium objects based on visual distortion correction according to the present invention; Figure 3 This is a schematic diagram of a drone performing cross-media inspection according to a specific embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the working state of the cross-media unmanned aerial vehicle platform in different media environments according to a specific embodiment of the present invention; Figure 5 This is a schematic diagram of the attitude and imaging geometry of a UAV in underwater navigation state provided in a specific embodiment of the present invention; Figure 6This is a schematic diagram of the structured light trans-medium imaging principle and optical path refraction provided in a specific embodiment of the present invention. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0016] Reference Figure 1 This invention provides a three-dimensional modeling method for water-air transmedia objects based on visual distortion correction, the method comprising the following steps: S100: Obtain the color information, depth information, and incident angle information of each pixel in the cross-medium image, and construct the mathematical equation of the air-water refraction plane; Taking offshore wind turbine inspection as an example, after taking off from land or the maintenance support vessel, the drone flies over the water surface, approaches the wind turbine tower foundation, and finally hovers in the transition area below the blades (see...). Figure 3 For different foundation types such as monopile, jacket, and tension leg, UAVs need to cross the air-water interface to complete the detailed modeling of underwater structures (such as pile foundation scour pits, anode blocks, and submarine cable J-tubes) to achieve integrated detection of blade surface defects and underwater foundation damage.

[0017] Cross-medium imaging refers to the process by which light rays sequentially pass through multiple optical media such as air, water surface, and water body to reach the imaging sensor. Due to the significant difference between the refractive index of air (approximately 1.0003) and the refractive index of water (approximately 1.333), light rays undergo nonlinear deflection at the interface, causing traditional pinhole camera models to fail. This embodiment targets the offshore wind power scenario, requiring the simultaneous handling of two cross-medium modes: UAV hovering above the water surface (camera-water surface-underwater target) and close-to-base operations (camera-underwater target). Unified refraction modeling is used to achieve integrated 3D reconstruction at different water depths (shallow water <30m, transition zone 30-60m, deep water >60m).

[0018] Specifically, the cross-medium UAV flies to the target operation area and hovers stably at the corresponding position, initializes the Gaussian scene, and projects a pre-coded structured light pattern onto the target operation area, acquiring a cross-medium image containing the pre-coded structured light pattern; it calculates the RGB information, light intensity information, and angle information of each pixel in the cross-medium image to construct an empty Gaussian primitive data structure set; based on the geometric features of the air-water refraction interface obtained by the visual sensor, the empty Gaussian primitive data structure set is regionalized to obtain the segmented Gaussian scene; invalid observation areas are removed from the segmented Gaussian scene, and the mathematical equation of the air-water refraction plane is constructed.

[0019] In this embodiment, as Figure 4 As shown, the cross-medium UAV flies to the operation area and hovers stably in the corresponding medium. Simultaneously, the system initializes the Gaussian scene and establishes a set of empty Gaussian primitive data structures.

[0020] The initialization of the Gaussian primitive set is represented as: ; In the above formula, For location, For covariance, For opacity, is the spherical harmonic coefficient.

[0021] The drone projects a pre-coded structured light pattern onto the target, artificially creating a recognizable texture in the underwater low-texture area. The imaging module synchronously acquires cross-media images containing the modulated pattern and records the RGB, light intensity, and angle information of each pixel.

[0022] Projected structured light encoded pattern: Or binary encoding: ,in For the first Bit-coded pattern.

[0023] More specifically, such as Figure 6 As shown, the structured light projection module is installed at the front or lower part of the UAV body. Its projection parameters need to be adaptively adjusted according to the resolution requirements of subsequent depth calculation steps and the reflection characteristics of the target surface. It is used to actively project a preset structured light coded pattern onto the target. The pattern includes, but is not limited to: parallel stripes, phase-shifted stripes, Gray code coded patterns, binary coded patterns, dot matrix patterns, or grid patterns. The density of the projected coded pattern directly determines the spatial resolution of the initial Gaussian primitive. For underwater areas with weak texture, a dense dot matrix pattern (spacing ≤ 5mm) is used to ensure that the initial distribution density of the Gaussian primitive is ≥ 200 dots / cm². 3 A sparse mesh pattern is used in areas with strong textures, and redundant Gaussian points are avoided through adaptive density control.

[0024] Furthermore, the three-dimensional position and orientation of the air-water refraction interface are detected in real time based on visual and sensor data. A Gaussian scene space is pre-divided according to the interface information, and invalid observation areas are eliminated to improve subsequent processing efficiency.

[0025] The equation of the air-water refraction plane is: ; Or parameterized representation: ; in It is the normal vector. Let be a point on the plane.

[0026] It should also be noted that for cross-medium image acquisition, the imaging acquisition module strictly and synchronously captures the cross-medium image modulated by the target surface during the structured light projection in step (I). The acquisition timing is precisely triggered by the data interface module according to the structured light projection period and the camera exposure time, ensuring that each frame contains complete structured light encoded information. In addition to capturing the structured light pattern, the imaging acquisition module needs to synchronously record the RGB-DA information (color, depth, and incident angle) of each pixel as the four-dimensional attribute input for Gaussian primitive initialization. The camera exposure time should be controlled within 1 / 200s to avoid initial errors in the Gaussian position caused by water surface ripples.

[0027] Depending on the medium between the drone and the target, image acquisition can be divided into the following two typical scenarios: 1) When the camera is positioned above the water surface, the light propagation path is: target → water body → water surface interface → air → camera; 2) When the camera is underwater, the light propagation path is: target → water body → camera.

[0028] In cross-water imaging, significant geometric distortion occurs in the image due to light refraction at the air-water interface. This distortion is the sole input source for compensation calculations. To improve subsequent registration accuracy, this embodiment employs multi-frame continuous acquisition or multi-view acquisition methods. The quality of the acquired image sequence directly determines the integrity and accuracy of the final model.

[0029] Cross-media unified projection model: This embodiment uses a unified ray tracing imaging model, embedding the law of refraction into the projection process to achieve single-path tracing.

[0030] Intersection point calculation, incident ray equation: ; in For the camera optical center, The direction of incidence; Solution for intersection parameters: ; in For the interface normal vector, These are interface parameters.

[0031] Virtual camera projection, after equivalent camera coordinate transformation, uses a standard pinhole model: ; S200. Based on each pixel of the cross-medium image, refraction distortion compensation and multi-scale phase shift analysis are performed to construct an underwater ray equation set and a matrix relating pixels to decoded projection coding values. Specifically, based on each pixel of the cross-medium image, reverse ray tracing is performed to obtain the direction of the light rays incident on the water surface for each pixel of the cross-medium image; the light ray-water surface intersection point is determined according to the light ray direction incident on the water surface, and the direction of the refracted light ray is calculated using Snell's law; a set of underwater ray equations is constructed by combining the light ray-water surface intersection point and the direction of the refracted light ray; refractive distortion compensation is performed on the cross-medium image to obtain the corrected cross-medium image; multi-scale pyramid decoding and multi-scale phase shift analysis are performed on the corrected cross-medium image to obtain the decoded projection coding value, and a relationship matrix between pixels and decoded projection coding values ​​is constructed.

[0032] In this embodiment, as Figure 5 As shown, the GPU performs parallel computation of the ray-water surface intersection point for each pixel of the camera. Calculate the direction of the light rays incident on the water surface in reverse. ; Calculate refracted rays using Snell's law The output is the set of underwater ray equations for each pixel.

[0033] Camera pixels Reverse ray tracing:

[0034] in For the camera's optical center, direction From camera internal parameters Decide: ; Calculate the intersection of the ray and the water surface. for: ; Snell's law for determining the direction of refraction: ; in ,when It was determined to be total internal reflection.

[0035] Output underwater ray equation set .

[0036] Furthermore, an anti-scattering structured light decoding algorithm is run on the original distorted image (to address the reduced contrast of underwater stripes); the output is pixels. With projection encoding value The correspondence.

[0037] For pixels Decode the projection encoding value: ; The decoding function is a multi-scale phase shift analysis: ; in For encoding-grayscale mapping function, It is a Gaussian filter.

[0038] The refractive distortion compensation needs to be explained: A transmedium refraction imaging model is established using the acquired distorted images and synchronously detected interface parameters. Snell's law is then applied. ; in The refractive index of air, Let be the refractive index of water. The angle of incidence on the air side. The angle of refraction on the water side.

[0039] For each 32×32 pixel block, calculate the refraction Jacobian matrix of the center pixel of the block. Propagation to the Gaussian covariance matrix via a first-order approximation: ; Achieve analytical compensation for refraction distortion to avoid rendering artifacts in subsequent steps.

[0040] The formula for calculating the refraction vector is: ; Among them, the applicable condition is that the incident angle is less than the critical angle for total internal reflection, that is... . Represents the unit vector of the incident ray. Represents the unit vector of the refracted ray. This represents the interface normal vector.

[0041] By employing reverse ray tracing, the refraction path of the imaging ray corresponding to each pixel is calculated, mapping the distorted image to equivalent "direct-view image" coordinates. To significantly reduce computational load, this step adopts an adaptive block-based parallel strategy: the image is divided into 32×32 pixel blocks, and complete ray tracing is performed on the center pixel of each block. The refraction offset of other pixels within the block is estimated using bilinear interpolation. Simultaneously, a pre-computed lookup table (LUT) is established to cache the refraction mapping relationships under common interface pose parameters. Under similar interface conditions, the mapping parameters are directly retrieved from the table, avoiding redundant calculations. Compared to pixel-by-pixel ray tracing, this improves efficiency. This compensation process must be coupled with the real-time output of the water surface's 3D position and normal from the interface perception module; otherwise, the intersection parameters between the ray and the interface cannot be accurately calculated, thus failing to restore the true geometric relationship of the structured light pattern. The accuracy of the pixel coordinates in the compensated, distortion-free image directly affects the quality of establishing the decoding correspondence, which in turn propagates to depth calculation errors.

[0042] The formula for finding the intersection of a ray and an interface is as follows: ; Find the intersection parameters: ; The mathematical equations of the S300 air-water refraction plane determine the water surface normal vector. The depth calculation and joint optimization are performed by combining the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values ​​to generate an integrated 3D model.

[0043] Specifically, the water surface normal vector is determined based on the mathematical equation of the air-water refraction plane, and the relationship matrix between Snell's law and the pixel and decoded projection encoding value is combined to construct the ray equation in the air. Depth calculation is performed by combining the ray equation in the air and the underwater ray equation set to determine the three-dimensional coordinates of each pixel in the cross-medium image. The air Gaussian primitive data structure set is updated based on the three-dimensional coordinates of each pixel in the cross-medium image to obtain the updated Gaussian primitive set. Based on the updated Gaussian primitive set, the UAV pose, water surface parameters, and Gaussian parameters are jointly optimized using 3DGS standard photometric loss and refractive geometric constraint loss as optimization objectives. Multi-frame registration is performed using a Gaussian-ICP variant, and a factor map is constructed to uniformly optimize the camera pose and Gaussian parameters, generating an integrated three-dimensional model.

[0044] In this embodiment, for pixels and projection encoding value Given the corresponding pixels of the projector, calculate its ray in the air. Based on the water surface normal vector n, use Snell's law to find the refracted ray. Solve the equations of the two underwater rays to find the intersection point and obtain the three-dimensional coordinates X.

[0045] Pixel-encode pairing Projector corresponding pixels .

[0046] Projector beam Using the same method, find the intersection point of the two underwater rays by combining the equations: ; Creating Gaussian primitives:

[0047] The covariance is initialized as a slender ellipsoid along the line of sight: ; This refers to the rotation of the camera to the world coordinate system.

[0048] Update the Gaussian set: .

[0049] Then input multi-frame Gaussian primitives Set optimization targets: 3DGS standard photometric loss (difference between rendered and actual images); refractive geometric constraint loss and reprojection error must satisfy Snell's law.

[0050] Joint optimization variable: UAV pose Water surface parameters Gaussian parameters It employs incremental optimization (similar to SLAM), achieving real-time performance of 5-10Hz.

[0051] Optimization objective after multi-frame Gaussian fusion: ; Photometric loss (3DGS standard): ; Geometric constraint loss: ; in For the projection function with refraction, This is a back projection.

[0052] By fusing multi-frame Gaussian primitive data, an improved ICP algorithm is used for registration. At the same time, the registration error is backpropagated to jointly optimize the position and shape parameters of the UAV pose and the Gaussian primitive, ensuring global model consistency.

[0053] Gaussian fusion optimization objective function: ; The optimization problem is modeled as a factor graph structure containing two types of nodes: Pose node: SE(3) pose of the UAV in each frame As a variable node; Gaussian nodes: the center of each Gaussian node As a variable node of the landmark.

[0054] Three types of constraint factors are introduced: 1) Reprojection factor: For observed Gaussian... Camera pose Construct reprojection error ,in This is a projection function that incorporates refraction effects. The error is calculated in pixel coordinates using the Huber robust kernel function and a threshold. Pixel; 2) Geometric consistency factor: For registered overlapping Gaussian pairs Calculate the overlap error The integral is solved analytically to ensure the consistency of the Gaussian spatial distribution between adjacent viewpoints; 3) Prior factors: Apply pose prior constraints to static feature points obtained from water surface detection to suppress UAV pose drift.

[0055] Factor graph optimization: Reprojection error: ; After refraction correction, the first The camera pose of the frame is The center point of Gauss. When projected onto this frame image, it should fall precisely on the decoded structured light coded coordinates. Place.

[0056] In cross-media scenarios, camera pose This is the virtual camera pose after distortion correction, not the original camera pose. Without this constraint, the reconstruction from different viewpoints would exhibit systematic drift.

[0057] This error implicitly assumes that the observation noise follows a Gaussian distribution, i.e. Therefore, minimizing the reprojection error is equivalent to maximizing the observation likelihood.

[0058] Gaussian overlap error: ; When fusing multiple views, if only reprojection error is used, each view will independently optimize the Gaussian parameters, resulting in the generation of multiple separate Gaussian clusters on the same physical surface under different views. Overlap error forces the reconstructed Gaussians from different views to coincide in the 3D spatial probability distribution.

[0059] Traditional point cloud ICP uses nearest-point hard association: This requires finding an explicit point-to-point match, but in cross-media weak texture scenarios, matching is prone to errors.

[0060] Gaussian overlap is a soft constraint that measures similarity through the product of probability densities: if two Gaussians represent the same surface region, their distributions should overlap significantly (the integral value should be close to 1); if they represent different regions, the overlap integral should be close to 0. No explicit matching is required, making it naturally suitable for dense but noisy point clouds generated by structured light.

[0061] In cross-medium scenarios, the confidence levels of observations differ depending on the perspective (underwater attenuation, refraction residuals). Automatic weighting of overlap integrals: High-confidence Gaussian (…). Small (gaussian) contributes significantly to the error; low confidence Gaussian ( Large contribution small contribution achieves adaptive robust fusion.

[0062] For two Gaussians and Its overlapping integral has a closed-form solution: in This formula can directly calculate the Jacobian matrix, achieving efficient optimization.

[0063] Optimization issues: ; The Gaussian point density is automatically adjusted based on the reconstruction quality, splitting points where detail is insufficient and merging points where redundancy is present. After quantization and compression, the model is uploaded to the management system via a communication link, supporting real-time rendering and engineering analysis.

[0064] Gaussian splitting condition: ; Iterate through all Gaussian elements and check their opacity. .like Furthermore, the Gaussian expression showed no improvement for 10 consecutive frames, indicating a negligible contribution to scene rendering, and was therefore removed. Simultaneously, depth outliers were removed: Forced pruning was performed on isolated Gaussian expressions more than 50 meters from the drone with zero gradients.

[0065] Gaussian pruning conditions: ; Based on the real-time distance d between the drone and the target area, the screen space projection size of Gaussian is dynamically adjusted during the rendering stage: Close-up mode Full-resolution rendering is used, and the Gaussian Splat size is limited to ≤2 pixels to ensure that millimeter-level details are visible. At this time, the active Gaussian number is about 500,000. Medium shot mode ( ): Implement quadtree space culling, set the Splat size to 2-8 pixels, downsample the Gaussians far from the camera, reduce the number of active Gaussians to 200,000, and improve the rendering speed by 2 times; Far View Mode Merge similar Gaussians with a covariance eigenvalue ratio of less than 0.1, Splat size ≥ 8 pixels, enable Gaussian fusion algorithm, compress the number of active Gaussians to below 50,000, and reduce transmission bandwidth requirements.

[0066] LOD level of detail: ; Furthermore, the structured light decoding and depth calculation are explained: The output corrected image is then subjected to structured light decoding to identify the coded value corresponding to each pixel, establishing a one-to-one correspondence matrix between image pixels and the coded values ​​of the projected pattern. This decoding process must be performed based on the aforementioned compensation; otherwise, refractive distortion will lead to coded crosstalk and misidentification.

[0067] To improve decoding speed, this embodiment employs a multi-scale pyramid decoding strategy: first, a rough correspondence is quickly determined on the low-resolution image, and then fine-tuned on the original resolution. Simultaneously, GPU parallel computing is utilized to accelerate the decoding process by more than 10 times. Based on the calibration parameters of the projector and camera, a projection-imaging geometric correspondence is constructed, and the three-dimensional coordinates of each point on the target surface are calculated using triangulation principles. ; ; ; in For depth, Focal length The baseline distance between the camera and the projector. To correspond to the disparity, obtain the three-dimensional spatial coordinates of each pixel. .

[0068] The pixel-to-encoded correspondence matrix established after decoding directly drives the generation of Gaussian parameters. For each valid correspondence... ,calculate: Location Obtained through triangulation.

[0069] covariance ,in ( (Baseline-distance scaling factor).

[0070] Opacity , For structured light intensity, This is the water body attenuation coefficient.

[0071] Spherical harmonic coefficient , The absorption coefficient of the water body.

[0072] The output Gaussian primitive set replaces the traditional dense point cloud, reducing memory usage by 40%.

[0073] The geometric accuracy of the dense point cloud data output in this embodiment is determined by both the correction residual and the decoding accuracy, and its quality directly determines the convergence speed and accuracy of point cloud registration. Finally, a dense or semi-dense depth map is generated.

[0074] Finally, point cloud registration and fusion are performed on the depth maps from multiple frames or multiple views. A GPU-accelerated Gaussian-ICP variant is used, with the objective function being: ; in Use a Gaussian kernel function to implement soft registration based on probability distribution 50.

[0075] A factor graph is constructed by unifying the camera pose nodes and Gaussian primitive nodes, and the optimization variables include: camera pose. Gaussian mean Gaussian covariance .

[0076] The residual terms include: Reprojection error: ; Gaussian overlap error: ; Solve problem 40 using the GPU version of the GTSAM library.

[0077] Real-time monitoring of the Gaussian gradient during the optimization process: like Split the Gaussian along the direction of the largest eigenvector.

[0078] like The Gaussian pruning transparency is below the threshold.

[0079] A global density redistribution is performed every 10 iterations to keep the total number of points between 300,000 and 800,000.

[0080] LOD (Level of Detail) rendering: dynamically adjusted based on the distance d between the drone and the target. Full resolution rendering, Splat size ≤ 2 pixels.

[0081] Medium resolution, Splat size 2-8 pixels, using quadtree culling.

[0082] Low resolution, Splat size ≥ 8 pixels, enable Gaussian aggregation.

[0083] Cross-media communication upload: The final optimized 3D Gaussian scene is compressed to ≤50MB after pruning and uploaded to the inspection management system in real time via 5G link. The cloud can then perform further refinement using Global Bundle Adjustment (Global BA).

[0084] The embodiments of the present invention have the following distinguishing technical features compared with the prior art: 1) Sparse ray tracing: Full resolution processing is applied to weak texture areas, while 50%-70% downsampling is applied to uniform strong texture areas, reducing the overall computational load by about 40% without affecting the reconstruction quality.

[0085] 2) Dynamic keyframe selection: Keyframes are automatically selected for 3D reconstruction based on motion amplitude and viewpoint changes. Non-keyframes are only used for model refinement, which greatly reduces redundant calculations.

[0086] 3) Mixed precision computation: FP16 half-precision computation is used in ray tracing and Gaussian optimization, while FP32 precision is maintained in critical geometry computation.

[0087] 4) Edge computing and cloud collaboration: The drone performs real-time distortion compensation and fast decoding, while the complex Gaussian splashing optimization is offloaded to the ground base station or cloud server, achieving a balance between millisecond-level response and high-quality reconstruction.

[0088] 5) Gaussian-specific optimization: Vector quantization (VQ) encoding is used for similar Gaussian primitives (covariance eigenvalue difference <5%), further reducing video memory usage by 25%; an independent Gaussian-optimized CUDA Stream is built and runs in parallel with the image acquisition pipeline, hiding a latency of 30ms; Gaussian position μ uses FP32, covariance Σ uses FP16, and spherical harmonic coefficients SH uses INT8 quantization, reducing the overall video memory usage to 2.1GB (runs smoothly on the Jetson AGX Orin platform).

[0089] Through the above system-level optimizations, this invention improves overall computational efficiency by more than 60% while maintaining reconstruction accuracy, thus meeting the real-time requirements of engineering inspection.

[0090] In summary, this invention generates textures through structured light projection, combines interface perception and refraction models to compensate for distortion in cross-medium images, generates point clouds through GPU-accelerated decoding and triangulation, and reconstructs a 3D model using Gaussian splashing technology after fusing multi-view data. This method solves the challenges of underwater weak texture matching and cross-interface distortion correction, improving computational efficiency by over 60%, and is suitable for integrated air-water intelligent inspection of marine engineering and water conservancy facilities.

[0091] This embodiment addresses the key technical challenge of unified 3D modeling in various media environments, including air, water surface, and underwater. It is applicable to underwater targets with weak textures, complex lighting environments, and cross-surface imaging scenarios with significant refractive distortion, enabling high-precision acquisition of target surface morphology, dimensional parameters, and spatial structure. This embodiment is not only suitable for 3D detection and evaluation of underwater structures, bridge foundations, offshore wind power facilities, port engineering, and water conservancy projects, but can also be extended to underwater archaeology, marine resource surveys, underwater pipeline inspections, aquatic ecological monitoring, and emergency rescue, representing an important component of cross-media intelligent sensing and 3D measurement technology.

[0092] Therefore, the embodiments of the present invention have the following advantages compared with the prior art: 1) Actively generate textures using structured light to solve the problem of weak target textures and difficulty in matching in underwater and cross-medium environments; 2) Geometrically accurate reconstruction of cross-medium imaging is achieved through refraction physics modeling and inverse correction; 3) Introducing Gaussian splashing technology for 3D scene representation significantly improves reconstruction efficiency and visual quality while reducing computational burden; 4) Deeply integrated with air-sea unmanned aerial vehicle (UAV) platforms, suitable for continuous operations in the air, on the surface, and underwater; 5) High modeling accuracy and strong stability, suitable for engineering-level applications; 6) System resource usage is optimized by adopting hybrid precision computing and an edge-cloud collaborative architecture, which reduces memory usage by 30%, making it suitable for the resource-constrained environment of UAV platforms; 7) Reconstructing the balance between quality and efficiency, adaptive density control and multi-scale processing strategies maximize computational efficiency while ensuring centimeter-level accuracy.

[0093] 8) Gaussian splashing technology is deeply integrated into each step, enabling end-to-end differentiable optimization from data acquisition to scene representation, avoiding the traditional separate processing of point cloud-mesh-texture.

[0094] Reference Figure 2 A 3D modeling system for water and air transmedia objects based on visual distortion correction includes: The first module 201 is used to obtain the color information, depth information and incident angle information of each pixel in the cross-medium image, and to construct the mathematical equation of the air-water refraction plane; The second module 202 is used to perform refraction distortion compensation and multi-scale phase shift analysis based on each pixel of the cross-medium image, and to construct the underwater ray equation set and the relationship matrix between pixels and decoded projection coding values. The third module 203 is used to determine the water surface normal vector by using the mathematical equations of the air-water refraction plane. It combines the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values ​​to perform depth calculation and joint optimization, generating an integrated 3D model.

[0095] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0096] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A three-dimensional modeling method for water-air transmedia objects based on visual distortion correction, characterized in that, Includes the following steps: Obtain the color and light intensity information of each pixel in the cross-medium image, estimate the initial parameters of the air-water refraction plane, and construct the mathematical equation of the refraction plane; Based on each pixel of the cross-medium image, refraction distortion compensation and multi-scale phase shift analysis are performed to construct an underwater ray equation set and a matrix relating pixels to decoded projection coding values. The mathematical equations of the air-water refraction plane determine the water surface normal vector. The depth calculation and joint optimization are performed by combining the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values ​​to generate an integrated 3D model.

2. The three-dimensional modeling method for water and air trans-medium objects based on visual distortion correction according to claim 1, characterized in that, The step of acquiring the RGB information and light intensity information of each pixel in the cross-medium image and constructing the mathematical equation for the air-water refraction plane specifically includes: The cross-medium UAV flies to the target operation area and hovers stably at the corresponding position, initializes the Gaussian primitive set, projects a pre-coded structured light pattern onto the target operation area, and acquires a cross-medium image containing the pre-coded structured light pattern; Calculate the RGB information and light intensity information of each pixel in the cross-medium image containing the preset coded structured light pattern, and construct the initial Gaussian primitive data structure set. Based on the geometric features of the air-water refraction interface obtained by the visual sensor, the imaging space is regionalized to obtain the division of air region, interface region and underwater region. By eliminating Gaussian primitives in invalid observation areas, the mathematical equation parameters of the air-water refraction plane are optimized.

3. The three-dimensional modeling method for water-air trans-medium objects based on visual distortion correction according to claim 2, characterized in that, The step of performing refractive distortion compensation and multi-scale phase shift analysis based on each pixel of the cross-medium image to construct the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values ​​specifically includes: Based on each pixel of the cross-medium image, reverse ray tracing is performed to calculate the direction and angle of incidence of the light rays incident on the water surface corresponding to each pixel; Determine the intersection point of the ray and the water surface based on the direction of the ray incident on the water surface, and determine whether total internal reflection exists. For non-total internal reflection rays, calculate the direction of the refracted ray using Snell's law. Refractive distortion compensation is performed on the cross-medium image to obtain the corrected cross-medium image; Multi-scale pyramid decoding and multi-scale phase shift analysis are performed on the corrected cross-media image to obtain the decoded projection coding value and construct the relationship matrix between pixels and decoded projection coding value; By combining the intersection of light rays and the water surface with the direction of refracted light rays, a set of underwater ray equations is constructed.

4. The three-dimensional modeling method for water and air trans-medium objects based on visual distortion correction according to claim 3, characterized in that, The step of performing refractive distortion compensation on the cross-medium image to obtain the corrected cross-medium image specifically includes: Based on the RGB information, light intensity information and angle information of each pixel in the cross-medium image, and combined with Snell's law, the refractive index of air, the refractive index of water and the corresponding angle relationship are determined. Based on the refractive index of air, the refractive index of water, and the corresponding angular relationship, an expression for calculating the refractive vector is constructed. The cross-medium image is divided into several pixel blocks. The refraction Jacobian matrix of the center pixel is calculated. The refraction offset of the pixels in the block is estimated by bilinear interpolation. Analytical compensation for the light refraction distortion is performed to obtain the compensated light refraction path. Reverse ray tracing is performed based on the compensated light refraction path to establish the correspondence between pixel positions and real geometry; The three-dimensional position and normal of the water surface are determined based on the expression for the refraction vector. Combining the correspondence between pixel position and real geometry, the corrected cross-medium image is output.

5. The three-dimensional modeling method for water-air trans-medium objects based on visual distortion correction according to claim 4, characterized in that, The specific expression for calculating the refraction vector is as follows: ； In the above formula, Represents the unit vector of the refracted ray. Indicates the refractive index of air. Indicates the refractive index of water. Indicates the air-side incident angle. Represents the interface normal vector. Represents the unit vector of the incident ray; Among them, when When total internal reflection occurs, the ray is left unprocessed.

6. The three-dimensional modeling method for water-air trans-medium objects based on visual distortion correction according to claim 5, characterized in that, The step of determining the water surface normal vector based on the mathematical equation of the air-water refraction plane, and performing depth calculation and joint optimization by combining the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values ​​to generate an integrated 3D model specifically includes: The water surface normal vector is determined based on the mathematical equation of the air-water refraction plane. The ray equation of the light emitted by the projector in the air is constructed by combining Snell's law with the correspondence between pixel-decoding projection encoding values. Triangulation was performed using the underwater ray equations from both the camera and projector sides to calculate the three-dimensional coordinates of each pixel in the cross-medium image. The Gaussian primitive data structure set is updated based on the calculated three-dimensional coordinates to obtain the updated Gaussian primitive set. Based on the updated set of Gaussian primitives, the UAV pose, water surface parameters and Gaussian parameters are jointly optimized with 3DGS photometric loss and refractive geometric constraint loss as optimization objectives. A soft correspondence registration method based on Gaussian distribution is used for multi-frame fusion, and a factor map is constructed to uniformly optimize the camera pose and Gaussian parameters to generate an integrated 3D model.

7. The three-dimensional modeling method for water-air trans-medium objects based on visual distortion correction according to claim 6, characterized in that, The expression for the depth calculation is as follows: For the structured light coding correspondence, depth calculation adopts the triangulation principle: ； ； ； In the above formula, The phase value or encoded offset obtained from decoding. This is the conversion factor between the encoding period and the pixel size. This is the fixed offset obtained from system calibration. Indicates the depth value. Indicates the camera's focal length. This indicates the baseline distance between the camera and the projection device. This represents the offset corresponding to the disparity or encoding. Represents pixel coordinates, Indicates the coordinates of the principal point. Indicates the equivalent focal length.

8. A 3D modeling system for water and air transmedia objects based on visual distortion correction, characterized in that, Includes the following modules: The first module is used to obtain the color information, depth information and incident angle information of each pixel in the cross-medium image, and to construct the mathematical equation of the air-water refraction plane; The second module is used to perform refraction distortion compensation and multi-scale phase shift analysis based on each pixel of the cross-medium image, and to construct the underwater ray equation set and the relationship matrix between pixels and decoded projection coding values. The third module uses the mathematical equations of the air-water refraction plane to determine the water surface normal vector, and combines the underwater ray equation set and the relationship matrix between pixels and decoded projection encoding values ​​to perform depth calculation and joint optimization, generating an integrated 3D model.

Images