A Three-Dimensional Gaussian-Medium Field Joint Model and Selective Super-Resolution Optimization Method for Underwater Scene Reconstruction

Abstract

This invention discloses a method for underwater scene reconstruction using a 3D Gaussian-medium field joint model and selective super-resolution optimization, relating to the fields of 3D scene reconstruction and computer graphics. The method first establishes an object Gaussian representation and a direction-conditional medium field, obtaining a stable decoupled representation through object-medium joint rendering and Stage I joint optimization. Then, based on geometric fidelity scores and medium reliability scores, a medium-aware Gaussian fidelity score and a per-view weight map are generated. Subsequently, a pseudo-high-resolution supervision is modulated using the per-view weight map to form a weighted selective super-resolution loss, which, together with the low-resolution consistency loss, constitutes the Stage II total loss. Under a fixed medium field, only the object Gaussian parameters are optimized. Finally, the demedium-restored image and underwater high-resolution new-view rendering are output. This invention unifies underwater medium modeling and medium-aware selective super-resolution optimization, taking into account the physical rationality of underwater imaging, high-resolution detail restoration, and multi-view consistency.

Description

Technical Field

[0001] This invention relates to the fields of 3D scene reconstruction and computer graphics technology, specifically to a 3D Gaussian-medium field joint model and a selective super-resolution optimization method for underwater scene reconstruction. Background Technology

[0002] Underwater 3D reconstruction has significant application value in underwater inspection, marine robot perception, marine archaeology, and immersive display. Unlike scene reconstruction in air, underwater imaging is affected by medium effects such as light absorption, attenuation, and backscattering, which can easily lead to problems such as color distortion, reduced contrast, and blurred details, posing significant challenges to scene geometry restoration, appearance modeling, and the synthesis of new perspectives.

[0003] For 3D reconstruction of underwater scenes, there are currently two main technical approaches. One is based on neural radiation fields, which models the participating medium through volume rendering and can effectively describe underwater light propagation effects. However, the training and rendering processes are computationally expensive, resulting in low efficiency. The other is based on 3D Gaussian splashing, which uses explicit scene representation and has high training and rendering efficiency. However, this type of method is essentially more suited to surface modeling of sharp scenes and has limited ability to characterize underwater scattering medium effects.

[0004] Regarding reconstruction quality, the output quality of existing efficient 3D reconstruction methods is usually closely related to the resolution of the training images. When the input is a low-resolution multi-view image, the reconstruction result is prone to texture blurring and loss of detail. To address this issue, selective super-resolution methods have been proposed, which, based on the sufficiency of multi-view geometric sampling, introduce high-frequency supervision signals only in undersampled regions to enhance details and preserve multi-view quality. Figure 1 Consistency. However, such methods are mainly designed for atmospheric environments, and their region selection criteria mostly rely solely on geometric information, failing to fully consider the impact of medium attenuation and scattering on the realism and recoverability of details in underwater scenes. This leads to the problem that in underwater scenes, some regions, although geometrically close and seemingly possessing high-frequency sampling conditions, carry unreliable visual information due to severe medium contamination. Including these regions in the super-resolution enhancement range solely based on geometric criteria can easily result in the media noise being mistakenly enhanced into pseudo-textures, thereby compromising the realism of the reconstruction and cross-viewpoint consistency.

[0005] In summary, existing underwater 3D reconstruction methods struggle with medium physical modeling, high-resolution detail recovery, and multi-view reconstruction. Figure 1Achieving a balance between physical accuracy and detail restoration is crucial. However, existing selective super-resolution methods, which only consider the sufficiency of geometric sampling and lack consideration for the effects of medium attenuation and scattering, are prone to mistakenly amplifying medium noise into false details in underwater scenes. How to achieve physically reasonable, high-resolution 3D reconstruction that balances medium modeling and detail restoration in underwater scenes is a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0006] This invention proposes a three-dimensional Gaussian-medium field joint model and a selective super-resolution optimization method for underwater scene reconstruction. Its objectives are: 1. To address the difficulty of existing underwater 3D reconstruction methods in simultaneously achieving medium physical modeling, high-resolution detail restoration, and multi-view functionality. Figure 1 1. Technical issues related to consistency; 2. Solving the problem that existing selective super-resolution methods rely solely on the sufficiency of geometric sampling for region selection, without considering the impact of medium attenuation and scattering on the availability of high-frequency information, thus easily amplifying medium noise into pseudo-details.

[0007] The technical solution of this invention is as follows:

[0008] A method for underwater scene reconstruction using a three-dimensional Gaussian-medium field joint model and selective super-resolution optimization includes:

[0009] Step S1: Construct a training set of multi-view low-resolution underwater observation images, and initialize sparse point clouds based on the multi-view low-resolution observation images;

[0010] Step S2: Based on the sparse point cloud, establish the Gaussian representation of the object and simultaneously establish the directional conditional medium field to decouple the object Gaussian representation from the medium scattering representation in function.

[0011] Step S3: Perform object-medium joint rendering and stage I joint optimization. Use low-resolution observation images to supervise the joint rendering results and obtain a preliminary decoupled and stable object Gaussian representation and orientation conditional medium field.

[0012] Step S4: Calculate the geometric fidelity score and the medium reliability score, then obtain the medium-aware Gaussian fidelity score, and then generate a view-by-view weighted map based on the medium-aware Gaussian fidelity score.

[0013] Step S5: Generate pseudo-high-resolution reference images corresponding to the training set images using the frozen single-image super-resolution model, and construct a weighted selective super-resolution loss based on the view-by-view weight map and the pseudo-high-resolution reference images;

[0014] Step S6: Construct the low-resolution consistency loss, then construct the stage II total loss based on the low-resolution consistency loss and the weighted selective super-resolution loss, optimize the Gaussian representation of the object based on the stage II total loss, and obtain the refined object Gaussian set.

[0015] Step S7: Output rendering based on the refined Gaussian set of objects.

[0016] As a further improvement to the aforementioned three-dimensional Gaussian-medium field joint model and selective super-resolution optimized underwater scene reconstruction method, in step S2, based on the sparse point cloud obtained in step S1, an object Gaussian set is established:

[0017]

[0018] in, Represents the total Gaussian count of the object; the first Gaussian object Characterized by the following parameters:

[0019]

[0020] in, For the first A Gaussian three-dimensional center; The covariance matrix is ​​related to spatial scale and orientation; This is the opacity parameter; For view-related color vectors;

[0021] During rendering, the first The 3D Gaussian projection is applied to the ... After considering the image plane from each viewpoint, a two-dimensional mean can be obtained. With the two-dimensional covariance matrix Therefore, the first Pixel coordinates in a projected image from a viewpoint The value of the two-dimensional Gaussian distribution function at point is denoted as Based on this, the first A Gaussian in a pixel The equivalent opacity weight at the location is denoted as:

[0022]

[0023] in, The Sigmoid function is used to limit the opacity parameter to a certain value. interval;

[0024] After depth sorting, the object component colors are represented in the cumulative forward transparency form of standard 3D Gaussian sputtering as follows:

[0025]

[0026] in, Indicates the first Pixels in a projected image from a viewpoint The clear object component color contributed by the object Gaussian.

[0027] As a further improvement to the aforementioned three-dimensional Gaussian-medium field joint model and selective super-resolution optimized underwater scene reconstruction method: In step S2, the established directional conditional medium field refers to the... and the corresponding line of sight Under the given conditions, the output of the directional conditional medium field is expressed as:

[0028]

[0029] in, Indicates the color of the medium at infinity; The attenuation coefficient representing the direct component of the object; This represents the backscattering coefficient.

[0030] As a further improvement to the aforementioned three-dimensional Gaussian-medium field joint model and selective super-resolution optimized underwater scene reconstruction method, the object-medium joint rendering process in step S3 is as follows:

[0031] Let the first Pixels in a projected image from a viewpoint The scene depth at that location is The transmittance of the medium at that pixel is expressed as:

[0032]

[0033] in, This represents the effective transmission ratio retained due to medium attenuation as light travels from an object to the camera; It is the attenuation coefficient of the direct component of the object in the directional condition medium field;

[0034] No. Pixels in a projected image from a viewpoint The backscattering term corresponding to this location is expressed as:

[0035]

[0036] in, This represents the backscattered color accumulated along the line of sight of the medium; It is the color term of the medium at infinity in the directional condition medium field; It is the backscattering coefficient in a directional condition medium field;

[0037] The underwater imaging model obtained by object-media co-rendering is represented as follows:

[0038]

[0039] in, Indicates the first Pixels in a projected image from a viewpoint Underwater prediction images; Indicates pixel-by-pixel multiplication; Indicates the first Pixels in a projected image from a viewpoint The sharp object component color contributed by the object Gaussian is obtained based on the object Gaussian representation.

[0040] As a further improvement to the aforementioned three-dimensional Gaussian-medium field joint model and selective super-resolution optimization underwater scene reconstruction method, in step S3, the total loss of the joint optimization in stage I is:

[0041]

[0042] in, For pixel-by-pixel L1 reconstruction loss, In order to be with the first Low-resolution observation images from various perspectives The corresponding object-media joint rendering graph; This represents the D-SSIM loss based on structural similarity; This is the balance coefficient;

[0043] By minimizing the total loss in stage I, the parameters of the object's Gaussian parameter set and the directional condition medium field are simultaneously optimized.

[0044] As a further improvement to the aforementioned three-dimensional Gaussian-medium field joint model and selective super-resolution optimized underwater scene reconstruction method, step S4 specifically includes:

[0045] Step S4.1: Calculate the geometric fidelity fraction based on the projection radius;

[0046] For the The Gaussian, and its number in the first... The two-dimensional covariance matrix projected from each perspective The two eigenvalues ​​are denoted as and Then the screen space radius of the Gaussian at this viewpoint is defined as:

[0047]

[0048] in, Indicates the first Gauss in the th... Maximum projection scale from a single viewpoint;

[0049] Let the first The set of all visible training views of a Gaussian is Then its maximum and minimum projection radii across the viewing angle are respectively:

[0050]

[0051] Based on this, the geometric sampling ratio is constructed as follows:

[0052] ;

[0053] Mapping the geometric sampling ratio to a geometric fidelity fraction:

[0054]

[0055] in, Indicates the first A geometrical fidelity fraction of Gaussian; For threshold parameters; For smoothing coefficients;

[0056] Step S4.2: Based on the directional condition medium field obtained in step S3, calculate the medium reliability score for each Gaussian. ;

[0057] Step S4.3: Calculate the medium-sensing Gaussian fidelity score;

[0058] The geometric fidelity score and the media reliability score are fused element-wise to obtain the media-aware Gaussian fidelity score:

[0059] ;

[0060] Step S4.4: Determine the maximum projection view set based on the medium-aware Gaussian fidelity score and generate a view-by-view weight map;

[0061] Define the Gaussian set of the maximum projected viewpoint: If the first... Gauss in perspective Projected radius below satisfy , For the first The maximum projection radius across the viewpoint of a Gaussian is then considered to be the current viewpoint. Belongs to the One of the maximum projection angles of a Gaussian, defining an indicator function:

[0062]

[0063] in, Used to indicate whether the current viewpoint is one of the closest observation viewpoints of that Gaussian;

[0064] By mapping Gaussian properties to pixel-by-pixel space using rasterization sputtering, the first Gaussian property is constructed. Initial weight map from each perspective:

[0065]

[0066] in, Indicates the first An operator that renders the Gaussian properties from a single perspective using 3D Gaussian sputtering and maps them to a pixel-wise image using transparency blending; This is the balance coefficient;

[0067] The initial weight map is normalized to obtain the final view-by-view weight map:

[0068]

[0069] in, Indicates the initial weight map at pixels The value at; This represents the maximum value of all pixels in the image; It is a constant.

[0070] As a further improvement to the aforementioned three-dimensional Gaussian-medium field joint model and selective super-resolution optimized underwater scene reconstruction method, step 4.2 involves calculating the reliability score. The process is as follows:

[0071] For the Gauss in the th... From each perspective, its depth is denoted as . Then its medium transmittance is:

[0072]

[0073] in, This represents the object attenuation coefficient along the path corresponding to the Gaussian curve.

[0074] Next, calculate the backscattering intensity index:

[0075] set up For the first Gauss in the th... The set of pixels effectively covered from each viewpoint For the Gaussian in pixels The equivalent opacity weight at the location; assuming all Gaussians are sorted by depth in ascending order, let [the value is missing]. For the first A depth of Gaussian units The depth of the previous Gaussian, For the light to reach the first The remaining transmittance before a Gaussian point indicates the amount of aggregation contributed by the object path. And the contribution of the medium path to the amount of polymerization They are respectively:

[0076]

[0077]

[0078] in, The attenuation coefficient for the direct component of the object. The backscattering coefficient, The color term of the medium at infinity is one of the three terms, all of which are obtained from the directional conditional medium field. For the first A Gaussian in a pixel The color contribution of the object at that location;

[0079] Define the normalized backscattering intensity index:

[0080]

[0081] in, Describing the L1 norm, It is a constant;

[0082] Therefore, the first Gauss in the th... The media reliability term from one perspective is:

[0083]

[0084] in, The transmittance of the medium along the path of the Gaussian; and Hyperparameters used to control the relative influence of transmittance and backscattering intensity;

[0085] For the first A Gaussian set in all visible views The average of the media reliability items is used to obtain the global media reliability score:

[0086]

[0087] in Indicates the first The number of training views visible to Gaussians.

[0088] As a further improvement to the aforementioned three-dimensional Gaussian-medium field joint model and selective super-resolution optimization underwater scene reconstruction method: In step S5, the frozen single-image super-resolution model is set as follows: Then for the first Underwater images from various perspectives Its pseudo-high resolution reference image is:

[0089]

[0090] After obtaining the high-resolution reference image, a selective super-resolution loss is constructed:

[0091] Let the first The high-resolution underwater prediction image obtained from the joint rendering based on step S3 from multiple perspectives is as follows: Then the first The weighted selective super-resolution loss under each viewpoint is:

[0092]

[0093] in, This represents the view-by-view weight map obtained in step S4; Indicates pixel-by-pixel multiplication; To calculate the L1 norm; Indicates D-SSIM loss; This is the balance coefficient.

[0094] As a further improvement to the aforementioned three-dimensional Gaussian-medium field joint model and selective super-resolution optimized underwater scene reconstruction method, the calculation method for the low-resolution consistency loss in step S6 is as follows:

[0095] Let the downsampling operator be ,in Describing the super-resolution ratio, then the first... The low-resolution consistency loss from any perspective is defined as:

[0096]

[0097] in, This represents the high-resolution underwater prediction image obtained based on the joint rendering process in step S3; This indicates that the first step obtained in step S1 is... Low-resolution underwater observation images from various perspectives and Correspondingly; This is the balance coefficient; To calculate the L1 norm.

[0098] As a further improvement to the underwater scene reconstruction method based on the three-dimensional Gaussian-medium field joint model and selective super-resolution optimization, in step S6, the weighted selective super-resolution loss is... and low resolution consistency loss Combining the results using a complementary weighted approach, we obtain the total loss for Stage II:

[0099]

[0100] in, This represents the balance coefficient between selective super-resolution supervision and low-resolution consistency supervision. Number of viewpoints;

[0101] Then, with the orientation condition medium field fixed, the parameters of the object Gaussian representation are optimized using the total loss in Stage II to obtain the refined object Gaussian set. .

[0102] Compared with the prior art, the present invention has the following beneficial effects:

[0103] 1. This invention decouples the appearance of scene objects from the scattering effects of the water medium at the representation level by establishing a joint model of the object's Gaussian representation and the directional conditional medium field. It employs a two-stage training strategy: first, joint object-medium reconstruction; then, under a fixed medium field, selective super-resolution refinement of only the object's Gaussian representation. This ensures that the physical modeling of underwater imaging and high-frequency detail restoration are completed at different stages without interference. Therefore, this invention can simultaneously ensure the accuracy of underwater medium physical modeling, high-resolution detail restoration for low-resolution input, and cross-view rendering consistency within a unified framework, solving the technical problem that existing methods struggle to simultaneously meet these three requirements.

[0104] 2. This invention addresses the limitation of existing selective super-resolution methods that rely solely on the sufficiency of geometric sampling for region selection by introducing a medium reliability constraint into the region determination process under super-resolution supervision. A medium reliability score is constructed by jointly calculating the medium transmittance and normalized backscattering intensity, and this score is then element-wise fused with the geometric fidelity score to form a medium-aware Gaussian fidelity score. This ensures that super-resolution enhancement only applies to geometrically undersampled local regions that are recoverable under the given medium conditions. This design effectively identifies and avoids unreliable regions that are geometrically close but heavily contaminated by the medium, suppressing the phenomenon of mistakenly enhancing medium noise into pseudo-texture.

[0105] 3. This invention further designs a medium-aware, view-by-view weighted map generation mechanism. After determining the maximum projection view set for each Gaussian, the initial weights are modulated using the medium-aware Gaussian fidelity score. This ensures that the current viewpoint assumes primary super-resolution supervision responsibility only when it is both the optimal geometric observation viewpoint and has reliable medium transmission conditions. This viewpoint-by-view, pixel-by-pixel adaptive weight allocation strategy avoids inter-view texture drift and false detail diffusion caused by strong scattering regions being incorrectly included in the high-frequency compensation range.

[0106] 4. By fixing all parameters of the directional conditional medium field in stage II and optimizing only the parameters of the Gaussian representation of the object, the high-frequency detail enhancement process in the second stage will not re-perturb the underwater imaging physical constraints (including the transmission attenuation and backscattering relationship) established in the first stage. This ensures the stability of the object-medium decoupling relationship and provides a structural guarantee for maintaining a physically consistent medium representation while enhancing high-resolution details.

[0107] In summary, this invention organically unifies underwater scattering medium modeling with selective super-resolution optimization for low-resolution inputs within the same framework. While inheriting the high rendering efficiency advantage of explicit Gaussian representation, it also possesses physically reasonable underwater imaging simulation capabilities, reliable media removal recovery capabilities, and media-aware high-frequency detail selective enhancement capabilities. This provides an effective 3D reconstruction approach for application scenarios such as underwater inspection and marine robot perception, which have high requirements for both computational efficiency and reconstruction accuracy. Attached Figure Description

[0108] Figure 1 The flowchart shows the underwater scene reconstruction method based on the three-dimensional Gaussian-medium field joint model and selective super-resolution optimization. Detailed Implementation

[0109] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0110] like Figure 1 As shown in this embodiment, an underwater scene reconstruction method using a three-dimensional Gaussian-medium field joint model and selective super-resolution optimization comprises four parts: low-resolution training data construction, object-medium joint reconstruction, view-by-view weight map generation, and selective high-resolution refinement of object parameters. The training process is divided into two stages: Stage I is the underwater joint reconstruction stage, and Stage II is the selective high-resolution refinement stage of object parameters. Figure 1 The upper part corresponds to Stage I, which mainly completes the processing chain of "multi-view low-resolution underwater image - key point extraction - sparse point cloud initialization - object Gaussian representation - orientation conditional medium field - object medium joint rendering - Stage I loss". Figure 1 The lower left part corresponds to the generation process of the per-view weight map, which mainly completes the processing chain of "learned medium field - medium reliability score - initial object Gaussian - geometric fidelity score - maximum projection view set - medium-aware Gaussian fidelity score - per-view weight map"; Figure 1 The lower right part corresponds to Stage II, which mainly completes the processing chain of "freezing the SISR model - pseudo-high resolution supervision - weighted selective super-resolution loss - low resolution consistency loss - Stage II total loss - object Gaussian parameter refinement"; Figure 1The bottom corresponds to the output links for "Image Demediation and Restoration" and "Underwater High-Resolution New Perspective Rendering Results".

[0111] Specifically, the following steps are included:

[0112] Step S1: Construct a training set of multi-view low-resolution underwater observation images and initialize a sparse point cloud based on the multi-view low-resolution observation images.

[0113] First, image sequences of the same underwater scene from multiple viewpoints and their corresponding camera parameters are acquired. Let the total number of training viewpoints be... , No. Low-resolution observation images from each perspective are denoted as follows: The corresponding camera parameters are denoted as Then the training set can be represented as:

[0114]

[0115] in, It can include information such as camera intrinsic parameter matrix, extrinsic parameter matrix, viewpoint center, and imaging direction.

[0116] In one embodiment, a low-resolution image can be constructed by downsampling the original image by a factor of 4. Let the... The original underwater images from various perspectives are The downsampling operator is denoted as Then we have:

[0117]

[0118] This process corresponds to Figure 1 The section on "Multi-view Underwater Image Sequence → 4x Downsampling" uses low-resolution observations during training to simulate the characteristics of low-frequency observations under the combined effects of long distances, turbid media, and hardware imaging conditions in real underwater scenarios, and to provide a unified training entry point for subsequent selective super-resolution optimization.

[0119] Subsequently, key points were extracted and cross-view matching were performed on the multi-view low-resolution observation images to restore the camera pose and initialize the sparse point cloud of the underwater scene.

[0120] Step S2: Establish the object Gaussian representation based on the sparse point cloud, and at the same time establish the directional conditional medium field to decouple the object Gaussian representation from the medium scattering representation in terms of function.

[0121] Based on the sparse point cloud obtained in step S1, establish an object Gaussian set:

[0122]

[0123] in, Represents the total Gaussian count of the object; the first Gaussian object Characterized by the following parameters:

[0124]

[0125] in, For the first A Gaussian three-dimensional center; The covariance matrix is ​​related to spatial scale and orientation; This is the opacity parameter; This is a view-dependent color vector. The object Gaussian set is used for explicitly modeling the object structure, surface details, and appearance information in the scene.

[0126] During rendering, the first The 3D Gaussian projection is applied to the ... After considering the image plane from each viewpoint, a two-dimensional mean can be obtained. With the two-dimensional covariance matrix Therefore, the first Pixel coordinates in a projected image from a viewpoint The two-dimensional Gaussian distribution function value at point can be denoted as: Based on this, the first A Gaussian in a pixel The equivalent opacity weight at the location is denoted as:

[0127]

[0128] in, The Sigmoid function is used to limit the opacity parameter to a certain value. Interval.

[0129] Furthermore, after depth sorting, the color of the object components can be represented in the form of cumulative forward transparency according to standard 3D Gaussian sputtering as follows:

[0130]

[0131] in, Indicates the first Pixels in a projected image from a viewpoint The color of the clear object component contributed by the object Gaussian; multiplication terms Indicates that it is located at the depth order. The residual transmittance resulting from each Gaussian before the first Gaussian reflects the explicit forward transparency mixing relationship.

[0132] Simultaneously, to explicitly model the influence of the water medium on the light propagation process, a directional conditional medium field is established. (Pixel) and the corresponding line of sight Under the given conditions, the output of the directional conditional medium field is expressed as:

[0133]

[0134] in, Indicates the color of the medium at infinity; The attenuation coefficient representing the direct component of the object; This represents the backscattering coefficient. Through this medium field, the Gaussian representation of the object and the medium scattering representation can be functionally decoupled: the object Gaussian represents the potentially sharp scene, while the medium field represents the water propagation effect.

[0135] Step S3: Perform object-medium joint rendering and stage I joint optimization. Use low-resolution observation images to supervise the joint rendering results and obtain a preliminary decoupled and stable object Gaussian representation and orientation conditional medium field.

[0136] After establishing the Gaussian representation of the object and the orientation-conditional medium field, perform object-medium joint rendering:

[0137] Let the first Pixels in a projected image from a viewpoint The scene depth at that location is Then the transmittance of the medium at that pixel can be expressed as:

[0138]

[0139] in, This represents the effective transmission ratio retained due to medium attenuation as light travels from an object to the camera; The larger or The larger the value, the more significantly the object component is attenuated.

[0140] No. Pixels in a projected image from a viewpoint The backscattering term corresponding to this location is expressed as:

[0141]

[0142] in, This represents the backscattered color accumulated along the line of sight of the medium; This represents the limiting color of the medium as the path approaches infinity. The terms in parentheses illustrate the physical process by which backscattering gradually accumulates to a saturation value as the propagation depth increases.

[0143] Therefore, the underwater imaging model obtained by object-medium joint rendering is represented as:

[0144]

[0145] in, Indicates the first Pixels in a projected image from a viewpoint Underwater prediction images; Indicates pixel-wise multiplication; first term Indicates the direct component of the object after medium attenuation; the second term This represents the backscattering component of the medium.

[0146] In this embodiment, the training objective of Phase I is to obtain a stable object-medium decoupled representation; therefore, low-resolution observation images are used to supervise the joint rendering results. The total loss of the Phase I joint optimization can be expressed as:

[0147]

[0148] in, For pixel-by-pixel L1 reconstruction loss, In order to be with the first Low-resolution observation images from various perspectives The corresponding object-media joint rendering graph; This represents the D-SSIM loss based on structural similarity; This is a balancing coefficient used to adjust the weights between the pixel fidelity term and the structural similarity term.

[0149] By minimizing the total loss in stage I, the object's Gaussian parameter set is simultaneously optimized. By combining the parameters of the directional conditional medium field, a stable object-medium joint representation can be obtained.

[0150] After training, a stable decoupled representation of the object component and the medium component can be obtained: the object Gaussian is responsible for modeling the scene geometry and appearance, and the directional condition medium field is responsible for modeling the transmission attenuation and backscattering caused by the water body. Figure 1 The “Stage I Loss ↔ Object-Media Joint Rendering” and its output “Stable Object-Media Joint Representation” correspond to this stage.

[0151] Step S4: Calculate the geometric fidelity score and the medium reliability score, then obtain the medium-aware Gaussian fidelity score, and then generate a view-by-view weighted map based on the medium-aware Gaussian fidelity score.

[0152] Step S4 is one of the key steps of this invention. Its purpose is to determine "where super-resolution enhancement is needed" and "where super-resolution enhancement is suitable" based on a stable object-medium representation.

[0153] Step S4.1: Calculate the geometric fidelity fraction based on the projection radius.

[0154] For the The Gaussian, and its number in the first... The two-dimensional covariance matrix projected from each perspective The two eigenvalues ​​are denoted as and Then the screen space radius of the Gaussian at this viewpoint is defined as:

[0155]

[0156] in, Indicates the first Gauss in the th... The maximum projection scale at a given viewpoint is used to measure the imaging scale at which the Gaussian is observed in that viewpoint.

[0157] Let the first The set of all visible training views of a Gaussian is Then its maximum and minimum projection radii across the viewing angle are respectively:

[0158]

[0159] Based on this, the geometric sampling ratio is constructed as follows:

[0160]

[0161] Geometric sampling ratio The larger the value, the greater the difference in the imaging scale observed by the Gaussian at different viewpoints, meaning that there are some closer viewpoints that are more likely to provide high-frequency information.

[0162] Furthermore, the geometric sampling ratio is mapped to a geometric fidelity fraction:

[0163]

[0164] in, Indicates the first A geometrical fidelity fraction of Gaussian; For threshold parameters; This is the smoothing coefficient. The larger the value, the more sufficient the "near-view high-frequency reference" information is in the multi-view geometric sense; The smaller the value, the more likely the Gaussian is to be in an undersampled region, requiring super-resolution for detail compensation. For Gaussians visible only in a very few views, due to their statistical instability, their geometric fidelity score can be set to zero or a low value to avoid unreliable estimations.

[0165] Step S4.2: Calculate the media reliability score.

[0166] Based on the directional condition medium field obtained in step S3, a medium reliability score is calculated for each Gaussian. For the first Gaussian... Gauss in the th... From each perspective, its depth is denoted as . Then its medium transmittance can be taken as:

[0167]

[0168] in, This represents the attenuation coefficient of the object along the path corresponding to this Gaussian curve. Transmittance. The larger the value, the more fully the target information is retained after passing through the medium.

[0169] Meanwhile, the normalized backscattering intensity index is obtained by combining the aggregated object path contribution and the medium path contribution:

[0170] set up For the first Gauss in the th... The set of pixels effectively covered from each viewpoint For the Gaussian in pixels The equivalent opacity weight at the location. Assuming all Gaussians are sorted by depth in ascending order, let... For the first A depth of Gaussian units Let be the depth of the previous Gaussian (for the first Gaussian, let be) ), For the light to reach the first The remaining transmittance before a Gaussian (i.e., the previous Gaussian transmittance) The cumulative transmittance after Gaussian contribution), then the aggregate amount of object path contribution. And the contribution of the medium path to the amount of polymerization They are respectively:

[0171]

[0172]

[0173] in, The attenuation coefficient for the direct component of the object. The backscattering coefficient, The color term of the medium at infinity is one of the three terms, all of which are obtained from the directional conditional medium field. For the first A Gaussian in a pixel The color contribution of the object at that location.

[0174] Based on this, a normalized backscattering intensity index is defined:

[0175]

[0176] in, Describing the L1 norm, To avoid dividing by zero, the smallest constant. It reflects the ratio of the intensity of the medium scattering contribution to the direct signal contribution from the object; the larger the ratio, the more severe the medium contamination.

[0177] Therefore, the first Gauss in the th... The media reliability term from this perspective can be expressed as:

[0178]

[0179] in, The transmittance of the medium along the path of the Gaussian; and This is a hyperparameter used to control the relative influence of transmittance and backscattering intensity. This comprehensively reflects the credibility of the Gaussian medium from the current perspective: higher transmittance and weaker backscattering. The closer to .

[0180] Furthermore, regarding the first A Gaussian set in all visible views The average of the media reliability items is used to obtain the global media reliability score:

[0181]

[0182] in Indicates the first The number of training viewpoints visible to a Gaussian; The larger the value, the more reliable the Gaussian is in imaging, and the less likely it is to misinterpret medium noise as real detail.

[0183] Step S4.3: Calculate the medium-sensing Gaussian fidelity score.

[0184] The geometric fidelity score and the media reliability score are fused element-wise to obtain the media-aware Gaussian fidelity score:

[0185]

[0186] in, Indicates the first The final medium-sensitivity fidelity score for a Gaussian is determined. This design ensures that a high final score is achieved only when a Gaussian has a geometrically reliable sampling basis and a high reliability in the medium; if either the geometric sampling or the medium reliability is insufficient, the final score will decrease.

[0187] Step S4.4: Determine the maximum projection view set based on the medium-aware Gaussian fidelity score and generate a view-by-view weight map.

[0188] To determine which perspective is more suitable as the primary high-frequency supervision perspective for a given Gaussian, we define the set of Gaussians with the maximum projection perspective: if the... Gauss in perspective Projected radius below satisfy , For the first The maximum projection radius across the viewpoint of a Gaussian is then considered to be the current viewpoint. Belongs to the One of the maximum projection angles of a Gaussian, defining an indicator function:

[0189]

[0190] in, Used to indicate whether the current viewpoint is one of the closest observation viewpoints of that Gaussian.

[0191] Subsequently, the Gaussian properties are mapped to the pixel-by-pixel space using rasterization sputtering to construct the first... Initial weight map from each perspective:

[0192]

[0193] in, Indicates the first An operator that renders the Gaussian properties from a single perspective using 3D Gaussian sputtering and maps them to a pixel-wise image using transparency blending; The balance coefficient; the first term Emphasis is placed on geometric undersampling and the reliable region of the medium; the second term It emphasizes the current perspective as the best perspective for supervision, and the area where the medium is reliable.

[0194] To ensure consistent weight scales across different viewpoints, the initial weight map is normalized to obtain the final viewpoint-specific weight map (pixel-specific normalization):

[0195]

[0196] in, Indicates the initial weight map at pixels The value at; This represents the maximum value of all pixels in the image; To prevent division by zero of extremely small constants, normalization is performed. It can be directly used for pixel-by-pixel weighting of subsequent high-resolution loss. Figure 1 The “Geometric Fidelity Score – Medium Reliability Score – Medium-Sensitive Gaussian Fidelity Score – Maximum Projection Viewpoint Set – Per-View Weighted Map” corresponds to the overall process of step S4.

[0197] Step S5: Generate pseudo-high-resolution reference images corresponding to the training set images using the Frozen Single Image Super-Resolution (SISR) model, and construct a weighted selective super-resolution loss based on the view-by-view weight map and the pseudo-high-resolution reference images.

[0198] In Phase II, a frozen single-image super-resolution model is introduced as a high-frequency prior source, such as the StableSR model.

[0199] Let the super-resolution model of a frozen single image be... Then for the first Underwater images from various perspectives Its pseudo-high-resolution reference image can be represented as:

[0200]

[0201] Since the parameters of the single-image super-resolution model are fixed at this stage, it only serves as a supervision signal generation module and does not participate in joint optimization. This process corresponds to... Figure 1 The branch is "multi-view underwater image sequence → frozen SISR model (StableSR) → super-resolution underwater image".

[0202] After obtaining the high-resolution reference image, a selective super-resolution loss is constructed:

[0203] Let the first The high-resolution underwater prediction image obtained from the joint rendering based on step S3 from multiple perspectives is as follows: Then the first The weighted selective super-resolution loss under each viewpoint is:

[0204]

[0205] in, This represents the view-by-view weight map obtained in step S4; This indicates pixel-wise multiplication; the first term is the weighted L1 reconstruction term; the second term is the weighted D-SSIM structural similarity term. Indicates D-SSIM loss; This is the balance coefficient.

[0206] By applying pixel-wise weight modulation to the loss, super-resolution supervision can be focused on geometrically undersampled regions with relatively reliable media, while automatically reducing constraint strength in regions with severe media contamination or sufficient sampling. This part corresponds to... Figure 1 The bottom right side shows "Per-view Weighted Map → Weighted Selective Super-Resolution Loss".

[0207] Step S6: Construct the low-resolution consistency loss, then construct the stage II total loss based on the low-resolution consistency loss and the weighted selective super-resolution loss, optimize the object Gaussian representation based on the stage II total loss, and obtain the refined object Gaussian set.

[0208] To prevent Phase II from compromising the consistency of the original low-resolution observations by merely following a pseudo-high-resolution reference, this embodiment also constructs a low-resolution consistency loss:

[0209] Let the downsampling operator be ,in Indicates the super-resolution ratio (in one embodiment) ), then the first The low-resolution consistency loss from any perspective is defined as:

[0210]

[0211] in, This represents the high-resolution underwater prediction image obtained based on the joint rendering process in step S3; This indicates that the first step obtained in step S1 is... Low-resolution underwater observation images from various perspectives; This is a balancing coefficient used to adjust the weights between the L1 fidelity term and the D-SSIM structural similarity term. This loss corresponds to... Figure 1 The "4x downsampling → low-resolution consistency loss" in the model ensures that the model after high-resolution refinement can still explain the original low-resolution observations without geometric drift or cross-view divergence.

[0212] Subsequently, the weighted selective super-resolution loss was applied. and low resolution consistency loss Combining the results using a complementary weighted approach, we obtain the total loss for Stage II:

[0213]

[0214] in, This represents the balance coefficient between selective super-resolution supervision and low-resolution consistency supervision. When When the size is large, the model focuses more on using local high-frequency priors for detail enhancement; when When the values ​​are smaller, the model more rigorously maintains consistency with the original low-resolution observations. This complementary weighted design ensures that the relative contributions of the two losses can be directly expressed through a single parameter. Linear adjustment avoids weight imbalance caused by differences in loss scale.

[0215] Then, with the orientation conditional medium field fixed, the parameters of the object's Gaussian representation are optimized using the total loss of Stage II. The rationale behind this approach is that Stage I has already learned a relatively stable medium representation. If the medium field is updated further in Stage II, the medium branches may absorb high-frequency information that should belong to the object's surface, thereby disrupting the object-medium decoupling relationship. Conversely, by fixing the medium field and refining only the object's Gaussian representation, the optimization objective of Stage II can be focused on scene detail recovery while maintaining the stability of the underwater imaging physical model.

[0216] After Phase II is completed, the refined Gaussian set of objects is obtained. .

[0217] Step S7: Output rendering based on the refined Gaussian set of objects.

[0218] Output rendering is divided into two categories:

[0219] (1) Output demediated image: The demediated image can be obtained by directly rendering the sharp scene object components using the refined Gaussian set of objects:

[0220]

[0221] in, Represents the refined Gaussian set of objects The sharpness component of the rendered object. These are the pixel coordinates in the rendered image. This output is used to demonstrate the appearance of a potentially sharp scene, corresponding to... Figure 1 "Restore image by removing media" on the bottom right.

[0222] (2) Outputting high-resolution underwater new perspective rendering results: The refined object Gaussian set is recombined with the directional condition medium field obtained in step S3, and object-medium joint rendering is performed to obtain high-resolution new perspective rendering results that conform to the real underwater imaging mechanism:

[0223]

[0224] in, and These represent the medium transmittance and backscattering terms given by the fixed medium field under the new viewing angle, respectively; Represents the Gaussian set of refined objects The sharpness component of the rendered object. This output represents the pixel coordinates on the rendered image. Figure 1 The bottom center displays "Underwater high-resolution new perspective rendering results".

[0225] This completes the entire process of this invention, starting from low-resolution multi-view underwater input, through object-medium joint modeling, medium-aware per-view weight generation, and selective high-resolution refinement of object parameters, and finally outputting the de-medium restoration result and the underwater high-resolution rendering result.

[0226] It should be noted that, as will be apparent to those skilled in the art, the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics thereof. The scope of the present invention is defined by the claims rather than the foregoing description.

Claims

1. A method for underwater scene reconstruction with three-dimensional Gaussian-medium field joint model and selective super-resolution optimization, characterized in that, include: Step S1: Construct a training set of multi-view low-resolution underwater observation images, and initialize sparse point clouds based on the multi-view low-resolution observation images; Step S2: Based on the sparse point cloud, establish the Gaussian representation of the object and simultaneously establish the directional conditional medium field to decouple the object Gaussian representation from the medium scattering representation in function. Step S3: Perform object-medium joint rendering and stage I joint optimization. Use low-resolution observation images to supervise the joint rendering results and obtain a preliminary decoupled and stable object Gaussian representation and orientation conditional medium field. Step S4: Calculate the geometric fidelity score and the medium reliability score, then obtain the medium-aware Gaussian fidelity score, and then generate a view-by-view weighted map based on the medium-aware Gaussian fidelity score. Step S4 specifically includes: Step S4.1: Calculate the geometric fidelity fraction based on the projection radius; For the The Gaussian, and its number in the first... The two-dimensional covariance matrix projected from each perspective The two eigenvalues ​​are denoted as and Then the screen space radius of the Gaussian at this viewpoint is defined as: ； wherein, represents the maximum projection scale of the th Gaussian at the th view angle; Let the first The set of all visible training views of a Gaussian is Then its maximum and minimum projection radii across the viewing angle are respectively: ； Based on this, the geometric sampling ratio is constructed as follows: ； Mapping the geometric sampling ratio to a geometric fidelity fraction: ； in, Indicates the first A geometrical fidelity fraction of Gaussian; For threshold parameters; For smoothing coefficients; Step S4.2: Based on the directional condition medium field obtained in step S3, calculate the medium reliability score for each Gaussian. ; Step S4.3: Calculate the medium-sensing Gaussian fidelity score; The geometric fidelity score and the media reliability score are fused element-wise to obtain the media-aware Gaussian fidelity score: ； Step S4.4: Determine the maximum projection view set based on the medium-aware Gaussian fidelity score and generate a view-by-view weight map; Define the Gaussian set of the maximum projected viewpoint: If the first... Gauss in perspective Projected radius below satisfy , For the first The maximum projection radius across the viewpoint of a Gaussian is then considered to be the current viewpoint. Belongs to the One of the maximum projection angles of a Gaussian, defining an indicator function: ； in, Used to indicate whether the current viewpoint is one of the closest observation viewpoints of that Gaussian; By mapping Gaussian properties to pixel-by-pixel space using rasterization sputtering, the first Gaussian property is constructed. Initial weight map from each perspective: ； in, Indicates the first An operator that renders the Gaussian properties from a single perspective using 3D Gaussian sputtering and maps them to a pixel-wise image using transparency blending; This is the balance coefficient; The initial weight map is normalized to obtain the final view-by-view weight map: ； in, Indicates the initial weight map at pixels The value at; This represents the maximum value of all pixels in the image; It is a constant; Step S5: Generate pseudo-high-resolution reference images corresponding to the training set images using the frozen single-image super-resolution model, and construct a weighted selective super-resolution loss based on the view-by-view weight map and the pseudo-high-resolution reference images; Step S6: Construct the low-resolution consistency loss, then construct the stage II total loss based on the low-resolution consistency loss and the weighted selective super-resolution loss, optimize the Gaussian representation of the object based on the stage II total loss, and obtain the refined object Gaussian set. Step S7: Output rendering based on the refined Gaussian set of objects.

2. The underwater scene reconstruction method based on the three-dimensional Gaussian-medium field joint model and selective super-resolution optimization as described in claim 1, characterized in that... In step S2, based on the sparse point cloud obtained in step S1, an object Gaussian set is established: ； in, Represents the total Gaussian count of the object; the first Gaussian object Characterized by the following parameters: ； in, For the first A Gaussian three-dimensional center; The covariance matrix is ​​related to spatial scale and orientation; This is the opacity parameter; For view-related color vectors; During rendering, the first The 3D Gaussian projection is applied to the ... After considering the image plane from each viewpoint, a two-dimensional mean can be obtained. With the two-dimensional covariance matrix Therefore, the first Pixel coordinates in a projected image from a viewpoint The value of the two-dimensional Gaussian distribution function at point is denoted as Based on this, the first A Gaussian in a pixel The equivalent opacity weight at the location is denoted as: ； in, The Sigmoid function is used to limit the opacity parameter to a certain value. interval; After depth sorting, the object component colors are represented in the cumulative forward transparency form of standard 3D Gaussian sputtering as follows: ； in, Indicates the first Pixels in a projected image from a viewpoint The clear object component color contributed by the object Gaussian.

3. The underwater scene reconstruction method based on the three-dimensional Gaussian-medium field joint model and selective super-resolution optimization as described in claim 2, characterized in that: In step S2, the established directional conditional medium field refers to the field established in the pixel. and the corresponding line of sight Under the given conditions, the output of the directional conditional medium field is expressed as: ； in, Indicates the color of the medium at infinity; The attenuation coefficient representing the direct component of the object; This represents the backscattering coefficient.

4. The underwater scene reconstruction method based on the three-dimensional Gaussian-medium field joint model and selective super-resolution optimization as described in claim 1, characterized in that... The object-media co-rendering process in step S3 is as follows: Let the first Pixels in a projected image from a viewpoint The scene depth at that location is The transmittance of the medium at that pixel is expressed as: ； in, This represents the effective transmission ratio retained due to medium attenuation as light travels from an object to the camera; It is the attenuation coefficient of the direct component of the object in the directional condition medium field; No. Pixels in a projected image from a viewpoint The backscattering term corresponding to this location is expressed as: ； in, This represents the backscattered color accumulated along the line of sight of the medium; It is the color term of the medium at infinity in the directional condition medium field; It is the backscattering coefficient in a directional condition medium field; The underwater imaging model obtained by object-media co-rendering is represented as follows: ； in, Indicates the first Pixels in a projected image from a viewpoint Underwater prediction images; Indicates pixel-by-pixel multiplication; Indicates the first Pixels in a projected image from a viewpoint The sharp object component color contributed by the object Gaussian is obtained based on the object Gaussian representation.

5. The underwater scene reconstruction method based on the three-dimensional Gaussian-medium field joint model and selective super-resolution optimization as described in claim 1, characterized in that... In step S3, the total loss of the joint optimization in stage I is: ； in, For pixel-by-pixel L1 reconstruction loss, In order to be with the first Low-resolution observation images from various perspectives The corresponding object-media joint rendering graph; This represents the D-SSIM loss based on structural similarity; This is the balance coefficient; By minimizing the total loss in stage I, the parameters of the object's Gaussian parameter set and the directional condition medium field are simultaneously optimized.

6. The underwater scene reconstruction method based on the three-dimensional Gaussian-medium field joint model and selective super-resolution optimization as described in claim 1, characterized in that... In step 4.2, the reliability score is calculated. The process is as follows: For the Gauss in the th... From each perspective, its depth is denoted as . Then its medium transmittance is: ； in, This represents the object attenuation coefficient along the path corresponding to the Gaussian curve. Next, calculate the backscattering intensity index: set up For the first Gauss in the th... The set of pixels effectively covered from each viewpoint For the Gaussian in pixels The equivalent opacity weight at the location; assuming all Gaussians are sorted by depth in ascending order, let [the value is missing]. For the first A depth of Gaussian units The depth of the previous Gaussian, For the light to reach the first The remaining transmittance before a Gaussian point indicates the amount of aggregation contributed by the object path. And the contribution of the medium path to the amount of polymerization They are respectively: ； ； in, The attenuation coefficient for the direct component of the object. The backscattering coefficient, The color term of the medium at infinity is one of the three terms, all of which are obtained from the directional conditional medium field. For the first A Gaussian in a pixel The color contribution of the object at that location; Define the normalized backscattering intensity index: ； in, Describing the L1 norm, It is a constant; Therefore, the first Gauss in the th... The media reliability term from one perspective is: ； in, The transmittance of the medium along the path of the Gaussian; and Hyperparameters used to control the relative influence of transmittance and backscattering intensity; For the first A Gaussian set in all visible views The average of the media reliability items is used to obtain the global media reliability score: ； in Indicates the first The number of training views visible to Gaussians.

7. The underwater scene reconstruction method based on the three-dimensional Gaussian-medium field joint model and selective super-resolution optimization as described in claim 1, characterized in that: In step S5, let the frozen single-image super-resolution model be... Then for the first Underwater images from various perspectives Its pseudo-high resolution reference image is: ； After obtaining the high-resolution reference image, a selective super-resolution loss is constructed: Let the first The high-resolution underwater prediction image obtained from the joint rendering based on step S3 from multiple perspectives is as follows: Then the first The weighted selective super-resolution loss under each viewpoint is: ； in, This represents the view-by-view weight map obtained in step S4; Indicates pixel-by-pixel multiplication; To calculate the L1 norm; Indicates D-SSIM loss; This is the balance coefficient.

8. The underwater scene reconstruction method based on the three-dimensional Gaussian-medium field joint model and selective super-resolution optimization as described in claim 1, characterized in that... The low-resolution consistency loss in step S6 is calculated as follows: Let the downsampling operator be ,in Describing the super-resolution ratio, then the first... The low-resolution consistency loss from any perspective is defined as: ； in, This represents the high-resolution underwater prediction image obtained based on the joint rendering process in step S3; This indicates that the first step obtained in step S1 is... Low-resolution underwater observation images from various perspectives and Correspondingly; This is the balance coefficient; To calculate the L1 norm.

9. The underwater scene reconstruction method based on the three-dimensional Gaussian-medium field joint model and selective super-resolution optimization as described in claim 1, characterized in that... In step S6, the weighted selective super-resolution loss is... and low resolution consistency loss Combining the results using a complementary weighted approach, we obtain the total loss for Stage II: ； in, This represents the balance coefficient between selective super-resolution supervision and low-resolution consistency supervision. Number of viewpoints; Then, with the orientation condition medium field fixed, the parameters of the object Gaussian representation are optimized using the total loss in Stage II to obtain the refined object Gaussian set. .

Images