Three-dimensional mapping method, device and equipment based on polarization information enhancement and medium

Abstract

Embodiments of the present disclosure disclose a polarization information enhancement-based three-dimensional mapping method, device, equipment and medium, relating to the technical field of computer vision and computer graphics. The specific implementation of the method comprises: determining polarization information of a polarization image in a polarization image sequence, wherein the polarization image sequence comprises a plurality of polarization images acquired at different time points; for each polarization image in the polarization image sequence, performing polarization chrominance enhancement processing on the polarization image according to the polarization information of the polarization image to generate a chrominance enhanced image, thereby obtaining a chrominance enhanced image sequence; performing camera pose estimation on the image frames in the chrominance enhanced image sequence to obtain target camera pose information; and performing three-dimensional mapping based on three-dimensional Gaussian sputtering according to the target camera pose information, thereby obtaining a target Gaussian global map. The implementation can improve the quality of three-dimensional scene surface reconstruction, and meet the high-quality mapping requirements in the fields of robot navigation, augmented reality, etc.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of computer vision and computer graphics, and in particular to a three-dimensional mapping method, apparatus, device, and medium based on polarization information enhancement. Background Technology

[0002] With the rapid development of technologies such as computer vision and computer graphics, intensive 3D scene reconstruction has become one of the key technologies in fields such as robot navigation and AR (Augmented Reality).

[0003] Simultaneous Localization and Mapping (SLAM) technology has made significant progress in recent years. SLAM is a technique used in robotics and computer vision to simultaneously estimate the camera's pose (localization) and create a 3D map of the environment (mapping). SLAM-based dense 3D reconstruction aims to generate detailed and accurate 3D representations of scenes or objects, updating in real time as the camera moves through the environment.

[0004] Currently, SLAM technology still faces many challenges when applied to indoor environment reconstruction, especially for objects lacking texture, with dark colors, or high reflectivity. These features severely affect the extraction of environmental features, leading to low positioning accuracy, poor mapping quality, and ultimately impacting the 3D reconstruction results. Even when depth sensors are integrated into the mapping system, they may fail in many scenarios, unable to detect depth information properly, thus limiting the system's robustness and accuracy. Summary of the Invention

[0005] Embodiments of this disclosure provide a three-dimensional mapping method, apparatus, electronic device, and computer-readable medium based on polarization information enhancement.

[0006] In a first aspect, embodiments of this disclosure provide a three-dimensional mapping method based on polarization information enhancement. The method includes: determining the polarization information of polarization images in a polarization image sequence, wherein the polarization image sequence includes multiple polarization images acquired at different times; performing polarization chromaticity enhancement processing on each polarization image in the polarization image sequence according to the polarization information of the polarization image to generate a chromaticity enhanced image, thereby obtaining a chromaticity enhanced image sequence; performing camera pose estimation on the image frames in the chromaticity enhanced image sequence to obtain target camera pose information; and performing three-dimensional mapping based on three-dimensional Gaussian sputtering according to the target camera pose information to obtain a target Gaussian global map.

[0007] In some embodiments, the polarization image has multiple polarization angles; determining the polarization information of the polarization image in the polarization image sequence includes: characterizing the polarization intensity of the polarization image at different polarization angles using a Stokes vector; determining the polarization information of the polarization image based on the Stokes vector, wherein the polarization information includes: total light intensity information, linear polarization angle information, and linear polarization degree information.

[0008] In some embodiments, for each polarized image in the polarized image sequence, polarization chromaticity enhancement processing is performed on the polarized image to generate a chromaticity enhanced image based on the polarization information of the polarized image, including: polarization separation of the diffuse reflection component and the specular reflection component in the polarized image based on the total light intensity information and linear polarization degree information of the polarized image; and generating the chromaticity enhanced image of the polarized image by suppressing the specular reflection component in the polarized image.

[0009] In some embodiments, camera pose estimation is performed on image frames in a chroma-enhanced image sequence to obtain target camera pose information, including: Traverse each frame in the color-enhanced image sequence and perform the following operations in sequence: perform camera pose estimation on the current frame to obtain the camera pose information of the current frame; in response to the spatial feature difference and / or polarization feature difference between the current frame and the previous key frame satisfying the preset conditions, determine the current frame as a key frame; obtain the target camera pose information based on the camera pose information of the key frame. Among them, the spatial feature difference and / or polarization feature difference between the current frame and the previous key frame meet the preset conditions, including at least one of the following: the relative movement distance between the current frame and the previous key frame exceeds the preset distance threshold; the co-view crossover ratio of the photometric information between the current frame and the previous key frame is lower than the preset visibility threshold; the polarization difference measurement between the current frame and the previous key frame in the co-view area exceeds the preset difference threshold.

[0010] In some embodiments, obtaining target camera pose information based on the camera pose information of the keyframe includes: using the camera pose information and color information of the keyframe as input, creating a Gaussian ellipsoid corresponding to the keyframe in an intermediate Gaussian global map; optimizing the camera pose information of the keyframe by minimizing the photometric error between the photometric image and the keyframe based on the photometric image rendered from the intermediate Gaussian global map, to obtain optimized camera pose information of the keyframe; and determining the optimized camera pose information as the target camera pose information. Based on the target camera pose information, a 3D map is constructed using 3D Gaussian sputtering to obtain the target Gaussian global map. This includes: for each keyframe, based on the optimized camera pose information of that keyframe, a Gaussian ellipsoid corresponding to that keyframe is created in the intermediate Gaussian global map using 3D Gaussian sputtering; in response to the completion of the creation of all Gaussian ellipsoids corresponding to all keyframes, the target Gaussian global map is obtained based on the current intermediate Gaussian global map.

[0011] In some embodiments, the obtained target Gaussian global map is further optimized as follows: for each polarized image in the polarized image sequence, the color information and polarization information in the polarized image are weighted and concatenated along the feature dimension to generate a polarization enhancement feature vector for each pixel in the polarized image; all polarization enhancement feature vectors of the polarized image are clustered, and the polarized image is segmented according to the clustering results to obtain multiple initial planar regions; the result of rendering the target Gaussian global map is constrained according to the determined multiple loss function, and after backpropagation, the optimized target Gaussian global map is obtained; wherein, the multiple loss function includes a weighted sum of the neighborhood consistency loss function, the global plane consistency loss function, the Manhattan world loss function, and the luminosity consistency loss function.

[0012] In some embodiments, the neighborhood consistency loss function is determined as follows: the normal of each pixel within each planar region is compared with the weighted average of the normals of other pixels in its neighborhood, and the neighborhood consistency loss function is determined based on the comparison result; the global planar consistency loss function is determined as follows: a representative normal vector is determined for each planar region, the similarity between the normal of each pixel within the planar region and the representative normal vector is compared, and the global planar consistency loss function is determined based on the similarity of all pixels within each planar region; the Manhattan world loss function is determined as follows: for multiple planar regions, the Manhattan world loss function is determined by defining the inter-planar structural loss.

[0013] Secondly, embodiments of this disclosure provide an apparatus comprising: a polarization enhancement module configured to determine polarization information of polarized images in a polarization image sequence, wherein the polarization image sequence includes multiple polarization images acquired at different times; for each polarization image in the polarization image sequence, performing polarization chromaticity enhancement processing on the polarization image according to the polarization information of the polarization image to generate a chromaticity enhanced image, thereby obtaining a chromaticity enhanced image sequence; a positioning module configured to perform camera pose estimation on image frames in the chromaticity enhanced image sequence to obtain target camera pose information; and a mapping module configured to perform three-dimensional mapping based on three-dimensional Gaussian sputtering according to the target camera pose information to obtain a target Gaussian global map.

[0014] Thirdly, embodiments of this disclosure provide an electronic device comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the polarization-information-enhanced three-dimensional mapping method as described in any embodiment of the first aspect.

[0015] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the polarization-information-enhanced three-dimensional mapping method as described in any embodiment of the first aspect.

[0016] The three-dimensional mapping method and apparatus based on polarization information enhancement provided in the embodiments of this disclosure can suppress the specular reflection effect in the reflective area during synchronous positioning and mapping by determining the polarization information in the polarized image and performing polarization-guided chromaticity enhancement processing on the polarized image according to the polarization information, thereby reducing specular reflection and achieving true geometric restoration and accurate spatial positioning of the reflective area. This ensures the accuracy of positioning using polarized images and the accuracy and continuity of Gaussian points in highly reflective areas.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0018] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a three-dimensional mapping method based on polarization information enhancement provided according to embodiments of the present disclosure; Figure 2 This is a flowchart of a three-dimensional mapping method based on polarization information enhancement according to another embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of a three-dimensional mapping device based on polarization information enhancement provided according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device suitable for implementing the polarization information enhancement-based three-dimensional mapping method of the embodiments of this disclosure. Detailed Implementation

[0019] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0020] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] Traditional SLAM technology employs differentiable rendering techniques, such as NeRF (Neural Radiance Fields), which provides a novel approach to 3D representation through implicit modeling of object information. However, NeRF requires significant computational overhead during rendering, greatly limiting its application in SLAM.

[0023] In recent years, the efficient scene representation and fast rendering capabilities offered by 3DGS (3D Gaussian Scattering) have led to significant success for SLAM methods based on 3DGS (hereinafter referred to as 3DGS SLAM) in many application scenarios. 3DGS is a Gaussian-based method for representing and rendering 3D scenes, primarily used for efficient modeling and rendering of complex 3D scenes, especially suitable for dynamic environments or applications requiring real-time performance. By representing the scene as a dense Gaussian ellipsoid and employing tile-based rasterization techniques, 3DGS achieves faster rendering speeds and higher-quality rendering results. Therefore, 3DGS-based SLAM is becoming a powerful direction for achieving real-time, robust mapping in complex environments.

[0024] However, the 3DGS SLAM method based on related technologies still has the following problems: 1) The problem of lack of geometric information in weak texture areas: It is difficult to effectively extract surface information for localization and mapping in weak texture areas. This is because such techniques rely on photometric information, and the photometric information in weak texture areas is insufficient, making it difficult to support accurate reconstruction of surface geometry. As a result, the reconstructed surface appears to have missing parts or holes in these areas.

[0025] 2) Incomplete reconstruction of reflective areas: In highly reflective areas, the photometric information in multi-view images varies significantly, making consistent geometric extraction difficult. Existing methods often result in holes or discontinuities in the mesh model due to high light reflection, which limits the integrity and accuracy of the 3D model.

[0026] 3) Problems with poor surface reconstruction: Existing SLAM systems often suffer from poor surface reconstruction quality when performing 3D reconstruction of indoor scenes. For example, the surface is rough, and there are problems such as holes and breaks. This is especially true for common low-texture furniture, such as sofas and tabletops. In particular, for dark or highly reflective objects, even when using depth sensors, it is very easy to fail to receive effective reflected light, resulting in the failure of depth measurement and thus poor reconstruction quality of the final 3D model.

[0027] The embodiments of this disclosure provide a three-dimensional mapping method and apparatus based on polarization information enhancement, which at least partially solves the aforementioned problems existing in the related art.

[0028] refer to Figure 1 , Figure 1 A flowchart 100 of a three-dimensional mapping method based on polarization information enhancement according to an embodiment of the present disclosure is shown. In this embodiment, the flowchart 100 includes the following steps: Step 101: Determine the polarization information of the polarization images in the polarization image sequence, wherein the polarization image sequence includes multiple polarization images acquired at different times.

[0029] This step aims to determine the polarization information of each polarization image in the polarization image sequence by an entity (e.g., a computing device) performing a 3D mapping method based on polarization information enhancement.

[0030] In some implementations, the aforementioned executing entity can obtain a polarization image sequence based on polarization images acquired at different times from multiple polarization angles (multi-viewpoints). These multi-viewpoint polarization images can have multiple polarization angles. For example, a camera combined with a rotating polarizer can be used to change the polarization direction of the incident light, thereby capturing polarization images with different polarization angles (e.g., 0 degrees, 45 degrees, 90 degrees, and 135 degrees). Simultaneously, color (luminosity) information, such as RGB (red, green, blue) information, can be captured. The RGB information, as part of the observation data for 3D mapping, can be used to provide color information and aid in texture mapping. In other words, the polarization images of this disclosure record the color information of the scene as well as polarization information from multiple polarization angles.

[0031] In some exemplary embodiments, a multi-channel polarization camera can be used to capture multiple polarization angles simultaneously through a beam splitter prism and fuse RGB images captured by a regular camera to obtain the aforementioned polarization image.

[0032] In some other exemplary embodiments, a polarized color image with multiple polarization angles and containing RGB information can be obtained by a polarized color sensor with integrated polarization-RGB functionality.

[0033] Step 102: For each polarization image in the polarization image sequence, perform polarization chromaticity enhancement processing on the polarization image according to the polarization information of the polarization image to generate a chromaticity enhanced image, thus obtaining a chromaticity enhanced image sequence.

[0034] Based on step 101, this step aims to have the aforementioned executing entity perform polarization chromaticity enhancement processing on each polarization image in the polarization image sequence to generate a corresponding chromaticity enhanced image, thereby obtaining a chromaticity enhanced image sequence.

[0035] In computer graphics and material surface property analysis, the diffuse component and specular component are two key physical components describing the interaction between light and an object's surface, together constituting the object's visual appearance. In some applications, such as indoor environments, specular reflection is common. When the specular component is too strong, it causes inconsistencies in the polarization image across multiple viewpoints, thus making Gaussian fitting difficult.

[0036] As an exemplary implementation, polarization-guided chromaticity enhancement processing can be performed on polarization images based on the polarization information of each polarization image. By effectively separating the diffuse reflection component in the polarization image and suppressing the specular reflection component in the polarization image, a chromaticity enhanced image is obtained. This can remove most of the specular highlights, enhance the photometric consistency under multiple viewing angles, and thus improve positioning accuracy and mapping quality.

[0037] Step 103: Perform camera pose estimation on the image frames in the chroma-enhanced image sequence to obtain the target camera pose information.

[0038] This step aims to have the aforementioned execution entity perform camera pose estimation based on the polarized image enhanced with polarization chromaticity. Specifically, by estimating the camera position and orientation, target camera pose information is obtained to generate a Gaussian distribution. This target camera pose information can be used to describe the scene's geometry and the camera's motion trajectory.

[0039] In this embodiment, the camera pose information may include spatial location information and attitude information. The camera pose information may also be referred to as the camera pose sequence.

[0040] In an exemplary embodiment, spatial location information may include a spatial location array representing the camera's position in the world coordinate system. Attitude information may include an attitude array representing the camera's pose changes at multiple timestamps, and may be stored as an array in frame order.

[0041] Step 104: Based on the target camera pose information, perform 3D mapping based on 3D Gaussian sputtering to obtain a global Gaussian map of the target.

[0042] This step aims to build upon step 103 by performing 3D mapping based on 3DGS using the obtained target camera pose information, thereby obtaining a global Gaussian map of the target.

[0043] The core idea of ​​3DGS is to represent objects or volumes in three-dimensional space as a set of multiple Gaussian functions. Each Gaussian function can be viewed as an ellipsoid with specific properties in three-dimensional space, including position, covariance matrix, color, and transparency. By generating and optimizing these Gaussian parameters to fit observation data (such as multi-view polarization images), high-quality rendering effects are ultimately achieved.

[0044] The Gaussian global map can be viewed as a set of all 3D Gaussian ellipsoids representing the scene, and each 3D Gaussian ellipsoid can be defined by the following parameters: Location: Usually a mean vector, defining the center coordinates of the Gaussian ellipsoid in three-dimensional space; Covariance matrix: Defines the shape, size, and orientation of the Gaussian distribution; Color: Defines RGB color information and lighting information, usually represented by RGB values, which determine the color and luminance of the Gaussian ellipsoid; Transparency: Controls the visibility of the Gaussian ellipsoid during the rendering process, enabling multiple Gaussian functions to blend naturally when superimposed, avoiding the appearance of residual boundaries.

[0045] The three-dimensional mapping method based on polarization information enhancement disclosed in the above embodiments of the present invention can suppress the specular reflection effect in the reflective area during synchronous positioning and mapping by determining the polarization information in the polarization image and performing polarization-guided chromaticity enhancement processing on the polarization image according to the polarization information, thereby reducing specular reflection and realizing the true geometric restoration and accurate spatial positioning of the reflective area. This ensures the accuracy of positioning using polarization images and the accuracy and continuity of Gaussian points in highly reflective areas.

[0046] Continue to refer to Figure 2 , Figure 2 A flowchart 200 of a three-dimensional mapping method based on polarization information enhancement according to another embodiment of the present disclosure is shown. In this embodiment, the flowchart 200 includes the following steps: Step 201: Obtain the polarization image sequence.

[0047] In this embodiment, the polarization image sequence can be obtained by the execution subject (e.g., electronic device) of the three-dimensional mapping method based on polarization information enhancement. For example, multi-view polarization images at different times can be acquired in real time by a polarization camera or polarization sensor, and after preprocessing such as noise removal, a polarization image sequence can be obtained; or, a polarization image sequence can be obtained by acquiring multi-view polarization images at different times that have been stored in advance.

[0048] Each polarization image in the polarization image sequence may include color information and has multiple polarization angles: 0 degrees, 45 degrees, 90 degrees, and 135 degrees. That is, the polarization image records the scene's color information and polarization information at multiple angles. Further details regarding the specific implementation of step 201 can be found in the description of step 101 above, and will not be repeated here.

[0049] Step 202: Determine the polarization information of the polarization images and the RGB images in the polarization image sequence.

[0050] In this embodiment, the polarization intensity of a polarization image at different polarization angles can be characterized by the Stokes vector. In 3D scene reconstruction, the Stokes vector serves as a bridge connecting polarization observation with surface geometry (normal, material).

[0051] The Stokes vector fully characterizes the intensity, polarization direction, and ellipticity of light using four parameters, which can be represented as follows:

[0052] in, For Stokes vectors, The total light intensity (unpolarized component) is the sum of the light intensities in all polarization directions. This represents the intensity difference of linearly polarized light at 0 degrees and 90 degrees. The intensity difference of linearly polarized light at 45 degrees and 135 degrees. This represents the circularly polarized component of light.

[0053] In this embodiment, the polarization information of the polarization image can be determined based on the Stokes vector. The polarization information may include: total light intensity information. Information such as AoLP (Angle of Linear Polarization) and DoLP (Degree of Linear Polarization).

[0054] Among them, the linear polarization angle Defined as:

[0055] linear polarization degree Defined as:

[0056] In this embodiment, the RGB image can also be determined based on the color information in the polarization image, where the color information can be image color values, such as RGB information: red, green, and blue.

[0057] Step 203: Perform polarization chromaticity enhancement processing on the polarization image to obtain a chromaticity enhanced image sequence.

[0058] In this step, based on step 202, the diffuse reflection component and the specular reflection component in the polarized image are polarized and the specular reflection component is suppressed to generate a chromaticity enhanced image of the polarized image.

[0059] As an exemplary implementation, the polarization image can be subjected to the following polarization chromaticity enhancement processing:

[0060] in, This is the reflection component after polarization chromaticity enhancement processing.

[0061] In the above method, by utilizing polarization AoLP and DoLP information and introducing polarization chromaticity enhancement algorithms, such as subtracting the high polarization degree (e.g., DoLP greater than a predetermined threshold) of the high light reflection component from the total light intensity, specular reflection can be effectively suppressed and reduced, maintaining consistency under multiple viewing angles. This avoids the problem of poor positioning and reconstruction quality caused by the specular effect in the reflection area during the 3D reconstruction process, thus providing a more complete reconstruction effect in scenes with specular or specular reflection, and meeting the requirements of high-quality reconstruction.

[0062] Step 204: Traverse the chroma-enhanced image sequence and perform camera pose estimation.

[0063] In this step, each frame in the chroma-enhanced image sequence obtained in step 203 is traversed, and spatial position tracking is performed, that is, camera pose estimation is performed on the current frame to obtain the camera pose information of the current frame.

[0064] As an exemplary implementation, camera pose information may include the camera's spatial position coordinates, as well as the camera's rotation matrix, translation vector, etc.

[0065] Step 205: Determine whether the current frame is a keyframe. If it is, proceed to step 206; otherwise, return to step 204.

[0066] In this embodiment, in order to save computing resources and improve computing efficiency, key frames are dynamically selected from the image frames in the chroma image sequence according to a certain key frame selection strategy based on the degree of scene change. During the mapping process, Gaussian ellipsoids are created for new observation perspectives or regions based on the camera pose estimation results of these key frames, thereby reducing redundant calculations.

[0067] As an exemplary implementation, the keyframe selection strategy may include: determining whether the current frame is a keyframe based on whether the spatial feature differences and / or polarization feature differences between the current frame and the previous keyframe meet preset conditions; if the preset conditions are met, the current frame is determined as a keyframe; otherwise, camera pose estimation continues for the next frame. The spatial feature differences may be, for example, the common visibility of relative displacement or photometric information, and the polarization feature differences may be the polarization differences within the common viewing area.

[0068] Specifically, the spatial feature differences and / or polarization feature differences between the current frame and the previous keyframe meet preset conditions, which may include one or more of the following: Preset condition 1: The relative movement distance between the current frame and the previous keyframe exceeds the preset distance threshold.

[0069] For example, if the relative movement distance from the previous keyframe k to the current frame i exceeds a preset distance threshold, indicating a large camera movement, it suggests that a new observation area has been entered, providing a wealth of previously unrecorded geometric structures or frame details. The preset distance threshold can be dynamically determined based on the actual situation; for instance, it can be calculated by multiplying a translation coefficient by the average depth of the scene observed in the current frame i. Here, "average depth" refers to the median distance from all feature points in the current frame to the camera, which can be roughly understood as the overall distance of the scene viewed in the current frame.

[0070] Preset condition 2: The co-view crossover ratio of luminance information between the current frame and the previous keyframe is lower than the preset visibility threshold.

[0071] Common visibility refers to how many "Gaussian objects" (which can be imagined as small regions or feature points in a scene represented by a set of Gaussian functions) in an RGB image are mutually visible.

[0072] In this embodiment, when comparing any current frame i and the previous keyframe k, their "shared visibility" can be measured in the following two ways: IOU (Intersection over Union) can be calculated as follows: divide the number of Gaussian objects that can be seen in both RGB images by the total number of Gaussian objects that can be seen in each of the two RGB images (where overlapping objects are counted only once).

[0073] OC (Overlap coefficient) can be calculated as follows: divide the number of Gaussian objects that can be seen in both RGB images by the smaller number of Gaussian objects that can be seen in each of the two RGB images.

[0074] For example, if the intersection-union ratio of the Gaussian objects that can be seen in the current frame i and the previous keyframe k is lower than a preset visibility threshold, it means that the scene information presented in the current frame has little overlap with the content covered by the previous keyframe, and it is very likely that the current frame has captured a significantly different perspective or new scene content.

[0075] Preset condition 3: The polarization difference between the current frame and the previous keyframe in the shared viewing area exceeds the preset difference threshold.

[0076] To address the characteristics of large areas with weak textures but rich polarization information in indoor scenes, the polarization difference between the current frame and the previous keyframe is considered. If the number of pixels in the shared viewing area between the current frame and the previous keyframe is less than a preset minimum, the polarization difference can be ignored to avoid mismatches caused by data sparsity. If the number of pixels in the shared viewing area meets the requirements, the polarization difference is calculated within that shared viewing area to evaluate the structural consistency between the current frame and the previous keyframe.

[0077] Specifically, the differences in linear polarization angle and linear polarization degree for each pixel within the shared viewing area are first calculated, and these differences are then weighted and averaged. Next, the weighted average of the linear polarization angle and linear polarization degree differences is combined to obtain a comprehensive polarization difference metric. This polarization difference metric can be used to reflect the consistency of polarization information between the current frame and the previous keyframe within the shared viewing area.

[0078] If the calculated polarization difference metric exceeds a preset difference threshold, the current frame is registered as a new keyframe and included in the map update window. This strategy helps to accurately identify keyframes with significant structural changes in dynamic scenes, thereby improving the robustness and accuracy of the tracking system.

[0079] For example, for the current frame i and the previous keyframe k, their respective Gaussian visibility masks are projected onto the image plane, and their common visible region mask is calculated:

[0080] in, This is the common visible region mask between the current frame i and the previous keyframe k, used to identify scene regions observed simultaneously from multiple viewpoints (or at different times). Let i be the Gaussian visibility mask for the current frame i. The Gaussian visibility mask for the previous keyframe k. for and The intersection, that is and Public areas.

[0081] If the number of pixels in the shared viewing area is less than a preset minimum, the polarization difference is ignored. Otherwise, within this shared viewing area, the Gaussian visible polarization difference is calculated using the following method. :

[0082] in,

[0083] in, and These represent the AoLP (linear polarization angle) of pixel p in the current frame i and the previous keyframe k, respectively. The difference in linear polarization angle between the current frame i and the previous keyframe k. and These represent the DoLP (linear polarization degree) of pixel p in the current frame i and the previous keyframe k, respectively. This represents the difference in linear polarization between the current frame i and the previous keyframe k. |M| represents the total number of pixels in this shared-view region. The custom weighting parameter, with a value between 0 and 1, is used to control the importance of differences in linear polarization angle and degree of linear polarization.

[0084] When Gaussian visible polarization difference When the difference exceeds a preset threshold, the current frame is registered as a new keyframe.

[0085] In this embodiment, for the current frame in the frame sequence, according to the keyframe selection strategy, if any of the preset conditions regarding relative displacement, co-view cross-union ratio of photometric information, and polarization difference are met, the current frame is registered as a new keyframe. By considering the correlation between the current frame and the immediately preceding keyframe in terms of spatial and / or polarization characteristics, the robustness and accuracy of keyframe selection are improved, enabling the entire system to achieve good mapping results even in environments with repetitive or weak textures.

[0086] Step 206: Create a Gaussian ellipsoid based on the camera pose and color information of the keyframes.

[0087] In this embodiment, the camera pose information and color information of the key frame are used as input (e.g., the camera pose information and RGB image are input simultaneously). Based on 3DGS, Gaussian parameters such as position, shape, size, and transparency of the Gaussian ellipsoid are generated, and the Gaussian ellipsoid corresponding to the key frame is created (added) in the intermediate Gaussian global map.

[0088] In some implementations, after creating the Gaussian ellipsoid, the camera pose information of the keyframe is optimized by minimizing the photometric error between the rendered photometric image (e.g., the rendered RGB image) obtained from the intermediate Gaussian global map, based on the photometric image. This optimized camera pose information is then used as the target camera pose information for 3D mapping.

[0089] As an exemplary implementation, in the camera pose tracking thread, for each keyframe, the camera pose information is continuously optimized by minimizing the photometric residual of the RGB image:

[0090] in, For photometric residuals, Represents the camera-world transformation matrix Gaussian mixture image generated by projection, This represents the observed actual image, i.e., the input RGB image.

[0091] In this step, keyframes are selected from each input frame according to a keyframe selection strategy, and these keyframes are used for 3D mapping, thereby saving computing resources and improving computing efficiency.

[0092] Step 207: Determine whether the traversal is complete. If the traversal is complete, continue to step 208; otherwise, return to step 204 to continue the camera pose estimation for the next frame.

[0093] Step 208: Obtain the target Gaussian global map.

[0094] For each keyframe, based on the optimized camera pose information of that keyframe, a Gaussian ellipsoid corresponding to that keyframe is created in the intermediate Gaussian global map. This process continues until the Gaussian ellipsoids corresponding to all keyframes are created. Then, the current intermediate Gaussian global map is used as the target Gaussian global map.

[0095] In an exemplary embodiment, after obtaining the target Gaussian global map, the target Gaussian global map can be optimized using the following steps 209-211.

[0096] Step 209, polarization plane segmentation.

[0097] In practical applications, polarization cameras, as passive sensors, are not limited by distance or angle and exhibit good stability across various colors and materials. Since most surfaces in indoor environments are planar, polarization information such as AoLP (angle of linear polarization) and DoLP (degree of linear polarization) can effectively distinguish objects and their geometric features. Based on this, this disclosure proposes a planar segmentation method based on hierarchical density clustering.

[0098] In an exemplary embodiment, based on the polarization information and RGB information of each polarized image in the polarization image sequence determined in step 202, the RGB information, linear polarization angle, and degree of linear polarization spectral information in the polarized image are weighted and concatenated along the feature dimension to generate a polarization enhancement feature vector for each pixel in the polarization image. Then, all polarization enhancement feature vectors of the polarization image are clustered, and the polarization image is segmented based on the clustering results to obtain multiple initial planar regions. For example, these polarization enhancement feature vectors can be input into a clustering algorithm (such as the HDBSCAN algorithm) to obtain preliminary planar segmentation results.

[0099] Step 210: Construct a multiple loss function.

[0100] In this embodiment, based on the preliminary plane segmentation results obtained in step 209, a multi-scale planar self-constrained framework for optimizing surface normal vectors is proposed.

[0101] As an exemplary implementation, a multiple loss function is defined. The weighted sum of the following loss functions:

[0102] in, Let the neighborhood consistency loss function be... Let the global plane consistency loss function be used. Let Manhattan World Loss Function be used. Let α be the photometric consistency loss function, and α be a hyperparameter used to balance the weights of different loss terms.

[0103] The following will explain each loss function in detail.

[0104] Neighborhood consistency loss function It is used to control the changes between local neighborhoods within a planar segmentation region, with the goal of making the reconstructed 3D surface smoother and more realistic.

[0105] As an example, we perform a weighted average of the neighborhood normals of each pixel's region, using a two-dimensional Gaussian weight distribution. Specifically, when calculating the neighborhood consistency loss, the normal of each pixel is compared to the weighted average of the normals of other pixels in its neighborhood to measure local normal differences. This process penalizes normal variations within the same plane of the image, encouraging consistency between normals of adjacent pixels and avoiding unsmooth normals caused by noise or high-frequency variations. Furthermore, by evaluating the overall local variations in each plane and normalizing the average differences across different plane regions, we construct the local neighborhood consistency loss. By respecting the inherent structure of smooth geometry in natural images, we can achieve smoother surface geometry in 3D Gaussian rendering.

[0106] Global plane consistency loss function This method aims to enhance the geometric consistency of planar regions in 3D reconstruction, ensuring that the normal directions of all pixels within each plane remain consistent. For example, the image is segmented into several planar regions, and a representative normal vector is calculated for each region. The overall consistency of the current prediction is measured by comparing the similarity between each pixel's normal and its region's representative normal (e.g., using cosine similarity), and by averaging the similarity differences among all pixels within each planar region. This global planar consistency loss effectively limits the deviation of pixel normal directions within a single plane, contributing to the generation of planar models with clearer structures and more coherent geometry.

[0107] Manhattan World Loss Function This is achieved by defining a structural loss between planes to penalize deviations of normal vectors from parallel or orthogonal planes. In indoor scenes, the "Manhattan world" assumption posits that dominant planes are either parallel or orthogonal. Let's assume... Given a set of plane normal vectors, weighted by incorporating the plane's area, it can be represented as follows:

[0108] in, This represents the total number of valid plane pairs. A valid plane pair is a set of planes that has practical representational value for the geometric structure of the scene and satisfies specific constraints. K The number of valid planes is given by m and n, where m and n represent two distinct valid planes. The unit normal direction of the corresponding plane. The cosine similarity of the normals is represented by a value of 0, indicating that planes m and n are orthogonal, and a value of 1, indicating that planes m and n are parallel. Weights This is an area weight used to enhance the impact of a larger plane on the loss function.

[0109] Photometric consistency loss function This is a consistency loss based on RGB luminance, used to measure the difference between the rendered result and the actual observed image (i.e., the input RGB image).

[0110] Construct a multiple loss function based on the above loss function. By minimizing this multiple loss function to apply multi-scale planar self-constraints, the surface quality of 3D reconstruction can be optimized.

[0111] Step 211: Constrain the result of the target Gaussian global map rendering.

[0112] In this embodiment, the constructed multiple loss function can be used as a basis. The Gaussian global map rendering result (image) is constrained, and the gradient of Gaussian parameters (position, shape, rotation, etc.) is calculated using differentiable rendering (such as α-blending rendering). The Gaussian distribution is then optimized through backpropagation to obtain the optimized target Gaussian global map. This makes the reconstruction of textureless and reflective areas of the Gaussian global map more accurate. Specifically, multi-scale planar self-constraints can be applied to highly polarized regions in the rendering result (such as regions with linear polarization higher than the average value) to optimize the scene surface, thereby obtaining a smoother and more accurate surface reconstruction result.

[0113] In an exemplary embodiment, based on the preliminary planar segmentation results, a region merging strategy can be used to remove excessively small regions resulting from over-segmentation, thus obtaining the final planar segmentation results.

[0114] Step 212, (optional) Export the 3D mesh model.

[0115] Optionally, in this step, a 3D mesh model can be exported based on the optimized target Gaussian global map. For example, using the depth map rendered from the optimized target Gaussian global map, a complete and compact mesh can be extracted using TSDF (Truncated Signed Distance Function) to generate dense triangular patches. TSDF is a technique used for 3D reconstruction and scene representation. In SLAM systems, TSDF is commonly used to reconstruct 3D scenes from continuous depth images or point cloud data.

[0116] In some applications of 3D scene reconstruction, such as robot navigation and augmented reality, a constructed Gaussian global map can be converted into a 3D mesh model. This allows the robot to better understand the environment and plan its path, while also making it easier for users to visually view the reconstructed scene.

[0117] The three-dimensional mapping method based on polarization information enhancement in the above embodiments of this disclosure can achieve at least one of the following beneficial effects: (1) Optimization of specular reflection region based on polarization information: The polarization image is enhanced in color and optimized in specular reflection region by using polarization information. The specular reflection is suppressed in the reflection region, and the consistency of spectral information in the SLAM process is improved. This enables the Gaussian point cloud to generate a complete and hole-free 3D model in a high-reflection scene, which significantly improves the integrity of the model surface reconstruction.

[0118] (2) Robust keyframe selection strategy based on polarization information enhancement to improve mapping performance: In response to the characteristics of a large number of repetitive textures and less RGB feature information in indoor scenes, polarization feature differences are introduced as information supplementation and fusion, thereby enhancing the robustness of keyframe selection and providing more accurate input for the mapping thread. End-to-end simultaneous localization and mapping are achieved through monocular RGB image loss, and the number, position, shape, size, transparency and other attributes of the Gaussian global map are continuously optimized in a way that minimizes image error, so that the spatial position is converged and determined. (3) Improve the surface reconstruction quality of the scene: By utilizing a large number of planar features in the indoor environment, and employing methods such as planar segmentation and multi-scale planar constraints, the surface reconstruction quality of the scene is significantly improved. Even on weakly textured surfaces with fewer feature points, good reconstruction results can be achieved. By fusing polarization features with RGB features, simple and effective planar segmentation can be achieved using traditional clustering algorithms, avoiding the dependence of traditional deep learning methods on a large amount of training data and effectively improving the generalization of planar recognition and segmentation. By aggregating the results of smaller regions to larger surrounding regions, unnecessary fragmented segmentation is reduced. Three loss functions, namely neighborhood consistency loss, planar consistency loss, and Manhattan world loss, are designed to construct a multi-scale planar self-constraint framework, thereby enabling the optimization of the scene surface even without prior knowledge of real normals, resulting in smoother and more accurate surface reconstruction results. Furthermore, the reconstructed 3D model has high smoothness and accuracy in weakly textured and highly reflective areas, which can meet the needs of 3D modeling applications such as robot navigation and virtual reality.

[0119] Next reference Figure 3 As an implementation of the methods shown in the above figures, this disclosure provides a three-dimensional mapping device 300 based on polarization information enhancement, which can be used with... Figure 1 , Figure 2 Corresponding to the method embodiments or processing flows shown, this device can be specifically applied to various servers or electronic devices.

[0120] like Figure 3As shown, the 3D mapping device 300 based on polarization information enhancement provided in this embodiment may include a polarization enhancement module 301, a positioning module 302, and a mapping module 303. The polarization enhancement module 301 is configured to determine the polarization information of polarized images in a polarization image sequence, wherein the polarization image sequence includes multiple polarization images acquired at different times; for each polarization image in the polarization image sequence, polarization chromaticity enhancement processing is performed on the polarization image based on its polarization information to generate a chromaticity enhanced image sequence. The positioning module 302 is configured to perform camera pose estimation on the image frames in the chromaticity enhanced image sequence to obtain target camera pose information. The mapping module 303 is configured to perform 3D mapping based on 3D Gaussian sputtering according to the target camera pose information to obtain a target Gaussian global map.

[0121] In some optional implementations of this embodiment, the polarization image has multiple polarization angles; the polarization enhancement module 301 is further configured to: characterize the polarization intensity of the polarization image at different polarization angles using Stokes vectors; and determine the polarization information of the polarization image based on the Stokes vectors, wherein the polarization information includes: total light intensity information, linear polarization angle information, and linear polarization degree information.

[0122] In some optional implementations of this embodiment, the polarization enhancement module 301 is further configured to: for each polarization image in the polarization image sequence, perform polarization separation on the diffuse reflection component and the specular reflection component in the polarization image according to the total light intensity information and linear polarization degree information of the polarization image; and generate a chromaticity enhanced image of the polarization image by suppressing the specular reflection component in the polarization image.

[0123] In some optional implementations of this embodiment, the positioning module 302 is further configured to traverse each frame in the chroma enhancement image sequence and perform the following operations in sequence: perform camera pose estimation on the current frame to obtain the camera pose information of the current frame; in response to the spatial feature difference and / or polarization feature difference between the current frame and the previous key frame satisfying a preset condition, determine the current frame as a key frame; and obtain the target camera pose information based on the camera pose information of the key frame.

[0124] In some optional implementations of this embodiment, the mapping module 303 is further configured to: for each keyframe, based on the optimized camera pose information of the keyframe, create a Gaussian ellipsoid corresponding to the keyframe in the intermediate Gaussian global map based on three-dimensional Gaussian sputtering; in response to the completion of the creation of all Gaussian ellipsoids corresponding to all keyframes, obtain the target Gaussian global map based on the current intermediate Gaussian global map.

[0125] In some optional implementations of this embodiment, the device 300 may further include an optimization module 304 (not shown), used to optimize the obtained target Gaussian global map as follows: for each polarized image in the polarized image sequence, the color information and polarization information in the polarized image are weighted and concatenated along the feature dimension to generate a polarization enhancement feature vector for each pixel in the polarized image; all polarization enhancement feature vectors of the polarized image are clustered, and the polarized image is segmented according to the clustering results to obtain multiple initial planar regions; the result of rendering the target Gaussian global map is constrained according to the determined multiple loss function, and after backpropagation, the optimized target Gaussian global map is obtained; wherein, the multiple loss function includes a weighted sum of the neighborhood consistency loss function, the global plane consistency loss function, the Manhattan world loss function, and the luminosity consistency loss function.

[0126] In the polarization information enhancement-based 3D mapping device 300 of this embodiment, the specific processing of the polarization enhancement module 301, the positioning module 302, the mapping module 303, and the optimization module 304, and the resulting technical effects, can be found in the respective references. Figure 1 , Figure 2 The relevant descriptions of the steps or processes in the corresponding implementation methods will not be repeated here.

[0127] The three-dimensional mapping apparatus based on polarization information enhancement disclosed in the above embodiments of this invention uses polarization information to separate specular and diffuse reflection regions of RGB images, suppressing highlights in specular reflection regions and enhancing photometric consistency under multiple viewing angles, thereby improving subsequent positioning accuracy and mapping quality. The positioning module combines the differences in polarization features between adjacent frames, and enhances the robustness of the key frame selection strategy by simultaneously considering relative displacement, photometric information co-view cross-interference-of-extension ratio, and polarization differences, enabling it to adapt to indoor weak texture environments. The optimization module uses polarization information to reflect the characteristics of object azimuth angles, and generates a feature vector for each pixel in the image by performing feature weighted fusion of AoLP, DoLP, and RGB information. Based on the feature vectors, a clustering algorithm is used for classification, thereby segmenting different planes. Using the plane segmentation results, self-constraints are applied at three scales: within the neighborhood, within the plane, and between planes, so that the reconstructed three-dimensional model can obtain excellent and robust three-dimensional mapping effects of scene surfaces even without real prior information.

[0128] In summary, the polarization-enhanced 3D mapping method and apparatus provided in the above embodiments of this disclosure address the problem of poor scene surface reconstruction quality in existing 3DGS-based SLAM systems. By introducing polarization cues to improve surface reconstruction quality, and through polarization segmentation and multi-scale planar self-constraint, a smoother and more accurate geometric representation is achieved, ultimately generating a high-quality 3D model. More specifically, the polarization-enhanced 3D mapping method and apparatus in the above embodiments of this disclosure may have one of the following characteristics: (a) Completeness and accuracy of specular and specular reflection region reconstruction: By utilizing information such as polarization AoLP and DoLP, a chromaticity map enhancement algorithm is introduced to reduce specular reflection and maintain consistency across multiple viewpoints, thereby eliminating the problem of poor localization and reconstruction quality caused by the Gaussian specular effect in reflective regions during SLAM. Compared to existing mapping methods, this approach provides more complete reconstruction results in specular or specular reflection scenes, meeting the requirements for high-quality reconstruction.

[0129] (b) Polarization plane segmentation mechanism with strong generalization ability without training: By calculating the feature vector after weighted fusion of RGB information and polarization information, it can be used to segment the plane using common clustering algorithms. Thus, it can achieve good generalization effect without a large amount of training data, providing a good prior foundation for 3D reconstruction and geometric recognition.

[0130] (c) Smooth and accurate 3D reconstruction effect of surface: By deeply fusing polarization information such as AoLP and DoLP, more accurate surface reconstruction effect can be achieved in the weak texture environment commonly found indoors; combined with polarization plane segmentation and multi-scale plane self-constraint strategies, the phenomena such as holes, grooves and faults on the reconstructed surface are reduced, so that the generated Gaussian points have better geometric representation ability.

[0131] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0132] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. It should be understood that... Figure 4The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of this disclosure described and / or claimed herein.

[0133] refer to Figure 4 Electronic device 400 may include a processor (e.g., CPU, Central Processing Unit) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage section 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of system 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0134] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0135] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined above in the methods of embodiments of this disclosure.

[0136] It should be noted that the computer-readable medium described in the embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0137] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: first, determine the polarization information of polarized images in a polarization image sequence, wherein the polarization image sequence includes multiple polarization images acquired at different times; then, for each polarization image in the polarization image sequence, perform polarization chromaticity enhancement processing on the polarization image according to the polarization information of that polarization image to generate a chromaticity enhanced image, thereby obtaining a chromaticity enhanced image sequence; next, perform camera pose estimation on the image frames in the chromaticity enhanced image sequence to obtain target camera pose information; finally, based on the target camera pose information, perform 3D mapping based on 3D Gaussian sputtering to obtain a target Gaussian global map.

[0138] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may include a polarization enhancement module, a positioning module, and a mapping module. The names of these modules do not necessarily limit the module itself; for example, the positioning module may also be described as a "pose estimation module" or a "pose tracking module," etc.

[0141] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A three-dimensional mapping method based on polarization information enhancement, comprising: Determine the polarization information of polarization images in a polarization image sequence, wherein the polarization image sequence includes multiple polarization images acquired at different times; For each polarization image in the polarization image sequence, polarization chromaticity enhancement processing is performed on the polarization image according to the polarization information of the polarization image to generate a chromaticity enhanced image, thus obtaining a chromaticity enhanced image sequence; Iterate through each frame of the chroma-enhanced image sequence and perform the following operations in sequence: Perform camera pose estimation on the current frame to obtain the camera pose information for the current frame; In response to the spatial feature difference and / or polarization feature difference between the current frame and the previous key frame meeting the preset conditions, the current frame is determined as a key frame; using the camera pose information and color information of the key frame as input, a Gaussian ellipsoid corresponding to the key frame is created in the intermediate Gaussian global map. Based on the photometric image rendered from the intermediate Gaussian global map, the camera pose information of the key frame is optimized by minimizing the photometric error between the photometric image and the key frame, thus obtaining the optimized camera pose information of the key frame. For each keyframe, based on the optimized camera pose information of the keyframe, a Gaussian ellipsoid corresponding to the keyframe is created in the intermediate Gaussian global map based on 3D Gaussian sputtering. Once all Gaussian ellipsoids corresponding to all keyframes have been created, the target Gaussian global map is obtained based on the current intermediate Gaussian global map. Wherein, the spatial feature difference and / or polarization feature difference between the current frame and the previous keyframe meets a preset condition, including at least one of the following: The relative movement distance between the current frame and the previous keyframe exceeds a preset distance threshold. The co-view crossover ratio of photometric information between the current frame and the previous keyframe is lower than the preset visibility threshold; The polarization difference between the current frame and the previous keyframe in the shared viewing area exceeds the preset difference threshold.

2. The method according to claim 1, wherein, The polarization image has multiple polarization angles; The determination of polarization information of polarization images in the polarization image sequence includes: The polarization intensity of the polarization image at different polarization angles is characterized by Stokes vectors. The polarization information of the polarization image is determined based on the Stokes vector, wherein the polarization information includes: total light intensity information, linear polarization angle information, and linear polarization degree information.

3. The method according to claim 2, wherein, The step of performing polarization chromaticity enhancement processing on each polarization image in the polarization image sequence, based on the polarization information of that polarization image, to generate a chromaticity enhanced image includes: Polarization separation is performed on the diffuse reflection component and the specular reflection component in the polarization image based on the total light intensity information and the linear polarization degree information of the polarization image; A chromaticity-enhanced image of the polarized image is generated by suppressing the specular reflection component in the polarized image.

4. The method according to claim 1 further includes optimizing the obtained target Gaussian global map as follows: For each polarized image in the polarized image sequence, the color information and polarization information in the polarized image are weighted and concatenated along the feature dimension to generate a polarization enhancement feature vector for each pixel in the polarized image; Cluster all the polarization enhancement feature vectors of the polarization image, and segment the polarization image based on the clustering results to obtain multiple initial planar regions; Based on the determined multiple loss function, the result of rendering the target Gaussian global map is constrained, and after backpropagation, the optimized target Gaussian global map is obtained. in, The multiple loss function includes a weighted sum of the neighborhood consistency loss function, the global plane consistency loss function, the Manhattan world loss function, and the luminosity consistency loss function.

5. The method according to claim 4, wherein, The neighborhood consistency loss function is determined as follows: the normal of each pixel in each planar region is compared with the weighted average of the normals of other pixels in its neighborhood, and the neighborhood consistency loss function is determined based on the comparison result. The global plane consistency loss function is determined as follows: a representative normal vector is determined for each plane region, the similarity between the normal vector of each pixel in the plane region and the representative normal vector is compared, and the global plane consistency loss function is determined based on the similarity of all pixels in each plane region. The Manhattan world loss function is determined as follows: for multiple planar regions, the Manhattan world loss function is determined by defining the inter-planar structural loss.

6. A three-dimensional mapping device based on polarization information enhancement, comprising: A polarization enhancement module is configured to determine the polarization information of polarization images in a polarization image sequence, wherein the polarization image sequence includes multiple polarization images acquired at different times; for each polarization image in the polarization image sequence, polarization chromaticity enhancement processing is performed on the polarization image according to the polarization information of the polarization image to generate a chromaticity enhanced image, thereby obtaining a chromaticity enhanced image sequence; The positioning module is configured to traverse each frame of the chroma-enhanced image sequence and perform the following operations in sequence: Perform camera pose estimation on the current frame to obtain the camera pose information for the current frame; In response to the spatial feature difference and / or polarization feature difference between the current frame and the previous key frame meeting the preset conditions, the current frame is determined as a key frame; using the camera pose information and color information of the key frame as input, a Gaussian ellipsoid corresponding to the key frame is created in the intermediate Gaussian global map. Based on the photometric image rendered from the intermediate Gaussian global map, the camera pose information of the key frame is optimized by minimizing the photometric error between the photometric image and the key frame, thus obtaining the optimized camera pose information of the key frame. The mapping module is configured to, for each keyframe, create a Gaussian ellipsoid corresponding to that keyframe in the intermediate Gaussian global map based on the optimized camera pose information of that keyframe and using 3D Gaussian sputtering; and in response to the completion of the creation of Gaussian ellipsoids for all keyframes, obtain the target Gaussian global map based on the current intermediate Gaussian global map. Wherein, the spatial feature difference and / or polarization feature difference between the current frame and the previous keyframe meets a preset condition, including at least one of the following: The relative movement distance between the current frame and the previous keyframe exceeds a preset distance threshold. The co-view crossover ratio of photometric information between the current frame and the previous keyframe is lower than the preset visibility threshold; The polarization difference between the current frame and the previous keyframe in the shared viewing area exceeds the preset difference threshold.

7. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

Images