Three-dimensional reconstruction method and system based on semantic guidance, terminal and storage medium

Abstract

The invention discloses a three-dimensional reconstruction method and system based on semantic guidance, a terminal and a storage medium, and relates to the technical field of computer graphic processing, and the method comprises the steps: obtaining RGB image data and semantic image data; initializing a Gaussian ellipsoid set, wherein parameters of each Gaussian ellipsoid comprise geometric attribute parameters, appearance attribute parameters and semantic attribute parameters; constructing a double-flow microrasterized pipeline, wherein the double-flow microrasterized pipeline comprises a luminosity rendering flow and a semantic rendering flow And alternately iteratively optimizing parameters corresponding to the Gaussian ellipsoids through the luminosity rendering flow and the semantic rendering flow until a preset iteration termination condition is met, obtaining a target Gaussian ellipsoid set, updating geometric attribute parameters and appearance attribute parameters according to the RGB image data and the generated luminosity rendering image when the luminosity rendering flow is activated, and obtaining a target Gaussian ellipsoid set. When the semantic rendering flow is activated, the geometric attribute parameters and the semantic attribute parameters are updated according to the semantic image data and the generated semantic rendering image. Therefore, the accuracy of three-dimensional reconstruction can be improved.

Description

Technical Field

[0001] This application relates to the field of computer graphics processing technology, and in particular to a semantically guided 3D reconstruction method, system, terminal, and storage medium. Background Technology

[0002] With the development of science and technology, especially computer vision technology, users' requirements for 3D reconstruction are gradually increasing.

[0003] In existing 3D reconstruction schemes, semantic features are typically applied as textures to the surface of the geometry after the 3D reconstruction is completed. The problem with existing technologies is that semantic features are only applied as textures to the surface of the reconstructed geometry, which cannot fully utilize semantic information to guide the optimization of the 3D geometry during the reconstruction process, thus hindering the improvement of the accuracy and effect of 3D reconstruction.

[0004] Therefore, the relevant technologies still need to be improved and developed. Summary of the Invention

[0005] The main purpose of this application is to provide a semantically guided 3D reconstruction method, system, terminal and storage medium, which aims to solve the technical problem in related technologies where semantic features are only used as textures pasted on the surface of the reconstructed geometry, and the semantic information cannot be fully used to guide the optimization of the 3D geometric structure during the reconstruction process, which is not conducive to improving the accuracy and effect of 3D reconstruction.

[0006] To achieve the above objectives, the first aspect of this application provides a semantically guided 3D reconstruction method, wherein the method includes: Obtain the RGB image data and semantic image data corresponding to the scene to be reconstructed; Initialize the Gaussian ellipsoid set, wherein the parameters corresponding to each Gaussian ellipsoid in the above Gaussian ellipsoid set include geometric attribute parameters, appearance attribute parameters, and semantic attribute parameters; A dual-stream differentiable rasterization pipeline is constructed, wherein the dual-stream differentiable rasterization pipeline includes a photometric rendering pipeline and a semantic rendering pipeline. The photometric rendering pipeline is used to perform rasterization rendering based on the geometric attribute parameters and appearance attribute parameters corresponding to the Gaussian ellipsoid to obtain a photometric rendering image. The semantic rendering pipeline is used to perform rasterization rendering based on the geometric attribute parameters and semantic attribute parameters corresponding to the Gaussian ellipsoid to obtain a semantic rendering image. Through the aforementioned photometric rendering stream and semantic rendering stream, the parameters corresponding to the aforementioned Gaussian ellipsoid are iteratively optimized alternately until a preset iteration termination condition is met, thereby obtaining a set of target Gaussian ellipsoids used to characterize the aforementioned scene to be reconstructed. Specifically, when the aforementioned photometric rendering stream is activated, the geometric and appearance attribute parameters corresponding to the aforementioned Gaussian ellipsoid are updated based on the aforementioned RGB image data and the generated aforementioned photometric rendering image. When the aforementioned semantic rendering stream is activated, the geometric and semantic attribute parameters corresponding to the aforementioned Gaussian ellipsoid are updated based on the aforementioned semantic image data and the generated aforementioned semantic rendering image.

[0007] Optionally, the acquisition of RGB image data and semantic image data corresponding to the scene to be reconstructed includes: Collect and obtain the multi-view RGB image sequence corresponding to the above-mentioned scene to be reconstructed, and use the above-mentioned multi-view RGB image sequence as the above-mentioned RGB image data; Using a pre-defined image segmentation model and a large language model, the multi-view RGB images in the above multi-view RGB image sequence are processed respectively to obtain the multi-view consistent semantic segmentation map corresponding to each of the above multi-view RGB images. For each multi-view consistent semantic segmentation map, a semantic mapping image is obtained by mapping the multi-view consistent semantic segmentation map according to the semantic label corresponding to each pixel in the multi-view consistent semantic segmentation map and the preset semantic label encoding method. The semantic image data described above is constructed based on all the semantic mapping images described above.

[0008] Optionally, the above-mentioned preset semantic tag encoding method is RGB encoding. The above-mentioned mapping process, based on the semantic labels corresponding to each pixel in the multi-view consistent semantic segmentation map and the preset semantic label encoding method, performs mapping processing on the multi-view consistent semantic segmentation map to obtain a semantic mapping image, including: Obtain a lookup table constructed based on the above RGB encoding method, wherein the lookup table is used to represent the mapping relationship between semantic tags and RGB encoded values; Based on the lookup table above, the semantic labels corresponding to each pixel in the multi-view consistent semantic segmentation image are mapped to the corresponding RGB encoded values ​​to obtain the semantic mapping image.

[0009] Optionally, the aforementioned geometric attribute parameters include position, rotation quaternion, scaling vector, and opacity; The aforementioned appearance attribute parameters include spherical harmonic coefficients; The semantic attribute parameters mentioned above include semantic color parameters.

[0010] Optionally, the above iteration termination condition is that the number of iterations reaches a preset total number of iterations; The parameters corresponding to the Gaussian ellipsoids are iteratively optimized alternately through the aforementioned photometric rendering stream and semantic rendering stream until a preset iteration termination condition is met, thereby obtaining a set of target Gaussian ellipsoids for characterizing the scene to be reconstructed, including: Obtain the iteration alternation period, the first iteration number used to control the number of times the photometric rendering stream is activated within one iteration alternation period, and the second iteration number used to control the number of times the semantic rendering stream is activated within one iteration alternation period; Within each iterative alternation cycle, the photometric rendering stream and the semantic rendering stream are activated sequentially according to the first iteration number and the second iteration number to perform alternating iterative optimization of the parameters corresponding to the Gaussian ellipsoid. When the photometric rendering stream is activated, the semantic attribute parameters are frozen, and the geometric and appearance attribute parameters corresponding to the Gaussian ellipsoid are updated according to the loss between the RGB image data and the generated photometric rendering image. When the semantic rendering stream is activated, the appearance attribute parameters are frozen, and the geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid are updated according to the loss between the semantic image data and the generated semantic rendering image. When the number of iterations reaches the preset total number of iterations, a set of target Gaussian ellipsoids is obtained to characterize the scene to be reconstructed.

[0011] Optionally, the above method further includes: Obtain the preset density control interval number and the preset gradient norm threshold; When the number of iterations reaches an integer multiple of the number of density control intervals, the mixed gradient norm corresponding to the Gaussian ellipsoid is determined based on the position gradient norm corresponding to the photometric rendering stream and the position gradient norm corresponding to the semantic rendering stream during the parameter alternation iteration optimization process. If the mixed gradient norm corresponding to the above Gaussian ellipsoid exceeds a preset gradient norm threshold, a preset density control operation is performed on the above Gaussian ellipsoid, wherein the preset density control operation includes splitting and / or cloning.

[0012] Optionally, if the mixing gradient norm corresponding to the Gaussian ellipsoid exceeds a preset gradient norm threshold, a preset density control operation is performed on the Gaussian ellipsoid, including: If the mixed gradient norm corresponding to the above Gaussian ellipsoid exceeds the preset gradient norm threshold, then the scaling vector corresponding to the above Gaussian ellipsoid is obtained. If the above scaling vector exceeds the preset first size threshold, then a splitting operation is performed on the above Gaussian ellipsoid. If the scaling vector does not exceed the preset second size threshold, then a cloning operation is performed on the Gaussian ellipsoid.

[0013] A second aspect of this application provides a semantically guided 3D reconstruction system, wherein the system comprises: The data acquisition module is used to acquire RGB image data and semantic image data corresponding to the scene to be reconstructed; The Gaussian ellipsoid initialization module is used to initialize a set of Gaussian ellipsoids, wherein the parameters corresponding to each Gaussian ellipsoid in the set include geometric attribute parameters, appearance attribute parameters, and semantic attribute parameters. A rasterization pipeline construction module is used to construct a dual-stream differentiable rasterization pipeline, wherein the dual-stream differentiable rasterization pipeline includes a photometric rendering stream and a semantic rendering stream. The photometric rendering stream is used to perform rasterization rendering based on the geometric and appearance attribute parameters corresponding to the Gaussian ellipsoid to obtain a photometric rendering image, and the semantic rendering stream is used to perform rasterization rendering based on the geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid to obtain a semantic rendering image. The 3D reconstruction module is used to iteratively optimize the parameters corresponding to the Gaussian ellipsoid through the aforementioned photometric rendering stream and semantic rendering stream until a preset iteration termination condition is met, thereby obtaining a set of target Gaussian ellipsoids representing the scene to be reconstructed. When the photometric rendering stream is activated, the geometric and appearance attribute parameters corresponding to the Gaussian ellipsoid are updated based on the aforementioned RGB image data and the generated photometric rendering image. When the semantic rendering stream is activated, the geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid are updated based on the aforementioned semantic image data and the generated semantic rendering image.

[0014] A third aspect of this application provides a terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements any of the steps of the semantically guided 3D reconstruction method described above.

[0015] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the steps of the semantically guided 3D reconstruction method described above.

[0016] As can be seen from the above, the present application provides a semantically guided 3D reconstruction method, which includes: acquiring RGB image data and semantic image data corresponding to the scene to be reconstructed; initializing a Gaussian ellipsoid set, wherein the parameters corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set include geometric attribute parameters, appearance attribute parameters, and semantic attribute parameters; constructing a two-stream differentiable rasterization pipeline, wherein the two-stream differentiable rasterization pipeline includes a photometric rendering stream and a semantic rendering stream, wherein the photometric rendering stream is used to perform rasterization rendering based on the geometric attribute parameters and appearance attribute parameters corresponding to the Gaussian ellipsoid to obtain a photometric rendered image, and the semantic rendering stream is used to perform rasterization rendering based on the geometric attribute parameters and appearance attribute parameters corresponding to the Gaussian ellipsoid to obtain a photometric rendered image. The geometric and semantic attribute parameters are rasterized and rendered to obtain a semantic rendering image. The parameters corresponding to the Gaussian ellipsoid are iteratively optimized alternately through the above photometric rendering stream and semantic rendering stream until the preset iteration termination condition is met, and a set of target Gaussian ellipsoids for representing the above scene to be reconstructed is obtained. When the above photometric rendering stream is activated, the geometric and appearance attribute parameters corresponding to the Gaussian ellipsoid are updated according to the above RGB image data and the generated above photometric rendering image. When the above semantic rendering stream is activated, the geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid are updated according to the above semantic image data and the generated above semantic rendering image.

[0017] Therefore, semantic features are not merely applied as textures to the reconstructed geometry; rather, semantic information guides the optimization of the 3D geometry during the reconstruction process. Specifically, a dual-stream differentiable rasterization pipeline is constructed, including a photometric rendering stream and a semantic rendering stream, both sharing the geometric property parameters of a Gaussian ellipsoid. The photometric rendering stream renders based on both geometric and appearance property parameters, while the semantic rendering stream renders based on both. In this application, the parameters of the Gaussian ellipsoid are iteratively optimized alternately based on the photometric and semantic rendering streams during 3D reconstruction. This achieves the optimization of the 3D geometry directly guided by RGB image data and semantic image data. The semantic image data, representing semantic information, can directly guide the optimization of the 3D geometry during reconstruction, which is beneficial for improving the accuracy and effect of 3D reconstruction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a semantically guided 3D reconstruction method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the specific process of a three-dimensional reconstruction method provided in an embodiment of this application; Figure 3 This is a comparative diagram of semantic rendering effects provided in an embodiment of this application; Figure 4 This is a schematic diagram of the constituent modules of a semantically guided 3D reconstruction system provided in an embodiment of this application; Figure 5 This is a block diagram illustrating the internal structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0021] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0027] With the development of science and technology, especially computer vision technology, users' requirements for 3D reconstruction are gradually increasing. For example, in some application scenarios, the demand for reconstructed 3D scenes has been upgraded from simply being visually realistic to not only being visually realistic but also providing semantic information.

[0028] In existing 3D reconstruction schemes, semantic features are typically applied as textures to the surface of the geometry after the 3D reconstruction is completed. However, simply applying semantic features as textures to the reconstructed geometry does not fully utilize semantic information to guide the optimization of the 3D geometry during the reconstruction process, which is detrimental to improving the accuracy and effectiveness of 3D reconstruction.

[0029] 3D reconstruction technology is a core research direction in the field of computer vision. Among them, 3D Gaussian Splatting (3DGS) has become one of the mainstream 3D reconstruction technologies due to its advantages such as real-time rendering, fast reconstruction speed, and high visual fidelity. However, with the development of technologies such as digital twins and autonomous robot operations, existing 3DGS reconstruction technology is gradually failing to meet the needs of practical applications. In some application scenarios (such as autonomous robot operations), robots not only need to see obstacles, but also need to know whether they are "chairs" or "people," and require precise object contours for grasping or obstacle avoidance. Existing 3DGS reconstruction often has artifacts or blurring at object edges and lacks semantic information, failing to meet the needs of refined operations.

[0030] Specifically, traditional 3DGS only focuses on scene geometry and appearance texture reconstruction, lacking semantic understanding capabilities. The reconstructed model can only achieve visual reproduction and cannot distinguish the categories of objects in the scene, making it difficult to meet the needs of robots for refined operations such as obstacle avoidance and precise grasping.

[0031] Existing 3DGS improvement methods that integrate semantic information mostly adopt a 2D-3D high-dimensional feature alignment paradigm. They extract high-dimensional image features and embed them into 3D Gaussian vectors through pre-trained visual language models, such as Contrastive Language-Image Pretraining (CLIP). This process requires feature dimensionality reduction and compression, which easily leads to the loss of fine-grained semantic information and significantly increases memory usage and computational burden, hindering edge deployment. To reduce the memory overhead of rendering high-dimensional features (e.g., 512-dimensional), existing methods typically perform dimensionality reduction and compression (e.g., compression to 3D or use autoencoders), which inevitably results in the loss of fine-grained semantic information.

[0032] Meanwhile, existing methods also suffer from the problem of semantic and geometric separation. Existing methods only attach semantic information as an additional texture to the surface of the geometric object, and the semantic field does not participate in the optimization of the 3D geometric structure. The semantic and geometric aspects are completely separated, and the edge information of the semantic mask cannot be used to sharpen the geometric edges. This results in blurred reconstruction of object edges and the boundaries between different objects, as well as the presence of artifacts. The geometric accuracy is difficult to meet the requirements of high-precision application scenarios.

[0033] Therefore, how to overcome the drawbacks of high-dimensional feature alignment, transform 2D semantic information into strong geometric constraints, and directly correct the spatial distribution of 3D Gaussians through optimization mechanisms to achieve deep integration of semantics and geometry has become an urgent problem to be solved in the current field of 3D reconstruction.

[0034] To address at least one of the aforementioned technical problems, this application proposes a semantically guided 3D reconstruction method. The method includes: acquiring RGB image data and semantic image data corresponding to the scene to be reconstructed; initializing a Gaussian ellipsoid set, wherein the parameters corresponding to each Gaussian ellipsoid in the set include geometric attribute parameters, appearance attribute parameters, and semantic attribute parameters; constructing a two-stream differentiable rasterization pipeline, wherein the two-stream differentiable rasterization pipeline includes a photometric rendering stream and a semantic rendering stream, wherein the photometric rendering stream is used to perform rasterization rendering based on the geometric attribute parameters and appearance attribute parameters corresponding to the Gaussian ellipsoids to obtain a photometric rendered image, and the semantic rendering stream is used to perform rasterization rendering based on the geometric attribute parameters and appearance attribute parameters corresponding to the Gaussian ellipsoids to obtain a photometric rendered image, and the semantic rendering stream is used to perform rasterization rendering based on the geometric attribute parameters and appearance attribute parameters corresponding to the Gaussian ellipsoids to obtain a photometric rendered image. The geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid are rasterized and rendered to obtain a semantic rendering image. Through the above-mentioned photometric rendering stream and semantic rendering stream, the parameters corresponding to the above-mentioned Gaussian ellipsoid are alternately iteratively optimized until a preset iteration termination condition is met, thereby obtaining a set of target Gaussian ellipsoids used to represent the above-mentioned scene to be reconstructed. When the above-mentioned photometric rendering stream is activated, the geometric and appearance attribute parameters corresponding to the above-mentioned Gaussian ellipsoid are updated according to the above-mentioned RGB image data and the generated above-mentioned photometric rendering image. When the above-mentioned semantic rendering stream is activated, the geometric and semantic attribute parameters corresponding to the above-mentioned Gaussian ellipsoid are updated according to the above-mentioned semantic image data and the generated above-mentioned semantic rendering image.

[0035] Therefore, semantic features are not merely applied as textures to the reconstructed geometry; rather, semantic information guides the optimization of the 3D geometry during the reconstruction process. Specifically, a dual-stream differentiable rasterization pipeline is constructed, including a photometric rendering stream and a semantic rendering stream, both sharing the geometric property parameters of a Gaussian ellipsoid. The photometric rendering stream renders based on both geometric and appearance property parameters, while the semantic rendering stream renders based on both. In this application, the parameters of the Gaussian ellipsoid are iteratively optimized alternately based on the photometric and semantic rendering streams during 3D reconstruction. This achieves the optimization of the 3D geometry directly guided by RGB image data and semantic image data. The semantic image data, representing semantic information, can directly guide the optimization of the 3D geometry during reconstruction, which is beneficial for improving the accuracy and effect of 3D reconstruction.

[0036] like Figure 1 As shown in the embodiments of this application, a semantically guided 3D reconstruction method is provided. Specifically, the method includes the following steps: Step S100: Obtain RGB image data and semantic image data corresponding to the scene to be reconstructed; Step S200: Initialize the Gaussian ellipsoid set, wherein the parameters corresponding to each Gaussian ellipsoid in the above Gaussian ellipsoid set include geometric attribute parameters, appearance attribute parameters, and semantic attribute parameters; Step S300: Construct a dual-stream differentiable rasterization pipeline, wherein the dual-stream differentiable rasterization pipeline includes a photometric rendering pipeline and a semantic rendering pipeline. The photometric rendering pipeline is used to perform rasterization rendering based on the geometric attribute parameters and appearance attribute parameters corresponding to the Gaussian ellipsoid to obtain a photometric rendering image. The semantic rendering pipeline is used to perform rasterization rendering based on the geometric attribute parameters and semantic attribute parameters corresponding to the Gaussian ellipsoid to obtain a semantic rendering image. Step S400: Through the above-mentioned photometric rendering stream and semantic rendering stream, the parameters corresponding to the above-mentioned Gaussian ellipsoid are iteratively optimized alternately until the preset iteration termination condition is met, thereby obtaining a set of target Gaussian ellipsoids used to characterize the above-mentioned scene to be reconstructed. When the above-mentioned photometric rendering stream is activated, the geometric attribute parameters and appearance attribute parameters corresponding to the above-mentioned Gaussian ellipsoid are updated according to the above-mentioned RGB image data and the generated above-mentioned photometric rendering image. When the above-mentioned semantic rendering stream is activated, the geometric attribute parameters and semantic attribute parameters corresponding to the above-mentioned Gaussian ellipsoid are updated according to the above-mentioned semantic image data and the generated above-mentioned semantic rendering image.

[0037] The area range corresponding to the above-mentioned scene to be reconstructed can be set and adjusted according to actual needs, and no specific limitation is made here.

[0038] Specifically, the acquisition of RGB image data and semantic image data corresponding to the scene to be reconstructed includes: Collect and obtain the multi-view RGB image sequence corresponding to the above-mentioned scene to be reconstructed, and use the above-mentioned multi-view RGB image sequence as the above-mentioned RGB image data; Using a pre-defined image segmentation model and a large language model, the multi-view RGB images in the above multi-view RGB image sequence are processed to obtain multi-view consistent semantic segmentation images corresponding to each of the above multi-view RGB images; wherein, in the above multi-view consistent semantic segmentation images, the semantic label of the same three-dimensional point remains consistent in each view. For each multi-view consistent semantic segmentation map, a semantic mapping image is obtained by mapping the multi-view consistent semantic segmentation map according to the semantic label corresponding to each pixel in the multi-view consistent semantic segmentation map and the preset semantic label encoding method. The semantic image data described above is constructed based on all the semantic mapping images described above.

[0039] In some applications, multi-view RGB image sequences corresponding to the scene to be reconstructed are acquired using image acquisition devices mounted on cameras or robots. The sequence is treated as RGB image data. A Segment Anything Model (SAM) combined with Large Language Models (LLMs) is used to process the multi-view RGB image sequence frame by frame, eliminating semantic biases caused by viewpoint differences and generating a multi-view consistent semantic segmentation map for each frame. This ensures the consistency of semantic information across different perspectives. Among these, , is the number of images with contextual relevance.

[0040] Furthermore, the aforementioned preset semantic tag encoding method is RGB encoding. The above-mentioned mapping process, based on the semantic labels corresponding to each pixel in the multi-view consistent semantic segmentation map and the preset semantic label encoding method, performs mapping processing on the multi-view consistent semantic segmentation map to obtain a semantic mapping image, including: Obtain a lookup table constructed based on the above RGB encoding method, wherein the lookup table is used to represent the mapping relationship between semantic tags and RGB encoded values; each semantic tag is uniquely mapped to a fixed RGB color value; Based on the lookup table above, the semantic labels corresponding to each pixel in the multi-view consistent semantic segmentation image are mapped to the corresponding RGB encoded values ​​to obtain the semantic mapping image.

[0041] It should be noted that the above-mentioned preset semantic tag encoding method can be set and adjusted according to actual needs. For example, one-hot encoding or hash encoding can also be used. In this embodiment, RGB encoding is used, which can directly reuse existing rasterizers and improve processing efficiency.

[0042] Specifically, in this embodiment, RGB encoding is used as the preset semantic tag encoding method. A semantic tag-RGB encoding value lookup table (LUT) is pre-constructed. The semantic category ID is used to represent the semantic tag in the table. The lookup table is used to map K semantic category IDs in the scene to different fixed RGB encoding values. .

[0043] For example, ID=1 represents the floor, mapped to RGB(128, 128, 128); ID=2 represents the chair, mapped to RGB(255, 0, 0). Thus, the semantic segmentation graph is used with a LUT. Convert to a semantic RGB image (i.e., a semantically mapped image). All semantic mapping images are integrated to form semantic image data, which serves as the supervised ground truth for subsequent semantic optimization.

[0044] Furthermore, initialize the Gaussian ellipsoid set. Each Gaussian ellipsoid... Among the corresponding parameters, the aforementioned geometric attribute parameters include position. Rotation Quaternions Scaling vector and opacity These are shared parameters for the dual-stream rendering; the aforementioned appearance attribute parameters include spherical harmonic coefficients. This is used to fit the realistic appearance texture of the scene; the semantic attribute parameters mentioned above include learnable semantic color parameters. , used to fit the RGB encoded values ​​of the semantic mapping image.

[0045] In this embodiment of the application, during initialization, the geometric attribute parameters are initialized conventionally based on the scene space distribution, while the appearance attribute parameters and semantic attribute parameters can be set to random values, and then gradually converged through subsequent optimization iterations.

[0046] In this embodiment, a dual-stream differentiable rasterization pipeline is constructed, including a photometric rendering stream for rendering realistic colors and a semantic rendering stream for rendering semantic tags, both sharing Gaussian geometric properties. Specifically, a dual-stream differentiable rasterization pipeline with both photometric and semantic rendering capabilities is built. This pipeline contains two parallel rendering streams that share Gaussian ellipsoidal geometric property parameters: one is the photometric rendering stream, which takes the geometric and appearance property parameters of the Gaussian ellipsoid as input, performs differentiable rasterization rendering, and outputs a photometric rendered image corresponding to the real RGB image. The first stream is used to restore the appearance and texture of the scene; the second stream is a semantic rendering stream, which takes the geometric and semantic attribute parameters of a Gaussian ellipsoid as input, renders them through differentiable rasterization, and outputs a semantic rendering image corresponding to the semantic image data. This is used to achieve semantic classification and geometric guidance. The two-stream design ensures geometric sharing: that is, regardless of which stream the error gradient is generated in, it can be backpropagated and the shared geometric property parameters can be updated.

[0047] In some application scenarios, an alternating optimization controller is constructed to cyclically execute the following two phases during 3D reconstruction (training): A photometric optimization phase: activating the photometric rendering stream, calculating the photometric loss using real RGB images, updating shared geometric attribute parameters and spherical harmonics, and freezing semantic attribute parameters; and a semantic optimization phase: activating the semantic rendering stream, calculating the semantic loss using semantic image data, updating shared geometric attribute parameters and semantic attribute parameters, and freezing spherical harmonics. The spatial distribution of the Gaussian ellipsoid (e.g., position, rotation quaternions, scaling vectors, opacity, etc.) is directly corrected using the gradient of the semantic boundary.

[0048] Specifically, the above iteration termination condition is that the number of iterations reaches the preset total number of iterations; The parameters corresponding to the Gaussian ellipsoids are iteratively optimized alternately through the aforementioned photometric rendering stream and semantic rendering stream until a preset iteration termination condition is met, thereby obtaining a set of target Gaussian ellipsoids for characterizing the scene to be reconstructed, including: Obtain the iteration alternation period, the first iteration number used to control the number of times the photometric rendering stream is activated within one iteration alternation period, and the second iteration number used to control the number of times the semantic rendering stream is activated within one iteration alternation period; Within each iterative alternation cycle, the photometric rendering stream and the semantic rendering stream are activated sequentially according to the first iteration number and the second iteration number to perform alternating iterative optimization of the parameters corresponding to the Gaussian ellipsoid. When the photometric rendering stream is activated, the semantic attribute parameters are frozen, and the geometric and appearance attribute parameters corresponding to the Gaussian ellipsoid are updated according to the loss between the RGB image data and the generated photometric rendering image. When the semantic rendering stream is activated, the appearance attribute parameters are frozen, and the geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid are updated according to the loss between the semantic image data and the generated semantic rendering image. When the number of iterations reaches the preset total number of iterations, a set of target Gaussian ellipsoids is obtained to characterize the scene to be reconstructed.

[0049] In some application scenarios, within an iterative alternation cycle, based on the first and second iteration counts mentioned above, the photometric rendering stream and the semantic rendering stream are activated sequentially. This can be done by activating the photometric rendering stream once, then switching to activating the semantic rendering stream once, then switching back to activating the photometric rendering stream once, and so on, until one iterative alternation cycle is completed. This can further improve the accuracy of 3D reconstruction.

[0050] In other application scenarios, within an iterative cycle, when activating the photometric rendering stream and the semantic rendering stream sequentially based on the first and second iteration counts mentioned above, the photometric rendering stream can be activated first to execute the first iteration count, and then the semantic rendering stream can be switched to execute the second iteration count. This reduces the number of rendering stream switching operations, thus improving processing efficiency.

[0051] It should be noted that the number of iterations corresponding to one iterative alternation cycle is the sum of the number of the first iteration and the number of the second iteration.

[0052] In this embodiment, the total number of iterations N, the iteration alternation period T, and the first iteration number corresponding to the photometric rendering stream are preset. The second iteration number corresponding to the semantic rendering stream Using iterative alternation cycles as the loop unit, photometric optimization and semantic optimization are performed cyclically until the number of iterations reaches the preset total number, thus obtaining the target Gaussian ellipsoid set.

[0053] Within a single iteration cycle, the photometric rendering stream is activated first according to the first iteration number, the semantic attribute parameters are frozen, the photometric loss between the real RGB image data and the photometric rendered image is calculated, and the geometric and appearance attribute parameters of the Gaussian ellipsoid are updated through backpropagation to restore the scene detail texture; then, the semantic rendering stream is activated according to the second iteration number, the appearance attribute parameters are frozen, the semantic loss between the semantic image data and the semantic rendered image is calculated, the geometric and semantic attribute parameters of the Gaussian ellipsoid are updated through backpropagation, and the spatial distribution of the Gaussian ellipsoid is corrected using the semantic boundary gradient to make the geometric structure fit the edge of the semantic mask.

[0054] For example, the total number of iterations N = 30000, and the iteration alternation period is defined as 10, meaning every 10 iterations constitute one cycle. Within each iteration alternation period, if the number of iterations performed in the current cycle is less than or equal to the number of the first iteration (e.g., 7), the photometric optimization phase begins. In this phase, the rendered image is obtained through the photometric rendering stream. The loss function for this phase is the photometric loss. The calculation method is shown in the following formula: ; in, This is a preset empirical parameter, for example, it can be set to 0.2. For the average absolute error loss, For structural similarity loss, and These are a set of corresponding photometric rendered images and the initially acquired RGB images, respectively.

[0055] With minimizing photometric loss as the optimization objective, the gradients of shared geometric and appearance attribute parameters are calculated through backpropagation. The optimizer (Adam) updates the parameters based on these gradients, yielding the updated parameters for the current iteration, which are then used in the next iteration. Simultaneously, the norm of the Gaussian ellipsoid parameter position gradient generated in this step is recorded and accumulated; this accumulated gradient will serve as the basis for subsequent adaptive density control (splitting or cloning). The photometric optimization phase aims to recover scene details using RGB texture information. During this process, semantic attribute parameters are frozen.

[0056] Within each iteration cycle, if the number of iterations performed in the current cycle exceeds the number of iterations in the first cycle, a semantic geometry alignment phase is entered. In this phase, the rendered image is obtained through the semantic rendering stream. The loss function for this phase is semantic loss. The calculation method is shown in the following formula: ; in, and These are a set of corresponding semantic rendering images and semantic image data. With minimizing semantic loss as the optimization objective, the gradients of shared geometric and semantic attribute parameters are calculated through backpropagation. The optimizer (Adam) updates the parameters based on these gradients, obtaining the updated parameters for the current iteration, which are then used in the next iteration. Simultaneously, the positional gradient norm of the Gaussian ellipsoid parameters generated in this step is recorded and accumulated; this accumulated gradient will serve as the basis for subsequent adaptive density control (splitting or cloning). This step utilizes the gradient flow generated by the semantic loss to force the Gaussian sphere located at the object's edge to adjust its position and shape, aligning its contour with the semantic mask, thereby eliminating geometric artifacts. During this process, the appearance attribute parameters are frozen.

[0057] Therefore, if a Gaussian sphere deviates slightly from the object in space, resulting in incorrect color pixels in the semantic rendering map, the gradient calculated through backpropagation will directly affect the shared geometric attribute parameters, forcing the Gaussian sphere to move back to the correct semantic region. This is the core of the dual optimizer strategy: using semantic flow to correct shared geometric attribute parameters.

[0058] In this embodiment, Gaussian density control is also performed to further improve the 3D reconstruction effect. Specifically, the above method also includes: Obtain the preset density control interval number and the preset gradient norm threshold; When the number of iterations reaches an integer multiple of the number of density control intervals, the mixed gradient norm corresponding to the Gaussian ellipsoid is determined based on the position gradient norm corresponding to the photometric rendering stream and the position gradient norm corresponding to the semantic rendering stream during the parameter alternation iteration optimization process. If the mixed gradient norm corresponding to the above Gaussian ellipsoid exceeds the preset gradient norm threshold, then a preset density control operation is performed on the above Gaussian ellipsoid, wherein the preset density control operation includes splitting and / or cloning.

[0059] Wherein, if the mixing gradient norm corresponding to the Gaussian ellipsoid exceeds a preset gradient norm threshold, a preset density control operation is performed on the Gaussian ellipsoid, including: If the mixed gradient norm corresponding to the above Gaussian ellipsoid exceeds the preset gradient norm threshold, then the scaling vector corresponding to the above Gaussian ellipsoid is obtained. If the above scaling vector exceeds the preset first size threshold, then a splitting operation is performed on the above Gaussian ellipsoid. If the scaling vector does not exceed the preset second size threshold, then a cloning operation is performed on the Gaussian ellipsoid.

[0060] Thus, a preset density control interval and gradient norm threshold are used. Whenever the number of iterations reaches an integer multiple of the density control interval, parameter updates are paused. The accumulated positional gradient norms of each Gaussian ellipsoid during the photometric optimization and semantic optimization stages are calculated, and the maximum value of the two is taken as the mixed gradient norm, which is used as the Gaussian ellipsoid density adjustment signal. This adaptively increases the density of Gaussian ellipsoids in regions with smooth textures but clear semantic boundaries. If the mixed gradient norm exceeds the preset gradient norm threshold, density control is further performed based on the Gaussian ellipsoid scaling vector: if the scaling vector is greater than the preset first size threshold, the Gaussian ellipsoid volume is determined to be too large, and a splitting operation is performed, removing the original Gaussian ellipsoid and generating two new Gaussian ellipsoids with halved scaling values ​​in the original position; if the scaling vector is less than or equal to the preset second size threshold, the region density is determined to be insufficient, and a cloning operation is performed to generate new Gaussian ellipsoids with identical parameters, improving local fitting accuracy; after density control is completed, the gradient accumulation of the new Gaussian ellipsoids is cleared, and subsequent iterations continue. It should be noted that the specific values ​​of the first size threshold and the second size threshold can be set and adjusted according to actual needs. For example, the second size threshold may be less than or equal to the first size threshold, but no specific limitation is made here.

[0061] In some specific application scenarios, the number of density control intervals is set. When the current iteration count reaches an integer multiple of the density control interval, parameter updates are paused, and density control is performed on the current Gaussian ellipsoid set. Specifically, the mixed gradient norm accumulated by each Gaussian point in the photometric phase (i.e., the optimization process based on the photometric rendering flow) and semantic phase (i.e., the rendering process based on the semantic rendering flow) is calculated. : ; in, The gradient generated by optimizing the phase for luminance represents "how much the position of this Gaussian ellipsoid needs to be moved in order to make the rendered colors look more like a real photograph." This value is usually large in areas with rich textures (such as carpet patterns or foliage). The gradient generated for semantic geometry alignment phase represents "how much the position of this Gaussian ellipsoid needs to be moved in order to make the semantic categories of the rendering more accurate (e.g., to distinguish 'chair' from 'ground')". This value is usually large at object edges and at the boundaries between different objects. The mixed gradient norm indicates that if either color or semantics deems a reconstruction area needing adjustment (large gradient), then that area is difficult to learn and requires adjustment of the Gaussian ellipsoid density. By taking the maximum of the two gradients, it ensures that regardless of the texture-rich region (where the gradient is large), the learning process is optimized. (Dominant) or regions with smooth textures but clear semantic boundaries (by) (Dominant), Gaussian ellipsoids can obtain sufficient density adjustment signals. If the mixing gradient norm of a Gaussian ellipsoid exceeds a preset gradient norm threshold, a splitting or cloning operation is triggered.

[0062] For a Gaussian ellipsoid that satisfies the above gradient conditions, further calculations are made based on its scaling vector. The size determination specifies which operation to perform: if the scaling vector of the Gaussian ellipsoid... If the size exceeds the set first size threshold (e.g., 1% of the scene range), it indicates that the Gaussian ellipsoid is too large and difficult to fit fine geometric or semantic boundaries. At this point, a split operation is performed to remove the Gaussian ellipsoid and generate two new Gaussian ellipsoids with half the original scaling value in the original position (the new Gaussian ellipsoids are based on the original center and are slightly randomly offset along the probability density direction of the Gaussian distribution; the scaling vector will become half of the previous one, and the other parameters remain unchanged).

[0063] If the scaling value of the Gaussian ellipsoid If the density is less than or equal to a set second size threshold (e.g., 0.01% of the scene range), it indicates that the region has subtle geometric features but insufficient density. In this case, the Gaussian sphere is cloned to generate a new Gaussian sphere with identical parameters to increase the Gaussian ellipsoid density of the region.

[0064] Each newly generated Gaussian sphere (whether obtained through splitting or cloning) is added to the Gaussian ellipsoid set as an independent entity, the gradient accumulation is cleared to zero, and the system continues to perform the next iteration of optimization. This mechanism ensures that even in regions with weak texture but clear semantic boundaries, sufficiently dense Gaussian points can be generated to fit the geometric edges. It utilizes the sensitivity of semantic boundaries to compensate for the blind spots of RGB textures in smooth regions, forcing the model to generate high-precision geometric structures at the edges of objects.

[0065] It should be noted that the first and second size thresholds can be set and adjusted according to actual needs, and no specific limitations are made here.

[0066] After the iteration optimization is terminated, a 3D reconstruction model with accurate geometric structure, realistic appearance texture and complete semantic information is output based on the target Gaussian ellipsoid set, thus completing the 3D reconstruction of the scene to be reconstructed.

[0067] This application provides a semantically guided 3D reconstruction method, which includes: acquiring RGB image data and semantic image data corresponding to the scene to be reconstructed; initializing a Gaussian ellipsoid set, wherein the parameters corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set include geometric attribute parameters, appearance attribute parameters, and semantic attribute parameters; constructing a two-stream differentiable rasterization pipeline, wherein the two-stream differentiable rasterization pipeline includes a photometric rendering stream and a semantic rendering stream, wherein the photometric rendering stream is used to perform rasterization rendering based on the geometric attribute parameters and appearance attribute parameters corresponding to the Gaussian ellipsoids to obtain a photometric rendered image, and the semantic ... Geometric and semantic attribute parameters are rasterized and rendered to obtain a semantically rendered image. The parameters corresponding to the Gaussian ellipsoid are iteratively optimized alternately through the above photometric rendering stream and semantic rendering stream until the preset iteration termination condition is met, and a set of target Gaussian ellipsoids for representing the above-mentioned scene to be reconstructed is obtained. When the above photometric rendering stream is activated, the geometric and appearance attribute parameters corresponding to the above Gaussian ellipsoid are updated according to the above RGB image data and the generated above photometric rendering image. When the above semantic rendering stream is activated, the geometric and semantic attribute parameters corresponding to the above Gaussian ellipsoid are updated according to the above semantic image data and the generated above semantic rendering image.

[0068] Therefore, semantic features are not merely applied as textures to the reconstructed geometry; rather, semantic information guides the optimization of the 3D geometry during the reconstruction process. Specifically, a dual-stream differentiable rasterization pipeline is constructed, including a photometric rendering stream and a semantic rendering stream, both sharing the geometric property parameters of a Gaussian ellipsoid. The photometric rendering stream renders based on both geometric and appearance property parameters, while the semantic rendering stream renders based on both. In this application, the parameters of the Gaussian ellipsoid are iteratively optimized alternately based on the photometric and semantic rendering streams during 3D reconstruction. This achieves the optimization of the 3D geometry directly guided by RGB image data and semantic image data. The semantic image data, representing semantic information, can directly guide the optimization of the 3D geometry during reconstruction, which is beneficial for improving the accuracy and effect of 3D reconstruction.

[0069] Specifically, this application's solution employs a dual-stream alternating optimization mechanism, alternately executing two operations during the 3D reconstruction process: determining shape based on color (photometric rendering stream) to ensure visual reconstruction quality; and adjusting edges based on semantic category (semantic rendering stream) to fine-tune the position and shape of the 3D Gaussian sphere using boundary information of object categories (such as the boundary between a table and the ground), resulting in sharper object edges and more accurate geometric structures. A new paradigm of semantic-guided geometric alignment is adopted, mapping semantic masks to low-dimensional RGB colors and using an alternating optimization strategy to directly update the position and shape of the Gaussian sphere using semantic consistency loss, thereby significantly improving reconstruction accuracy.

[0070] By employing a dual-stream alternating optimization strategy, the RGB texture reconstruction and semantic segmentation tasks are effectively decoupled, avoiding gradient interference that occurs when both are optimized simultaneously, thus ensuring high accuracy for each task. Innovatively, the gradient of the semantic phase is used to update shared geometric parameters. This makes the 2D semantic mask not merely a surface color label, but a strong prior constraining the 3D geometry, significantly improving the sharpness of object edge reconstruction. Abandoning high-dimensional feature distillation, a low-dimensional RGB color mapping approach is adopted, combined with dual-stream rasterization, resulting in extremely low training memory usage and inference speed consistent with native 3DGS, making it suitable for robot edge deployment. This provides a high-quality semantic foundation for robot environmental perception, augmented reality, and other applications.

[0071] A new paradigm of semantic-guided geometric alignment is adopted, which maps the semantic segmentation map to RGB colors and uses an alternating optimization strategy to directly update the position and shape of the Gaussian sphere using semantic consistency loss, thereby significantly improving the reconstruction accuracy.

[0072] It should be noted that the proposed solution can also be applied to 3D reconstruction scenes in fields such as augmented reality (AR) and mixed reality (MR), without specific limitations.

[0073] In this embodiment of the application, the above-mentioned semantically guided 3D reconstruction method is further described in detail based on a specific application scenario. Figure 2 This is a schematic diagram illustrating the specific process of a three-dimensional reconstruction method provided in an embodiment of this application, such as... Figure 2 As shown, in this application's scheme, RGB image data and semantic image data are first acquired, then a Gaussian ellipsoid set is initialized, and a dual-stream differentiable raster pipeline is constructed, including a photometric rendering stream and a semantic rendering stream, followed by alternating iterative optimization. Specifically, based on the current iteration count and iteration alternation period, it is determined whether to enter the semantic phase (i.e., whether to activate the semantic rendering stream). If so, parameter optimization is performed through the semantic rendering stream; otherwise, parameter optimization is performed through the photometric rendering stream. Further, based on the iteration count, it is determined whether to trigger density control operations. If so, adaptive density control is performed, and then it is determined whether the total number of iterations has been reached; otherwise, it is directly determined whether the total number of iterations has been reached. If the total number of iterations has not been reached, the process returns to the step of re-determining whether to enter the semantic phase and proceeds to the next iteration; otherwise, the 3D reconstruction is completed, and the reconstructed 3D model is output.

[0074] Figure 3 This is a comparative diagram of semantic rendering effects provided in an embodiment of this application. Figure 3The results of qualitative comparative experiments on the Replica dataset are presented, based on the scheme of this application, comparing scheme 1 (LEGaussians) and scheme 2 (Feature 3DGS). Figure 3 In the image, the first column represents the initially acquired RGB image, the second column is the corresponding manually labeled semantic ground truth, and each row represents a different viewpoint. Figure 3 In the diagram, the first semantic rendering image in the third column is a semantic rendering image generated based on the scheme of this application; the second semantic rendering image in the fourth column is a semantic rendering image generated based on LEGaussians; and the third semantic rendering image in the fifth column is a semantic rendering image generated based on Feature 3DGS. It should be noted that... Figure 3 The illustrations also show the semantic labels corresponding to different colors in the first semantic rendering image generated based on the solution of this application, such as switch, sofa, etc., that is, in the semantic rendering image generated by the solution of this application, different colors are used to represent different semantic categories. Figure 3 It is known that the rendering results of LEGaussians and Feature 3DGS exhibit significant semantic ambiguity at object edges and show noise or semantic classification errors in smooth areas (e.g., misidentifying some objects as others), making it difficult to accurately reproduce fine semantic structures. In contrast, the semantic rendering images generated by the proposed solution maintain a high degree of consistency with the semantic ground truth. Especially at the boundaries of object contours (such as corners and table leg edges), it can generate clear and sharp semantic boundaries and maintain a high degree of semantic continuity within similar objects, without obvious noise. Therefore, the proposed solution, through dual-stream alternating optimization and adaptive density control strategies, effectively overcomes the shortcomings of traditional methods in geometric and semantic alignment, significantly improving the accuracy and visual fidelity of 3D scene semantic segmentation.

[0075] like Figure 4 As shown, corresponding to the semantically guided 3D reconstruction method described above, this application embodiment also provides a semantically guided 3D reconstruction system, which includes: Data acquisition module 410 is used to acquire RGB image data and semantic image data corresponding to the scene to be reconstructed; The Gaussian ellipsoid initialization module 420 is used to initialize a set of Gaussian ellipsoids, wherein the parameters corresponding to each Gaussian ellipsoid in the set include geometric attribute parameters, appearance attribute parameters, and semantic attribute parameters. The rasterization pipeline construction module 430 is used to construct a dual-stream differentiable rasterization pipeline, wherein the dual-stream differentiable rasterization pipeline includes a photometric rendering stream and a semantic rendering stream. The photometric rendering stream is used to perform rasterization rendering based on the geometric attribute parameters and appearance attribute parameters corresponding to the Gaussian ellipsoid to obtain a photometric rendering image. The semantic rendering stream is used to perform rasterization rendering based on the geometric attribute parameters and semantic attribute parameters corresponding to the Gaussian ellipsoid to obtain a semantic rendering image. The 3D reconstruction module 440 is used to iteratively optimize the parameters corresponding to the Gaussian ellipsoid through the photometric rendering stream and the semantic rendering stream until a preset iteration termination condition is met, thereby obtaining a set of target Gaussian ellipsoids to represent the scene to be reconstructed. When the photometric rendering stream is activated, the geometric and appearance attribute parameters corresponding to the Gaussian ellipsoid are updated according to the RGB image data and the generated photometric rendering image. When the semantic rendering stream is activated, the geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid are updated according to the semantic image data and the generated semantic rendering image.

[0076] Therefore, semantic features are not merely applied as textures to the reconstructed geometry; rather, semantic information guides the optimization of the 3D geometry during the reconstruction process. Specifically, a dual-stream differentiable rasterization pipeline is constructed, including a photometric rendering stream and a semantic rendering stream, both sharing the geometric property parameters of a Gaussian ellipsoid. The photometric rendering stream renders based on both geometric and appearance property parameters, while the semantic rendering stream renders based on both. In this application, the parameters of the Gaussian ellipsoid are iteratively optimized alternately based on the photometric and semantic rendering streams during 3D reconstruction. This achieves the optimization of the 3D geometry directly guided by RGB image data and semantic image data. The semantic image data, representing semantic information, can directly guide the optimization of the 3D geometry during reconstruction, which is beneficial for improving the accuracy and effect of 3D reconstruction.

[0077] It should be noted that the specific structure and implementation of the semantically guided 3D reconstruction system and its various modules or units can be referred to the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0078] It should be noted that the division of the modules in the semantically guided 3D reconstruction system described above is not unique and is not intended as a specific limitation.

[0079] Based on the above embodiments, this application also provides a terminal, the principle block diagram of which can be as follows: Figure 5As shown. The terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps of any of the semantically guided 3D reconstruction methods described above. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0080] Those skilled in the art will understand that Figure 5 The block diagram shown is only a partial structural diagram related to the solution of this application and does not constitute a limitation on the terminal on which the solution of this application is applied. The specific terminal may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0081] In one embodiment, a terminal is provided, the terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of any of the semantically guided 3D reconstruction methods provided in the embodiments of this application.

[0082] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any semantically guided 3D reconstruction method provided in this application.

[0083] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0084] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0085] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0086] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0087] In the embodiments provided in this application, it should be understood that the disclosed systems / terminal devices and methods can be implemented in other ways. For example, the system / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0088] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0089] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions are not in essence a departure from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A semantically guided 3D reconstruction method, characterized in that, The method includes: Obtain the RGB image data and semantic image data corresponding to the scene to be reconstructed; Initialize a set of Gaussian ellipsoids, wherein the parameters corresponding to each Gaussian ellipsoid in the set of Gaussian ellipsoids include geometric attribute parameters, appearance attribute parameters, and semantic attribute parameters; A dual-stream differentiable rasterization pipeline is constructed, wherein the dual-stream differentiable rasterization pipeline includes a photometric rendering pipeline and a semantic rendering pipeline. The photometric rendering pipeline is used to perform rasterization rendering based on the geometric attribute parameters and appearance attribute parameters corresponding to the Gaussian ellipsoid to obtain a photometric rendering image. The semantic rendering pipeline is used to perform rasterization rendering based on the geometric attribute parameters and semantic attribute parameters corresponding to the Gaussian ellipsoid to obtain a semantic rendering image. The parameters corresponding to the Gaussian ellipsoid are iteratively optimized alternately through the photometric rendering stream and the semantic rendering stream until a preset iteration termination condition is met, thereby obtaining a set of target Gaussian ellipsoids to characterize the scene to be reconstructed. When the photometric rendering stream is activated, the geometric and appearance attribute parameters corresponding to the Gaussian ellipsoid are updated according to the RGB image data and the generated photometric rendering image. When the semantic rendering stream is activated, the geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid are updated according to the semantic image data and the generated semantic rendering image.

2. The semantically guided 3D reconstruction method according to claim 1, characterized in that, The acquisition of RGB image data and semantic image data corresponding to the scene to be reconstructed includes: Acquire a multi-view RGB image sequence corresponding to the scene to be reconstructed, and use the multi-view RGB image sequence as the RGB image data; By using a preset image segmentation model and a large language model, the multi-view RGB images in the multi-view RGB image sequence are processed respectively to obtain the multi-view consistent semantic segmentation map corresponding to each multi-view RGB image. For each multi-view consistent semantic segmentation map, a semantic mapping image is obtained by mapping the multi-view consistent semantic segmentation map according to the semantic label corresponding to each pixel in the multi-view consistent semantic segmentation map and the preset semantic label encoding method. The semantic image data is constructed based on all the semantic mapping images.

3. The semantically guided 3D reconstruction method according to claim 2, characterized in that, The preset semantic tag encoding method is RGB encoding. The step of mapping the multi-view consistent semantic segmentation map to obtain a semantic mapping image based on the semantic labels corresponding to each pixel in the multi-view consistent semantic segmentation map and a preset semantic label encoding method includes: Obtain a lookup table constructed based on the RGB encoding method, wherein the lookup table is used to represent the mapping relationship between semantic tags and RGB encoded values; Based on the lookup table, the semantic labels corresponding to each pixel in the multi-view consistent semantic segmentation image are mapped to the corresponding RGB encoded values ​​to obtain a semantic mapping image.

4. The semantically guided 3D reconstruction method according to claim 1, characterized in that, The geometric attribute parameters include position, rotation quaternion, scaling vector, and opacity; The appearance attribute parameters include spherical harmonic coefficients; The semantic attribute parameters include semantic color parameters.

5. The semantically guided 3D reconstruction method according to claim 4, characterized in that, The iteration termination condition is when the number of iterations reaches the preset total number of iterations; The step involves iteratively optimizing the parameters corresponding to the Gaussian ellipsoid through the photometric rendering stream and the semantic rendering stream until a preset iteration termination condition is met, thereby obtaining a set of target Gaussian ellipsoids for characterizing the scene to be reconstructed, including: Obtain the iteration alternation period, the first iteration number used to control the number of times the photometric rendering stream is activated within one iteration alternation period, and the second iteration number used to control the number of times the semantic rendering stream is activated within one iteration alternation period; Within each iteration cycle, the photometric rendering stream and the semantic rendering stream are activated sequentially according to the first iteration number and the second iteration number to perform alternating iterative optimization of the parameters corresponding to the Gaussian ellipsoid. When the photometric rendering stream is activated, the semantic attribute parameters are frozen, and the geometric and appearance attribute parameters corresponding to the Gaussian ellipsoid are updated according to the loss between the RGB image data and the generated photometric rendering image. When the semantic rendering stream is activated, the appearance attribute parameters are frozen, and the geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid are updated according to the loss between the semantic image data and the generated semantic rendering image. When the number of iterations reaches the preset total number of iterations, a set of target Gaussian ellipsoids is obtained to characterize the scene to be reconstructed.

6. The semantically guided 3D reconstruction method according to claim 5, characterized in that, The method further includes: Obtain the preset density control interval number and the preset gradient norm threshold; When the number of iterations reaches an integer multiple of the number of density control intervals, the mixed gradient norm corresponding to the Gaussian ellipsoid is determined based on the position gradient norm corresponding to the photometric rendering stream and the position gradient norm corresponding to the semantic rendering stream during the parameter alternating iteration optimization process. If the mixed gradient norm corresponding to the Gaussian ellipsoid exceeds a preset gradient norm threshold, a preset density control operation is performed on the Gaussian ellipsoid, wherein the preset density control operation includes splitting and / or cloning.

7. The semantically guided 3D reconstruction method according to claim 6, characterized in that, If the mixed gradient norm corresponding to the Gaussian ellipsoid exceeds a preset gradient norm threshold, then a preset density control operation is performed on the Gaussian ellipsoid, including: If the mixed gradient norm corresponding to the Gaussian ellipsoid exceeds a preset gradient norm threshold, then the scaling vector corresponding to the Gaussian ellipsoid is obtained. If the scaling vector exceeds a preset first size threshold, a splitting operation is performed on the Gaussian ellipsoid. If the scaling vector does not exceed the preset second size threshold, then a cloning operation is performed on the Gaussian ellipsoid.

8. A semantically guided 3D reconstruction system, characterized in that, The system includes: The data acquisition module is used to acquire RGB image data and semantic image data corresponding to the scene to be reconstructed; A Gaussian ellipsoid initialization module is used to initialize a set of Gaussian ellipsoids, wherein the parameters corresponding to each Gaussian ellipsoid in the set of Gaussian ellipsoids include geometric attribute parameters, appearance attribute parameters, and semantic attribute parameters; A rasterization pipeline construction module is used to construct a dual-stream differentiable rasterization pipeline, wherein the dual-stream differentiable rasterization pipeline includes a photometric rendering stream and a semantic rendering stream. The photometric rendering stream is used to perform rasterization rendering based on the geometric and appearance attribute parameters corresponding to the Gaussian ellipsoid to obtain a photometric rendering image, and the semantic rendering stream is used to perform rasterization rendering based on the geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid to obtain a semantic rendering image. The 3D reconstruction module is used to iteratively optimize the parameters corresponding to the Gaussian ellipsoid through the photometric rendering stream and the semantic rendering stream until a preset iteration termination condition is met, thereby obtaining a set of target Gaussian ellipsoids to characterize the scene to be reconstructed. When the photometric rendering stream is activated, the geometric and appearance attribute parameters corresponding to the Gaussian ellipsoid are updated according to the RGB image data and the generated photometric rendering image. When the semantic rendering stream is activated, the geometric and semantic attribute parameters corresponding to the Gaussian ellipsoid are updated according to the semantic image data and the generated semantic rendering image.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of the semantically guided 3D reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the semantically guided 3D reconstruction method as described in any one of claims 1 to 7.

Images