Monocular semantic reconstruction method based on implicit neural representation

Through the method based on implicit neural representation, continuous three-dimensional functions are learned to represent scene information, solving the problems of low precision and high memory overhead in monocular semantic reconstruction, and achieving high-quality three-dimensional semantic map reconstruction and low memory footprint.

CN118691756BActive Publication Date: 2024-11-08NAT UNIV OF DEFENSE TECH
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202411171367.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-08
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

Existing monocular semantic reconstruction methods face the problems of low-precision semantic quality and high memory overhead, especially in complex scenarios and high resolution situations, it is difficult to effectively reconstruct high-quality three-dimensional semantic maps.

Method used

Using an implicit neural representation method, a continuous three-dimensional function is learned through a neural network to represent the texture, geometric and semantic information of the scene. The MLP module is used to learn geometric, texture and semantic functions respectively, and the volume rendering principle is used to generate the Light color and semantic information reduce storage needs.

Benefits of technology

Reconstruction of high-resolution semantic maps is realized, while reducing memory overhead, improving semantic-geometric-texture consistency, enhancing semantic reconstruction quality, and reducing training costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691756B_ABST
    Figure CN118691756B_ABST
Patent Text Reader

Abstract

The present application relates to a monocular semantic reconstruction method based on implicit neural representation. The method comprises: performing implicit neural scene representation on a monocular image sequence, learning a specific initial scene representation function; obtaining the light color, normal vector and semantic information at any viewing angle according to the scene representation function and the volume rendering principle in graphics, and respectively designing a geometric loss function to supervise the learning process of the geometric function, a texture loss function to supervise the learning process of the texture function, and a semantic loss function to supervise the learning process of the semantic function, and at the same time designing a consistency loss function to reconstruct the optimized semantic function; setting regular constraints to supervise the optimization of the geometric function after the initial optimization, and using the optimized texture function, the reconstructed semantic function and the final geometric function to reconstruct a three-dimensional semantic scene map. This method can reconstruct a high-resolution semantic map while ensuring low memory overhead.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of semantic reconstruction, and in particular to a monocular semantic reconstruction method based on implicit neural representation. Background Art

[0002] Three-dimensional semantic reconstruction of the real-world environment is one of the key tasks that robots must complete in order to move towards automated and intelligent general spatial artificial intelligence. This technology has been widely used in many fields, such as autonomous driving, VR / AR, embodied intelligence, etc. Monocular cameras, which only contain one optical camera, are ubiquitous in people's lives (such as mobile phones, SLR cameras, drones, industrial robots, etc.) due to their low cost, small size, and easy access. How to scan the environment with a monocular optical camera and reconstruct a three-dimensional semantic map of the scene has always been a technical problem that academia and industry have been committed to solving.

[0003] Existing monocular semantic reconstruction methods usually adopt two methods. The first is the semantic fusion method, which takes the monocular image sequence as input. This type of method first uses the existing SFM / SLAM system to perform geometric reconstruction, then uses the pre-trained convolutional neural network to perform semantic segmentation on the two-dimensional image sequence, and finally uses the restored geometric information to back-project the 2D semantics into the three-dimensional space to establish a three-dimensional semantic map of the scene. This type of method usually faces low-precision semantic quality problems and memory overhead problems: 1. When there is an error in geometric reconstruction, the back-projection of semantic information will also be wrong, which will greatly reduce the quality of the three-dimensional semantic map; 2. The semantics predicted by the neural network will be wrong, which will also affect the quality of the three-dimensional semantic map; 3. Since this type of method usually adopts explicit representation forms such as point clouds and voxels, when the scene is larger and the resolution is higher, the number of points and voxels that need to be stored will also increase, which will bring a large memory overhead.

[0004] The second method is the semantic prediction method, which trains a semantic reconstruction network through a large amount of scene corpus, and directly predicts the geometric shape and semantic information of the three-dimensional scene from the input monocular image sequence. This type of method usually faces the following problems: 1. Since the semantic reconstruction network requires a large amount of training data and GPU computing power consumption, the training cost is high; 2. The semantic reconstruction network is difficult to maintain accurate prediction accuracy in unseen scenes, and the generalization ability is poor, and it cannot cope with complex real-world scenes; 3. The semantic prediction method also usually takes the form of display representation to predict the feature information of each point or voxel, so it also faces the problem of memory overhead. Summary of the invention

[0005] Based on this, it is necessary to provide a monocular semantic reconstruction method based on implicit neural representation that can reconstruct high-resolution semantic maps while ensuring low memory overhead to address the above technical problems.

[0006] A monocular semantic reconstruction method based on implicit neural representation, the method comprising:

[0007] Obtain a monocular image sequence obtained by scanning the scene with a camera; perform implicit neural scene representation on the monocular image sequence, model the texture, geometry, and semantic information of the scene as continuous three-dimensional functions, and use a neural network to learn a specific initial scene representation function; the scene representation function includes a geometric function, a texture function, and a semantic function; based on the scene representation function and the volume rendering principle in graphics, obtain the light color, normal vector, and semantic information at any viewing angle;

[0008] Construct a monocular semantic reconstruction model; the monocular semantic reconstruction model includes a geometry learning module, a texture learning module and a semantic learning module;

[0009] In the geometry learning module, the geometric functions and feature vectors of the scene are learned according to the 8-layer MLP. The geometric loss function is designed using the normal vector at any viewing angle to supervise the learning process of the geometric function and obtain the geometric function after the initial optimization.

[0010] In the texture learning module, the texture function of the scene is learned based on the 4-layer MLP. The texture loss function is designed using the light color at any viewing angle to supervise the learning process of the texture function and obtain the optimized texture function.

[0011] In the semantic learning module, the semantic function of the scene is learned according to the 4-layer MLP, and the semantic loss function is designed to supervise the learning process of the semantic function using the semantic information under any perspective to obtain the optimized semantic function; the monocular image sequence is segmented to obtain the segmentation block; the consistency loss function is designed by making the semantics of the sampled light in the segmentation block consistent with the maximum probability semantics; the optimized semantic function is reconstructed according to the consistency loss function to obtain the reconstructed semantic function; the semantic label is output using the reconstructed semantic function;

[0012] According to the semantic labels, regular constraints of different strengths are adaptively applied to the weak texture plane area and the object area to supervise the optimization of the geometric function after the initial optimization to obtain the final geometric function;

[0013] The 3D semantic scene map is reconstructed using the optimized texture function, the reconstructed semantic function and the final geometric function.

[0014] The above-mentioned monocular semantic reconstruction method based on implicit neural representation, the present application uses a neural network to regress a continuous function based on implicit neural representation to represent the target attributes of the scene. In theory, it can output scene information of any point in the space. Its actual resolution of three-dimensional space depends on the expression ability of the neural network itself. At the same time, the three-dimensional space is stored in the weights of the neural network, and the memory occupancy is low. At the same time, the advantage of implicit neural representation is that it can directly fit the semantic function of the scene through the neural network, and learn texture, geometry and semantic information at the same time in an end-to-end manner. It can mine the relationship between different information, remove erroneous and inconsistent 2D semantic prediction semantics, ensure the semantic-geometry-texture consistency of the scene, and improve the quality of the semantic reconstruction map. During the camera scanning process, the present application only uses three MLPs to learn the representation function of each scene, does not require a large amount of data sets and corpus information, and therefore has a lower training cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flowchart of a monocular semantic reconstruction method based on implicit neural representation in one embodiment;

[0016] Figure 2 A schematic diagram of constructing a monocular semantic reconstruction model in one embodiment;

[0017] Figure 3 A schematic diagram of the overall flow chart of the model in one embodiment;

[0018] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] In one embodiment, Figure 1 As shown, a monocular semantic reconstruction method based on implicit neural representation is provided, including the following steps:

[0021] Step 102, obtaining a monocular image sequence obtained by scanning a scene with a camera; performing implicit neural scene representation on the monocular image sequence, modeling the texture, geometry, and semantic information of the scene as continuous three-dimensional functions, and using a neural network to learn a specific initial scene representation function; the scene representation function includes a geometric function, a texture function, and a semantic function; according to the scene representation function and the volume rendering principle in graphics, obtaining the light color, normal vector, and semantic information at any viewing angle.

[0022] Based on implicit neural scene representation, the present invention models the texture, geometry and semantic information of the scene as continuous three-dimensional functions respectively, and uses neural networks to learn specific scene representation functions. :

[0023] ;

[0024] in, and are the 3D reconstruction position and viewing angle vector respectively, is the scene volume density, and the scene geometry is represented as a signed distance field , and They are the texture function and semantic function of the scene respectively.

[0025] Based on the learned scene representation function and drawing on the volume rendering principle in graphics, this application can render scene attributes at any perspective in the image domain: by projecting rays through pixels , for spatial sampling points The color of the light can be obtained by accumulating the product of the scene attribute and the contribution size of the point , the normal vector , semantic information .

[0026] Step 104, constructing a monocular semantic reconstruction model; the monocular semantic reconstruction model includes a geometry learning module, a texture learning module and a semantic learning module.

[0027] The monocular semantic reconstruction model constructed in this application consists of three modules: geometry learning module, texture learning module and semantic learning module. The three modules use MLP to learn scene texture, geometry and semantic information respectively. The network structure of the model is as follows: Figure 2 The overall idea of ​​the invention is: first input the spatial coordinates and the view vector , use geometric MLP to learn the geometric function of the scene and the eigenvector In order to improve the accuracy of geometric reconstruction, the invention uses the prediction results of the existing normal vector prediction network to supervise the learning process of the geometric function. In the texture learning module, the color MLP is used to learn the texture function of the scene. By taking the difference and gradient backpropagation between the rendered image and the input true value image, the representation learning process of texture function and geometric function is established. In the semantic learning module, semantic MLP is used to learn the semantic function of the scene. , and uses the 2D semantic network DeeplabV3+ and image segmentation technology (such as SAM or superpixel segmentation technology) to assist in the reconstruction of 3D semantics. A semantically guided geometric regularization strategy is also designed to improve both geometric accuracy and semantic quality. The overall flow chart of the model is as follows Figure 3 shown.

[0028] Step 106, in the geometry learning module, the geometric function and feature vector of the scene are learned according to the 8-layer MLP, and the geometric loss function is designed using the normal vector at any viewing angle to supervise the learning process of the geometric function, so as to obtain the geometric function after the initial optimization.

[0029] This module uses spatial coordinates As input, an 8-layer MLP is used to learn the geometric function of the scene and the eigenvector : In order to help the learning of geometric functions, the normal vector information of each image is estimated with the help of the existing normal vector prediction network SNU , forcing the light normal vector obtained by and To align:

[0030] ;

[0031] By back-propagating the gradient to the geometric MLP, the loss function can optimize the geometric function of the scene and the volume density function , which helps the scene's geometric reconstruction process. Through the learned geometric continuous function - signed distance field , you can create any resolution size (for example ), calculate the signed distance value of each voxel, and then use the Marching Cube algorithm to extract the geometric surface with a signed distance of 0, and then you can get the 3D grid map of the scene. Since the geometric properties of the 3D scene are stored in the weights of the neural network, and the geometric function of the scene is represented as a continuous spatial function, it is possible to obtain a scene geometric grid of any resolution with low memory overhead.

[0032] Step 108, in the texture learning module, the texture function of the scene is learned according to the 4-layer MLP, and the texture loss function is designed by using the light color at any viewing angle to supervise the learning process of the texture function, so as to obtain the optimized texture function.

[0033] This module uses spatial coordinates , view vector , spatial geometric gradient And the feature vector output by the geometry module As input, a 4-layer MLP is used to learn the texture function of the scene: The images obtained under the training perspective , and compare it with the input image To align:

[0034] ;

[0035] Backpropagating the gradient, the color loss function can optimize the texture function of the scene and the feature vector output by the geometric MLP , which can also optimize the scene's geometric function and the volume density function .

[0036] Step 110, in the semantic learning module, the semantic function of the scene is learned according to the 4-layer MLP, and the semantic loss function is designed by using the semantic information under any perspective to supervise the learning process of the semantic function to obtain an optimized semantic function; the monocular image sequence is segmented to obtain segmentation blocks; a consistency loss function is designed by making the semantics of the sampling light in the segmentation block consistent with the maximum probability semantics; the optimized semantic function is reconstructed according to the consistency loss function to obtain a reconstructed semantic function; and the semantic label is output using the reconstructed semantic function.

[0037] Step 112, according to the semantic labels, regular constraints of different strengths are adaptively applied to the weak texture plane area and the object area to supervise the optimization of the geometric function after the initial optimization, so as to obtain the final geometric function.

[0038] This application uses the universal NYU-40 tag to represent the semantic attributes of spatial points as a probability distribution of 40 predefined semantic categories. Input spatial coordinates and geometric eigenvectors , the semantic learning module uses a 4-layer MLP to learn the semantic attributes of the scene: This application uses the pre-trained two-dimensional semantic prediction network DeeplabV3+ to predict the image sequence. As a supervisory signal, the semantic information and To align:

[0039] ;

[0040] After learning the three-dimensional semantic function of the scene, we can find the semantic information of any point in the space and render the corresponding two-dimensional semantic information in the image domain.

[0041] Since the two-dimensional semantic prediction results usually have large noise and show obvious local semantic inconsistency and object semantic discontinuity, in order to further improve the quality of semantic fusion, this application uses image segmentation technology, such as SAM and superpixel segmentation technology, to process the input image sequence to obtain the corresponding segmentation results. , constraining the semantic labels of the sampling points in each segmentation block to be consistent: for each segmentation block Vote for the most probable semantic label using an appropriate voting strategy (e.g., quantity, uncertainty entropy) , forcing the semantics of sampling rays within the segmentation block to be consistent with the maximum probability semantics:

[0042] ;

[0043] The consistency loss function constrains the semantic consistency of local segments and removes high-frequency noise from the two-dimensional semantic network, thereby enhancing the semantic network's perception of the surrounding environment, ensuring the continuity and smoothness of three-dimensional semantic labels, and improving the reconstruction quality of the semantic map.

[0044] In addition, this application also designs a semantically guided geometric regularization strategy. The core idea of ​​this strategy is that the reconstructed geometric function needs to satisfy the specific property of gradient unity: . However, the photometric consistency error that implicit neural representation learning relies on usually fails in weak texture areas (such as walls, floors, and desktops). This leads to a lack of sufficient constraints in geometric representation learning, and serious degradation problems in geometric reconstruction. It does not meet the inherent properties of the signed distance field. However, if sufficient constraints are imposed to force the entire scene to meet this property, the object area will be too smooth and lose high-frequency details. Therefore, this application considers using semantic labels to For weakly textured flat areas and object area Adaptively apply regularization constraints of different strengths while maintaining plane quality and object details:

[0045] ;

[0046] ;

[0047] Through this loss function, the present application can impose stronger regularization constraints in the plane area to ensure geometric smoothness, and impose weaker regularization constraints in the object area to retain more object details, achieve dynamic adjustment of the geometric structure, and establish a more accurate scene representation function, thereby further helping the learning process of the semantic function.

[0048] Step 114, reconstructing the three-dimensional semantic scene map using the optimized texture function, the reconstructed semantic function and the final geometric function.

[0049] The process of reconstructing a three-dimensional semantic scene map after determining the texture function, semantic function and geometric function is prior art and will not be elaborated in this application. The innovation of this application lies in the process of determining the texture function, semantic function and geometric function.

[0050] The above-mentioned monocular semantic reconstruction method based on implicit neural representation, the present application uses a neural network to regress a continuous function based on implicit neural representation to represent the target attributes of the scene. In theory, it can output scene information of any point in the space. Its actual resolution of three-dimensional space depends on the expression ability of the neural network itself. At the same time, the three-dimensional space is stored in the weights of the neural network, and the memory occupancy is low. At the same time, the advantage of implicit neural representation is that it can directly fit the semantic function of the scene through the neural network, and learn texture, geometry and semantic information at the same time in an end-to-end manner. It can mine the relationship between different information, remove erroneous and inconsistent 2D semantic prediction semantics, ensure the semantic-geometry-texture consistency of the scene, and improve the quality of the semantic reconstruction map. During the camera scanning process, the present application only uses three MLPs to learn the representation function of each scene, does not require a large amount of data sets and corpus information, and therefore has a lower training cost.

[0051] In one embodiment, a monocular image sequence is represented based on an implicit neural scene representation, the texture, geometry, and semantic information of the scene are modeled as continuous three-dimensional functions, and a neural network is used to learn a specific scene representation function, including:

[0052] Based on implicit neural scene representation, the texture, geometry and semantic information of the scene are modeled as continuous three-dimensional functions respectively, and a neural network is used to learn a specific scene representation function.

[0053] ;

[0054] in, and are the 3D reconstruction position and viewing angle vector respectively, is the scene volume density, is a geometric function, and They are texture function and semantic function respectively.

[0055] In one embodiment, according to the scene representation function and the volume rendering principle in graphics, the light color, normal vector and semantic information at any viewing angle are obtained, including:

[0056] According to the scene representation function and the volume rendering principle in graphics, the light color, normal vector and semantic information at any viewing angle are obtained respectively:

[0057] ;

[0058] ;

[0059] ;

[0060] in, Indicates the color of light. Represents the light normal vector information, Represents the spatial sampling point index, represents the sampling point, Indicates sampling point The cumulative opacity of express The opacity, represents the spatial distance between two sampling points, express Color, represents the normal vector, express The normal vector of Represents semantic information, express The semantic information of Represents the spatial sampling point.

[0061] In one embodiment, a geometric loss function is designed using normal vectors at any viewing angle, including:

[0062] The geometric loss function is designed using the normal vector at any viewing angle:

[0063] ;

[0064] in, represents the normal vector, Indicates that the existing normal vector prediction network SNU estimates the normal vector information of each image. represents the light projected into space by the camera, Represents the collection of total rays.

[0065] In one embodiment, a texture loss function is designed using the color of light at any viewing angle, including:

[0066] The texture loss function is designed using the color of light at any viewing angle:

[0067] ;

[0068] in, Indicates the color of light. represents the input image, represents the light projected into space by the camera, Represents the collection of total rays.

[0069] In one embodiment, the semantic loss function is designed using semantic information at any perspective:

[0070] ;

[0071] in, represents the predicted semantic probability density function, represents the true probability density value, represents the light projected into space by the camera, represents the total set of rays, Through Obtained by performing argmax operation.

[0072] In one embodiment, a consistency loss function is designed by making the semantics of the sampled rays in the segmented block consistent with the maximum probability semantics, including:

[0073] By making the semantics of the sampled light in the segmentation block consistent with the maximum probability semantics, the consistency loss function is designed as

[0074] ;

[0075] in, Represents the split block index, Indicates the maximum number of split blocks. represents the light projected into space by the camera, represents the set of total rays within the partition block, represents the maximum probability semantic class, Represents the predicted semantic probability function In the maximum probability semantic class The probability value of .

[0076] In one embodiment, regular constraints of different strengths are adaptively applied to weak texture plane regions and object regions according to semantic labels to supervise the optimization of the geometric function after the initial optimization, including:

[0077] The loss function in the optimization process of the geometric function after the initial optimization is

[0078] ;

[0079] ;

[0080] in, represents the geometric function after the initial optimization, represents the gradient operation, represents the weight adjustment function, represents spatially uniform sampling points, represents the set of sampling points, represents the plane semantic category, express Predicted semantic category.

[0081] In a specific embodiment, after multiple experimental verifications, Tables 1-2 compare the three-dimensional reconstruction semantics, two-dimensional rendering semantics and geometric reconstruction results of the prior art and the present application. The results show that this method is better than previous methods in terms of semantic quality and geometric accuracy, and has the ability to reconstruct high-quality semantic maps. Table 1 is a comparison of the results of three-dimensional reconstruction semantics and two-dimensional rendering semantics. Table 2 is a comparison of geometric reconstruction results, where DeeplabV3+ represents the DeeplabV3+ semantic segmentation neural network, Manhattan-SDF represents the name of the semantic reconstruction method, the original Manhattan semantic reconstruction system, Manhattan-SDF* represents the name of the semantic reconstruction method, the improved Manhattan semantic reconstruction system, NeuRIS* represents the name of the semantic reconstruction method, the improved NeuRIS semantic reconstruction system, Acc represents Accuracy, Comp represents Completion, Prec represents Precision, Recall represents Recall, F-score represents F index, and Chamfer represents ChamferDistance, chamfer distance. 3D geometry evaluation is to calculate the distance between the 3D point cloud in GT and the reconstructed 3D point cloud. Acc and Compl are respectively calculated from the reconstructed point cloud->GT point cloud and the distance from the reconstructed point cloud->GT point cloud. Since it is actually necessary to calculate the matching relationship of 3D points in the two point clouds, their values ​​are not the same. Prec and Recall are the percentages of Acc<0.5 and Compl<0.5 respectively. F-score is 2*prec*recal / (prec+recall), and Chamfer is (Acc+Compl) / 2.

[0082] Table 1

[0083]

[0084] Table 2

[0085]

[0086] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0087] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a monocular semantic reconstruction method based on implicit neural representation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0088] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0089] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A monocular semantic reconstruction method based on implicit neural representation, characterized in that: The method comprises: Obtain a monocular image sequence obtained by scanning a scene with a camera; perform implicit neural scene representation on the monocular image sequence, model the texture, geometry, and semantic information of the scene as continuous three-dimensional functions, and use a neural network to learn a specific initial scene representation function; the scene representation function includes a geometric function, a texture function, and a semantic function; according to the scene representation function and the volume rendering principle in graphics, obtain the light color, normal vector, and semantic information at any viewing angle; Constructing a monocular semantic reconstruction model; the monocular semantic reconstruction model includes a geometry learning module, a texture learning module and a semantic learning module; In the geometric learning module, the geometric function and feature vector of the scene are learned according to the 8-layer MLP, and the geometric loss function is designed by using the normal vector under the arbitrary viewing angle to supervise the learning process of the geometric function, so as to obtain the geometric function after the initial optimization; In the texture learning module, the texture function of the scene is learned according to the 4-layer MLP, and the texture loss function is designed by using the light color under the arbitrary viewing angle to supervise the learning process of the texture function, so as to obtain an optimized texture function; In the semantic learning module, the semantic function of the scene is learned according to the 4-layer MLP, and the semantic loss function is designed by using the semantic information under the arbitrary perspective to supervise the learning process of the semantic function to obtain an optimized semantic function; the monocular image sequence is segmented to obtain segmentation blocks; a consistency loss function is designed by making the semantics of the sampled light in the segmentation block consistent with the maximum probability semantics; the optimized semantic function is reconstructed according to the consistency loss function to obtain a reconstructed semantic function; and a semantic label is output using the reconstructed semantic function; Adaptively applying regular constraints of different strengths to the weak texture plane region and the object region according to the semantic label to supervise the optimization of the geometric function after the initial optimization to obtain a final geometric function; The optimized texture function, the reconstructed semantic function and the final geometric function are used to reconstruct a three-dimensional semantic scene map.

2. The method according to claim 1, characterized in that The monocular image sequence is represented based on an implicit neural scene representation, the texture, geometry, and semantic information of the scene are modeled as continuous three-dimensional functions, and a neural network is used to learn a specific scene representation function, including: The monocular image sequence is represented based on implicit neural scene representation, the texture, geometry and semantic information of the scene are modeled as continuous three-dimensional functions respectively, and the specific scene representation function is learned using neural network: ; in, and are the 3D reconstruction position and viewing angle vector respectively, is the scene volume density, is a geometric function, and They are texture function and semantic function respectively.

3. The method according to claim 1, characterized in that: According to the scene representation function and the volume rendering principle in graphics, the light color, normal vector and semantic information at any viewing angle are obtained, including: According to the scene representation function and the volume rendering principle in graphics, the light color, normal vector and semantic information at any viewing angle are obtained as follows: ; ; ; in, Indicates the color of light. Represents the light normal vector information, Represents the spatial sampling point index, represents the sampling point, Indicates sampling point The cumulative opacity of express The opacity, represents the spatial distance between two sampling points, express Color, represents the normal vector, express The normal vector of Represents semantic information, express The semantic information of Represents the spatial sampling point.

4. The method according to claim 1, characterized in that: The geometric loss function is designed using the normal vector at any viewing angle, including: The geometric loss function is designed using the normal vector at any viewing angle: ; in, represents the normal vector, Represents the existing normal vector prediction network Estimate the normal vector information of each image, represents the light projected into space by the camera, Represents the collection of total rays.

5. The method according to claim 1, characterized in that The texture loss function is designed using the light color at any viewing angle, including: The texture loss function designed using the light color at any viewing angle is: ; in, Indicates the color of light. represents the input image, represents the light projected into space by the camera, Represents the collection of total rays.

6. The method according to claim 1, characterized in that The semantic loss function is designed using the semantic information under any perspective: ; in, represents the predicted semantic probability density function, represents the true probability density value, Represents light, Represents the collection of total rays.

7. The method according to claim 1, characterized in that The consistency loss function is designed by making the semantics of the sampled light in the segmentation block consistent with the maximum probability semantics, including: The consistency loss function is designed by making the semantics of the sampled light in the segmentation block consistent with the maximum probability semantic class: ; in, Represents the split block index, Indicates the maximum number of split blocks. Represents light, represents the set of total rays within the partition block, represents the maximum probability semantic class, Represents the predicted semantic probability function In the maximum probability semantic class The probability value of .

8. The method according to claim 1, characterized in that Adaptively applying regular constraints of different strengths to the weak texture plane area and the object area according to the semantic label to supervise the optimization of the geometric function after the initial optimization, including: The loss function in the optimization process of the geometric function after the initial optimization is ; ; in, represents the geometric function after the initial optimization, represents the gradient operation, , , represents the set of sampling points, represents the plane semantic category, express Predicted semantic category.

Citation Information

Patent Citations

  • Block sparse aperture ISAR imaging method based on PCSBL-GAMP-Net

    CN112965064A

  • Image-based three-dimensional scene reconstruction method and device

    CN114742966A

  • Large-scene elastic semantic representation and self-supervision light field reconstruction method and device

    CN115423946A

  • Semantic segmentation-based NeRF three-dimensional target object model reconstruction method

    CN117994508A

  • Three-dimensional point cloud semantic map construction method based on neural radiation field

    CN118379450A