An efficient semantic 3D reconstruction method based on feature mesh mapping

Abstract

The present invention discloses an efficient semantic three-dimensional reconstruction method based on feature grid mapping, which is characterized by using a three-dimensional Gaussian model to render low-dimensional features, using low-dimensional feature distances to match target feature grids to restore high-dimensional semantic features, and realizing pixel-level target open semantic search, specifically including: semantic feature initialization extraction, construction of feature grid mapping, feature training and result query and other steps. Compared with the prior art, the present invention has efficient and feasible three-dimensional reconstruction results, realizes accurate query based on natural language, solves the problem that many semantic three-dimensional reconstruction methods can only be based on a few fixed label words, low efficiency and cannot truly interact in real time, and lack accuracy and stability. It can further enable environmental perception and interaction in smart home, industrial robots and autonomous driving scenarios, and has broad application prospects and commercial development value.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction and cross-modal technology, and in particular to an efficient semantic three-dimensional reconstruction method based on feature grid mapping. Background Art

[0002] 3D reconstruction refers to the reconstruction of the complete 3D scene content based on a series of RGB images. Combined with different post-processing processes, it can also directly export surface meshes or models, etc., without the need for professional sampling tools to obtain 3D representation data of the scene, and can achieve fast and easy modeling in the fields of film and television industry, digital twins, etc. Some other types of 3D reconstruction are also widely used in robot environment perception and automatic path planning. Common technologies in this field include neural radiation field and 3D Gaussian.

[0003] Modality refers to the form in which data exists, such as text, audio, image, video and other file formats. Cross-modality refers to technologies such as mapping content from different modalities to the same modality and using data from other modalities to enrich the current modality. It is an important interaction and knowledge transfer technology. Cross-modality technology is widely used in applications such as text-image generation, image understanding and semantic query. Due to the considerable application prospects of 3D reconstruction, related technologies have developed rapidly in recent years, with a very huge improvement in reconstruction accuracy and speed.

[0004] The reconstruction results of the existing technology do not contain semantic information, can only be based on a few fixed label words, and are inefficient, so they cannot be directly applied to the interaction process. There are a few reconstruction technologies with semantic information that cannot simultaneously meet the accuracy, openness of query words, and high efficiency under high resolution. This makes it difficult to apply the reconstruction results to the semantic perception of intelligent environments in fields such as robots or autonomous driving, and there are problems such as the inability to truly interact in real time and lack of accuracy and stability. Summary of the invention

[0005] The purpose of the present invention is to provide an efficient semantic three-dimensional reconstruction method based on feature grid mapping, which adopts the method of "segment everything" model, extracts object blocks and uses the contrast language-image pre-training model to obtain semantic coding, and uses feature grid mapping to map high-dimensional semantic coding to low-dimensional vectors, so as to achieve efficient training and restoration of semantic features during query, and realize interactive open semantic search with high precision, high quality and high efficiency in three-dimensional reconstruction scenes. The method includes four steps. For the input labeled pose picture set, firstly, all complete object images in the picture are segmented using the "segment everything" model, and then the image is put into the contrast language-image pre-training model to obtain the semantic coding of the object; then, the feature grid is initialized for storing high-dimensional semantic features, and then cross-view matching based on the comprehensive evaluation criteria of feature points, semantic features and color features is performed to store different features of the same object to the same feature grid and complete the mapping of low-dimensional features of the grid; after constructing the feature grid and mapping, the low-dimensional features are reconstructed in parallel with the scene to obtain the original scene and additional semantic information at the same time; finally, the low-dimensional features are projected to the observation plane during query, and the pixel-level high-dimensional semantic features are quickly restored from the feature grid, so that accurate query can be performed based on natural language. The present invention solves the problems that many semantic 3D reconstruction methods can only be based on a few fixed label words, are inefficient and cannot truly interact in real time, and lack accuracy and stability. It provides a more effective and feasible method for realizing extensive semantic interaction of 3D reconstruction results, further enabling environmental perception and interaction in smart home, industrial robots and autonomous driving scenarios, and has broad application prospects and commercial development value.

[0006] The specific technical scheme for achieving the purpose of the present invention is: an efficient semantic 3D reconstruction method based on feature grid mapping, which is characterized by using a 3D Gaussian model to render low-dimensional features, using low-dimensional feature distance to match the target feature grid to restore high-dimensional semantic features, and realizing pixel-level target open semantic search, which specifically includes the following steps:

[0007] Step 1: Initialize semantic feature extraction

[0008] 1.1) For the input calibrated RGB image sequence, use the "segment everything" model to segment the model input 32x32 dot matrix and only select the "complete" scale results to obtain the segmented item image sequence Seg_Items;

[0009] 1.2) The segmentation results are fed into the comparative language-image pre-training model to obtain accurate high-dimensional semantic encodings High_Embeds for all objects from all perspectives.

[0010] Step 2: Construct the feature grid map

[0011] 2.1) Initialize the feature grid Grid with the first image in the sequence. Assuming there are K item segments in the image, the initialization grid scale is d is the number of low-dimensional feature dimensions, and the size of each grid item in the Grid is 512 dimensions;

[0012] 2.2) Continue to traverse the input image sequence in sequence, and for each image and its processed segmentation and semantic information, match it with the previous image to assign the features of the same item to the same Grid. This step performs item-level matching based on both image feature matching and color-semantic comprehensive feature similarity. Image feature matching uses SIFT descriptor and K-nearest neighbor algorithm. For images i and j, their color-semantic comprehensive feature similarity sim i,j It is expressed by the following formula (a):

[0013]

[0014] in, and are the RGB color data vectors of the two objects and the feature vector similarity encoded by the contrast language-image pre-training model; for the color feature C i ,C j ,have For the contrastive language-image pre-trained model encoding L i ,L j ,have The weight α is 0.3;

[0015] 2.3) The specific matching algorithm is as follows: first initialize the first image, then perform feature matching on each subsequent item in the previous image. If the number of matching feature point pairs is greater than the threshold value τ, the match is directly determined, the current item feature is stored in the matching Grid, and the pixel coordinates of the current item are assigned the mapped low-dimensional features. If the number of matching feature points is insufficient, the color-semantic comprehensive feature similarity is calculated, and matching is performed in all previous images. When the similarity is greater than the threshold value θ, the match is determined, otherwise the match is not satisfied; assign new Grid initialization semantic features to the unmatched objects and add new corresponding low-dimensional features, and repeat the above matching operation until all objects are completely traversed.

[0016] Step 3: Feature training

[0017] 3.1) Given the camera pose v, define the low-dimensional feature F of each pixel under the three-dimensional Gaussian v,p is a mixed modulation of all Gaussian balls N superimposed on the pixel, and its rendering feature F v,p It is expressed by the following formula (b):

[0018]

[0019] Among them, f represents low-dimensional features, a i It is obtained by the 2D Gaussian covariance and the learned opacity of each Gaussian sphere.

[0020] 3.2) Loss function L of low-dimensional features f It is composed of the L1 function of formula (c) and the D-SSIM structure:

[0021] L f =(1-λ)L1+λL D-SSIM (c).

[0022] Here, λ is set to 0.2.

[0023] Step 4: Query results

[0024] 4.1) Given the camera pose v, first project the low-dimensional features onto the observation plane, and calculate the low-dimensional features of the pixels at that viewing angle according to the rendering formula given in step 3;

[0025] 4.2) After a natural language query is given, the text encoding of the query is first obtained using the comparative language-image pre-training model, and then the similarity is calculated between it and all the encodings in the Grid to find the Grid with the highest similarity and its mapped low-dimensional features;

[0026] 4.3) Use the low-dimensional feature and rendering feature F v,p The Euclidean distance Dis is calculated by the following formula (d): gf :

[0027]

[0028] Among them, Dis gf Less than the activation threshold τ ac The pixels constitute the mask of the query result.

[0029] The semantic feature initialization extraction only selects the segmentation results of the "complete" scale and puts them into feature encoding to ensure the consistency of cross-view distribution and the accuracy of features.

[0030] The feature grid and its low-dimensional mapping are constructed, and the construction and allocation process is completed based on feature point matching and the proposed color-semantic comprehensive feature similarity, thereby obtaining a highly consistent cross-view allocation and accurate semantic grouping of the same object.

[0031] Compared with the prior art, the present invention has efficient and feasible 3D reconstruction results, realizes more effective and feasible 3D reconstruction results and extensive semantic interaction, and better solves the problems that many semantic 3D reconstruction methods can only be based on a few fixed label words, have low efficiency and cannot truly interact in real time, and lack accuracy and stability. It can further enable environmental perception and interaction in smart home, industrial robots and autonomous driving scenarios, and has broad application prospects and commercial development value. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the present invention;

[0033] Figure 2 Mapping flow chart for feature grid;

[0034] Figure 3 Flowchart for prediction query. DETAILED DESCRIPTION

[0035] In order to facilitate the understanding of the present invention, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0036] Example 1

[0037] See also Figure 1 The present invention adopts a three-dimensional Gaussian model to render low-dimensional features, uses low-dimensional feature distance to match the target feature grid to restore high-dimensional semantic features, and realizes pixel-level target open semantic search, which specifically includes four steps: for the input labeled pose picture set, first use the "segment everything" model to segment all complete object images in the picture, and then put it into the comparative language-image pre-training model to obtain the semantic encoding of the object; then initialize the feature grid for storing high-dimensional semantic features, and then perform cross-view matching based on the comprehensive evaluation criteria of feature points, semantic features, and color features to store different features of the same object to the same feature grid and complete the mapping of the grid low-dimensional features; after constructing the feature grid and mapping, reconstruct the low-dimensional features and the scene in parallel to obtain the original scene and additional semantic information at the same time; finally, when querying, project the low-dimensional features to the observation plane, and quickly restore the pixel-level high-dimensional semantic features from the feature grid, so that accurate queries based on natural language can be performed, which specifically includes the following steps:

[0038] S100: Item Target Segmentation

[0039] For the input calibrated RGB image sequence, it is put into the "segment everything" model for target segmentation, that is, the model input 32x32 dot matrix is ​​segmented, and the segmentation result of the "complete" scale is taken to obtain the segmented object image sequence Seg_Items.

[0040] S110: The segmentation result is input into the comparison-language pre-training model to obtain the block semantic feature encoding, that is, to obtain the accurate high-dimensional semantic encoding High_Embeds of all objects under all perspectives to ensure the consistency of cross-perspective distribution and the accuracy of features.

[0041] S120: Feature Grid Mapping

[0042] See also Figure 2 The results obtained in steps S100 and S110 are used to construct a feature grid and its low-dimensional mapping, and the construction and allocation process is completed based on feature point matching and the proposed color-semantic comprehensive feature similarity, thereby obtaining a highly consistent cross-view allocation and accurate semantic grouping of the same object.

[0043] See also Figure 2 , construct feature grid mapping

[0044] S200: Feature point matching. If the number of matches reaches the standard, it is determined to belong to the previous feature grid, the features are stored and the corresponding low-dimensional mapping is assigned to the area, otherwise comprehensive feature matching continues.

[0045] S210: Color-semantic comprehensive feature matching. If there is a match, it is also determined to belong to the previous feature grid, the same as step S200. Otherwise, it is considered that there is no match, and a new feature grid and low-dimensional mapping are created for it and assigned to the corresponding area.

[0046] The constructing of the feature grid mapping specifically includes:

[0047] 1) Initialize the feature grid Grid with the first image in the sequence. Assuming there are K item segments in the image, the initialization grid scale is Where d is the number of low-dimensional feature dimensions, and the size of each grid item in the Grid is 512 dimensions.

[0048] 2) Continue to traverse the input image sequence in sequence, match each image and its processed segmentation and semantic information with the previous image, and assign the features of the same items to the same Grid. At the same time, perform item-level matching based on image feature matching and color-semantic comprehensive feature similarity. The image feature matching uses SIFT descriptor and K-nearest neighbor algorithm. For images i and j, their color-semantic comprehensive feature similarity sim i,j It is expressed by the following formula (a):

[0049]

[0050] in, and are the RGB color data vectors of the two objects and the feature vector similarity encoded by the contrast language-image pre-training model, Ci ,C j are the color characteristics of the two objects; L i ,L j are the language-image pre-training model for comparing two items respectively; α is the weight;

[0051] 3) The specific matching algorithm is as follows: first initialize the first image, then perform feature matching on each subsequent item in the previous image. If the number of matching feature point pairs is greater than the threshold τ, the match is directly determined, the current item feature is stored in the matching Grid, and the pixel coordinates of the current item are assigned the mapped low-dimensional features; if the number of matching feature points is insufficient, the color-semantic comprehensive feature similarity is calculated, and matching is performed on all previous images. When the similarity is greater than the threshold θ, the match is determined and the above allocation operation is repeated. Otherwise, it is considered that there is no satisfactory match, and the object appears for the first time in the sequence. A new Grid initialization semantic feature is allocated and a new corresponding low-dimensional feature is added.

[0052] S130: 3D reconstruction model training

[0053] Reconstruct the low-dimensional mapping features obtained through training, including:

[0054] 1) Given the camera pose v, define the low-dimensional feature F of each pixel under the three-dimensional Gaussian v,p is a mixed modulation of all Gaussian balls N superimposed on the pixel, and its rendering feature F v,p It is expressed by the following formula (b):

[0055]

[0056] Among them, f represents low-dimensional features, a i It is obtained by the 2D Gaussian covariance and the learned opacity of each Gaussian sphere.

[0057] 2) Loss function L of low-dimensional features f It is composed of the L1 function of formula (c) and the D-SSIM structure:

[0058] L f =(1-λ)L1+λL D-SSIM (c).

[0059] Here, λ is set to 0.2.

[0060] S140: Semantic Query

[0061] See also Figure 3 , the semantic query results include:

[0062] 1) Given a camera pose v, first project the low-dimensional features onto the observation plane, and calculate the low-dimensional features of the pixels at that viewing angle according to the above formula (b).

[0063] 2) After a natural language query is given, the text encoding of the query is first obtained using the contrastive language-image pre-training model, and then the similarity is calculated between it and all the encodings in the Grid to find the Grid with the highest similarity and its mapped low-dimensional features.

[0064] 3) Use the low-dimensional feature and rendering feature F v,p The Euclidean distance Dis is calculated by the following formula (d): gf :

[0065]

[0066] Among them, Dis gf Less than the activation threshold τ ac The pixels constitute the mask of the query result.

[0067] See also Figure 3 , prediction queries specifically include:

[0068] S300: Rendering low-dimensional features

[0069] Given any observation perspective, low-dimensional features of the perspective are rendered pixel by pixel, as well as semantic features after the perspective is mapped.

[0070] S310: Text feature encoding

[0071] For the input query word, the contrastive-linguistic pre-training model is used to obtain the semantic feature encoding of the query word.

[0072] S320: Semantic feature recovery and similarity calculation

[0073] First, the text encoding of the query word is used to calculate the similarity with all the high-dimensional encodings in the Grid, and the Grid with the highest similarity and its mapped low-dimensional features are found. Then, the Euclidean distance Dis expressed by the following formula (d) is calculated between the target low-dimensional feature and the rendered low-dimensional feature. gf :

[0074]

[0075] Among them, Dis gf Less than the activation threshold τ ac The pixels constitute the mask of the query result.

[0076] The present invention uses a three-dimensional Gaussian model to render low-dimensional features, and uses low-dimensional feature distance to match the target feature grid to recover high-dimensional semantic features without the need for a neural network, thereby ultimately enabling fast pixel-level open semantic search of targets. This solves the problem that many semantic three-dimensional reconstruction methods can only be based on a few fixed label words, are inefficient and cannot truly interact in real time, and lack accuracy and stability, and can further enable environmental perception and interaction in smart home, industrial robots, and autonomous driving scenarios.

[0077] The above is only a further explanation of the present invention and is not intended to limit this patent. Any equivalent implementation of the present invention should be included in the scope of the claims of this patent.

Claims

1. An efficient semantic 3D reconstruction method based on feature grid mapping, characterized in that: A three-dimensional Gaussian model is used to render low-dimensional features, and low-dimensional feature distance is used to match the target feature grid to restore high-dimensional semantic features, realizing pixel-level target open semantic search, which specifically includes the following steps: Step 1: Initialize semantic feature extraction 1.1) For the input calibrated RGB image sequence, put it into the pre-trained model for segmentation, use the 32x32 dot matrix as an indicator for segmentation, select the result of the full scale, and obtain the segmented item image sequence Seg_Items; 1.2) The segmentation results are fed into the comparative language-image pre-training model to obtain accurate high-dimensional semantic encodings High_Embeds for all objects from all perspectives; Step 2: Construct the feature grid map 2.1) Initialize the feature grid Grid with the first image in the sequence. Assuming there are K item segments in the image, the initialization grid scale is Where d is the number of low-dimensional feature dimensions, and the size of each grid item in the Grid is 512 dimensions; 2.2) Traverse the input image sequence in order, match each image and its acquired segmentation and semantic information with the previous image, assign the features of the same object to the same Grid, and perform item-level matching based on image feature matching and color-semantic comprehensive feature similarity. The image feature matching uses SIFT descriptor and K-nearest neighbor algorithm; the color-semantic comprehensive feature similarity sim i,j It is expressed by the following formula (a): in, and are the RGB color data vectors of the two objects and the feature vector similarity encoded by the contrasting language-image pre-training model, C i ,C j are the color characteristics of the two objects; L i ,L j are the comparative language-image pre-trained model encodings of the two items respectively; α is the weight; 2.3) After the first image is initialized, feature matching is performed on each subsequent item in the previous image. If the number of matching feature points is greater than the threshold τ, the match is directly determined, the current item feature is stored in the matching Grid, and the mapped low-dimensional features are assigned to the pixel coordinates of the current item; if the number of matching feature points is insufficient, the color-semantic comprehensive feature similarity is calculated, and matching is performed in all previous images. When the similarity is greater than the threshold θ, the match is determined and the mapped low-dimensional features are assigned to the matching items. Otherwise, the match is not satisfied; new Grid initialization semantic features are assigned to unmatched objects and new corresponding low-dimensional features are added. The above matching operation is repeated until all objects are completely traversed; Step 3: Feature training 3.1) Given the camera pose v, define the low-dimensional feature F of each pixel under the three-dimensional Gaussian v,p is a mixed modulation of all Gaussian balls N superimposed on the pixel, and its rendering feature F v,p It is expressed by the following formula (b): Among them, f represents low-dimensional features; a i is the opacity of each Gaussian sphere learned by 2D Gaussian covariance; 3.2) Loss function L of low-dimensional features f Formula (c) composed of the L1 function and the D-SSIM structure is expressed as: 50 f =(1-λ)L1+λL D-SSIM (c); Among them, λ is set as a hyperparameter; Step 4: Query results 4.1) Given the camera pose v, project the low-dimensional features onto the observation plane, and calculate the low-dimensional features of the pixels at that viewing angle according to formula (b); 4.2) After a natural language query is given, the text encoding of the query is obtained by comparing the language-image pre-training model, and the similarity between the text encoding and all the encodings in the Grid is calculated to find the Grid with the highest similarity and its mapped low-dimensional features; 4.3) Use the low-dimensional feature and rendering feature F v,p The Euclidean distance Dis is calculated by the following formula (d): gf : Among them, Dis gf Less than the activation threshold τ ac The pixels constitute the mask of the query result.

2. The efficient semantic 3D reconstruction method based on feature grid mapping according to claim 1, characterized in that: The step 1 selects the segmentation results of the entire scale and puts them into feature encoding to ensure the consistency of cross-view distribution and the accuracy of features.

3. The efficient semantic 3D reconstruction method based on feature grid mapping according to claim 1, characterized in that: The step 2 constructs a feature grid and its low-dimensional mapping, and the construction and allocation process is based on feature point matching and color-semantic comprehensive feature similarity, thereby obtaining a highly consistent cross-view allocation and accurate semantic grouping of the same object.

4. The efficient semantic 3D reconstruction method based on feature grid mapping according to claim 1, characterized in that: The step 3 uses a three-dimensional Gaussian model to render low-dimensional features, and uses low-dimensional feature distance to match the target feature grid to restore high-dimensional semantic features, thereby achieving pixel-level target open semantic search.

Images