Method for storing scene layout and object information based on implicit representation and application method

Abstract

The application provides a method for implicitly expressing perception storage scene layout and object information, a comparison loss between features obtained by scene implicit coding prediction and features obtained by a large model is established through input picture sequence data, knowledge of the large model trained in a mass large-scale Internet data set is used, and an end-to-end training mode is used to enable the scene implicit coding to learn the relationship between positions, regions and objects in the scene, so that the mapping of the spatial coordinates and the object semantic information and the region information can be established simultaneously by using the scene implicit coding, the problem that the traditional scene understanding algorithm cannot understand the scene layout information is solved, and therefore the scene region understanding reasoning related task that cannot be completed before can be completed by using the region information. In addition, the target features used for training the scene implicit coding are object semantic information and region information obtained by predicting the existing visual language large model, and therefore the labeling cost required for training is extremely low.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method for implicitly expressing and storing scene layout and object information, as well as a method for applying this information. Background Technology

[0002] Spatial cognition is a fundamental ability for humans and animals to achieve long-term autonomous activity in their environment. Spatial information can be divided by scale into scene-level layout information, such as the location, orientation, and connectivity of elements like rooms, buildings, and roads; and object-level content information, such as the attributes, shape, size, and location of various objects. Within spatial information, region information is a crucial way for humans and animals to recognize, understand, and express spatial information, and it is also a prerequisite for many downstream tasks, such as association, reasoning, localization, and navigation. If robots can understand the spatial region relationships within a scene like humans, they will be able to perform tasks such as spatial region reasoning, establishing connections between objects and regions, and distinguishing objects existing in different spatial regions, thus enabling robots to possess better spatial cognition and comprehension abilities.

[0003] Current research on spatial cognition and understanding largely focuses on 3D spatial reconstruction, object or target detection and segmentation. However, few studies address the modeling and understanding of scene region information. This lack of scene region information leads to insufficient understanding of the scene by the robot, making it difficult to comprehend related commands.

[0004] The rapid development of large-scale models has recently provided new solutions for spatial cognition and understanding. Large models trained on massive datasets encompassing various scenes are considered capable of reasoning using general knowledge and even zero-shot reasoning. Some research utilizes large models to process visual text features to represent spatial scenes, establishing connections between scene coordinates and visual text features. These works provide preliminary methods for linking object features with 3D spatial coordinates; however, most studies have not yet established correlations between object-level information, scene region-level information, and 3D spatial coordinates. Summary of the Invention

[0005] To address the aforementioned issues, a method for scene layout and object information based on implicit representation perceptual storage is provided. In this method, an implicit representation of the scene is established, which includes object-level semantic information and scene region information. This is combined with object-level 3D reasoning capabilities and scene region reasoning and region information application capabilities. The method mainly utilizes scene encoding to synchronously establish a mapping between spatial coordinates and object semantic information and region information. The training target features are the prediction of object semantic information and region information obtained through existing large-scale visual language models.

[0006] Specifically, the present invention adopts the following technical solution:

[0007] This invention provides a method for implicitly representing perceptual storage of scene layout and object information, characterized by the following steps: Step S1, using an object detection network model to perform object detection on the input depth map to obtain object detection boxes, with the background outside the detection boxes; Step S2, based on the detection boxes, combined with the regional information labels of the spatial location corresponding to the depth map, using a visual language feature extraction model and a text feature extraction model to extract visual language features and text semantic features of the object and the background respectively, as target features; Step S3, back-projecting the depth map to three-dimensional space through camera pose to obtain a three-dimensional point cloud corresponding to the scene in the depth map; Step S4, initializing a multi-scale hash code as a scene implicit code to represent the relationship between the three-dimensional point cloud and the corresponding features; Step S5, encoding the three-dimensional point cloud through the scene implicit code to obtain encoded features, comparing the encoded features with the target features to construct a loss function, and optimizing the scene implicit code based on the loss function, the optimized scene implicit code being the scene layout and object information in the depth map based on implicitly representing perceptual storage.

[0008] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have the following technical features: in step S5, the three-dimensional coordinate sequence of the three-dimensional point cloud is encoded into a spatial vector representing its spatial features through the scene implicit encoding, and the spatial vector is mapped to the space where the target feature is located through the fully connected layer of the multilayer perceptron, thereby obtaining the encoded feature.

[0009] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have the following technical features: in step S5, the number of training rounds is 100, the batch size is 12544, the number of threads is 10, the loss function is calculated using the cross-entropy function, the initial learning rate is set to 1e-4, and the learning rate decay weight is 0.003.

[0010] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have the following technical features: In step S1, the target detection network model outputs the detection box and corresponding object label for each object; in step S2, for the object, the visual language features of the object are encoded using the image within the detection box, and the semantic features of the object are encoded using the object label; for the background, the visual language features of the background are encoded using the image outside the detection box, and the textual semantic features of the background are encoded using the region label; and the encoded features of the object and the background are used as the target features.

[0011] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have the following technical features: the visual language feature extraction model is a pre-trained CLIP with a ViT-B / 32 encoder, and the text feature extraction model is a pre-trained SentenceBERT with an mpnet-base-v2 encoder.

[0012] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have the following technical features, wherein, in step S1, the target detection network model is a pre-trained Detic, and its confidence threshold for judging objects is 0.4.

[0013] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have the following technical features, wherein in step S4, the number of layers of hash encoding is 18, and the dimension of each layer is 8.

[0014] This invention provides an application method for scene layout and object information based on implicit expression perceptual storage, used for reasoning scene regions and / or objects corresponding to spatial locations. The method has the following technical features, including the following steps: inputting the three-dimensional coordinates corresponding to the spatial location into the above-mentioned scene implicit encoding, the scene implicit encoding outputs the semantic features of the scene region and / or object, comparing the features of the scene region and object in the scene at the spatial location, and obtaining the scene region and / or object corresponding to the spatial location.

[0015] This invention provides an application method for scene layout and object information based on implicit representation perceptual storage, used to locate a specific scene region or a specific object within a specific scene region based on input text. The method includes the following steps: Step SA1, inputting the text into a visual language feature extraction model and a text feature extraction model respectively to extract the visual language features and text semantic features corresponding to the text, as text features; Step SA2, randomly sampling the three-dimensional coordinate sequence used to train the aforementioned scene implicit encoding, inputting multiple sampling results into the scene implicit encoding to obtain the visual language features and text semantic features of each sampling result, as multiple sampling features; Step SA3, calculating the similarity between the text features and each of the sampling features, finding the sampling feature with the highest similarity, where the three-dimensional coordinates corresponding to this sampling feature are the location of the specific scene region or the specific object within the specific scene region corresponding to the text.

[0016] This invention provides an application method for scene layout and object information based on implicit representation perceptual storage, used to locate a specific scene region or a specific object within a specific scene region based on an input depth map. The method includes the following steps: Step SB1, inputting the depth map into a visual language feature extraction model and a text feature extraction model respectively to extract visual language features and text semantic features corresponding to the depth map, as image features; Step SB2, randomly sampling the three-dimensional coordinate sequence used to train the aforementioned scene implicit encoding, inputting multiple sampling results into the scene implicit encoding to obtain the visual language features and text semantic features of each sampling result, as multiple sampling features; Step SB3, calculating the similarity between the image features and each of the sampling features, finding the sampling feature with the highest similarity, where the three-dimensional coordinates corresponding to this sampling feature are the location of the specific scene region or the specific object within the specific scene region corresponding to the depth map.

[0017] Invention Function and Effect

[0018] The method based on implicit representation of scene layout and object information according to the present invention establishes a comparative loss between features predicted by scene implicit encoding and features obtained through a large model using input depth map sequence data. Utilizing the knowledge of a large model trained on massive internet datasets, the scene implicit encoding learns the relationships between positions, regions, and objects in the scene through end-to-end training. This enables the scene implicit encoding to simultaneously establish a mapping between spatial coordinates and object semantic information and region information, solving the problem that traditional scene understanding algorithms cannot understand scene layout information. Therefore, it can leverage region information to complete previously difficult tasks related to scene region understanding and reasoning. Furthermore, since the target features used to train the scene implicit encoding are the predicted object semantic information and region information obtained through existing large visual language models, the annotation cost required for training is extremely low.

[0019] After training, the scene implicit encoding can be used for various applications such as inferring the spatial location of objects and the layout information of regions, locating specific regions / objects based on input text, and locating specific locations in the environment based on input images. Attached Figure Description

[0020] Figure 1 This is a flowchart of a method for implicitly expressing and storing scene layout and object information in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the data processing and method flow in an embodiment of the present invention;

[0022] Figure 3This is a flowchart of the application method of the object and region layout information corresponding to the reasoning spatial location in this embodiment of the invention;

[0023] Figure 4 This is a flowchart of an application method for locating a specific object in a specific area based on input text, as described in an embodiment of the present invention.

[0024] Figure 5 This is an example diagram illustrating the results of calculating the similarity between visual language features and text semantic features corresponding to an object and a scene region in an embodiment of the present invention;

[0025] Figure 6 This is an example diagram showing the result of locating a specific object in a specific area based on the input text in an embodiment of the present invention;

[0026] Figure 7 This is a flowchart of an application method for locating a specific location in the environment based on an input image, as described in an embodiment of the present invention.

[0027] Figure 8 This is an example image showing the result of locating a specific location in the environment based on an input image in an embodiment of the present invention. Detailed Implementation

[0028] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following describes in detail the method of implicit expression perception storage scene layout and object information based on the present invention with reference to embodiments and accompanying drawings.

[0029] <Example>

[0030] Figure 1 This is a flowchart of the method for implicitly expressing and storing scene layout and object information in this embodiment. Figure 2 This is a schematic diagram of the data processing and method flow in this embodiment.

[0031] like Figure 1 and Figure 2 As shown, the method includes the following steps:

[0032] Step S1: Use a pre-trained open-set object detection network model to perform object detection on the input depth map to obtain the object detection box, and the area outside the detection box is the background.

[0033] Step S2: Based on the detection box, and combined with the region information label of the spatial location corresponding to the image, the visual language feature extraction model and the text feature extraction model, respectively, are used to extract the visual language features and semantic features of the object and the background as target features.

[0034] Step S3: The depth map corresponding to the image is back-projected into the three-dimensional space through the camera pose to obtain the 3D point cloud corresponding to the scene in the depth map.

[0035] Step S4: Initialize a multi-scale hash code as an implicit scene code to represent the relationship between 3D point clouds and corresponding features.

[0036] Step S5: Encode the 3D point cloud through scene implicit encoding, compare the encoded features with the target features to construct a loss function, and optimize the scene implicit encoding based on the loss function. The optimized scene implicit encoding is the scene layout and object information in the image stored based on implicit expression perception.

[0037] The steps described above will be explained in detail below.

[0038] Step S1: Use a pre-trained open-set object detection network model to perform object detection on the input image to obtain the object detection box, and the area outside the detection box is the background.

[0039] In step S1, the object detection network model is Detic, and its confidence threshold for judging objects is 0.4. RGB images and corresponding depth maps are obtained from RGB-D images. The information sequence of the RGB images is input into the model for object detection, and the model outputs the detection box surrounding the object and the object label.

[0040] like Figure 2 As shown, taking an image of an indoor environment as an example, the objects include a coffee table, cushions, and an air conditioner vent, with the background scene being a living room. Step S1 obtains the bounding boxes for each object and their corresponding object labels.

[0041] Step S2: Based on the detection box, and combined with the regional information label of the spatial location corresponding to the depth map, the visual language feature extraction model and the text feature extraction model, respectively, are used to extract the visual language features and text semantic features of the object and the background as target features.

[0042] In step S2, the visual language feature extraction model is CLIP, with a ViT-B / 32 encoder. The text feature extraction model is SentenceBERT, with an mpnet-base-v2 encoder. For objects, the visual language features of the object are encoded using the image within the detection box, and the semantic features of the object are encoded using the object label. For background scene regions, the visual language features of the background are encoded using the image outside the detection box, and the textual semantic features of the background are encoded using the region label. The encoded features of the object and the background are used as the target features.

[0043] Step S3: The depth map corresponding to the image is back-projected into the three-dimensional space through the camera pose to obtain the 3D point cloud corresponding to the scene in the depth map.

[0044] Step S4: Initialize a multi-scale hash code as an implicit scene code to represent the relationship between 3D point clouds and corresponding features.

[0045] In step S4, the hash encoding has 18 layers, and each layer has a dimension of 8.

[0046] Step S5: Encode the 3D point cloud coordinates through scene implicit encoding, compare the encoded features with the target features to construct a loss function, and optimize the scene implicit encoding based on the loss function. The optimized scene implicit encoding is the scene layout and object information in the image based on implicit expression perception storage.

[0047] That is, in step S5, the scene implicit encoding (scene implicit representation) is trained. The 3D coordinate sequence in the 3D point cloud obtained in step S3 is used as input. The scene implicit encoding encodes the 3D coordinate sequence into a spatial vector representing its spatial features. The spatial vector is then mapped to the space of the target features (including target visual language features and target text semantic features) obtained in step S2 through the fully connected layer of the multilayer perceptron (MLP) to obtain the encoded features. Then, the encoded features are compared with the target features to construct a loss function. The scene implicit encoding is then optimized based on the loss function.

[0048] In this embodiment, in step S5, the number of training rounds is 100, the batch size is 12544, the number of threads is 10, the loss function is calculated using the cross-entropy function, the initial learning rate is set to 1e-4, and the learning rate decay weight is 0.003.

[0049] Figure 3 This is a flowchart of the application method of the object and area layout information corresponding to the inference spatial location in this embodiment.

[0050] like Figure 3 As shown, after obtaining the implicit encoding of the above scene, one application method is to use it for inference of scene regions and / or objects corresponding to spatial locations. This application method includes the following steps:

[0051] The 3D coordinates of the spatial location are input into the trained scene implicit encoding. The scene implicit encoding outputs the semantic features of the scene region and / or object. The text of the known regions and objects contained in the environment are obtained by visual language feature encoding and text feature encoding, respectively, to obtain the features to be compared. The similarity between the features and the features obtained by implicit encoding is calculated. The text (i.e., the label) corresponding to the feature with the highest similarity is taken as the scene region and / or object corresponding to the spatial location.

[0052] Figure 4 This is a flowchart of the application method for locating a specific object in a specific area based on input text in this embodiment.

[0053] like Figure 4 As shown, after obtaining the implicit encoding of the above scene, another application method is to use it to locate a specific scene region or a specific object in a specific scene region based on the input text. This application method includes the following steps:

[0054] Step SA1 involves inputting the text into a pre-trained visual language feature extraction model and a pre-trained text feature extraction model to extract the corresponding visual language features and text semantic features as text features.

[0055] Step SA2 involves randomly sampling the 3D coordinate sequence used for training the scene implicit encoding, inputting multiple sampling results into the trained scene implicit encoding to obtain the visual language features and text semantic features of each sampling result, which are then used as multiple sampling features.

[0056] Step SA3: Compare the similarity between the text features and each sampled feature, and find the sampled feature with the highest similarity (closest). The 3D coordinates corresponding to this sampled feature are the location of the specific scene area or the specific object in the specific scene area corresponding to the input text.

[0057] Figure 5 This is an example diagram showing the results of calculating the similarity between visual language features and text semantic features corresponding to objects and scene regions in this embodiment.

[0058] like Figure 5 As shown, the input text can be text from an object or scene area. The similarity between the text features and multiple sampled features is calculated to obtain the results shown in the figure. Then, the sampled feature with the highest similarity is found.

[0059] Figure 6 This is an example diagram showing the result of locating a specific object in a specific area based on the input text in this embodiment.

[0060] like Figure 6 As shown, taking the multi-room interior environment provided by the open-source standard dataset Matterport3D as an example, in one example in the first row of the figure, the input text is "Table in the bedroom", which means that the table in the bedroom needs to be located. Through the method of this embodiment, the correct location in the figure is accurately located (the star in the bedroom area in the far right column of the first row of the figure).

[0061] Figure 6 The document also shows examples of two other input texts, and for each example, Figure 6The document also shows the search results of two other methods in the prior art (CLIP-Field and VLMaps). It can be seen that these two methods will find similar objects in other areas, that is, identification errors will occur; while the method of this embodiment can locate objects in the correct scene area.

[0062] Figure 7 This is a flowchart of the application method for locating a specific location in the environment based on an input image in this embodiment.

[0063] like Figure 7 As shown, after obtaining the implicit encoding of the above scene, another application method is to use it to locate a specific scene region or a specific object in a specific scene region based on the input depth map. This application method includes the following steps:

[0064] Step SB1: Input the depth map into the pre-trained visual language feature extraction model and the pre-trained text feature extraction model respectively to extract the visual language features and text semantic features corresponding to the depth map as image features.

[0065] Step SB2 involves randomly sampling the 3D coordinate sequence used for training the scene implicit encoding, and inputting the multiple sampling results into the trained scene implicit encoding to obtain the visual language features and text semantic features of each sampling result, which are then used as multiple sampling features.

[0066] Step SB3: Compare the similarity between the image features and each sampled feature, and find the sampled feature with the highest similarity (closest). The 3D coordinates corresponding to this sampled feature are the location of the specific scene area or the specific object in the specific scene area corresponding to the input depth map.

[0067] Figure 8 This is an example image showing the result of locating a specific location in the environment based on the input image in this embodiment.

[0068] like Figure 8 As shown, taking the multi-room interior environment provided by the open-source Matterport3D as an example, in one example in the first row of the figure, the input image is a picture taken facing the living room. The picture contains objects such as a television. Using the method of this embodiment, the location of the scene area and the objects is accurately located (the area marked by the box in the far right column of the first row of the figure).

[0069] Figure 8 The document also shows examples of two other input images, and for each example, Figure 8The image also shows the search results of two other methods in the prior art (CLIP-Field and VLMaps). It can be seen that these two methods will find similar areas in multiple scene areas (areas marked by boxes in the image), that is, the positioning accuracy is not high; while the method of this embodiment can accurately locate the closest area.

[0070] The tests conducted on the above examples demonstrate that the method of this embodiment can successfully recognize and infer spatial region information in a series of scenarios containing multiple regions and multiple objects.

[0071] The role and effect of the embodiments

[0072] According to the method for implicitly representing and storing scene layout and object information provided in this embodiment, a comparative loss is established between the features predicted by the scene implicit encoding and the features obtained through a large model, using the input depth map sequence data. Utilizing the knowledge of a large model trained on massive internet datasets, the scene implicit encoding learns the relationships between positions, regions, and objects in the scene through end-to-end training. This enables the scene implicit encoding to simultaneously establish a mapping between spatial coordinates and object semantic information and region information, solving the problem that traditional scene understanding algorithms cannot understand scene layout information. Therefore, it can leverage region information to complete previously difficult tasks related to scene region understanding and reasoning. Furthermore, since the target features used to train the scene implicit encoding are the predicted object semantic information and region information obtained through existing large visual language models, the annotation cost required for training is extremely low.

[0073] After training, the implicit scene encoding can be used for various applications, including reasoning about the spatial location of objects and the layout of areas, locating specific areas / objects based on input text, and locating specific locations in the environment based on input images. Furthermore, the implicit scene encoding can serve as the robot's spatial scene memory, enabling the robot to recognize and reason about spatial areas within these scenes. This allows the robot to establish relationships between destinations, target objects, and areas, and to perform scene-area understanding and reasoning tasks that were previously difficult to accomplish with the help of area information.

[0074] The above embodiments are merely illustrative of specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are only for illustrating the principles of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope thereof, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for implicitly expressing and storing scene layout and object information, characterized in that, Includes the following steps: Step S1: Use an object detection network model to perform object detection on the input image to obtain the object detection box, and the area outside the detection box is the background; Step S2: Based on the detection box, and combined with the region information label of the spatial location corresponding to the image, the visual language feature extraction model and the text semantic feature extraction model are used to extract the visual language features and text semantic features of the object and the background, respectively, as target features; Step S3: The depth map corresponding to the image is back-projected into the three-dimensional space through the camera pose to obtain the three-dimensional point cloud corresponding to the scene in the depth map; Step S4: Initialize a multi-scale hash code as an implicit scene code to represent the relationship between the 3D point cloud and the corresponding features; Step S5: Encode the 3D point cloud using the scene implicit encoding to obtain encoded features. Compare and learn the encoded features with the target features to construct a loss function. Optimize the scene implicit encoding based on the loss function. The optimized scene implicit encoding is the scene layout and object information in the image stored based on implicit representation perception.

2. The method for storing scene layout and object information based on implicit expression perception according to claim 1, characterized in that: in, In step S5, the three-dimensional coordinate sequence of the three-dimensional point cloud is encoded into a spatial vector representing its spatial features through the scene implicit encoding, and the spatial vector is mapped to the space where the target feature is located through the fully connected layer of the multilayer perceptron, thereby obtaining the encoded feature.

3. The method for implicitly expressing and storing scene layout and object information based on claim 2, characterized in that: in, In step S5, the training rounds are 100, the batch size is 12544, the number of threads is 10, the loss function is calculated using the cross-entropy function, the initial learning rate is set to 1e-4, and the learning rate decay weight is 0.

003.

4. The method for implicitly expressing and storing scene layout and object information according to claim 1, characterized in that: in, In step S1, the object detection network model outputs the detection bounding box and corresponding object label for each object. In step S2, for the object, the visual language features of the object are encoded using the image within the detection box, and the semantic features of the object are encoded using the object label. For the background, the visual language features of the background are encoded using the image outside the detection box, and the textual semantic features of the background are encoded using the region labels. The encoded features of the object and the background are used as the target features.

5. The method for implicitly expressing and storing scene layout and object information according to claim 4, characterized in that: in, The visual language feature extraction model is a pre-trained CLIP, with a ViT-B / 32 encoder. The text feature extraction model is a pre-trained SentenceBERT with an encoder of mpnet-base-v2.

6. The method for storing scene layout and object information based on implicit expression perception according to claim 1, characterized in that: in, In step S1, the target detection network model is a pre-trained Detic, and its confidence threshold for judging objects is 0.

4.

7. The method for storing scene layout and object information based on implicit expression perception according to claim 1, characterized in that: in, In step S4, the hash encoding has 18 layers, and each layer has a dimension of 8.

8. A method for applying scene layout and object information based on implicit representation perceptual storage, used for reasoning about scene regions and / or objects corresponding to spatial locations, characterized in that, Includes the following steps: The three-dimensional coordinates corresponding to the spatial location are input into the scene implicit encoding. The scene implicit encoding outputs the semantic features of the scene region and / or object. The text of the known region and the text of the object contained in the environment of the spatial location are respectively encoded using visual language features and text features to obtain multiple features to be compared. The similarity between the semantic features and each of the features to be compared is calculated. The text corresponding to the feature with the highest similarity is the scene region and / or object corresponding to the spatial location. The scene implicit encoding is obtained according to the method of implicit expression-based perception storage of scene layout and object information as described in any one of claims 1-7.

9. A method for applying scene layout and object information based on implicit representation-aware storage, used to locate a specific scene region or a specific object within a specific scene region based on input text, characterized in that, Includes the following steps: Step SA1: Input the text into the visual language feature extraction model and the text feature extraction model respectively to extract the visual language features and text semantic features corresponding to the text, as text features; Step SA2: Randomly sample the three-dimensional coordinate sequence used for training the scene implicit encoding, and input multiple sampling results into the scene implicit encoding to obtain the visual language features and text semantic features of each sampling result, as multiple sampling features; Step SA3: Calculate the similarity between the text feature and each of the sampled features, and find the sampled feature with the highest similarity. The three-dimensional coordinates corresponding to this sampled feature represent the location of the specific scene region or the specific object within the specific scene region corresponding to the text. The scene implicit encoding is obtained according to the method of implicit expression-based perception storage of scene layout and object information as described in any one of claims 1-7.

10. A method for applying scene layout and object information based on implicit representation-aware storage, used to locate a specific scene region or a specific object within a specific scene region based on an input depth map, characterized in that, Includes the following steps: Step SB1: Input the depth map into the visual language feature extraction model and the text feature extraction model respectively to extract the visual language features and text semantic features corresponding to the depth map as image features; Step SB2: Randomly sample the three-dimensional coordinate sequence used for training the scene implicit encoding, and input multiple sampling results into the scene implicit encoding to obtain the visual language features and text semantic features of each sampling result, as multiple sampling features; Step SB3: Calculate the similarity between the image features and each of the sampled features, and find the sampled feature with the highest similarity. The three-dimensional coordinates corresponding to this sampled feature are the location of a specific scene region or a specific object within that scene region corresponding to the depth map. The scene implicit encoding is obtained according to the method of implicit expression-based perception storage of scene layout and object information as described in any one of claims 1-7.

Images