Method for storing scene layout and object information based on implicit expression and application method

Through the implicit expression perception method, implicit encoding of scenes is established to represent the relationship between three-dimensional point clouds and features, which solves the problem of insufficient understanding of scene area information by the robot, realizes scene area reasoning and positioning capabilities with low labeling costs, and enhances the robot space Cognitive ability.

CN119068476AActive Publication Date: 2024-12-03FUDAN UNIVERSITY +1

Patent Information

Application Number
CN202411201560.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-03
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

The lack of modeling and cognition of scene area information in the prior art makes it difficult for robots to understand scene layout and related instructions, and most of the work fails to establish the correlation between object level, scene area level information and three-dimensional spatial coordinates.

Method used

Using an implicit expression perception method, through the object detection network, visual language feature extraction model and text feature extraction model, combined with multi-scale hash coding, the implicit scene encoding is established, and the implicit scene encoding is optimized to represent three-dimensional point clouds and features. , training target features to predict object semantic information and area information, and realizing the mapping of spatial coordinates and object semantic information.

Benefits of technology

Through end-to-end training, the scene implicitly encodes the relationship between positions, regions, and objects in the scene, solving the problem that traditional algorithms cannot understand scene layout information, realizing the scenario area understanding inference task with low labeling costs, and being able to reason space The object and area layout information corresponding to the position, and position it to a specific scene area or object based on the input text or picture.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068476A_ABST
    Figure CN119068476A_ABST
Patent Text Reader

Abstract

The invention provides a method for perceiving and storing scene layout and object information based on implicit expression, which comprises the following steps of: establishing comparison loss of characteristics obtained by scene implicit coding prediction and characteristics obtained by a large model through input picture sequence data, and utilizing the knowledge of the large model trained in a massive large-scale internet data set to obtain the object layout and object information. Through an end-to-end training mode, a scene implicit code learns a relationship among a position, a region and an object in a scene, so that mapping of space coordinates, object semantic information and region information can be synchronously established by utilizing the scene implicit code, and the problem that a traditional scene understanding algorithm cannot understand scene layout information is solved; therefore, scene area understanding and reasoning related tasks which are difficult to achieve before can be completed by means of the area information. In addition, the target features used for training scene implicit coding are object semantic information and regional information obtained by predicting an existing visual language large model, so that the marking cost required by training is extremely low.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a method for storing scene layout and object information based on implicit expression perception and an application method of the information. Background Art

[0002] Spatial cognition is a basic ability for humans and animals to achieve long-term autonomous activities in the environment. Spatial information can be divided into scene-level layout information, such as the position, direction, and connection relationship of elements such as rooms, buildings, and roads; object-level content information, such as the attributes, shape, size, and position of various objects. In spatial information, regional information is an important way for humans and animals to recognize, understand, and express spatial information, and it is also a prerequisite for many downstream tasks, such as association, reasoning, positioning, and navigation. If robots can understand the spatial regional relationship in the scene like humans, they will be able to complete tasks such as spatial regional reasoning, establishing connections between objects and regions, and distinguishing objects in different spatial regions, enabling robots to have better spatial cognition and understanding capabilities.

[0003] In the study of spatial cognition and understanding, a large number of current research works focus on three-dimensional space reconstruction, object or target detection and segmentation, etc. However, few of these works focus on the modeling and cognition of the regional information of the scene. The lack of regional information of the scene will lead to insufficient understanding of the robot and difficulty in understanding related instructions.

[0004] The recent rapid development of large models has provided new solutions for spatial cognition and understanding. Large models trained on large-scale data covering various scenes are believed to have the ability to reason using general knowledge and zero-sample reasoning. Some research works use large models to process visual text features to express spatial scenes and establish a connection between scene coordinates and visual text features. These works provide preliminary methods to establish a connection between object features and three-dimensional spatial position coordinates, but most of the work has not yet established a relationship between object-level information, scene area-level information, and three-dimensional spatial coordinates. Summary of the invention

[0005] To solve the above problems, a method for scene layout and object information based on implicit expression perception storage is provided. In this method, an implicit scene expression including object-level semantic information and scene area information is established, combined with object-level three-dimensional reasoning capability, scene area reasoning and area information application capability. This method mainly utilizes scene encoding to synchronously establish a mapping between spatial coordinates and object semantic information and area information. The training target features are used to predict the object semantic information and area information obtained through the existing visual language large model.

[0006] Specifically, the present invention adopts the following technical solutions:

[0007] The present invention provides a method for storing scene layout and object information based on implicit expression perception, which is characterized by comprising the following steps: step S1, using a target detection network model to perform target detection on an input depth map to obtain a detection frame of the object, and the outside of the detection frame is the background; step S2, based on the detection frame, combined with the regional information label of the spatial position corresponding to the depth map, using a visual language feature extraction model and a text feature extraction model to respectively extract the visual language features and text semantic features of the object and the background as target features; step S3, back-projecting the depth map into a three-dimensional space through a camera pose to obtain a three-dimensional point cloud corresponding to the scene in the depth map; step S4, initializing a multi-scale hash code as a scene implicit code for representing the relationship between the three-dimensional point cloud and the corresponding feature; step S5, encoding the three-dimensional point cloud through the scene implicit code to obtain an encoded feature, comparing the encoded feature with the target feature to learn and construct a loss function, and optimizing the scene implicit code based on the loss function, the optimized scene implicit code is the scene layout and object information in the depth map stored based on implicit expression perception.

[0008] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have such a technical feature, wherein, in step S5, the three-dimensional coordinate sequence of the three-dimensional point cloud is encoded into a spatial vector representing its spatial features through the scene implicit encoding, and the spatial vector is mapped to the space where the target feature is located through the fully connected layer of the multi-layer perceptron, thereby obtaining the encoded feature.

[0009] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have such technical features, wherein, in step S5, the training rounds are 100, the batch size is 12544, the number of threads is 10, the loss function is calculated using the cross entropy function, the initial learning rate is set to 1e-4, and the learning rate decay weight is 0.003.

[0010] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have such technical features, wherein, in step S1, the target detection network model outputs the detection frame of each object and the corresponding object label, and in step S2, for the object, the image within the detection frame is used to encode the visual language features of the object, and the object label is used to encode the semantic features of the object, and for the background, the image outside the detection frame is used to encode the visual language features of the background, and the region label is used to encode the textual semantic features of the background, and the encoded features of the object and the background are used as the target features.

[0011] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have such technical features, wherein the visual language feature extraction model is a pre-trained CLIP, whose encoder is ViT-B / 32, and the text feature extraction model is a pre-trained SentenceBERT, whose encoder is mpnet-base-v2.

[0012] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have such a technical feature, wherein, in step S1, the target detection network model is a pre-trained Detic, and its confidence threshold for judging the object is 0.4.

[0013] The method for storing scene layout and object information based on implicit expression perception provided by the present invention may also have such a technical feature, wherein, in step S4, the number of layers of the hash code is 18, and the dimension of each layer is 8.

[0014] The present invention provides an application method of scene layout and object information based on implicit expression perception storage, which is used to infer scene areas and / or objects corresponding to spatial positions. The method has the following technical features, which include the following steps: inputting the three-dimensional coordinates corresponding to the spatial position into the above-mentioned scene implicit coding, the scene implicit coding outputs the semantic features of the scene area and / or object, comparing the features of the scene area and object in the scene of the spatial position, and obtaining the scene area and / or object corresponding to the spatial position.

[0015] The present invention provides an application method of scene layout and object information based on implicit expression perception storage, which is used to locate a specific scene area or a specific object in a specific scene area according to an input text. The method has the following technical features, which include the following steps: Step SA1, inputting the text into a visual language feature extraction model and a text feature extraction model respectively to extract the visual language features and text semantic features corresponding to the text as text features; Step SA2, randomly sampling the three-dimensional coordinate sequence used for training the above-mentioned scene implicit coding, inputting multiple sampling results into the scene implicit coding to obtain the visual language features and text semantic features of each sampling result as multiple sampling features; Step SA3, calculating the similarity between the text feature and each sampling feature, and finding the sampling feature with the highest similarity, the three-dimensional coordinates corresponding to the sampling feature are the location of the specific scene area corresponding to the text or the specific object in the specific scene area.

[0016] The present invention provides an application method for scene layout and object information based on implicit expression perception storage, which is used to locate a specific scene area or a specific object in a specific scene area according to an input depth map. The method has the following technical features, which include the following steps: Step SB1, inputting the depth map into a visual language feature extraction model and a text feature extraction model respectively to extract the visual language features and text semantic features corresponding to the depth map as picture features; Step SB2, randomly sampling the three-dimensional coordinate sequence used for training the above-mentioned scene implicit coding, inputting multiple sampling results into the scene implicit coding to obtain the visual language features and text semantic features of each sampling result as multiple sampling features; Step SB3, calculating the similarity between the picture feature and each of the sampling features, and finding the sampling feature with the highest similarity, the three-dimensional coordinates corresponding to the sampling feature are the location of the specific scene area or the specific object in the specific scene area corresponding to the depth map.

[0017] Function and Effect of the Invention

[0018] According to the method of storing scene layout and object information based on implicit expression perception of the present invention, the contrast loss between the features predicted by scene implicit coding and the features obtained by the large model is established through the input depth map sequence data, and the knowledge of the large model trained in a massive large-scale Internet data set is used to make the scene implicit coding learn the relationship between the position, area, and object in the scene through an end-to-end training method, so that the scene implicit coding can be used to synchronously establish a mapping between spatial coordinates and object semantic information and regional information, solving the problem that traditional scene understanding algorithms cannot understand scene layout information. Therefore, the scene region understanding reasoning-related tasks that were previously difficult to achieve can be completed with the help of regional information. In addition, since the target features used for training scene implicit coding are to predict the object semantic information and regional information obtained through the existing visual language large model, the annotation cost required for training is extremely low.

[0019] After the implicit scene encoding is trained, it can be used for a variety of applications such as inferring the objects and regional layout information corresponding to the spatial position, locating specific areas / specific objects based on input text, and locating specific locations in the environment based on input images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a method for storing scene layout and object information based on implicit expression perception in an embodiment of the present invention;

[0021] Figure 2 It is a schematic diagram of the data processing and method flow in an embodiment of the present invention;

[0022] Figure 3is a flow chart of an application method for reasoning object and regional layout information corresponding to spatial positions in an embodiment of the present invention;

[0023] Figure 4 is a flow chart of an application method for locating a specific object in a specific area according to input text in an embodiment of the present invention;

[0024] Figure 5 is an example diagram of the similarity results of calculating the visual language features and text semantic features corresponding to the object and the scene area in an embodiment of the present invention;

[0025] Figure 6 is an example diagram of a result of locating a specific object in a specific area according to input text in an embodiment of the present invention;

[0026] Figure 7 is a flow chart of an application method for locating a specific location in an environment according to an input image in an embodiment of the present invention;

[0027] Figure 8 This is an example diagram of the result of locating a specific location in an environment based on an input image in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the method of storing scene layout and object information based on implicit expression perception of the present invention is specifically described below in combination with embodiments and drawings.

[0029] <Example>

[0030] Figure 1 is a flowchart of a method for storing scene layout and object information based on implicit expression perception in this embodiment, Figure 2 It is a schematic diagram of the data processing and method flow in this embodiment.

[0031] like Figure 1 and Figure 2 As shown, the method comprises the following steps:

[0032] Step S1, use a pre-trained open set target detection network model to perform target detection on the input depth map to obtain the detection frame of the object, and the outside of the detection frame is the background.

[0033] Step S2, based on the detection frame and in combination with the regional information label of the spatial position corresponding to the image, a pre-trained visual language feature extraction model and a text feature extraction model are used to extract the visual language features and semantic features of the object and the background respectively as target features.

[0034] Step S3, back-projecting the depth map corresponding to the image into three-dimensional space through the camera posture to obtain a 3D point cloud corresponding to the scene in the depth map.

[0035] Step S4, initialize a multi-scale hash code as a scene implicit code to represent the relationship between the 3D point cloud and the corresponding features.

[0036] Step S5, encode the 3D point cloud through scene implicit coding, compare the encoded features with the target features to learn and construct a loss function, and optimize the scene implicit coding based on the loss function. The optimized scene implicit coding is the scene layout and object information in the picture stored based on implicit expression perception.

[0037] The above steps will be described in detail below.

[0038] Step S1, use a pre-trained open set target detection network model to perform target detection on the input image to obtain the detection frame of the object, and the outside of the detection frame is the background.

[0039] In step S1, the target detection network model is Detic, and its confidence threshold for judging objects is 0.4. The RGB image and the corresponding depth map are obtained through the RGB-D image. The information sequence of the RGB image is input into the model for object detection, and the model outputs the detection box surrounding the object and the object label.

[0040] like Figure 2 As shown, a picture of an indoor environment is taken as an example, in which objects include a coffee table, a cushion, an air-conditioning outlet, etc., and the background scene area is a living room. Through step S1, the detection frame of each object and the corresponding object label can be obtained.

[0041] Step S2, based on the detection frame and in combination with the regional information label of the spatial position corresponding to the depth map, a pre-trained visual language feature extraction model and a text feature extraction model are used to extract the visual language features and text semantic features of the object and the background respectively as target features.

[0042] In step S2, the visual language feature extraction model is CLIP, and its encoder is ViT-B / 32. The text feature extraction model is SentenceBERT, and its encoder is mpnet-base-v2. For objects, the image within the detection frame is used to encode the visual language features of the object, and the object label is used to encode the semantic features of the object. For the background scene area, the image outside the detection frame is used to encode the visual language features of the background, and the region label is used to encode the text semantic features of the background. The encoded object and background features are used as target features.

[0043] Step S3, back-projecting the depth map corresponding to the image into three-dimensional space through the camera posture to obtain a 3D point cloud corresponding to the scene in the depth map.

[0044] Step S4, initialize a multi-scale hash code as a scene implicit code to represent the relationship between the 3D point cloud and the corresponding features.

[0045] In step S4, the number of hash coding layers is 18, and the dimension of each layer is 8.

[0046] Step S5, encode the 3D point cloud coordinates through scene implicit coding, compare the encoded features with the target features to learn and construct a loss function, and optimize the scene implicit coding based on the loss function. The optimized scene implicit coding is the scene layout and object information in the picture stored based on implicit expression perception.

[0047] That is, in step S5, the scene implicit coding (scene implicit expression) is trained, and the 3D coordinate sequence in the 3D point cloud obtained in step S3 is taken as input. The 3D coordinate sequence is encoded into a spatial vector representing its spatial features through scene implicit coding, and the spatial vector is mapped to the space where the target features (including target visual language features and target text semantic features) obtained in step S2 are located through the fully connected layer of the multi-layer perceptron (MLP) to obtain the encoded features. Then, the encoded features are compared with the target features to learn and construct a loss function, and then the scene implicit coding is optimized based on the loss function.

[0048] In this embodiment, in step S5, the number of training rounds is 100, the batch size is 12544, the number of threads is 10, the loss function is calculated using the cross entropy function, the initial learning rate is set to 1e-4, and the learning rate decay weight is 0.003.

[0049] Figure 3 It is a flow chart of the application method of inferring the object and regional layout information corresponding to the spatial position in this embodiment.

[0050] like Figure 3 As shown, after obtaining the above scene implicit coding, one application method is to use it to infer the scene area and / or object corresponding to the spatial position, and the application method includes the following steps:

[0051] The 3D coordinates of the spatial position are input into the trained scene implicit coding, and the scene implicit coding outputs the semantic features of the scene area and / or object. The texts of the known areas and objects contained in the environment are respectively encoded by visual language features and text features to obtain the features to be compared. The similarity is calculated with the features obtained by implicit coding, and the text (i.e., label) corresponding to the feature to be compared with the highest similarity is used as the scene area and / or object corresponding to the spatial position.

[0052] Figure 4 It is a flow chart of an application method for locating a specific object in a specific area according to input text in this embodiment.

[0053] like Figure 4 As shown, after obtaining the above scene implicit coding, another application method is to use it to locate a specific scene area or a specific object in a specific scene area according to the input text. The application method includes the following steps:

[0054] Step SA1, inputting the text into a pre-trained visual language feature extraction model and a pre-trained text feature extraction model respectively to extract visual language features and text semantic features corresponding to the text as text features.

[0055] Step SA2, randomly sampling the 3D coordinate sequence used for training scene implicit coding, inputting multiple sampling results into the trained scene implicit coding to obtain the visual language features and text semantic features of each sampling result as multiple sampling features.

[0056] Step SA3, compare the similarity between the text feature and each sampling feature, and find the sampling feature with the highest (closest) similarity. The 3D coordinates corresponding to the sampling feature are the specific scene area corresponding to the input text or the location of a specific object in the specific scene area.

[0057] Figure 5 This is an example diagram of the similarity results of calculating the visual language features and text semantic features corresponding to the object and the scene area in this embodiment.

[0058] like Figure 5 As shown, the input text can be the text of an object or a scene area, and the similarities between the text features and multiple sampling features are calculated respectively to obtain the results shown in the figure, and then the sampling features with the highest similarity are found.

[0059] Figure 6 This is an example diagram of the result of locating a specific object in a specific area according to input text in this embodiment.

[0060] like Figure 6 As shown, taking the multi-room indoor environment provided by the open source standard data set Matterport3D as an example, in an example in the first row of the figure, the input text is "Table in the bedroom", that is, it is necessary to locate the table in the bedroom. Through the method of this embodiment, the correct position in the figure is accurately located (the rightmost column of the first row in the figure, the star in the bedroom area).

[0061] Figure 6 Two other examples of input text are also shown in , and for each example, Figure 6The figure also shows the search results of two other methods in the prior art (CLIP-Field and VLMaps). It can be seen that these two methods will find similar objects in other areas, that is, recognition errors will occur; while the method of this embodiment can locate objects in the correct scene area.

[0062] Figure 7 It is a flow chart of an application method of locating a specific location in an environment according to an input image in this embodiment.

[0063] like Figure 7 As shown, after obtaining the above scene implicit coding, another application method is to use it to locate a specific scene area or a specific object in a specific scene area in the environment according to the input depth map. The application method includes the following steps:

[0064] Step SB1, input the depth map into a pre-trained visual language feature extraction model and a pre-trained text feature extraction model respectively to extract the visual language features and text semantic features corresponding to the depth map as image features.

[0065] Step SB2, randomly sampling the 3D coordinate sequence used for training scene implicit coding, and inputting multiple sampling results into the trained scene implicit coding to obtain visual language features and text semantic features of each sampling result as multiple sampling features.

[0066] Step SB3, compare the similarity between the image feature and each sampling feature, find the sampling feature with the highest (closest) similarity, and the 3D coordinates corresponding to the sampling feature are the specific scene area corresponding to the input depth map or the location of a specific object in the specific scene area.

[0067] Figure 8 This is an example diagram of the result of locating a specific location in an environment based on an input image in this embodiment.

[0068] like Figure 8 As shown, taking the multi-room indoor environment provided by the open source Matterport3D as an example, in an example in the first row of the figure, the input image is a picture taken toward one side of the living room, and the picture contains objects such as a television. Through the method of this embodiment, the scene area and the location of the object are accurately located (the rightmost column of the first row in the figure, the area marked by the box).

[0069] Figure 8 Examples of two other input images are also shown in , and for each example, Figure 8The figure also shows the search results of two other methods in the prior art (CLIP-Field and VLMaps). It can be seen that these two methods will find similar areas in multiple scene areas (areas marked with boxes in the figure), that is, the positioning accuracy is not high; while the method of this embodiment can accurately locate the closest area.

[0070] The test of the above examples proves that the method of this embodiment can successfully recognize and infer spatial area information in a series of scenes containing multiple areas and multiple objects.

[0071] Functions and Effects of the Embodiments

[0072] According to the method for storing scene layout and object information based on implicit expression perception provided by this embodiment, the contrast loss between the features predicted by scene implicit coding and the features obtained by the large model is established through the input depth map sequence data, and the knowledge of the large model trained in massive large-scale Internet data sets is used to enable the scene implicit coding to learn the relationship between the position, area, and object in the scene through an end-to-end training method, so that the scene implicit coding can be used to synchronously establish a mapping between spatial coordinates and object semantic information and regional information, solving the problem that traditional scene understanding algorithms cannot understand scene layout information. Therefore, it is possible to use regional information to complete scene region understanding reasoning-related tasks that were previously difficult to do. In addition, since the target features used for training scene implicit coding are to predict the object semantic information and regional information obtained through the existing visual language large model, the annotation cost required for training is extremely low.

[0073] After training, scene implicit coding can be used to infer objects and regional layout information corresponding to spatial positions, locate specific areas / specific objects based on input text, and locate specific places in the environment based on input images. In addition, scene implicit coding can be used as a robot's spatial scene memory storage, enabling the robot to recognize and infer spatial regional information in the above scenes, thereby enabling the robot to establish the relationship between the destination, target object and region, and can use regional information to complete scene region understanding and reasoning related tasks that were previously difficult to do.

[0074] The above embodiments are only used to illustrate the specific implementation of the present invention, and the present invention is not limited to the description scope of the above embodiments. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for storing scene layout and object information based on implicit expression perception, characterized in that: The following steps are involved: Step S1, using a target detection network model to perform target detection on the input image to obtain a detection frame of the object, and the outside of the detection frame is the background; Step S2, based on the detection frame and in combination with the region information label of the spatial position corresponding to the image, a visual language feature extraction model and a text feature extraction model are used to respectively extract the visual language features and text semantic features of the object and the background as target features; Step S3, back-projecting the depth map corresponding to the image into a three-dimensional space through the camera posture to obtain a three-dimensional point cloud corresponding to the scene in the depth map; Step S4, initializing a multi-scale hash code as a scene implicit code to represent the relationship between the three-dimensional point cloud and the corresponding features; Step S5, encode the three-dimensional point cloud through the scene implicit coding to obtain the encoded features, compare the encoded features with the target features to learn and construct a loss function, and optimize the scene implicit coding based on the loss function. The optimized scene implicit coding is the scene layout and object information in the picture stored based on implicit expression perception.

2. The method for storing scene layout and object information based on implicit expression perception according to claim 1, characterized in that: in, In step S5, the three-dimensional coordinate sequence of the three-dimensional point cloud is encoded into a spatial vector representing its spatial features through the scene implicit coding, and the spatial vector is mapped to the space where the target feature is located through the fully connected layer of the multi-layer perceptron, so as to obtain the encoded feature.

3. The method for storing scene layout and object information based on implicit expression perception according to claim 2, characterized in that: in, In step S5, the number of training rounds is 100, the batch size is 12544, the number of threads is 10, the loss function is calculated using the cross entropy function, the initial learning rate is set to 1e-4, and the learning rate decay weight is 0.

003.

4. The method for storing scene layout and object information based on implicit expression perception according to claim 1, characterized in that: in, In step S1, the target detection network model outputs the detection frame of each object and the corresponding object label. In step S2, for the object, the image in the detection frame is used to encode the visual language features of the object, and the object label is used to encode the semantic features of the object. For the background, the image outside the detection frame is used to encode the visual language features of the background, and the region label is used to encode the textual semantic features of the background. The encoded features of the object and the background are used as the target features.

5. The method for storing scene layout and object information based on implicit expression perception according to claim 4, characterized in that: in, The visual language feature extraction model is the pre-trained CLIP, whose encoder is ViT-B / 32. The text feature extraction model is the pre-trained SentenceBERT, and its encoder is mpnet-base-v2.

6. The method for storing scene layout and object information based on implicit expression perception according to claim 1, characterized in that: in, In step S1, the target detection network model is a pre-trained Detic, and its confidence threshold for judging objects is 0.

4.

7. The method for storing scene layout and object information based on implicit expression perception according to claim 1, characterized in that: in, In step S4, the number of layers of the hash code is 18, and the dimension of each layer is 8.

8. An application method of scene layout and object information based on implicit expression perception storage, used to infer scene areas and / or objects corresponding to spatial positions, characterized in that: The following steps are involved: The three-dimensional coordinates corresponding to the spatial position are input into scene implicit coding, and the scene implicit coding outputs semantic features of the scene area and / or the object. The text of the known area and the text of the object contained in the environment of the spatial position are respectively encoded by visual language features and text features to obtain multiple features to be compared. The similarity between the semantic feature and each of the features to be compared is calculated, and the text corresponding to the feature to be compared with the highest similarity is the scene area and / or object corresponding to the spatial position. The scene implicit coding is obtained according to the method for storing scene layout and object information based on implicit expression perception as described in any one of claims 1 to 7.

9. An application method of scene layout and object information based on implicit expression perception storage, used to locate a specific scene area or a specific object in a specific scene area according to input text, characterized in that: The following steps are involved: Step SA1, inputting the text into a visual language feature extraction model and a text feature extraction model respectively to extract visual language features and text semantic features corresponding to the text as text features; Step SA2, randomly sampling the three-dimensional coordinate sequence used for training scene implicit coding, inputting multiple sampling results into the scene implicit coding to obtain visual language features and text semantic features of each of the sampling results as multiple sampling features; Step SA3, calculating the similarity between the text feature and each of the sampling features, finding the sampling feature with the highest similarity, the three-dimensional coordinates corresponding to the sampling feature are the location of the specific scene area corresponding to the text or the specific object in the specific scene area, The scene implicit coding is obtained according to the method for storing scene layout and object information based on implicit expression perception as described in any one of claims 1 to 7.

10. An application method of scene layout and object information based on implicit expression perception storage, used to locate a specific scene area or a specific object in a specific scene area according to an input depth map, characterized in that: The following steps are involved: Step SB1, inputting the depth map into a visual language feature extraction model and a text feature extraction model respectively to extract visual language features and text semantic features corresponding to the depth map as picture features; Step SB2, randomly sampling the three-dimensional coordinate sequence used for training scene implicit coding, inputting multiple sampling results into the scene implicit coding to obtain visual language features and text semantic features of each of the sampling results as multiple sampling features; Step SB3, calculating the similarity between the image feature and each of the sampling features, finding the sampling feature with the highest similarity, the three-dimensional coordinates corresponding to the sampling feature are the location of the specific scene area corresponding to the depth map or the specific object in the specific scene area, The scene implicit coding is obtained according to the method for storing scene layout and object information based on implicit expression perception as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Cross-modal visual positioning method for low-altitude security scene

    CN117911513A

  • Representation learning using joint semantic vectors

    GB201918277D0

  • Video rendering method and system based on multi-scale spatial delta encoding

    WO2024138350A1

Cited By

  • Scene representation method and device for enhancing three-dimensional space understanding of large language model

    CN119942027A

  • Training method of implicit problem inference model and implicit problem inference method

    CN120409712A