A method for constructing a scene graph, a method for retrieving a target, and a related apparatus.

Abstract

This application discloses a scene graph construction method, a target retrieval method, and related apparatus, comprising: constructing a set of topological nodes and a set of topological edges; for each topological node in the set of topological nodes, using the set of visual information, textual information, and spatial location information of the topological node as the multimodal data of the topological node; for each topological node in the set of topological nodes, determining the multimodal data of the object node corresponding to the topological node and the set of connecting edges corresponding to the topological node based on the visual information of the topological node; and constructing a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the set of topological edges, and the set of connecting edges corresponding to each topological node. This target scene graph possesses semantic consistency and spatial continuity, significantly improving the expressive power of the scene graph in complex environments and its generalization performance across scenes.

Description

Technical Field

[0001] This application relates to the field of embodied intelligent environmental perception, and in particular to a scene graph construction method, a target retrieval method, and related apparatus. Background Technology

[0002] In the specific field of intelligence, scene graphs serve as information carriers representing the structure of indoor environments, providing effective support for downstream tasks such as object localization, embodied question answering, and target navigation. This enables intelligent agents to make reasonable decisions based on multimodal sensory inputs and contextual cues when they lack explicit task experience.

[0003] However, traditional scene graphs commonly used in solutions have several problems. For example, room nodes in tree-structured scene graphs with a "building-room-object" structure are usually assigned an abstract semantic label and aggregated with a large number of object nodes, resulting in complex and chaotic semantic features of the room hierarchy, weakening salient features and expressive power; the edges between object nodes usually lack explicit modeling of spatial relationships such as proximity and visibility; and individual object nodes lack effective contextual interaction mechanisms, causing tree-structured scene graphs to suffer from semantic confusion and hierarchical disconnect.

[0004] For example, scene graphs in traditional solutions typically use independently constructed topological graphs that lack semantic connections to represent the spatial reachability of the environment. This results in navigable pathpoints in the scene graph not providing complete semantics of the surrounding environment, limiting the agent's ability to understand local details. If the observation information only contains semantic information but lacks topological structure, the agent cannot capture complete spatial connectivity, which reduces the agent's spatial reasoning and planning capabilities, causing the agent to get stuck in problems such as regional closure or action failure, making it difficult to smoothly advance the task process.

[0005] How to construct a scene graph that can collaboratively express the semantics and topological structure of complex scenes has become one of the urgent technical problems to be solved in the field of embodied intelligent environmental perception. Summary of the Invention

[0006] To address the aforementioned issues, this application provides a scene graph construction method that integrates the semantic contextual relationships and spatial reachability information of the environment into the graph structure, thereby constructing a scene graph capable of coordinating semantic and topological representations in complex scenes.

[0007] The embodiments of this application disclose the following technical solutions:

[0008] The first aspect of this application discloses a method for constructing a scene graph, including:

[0009] Based on the scene structure data, construct a set of topological nodes and a set of topological edges;

[0010] For each topological node in the set of topological nodes, the set of visual information, text information and spatial location information of the topological node is used as the multimodal data of the topological node;

[0011] For each topological node in the set of topological nodes, based on the visual information of the topological node, the multimodal data of the object node corresponding to the topological node and the set of connecting edges corresponding to the topological node are determined; the multimodal data of the object node includes the visual information, text information and spatial location information of the object node; the set of connecting edges is a set composed of the connecting edges between the topological node and the corresponding object node.

[0012] A target scene graph is constructed based on the multimodal data of each topological node, the multimodal data of each object node, the set of topological edges, and the set of connection edges corresponding to each topological node.

[0013] In one optional implementation, for each topological node in the set of topological nodes, determining the multimodal data of the object node corresponding to the topological node and the set of connecting edges corresponding to the topological node based on the visual information of the topological node includes:

[0014] Based on the visual information and instance segmentation model of the topological node, multiple sub-visual information of the topological node are determined; each sub-visual information corresponds one-to-one with the object node; each visual information corresponds to multiple sub-visual information.

[0015] The sub-visual information of the topological node is input into the visual language generation model to determine the text information corresponding to the sub-visual information;

[0016] For each sub-visual information corresponding to the topological node, the set constructed from the sub-visual information, the text information corresponding to the sub-visual information, and the spatial location information of the object node corresponding to the sub-visual information is used as the multimodal data of the object node corresponding to the topological node.

[0017] Connect the topological node to each corresponding object node to obtain multiple connecting edges; the set of these multiple connecting edges is taken as the connecting edge set corresponding to the topological node.

[0018] In one alternative implementation, the method further includes:

[0019] For each topological node, the visual similarity and spatial distance between any two object nodes are calculated sequentially. The visual similarity is generated based on the two sub-visual information corresponding to the two object nodes. The spatial distance is generated based on the two spatial position information corresponding to the two object nodes.

[0020] For every two object nodes among all the object nodes, if the visual similarity between the two object nodes is greater than a preset first threshold and the spatial distance is less than a preset second threshold, then one of the two object nodes is deleted, resulting in multiple object nodes corresponding to the topology node.

[0021] In one optional implementation, constructing the topological node set and topological edge set based on scene structure data includes:

[0022] Based on the scene structure data, a skeleton diagram is constructed;

[0023] Based on multiple candidate topology nodes that meet the first screening condition selected from multiple candidate topology nodes corresponding to the skeleton graph, the topology node set is constructed; the first screening condition indicates that the node spacing between two candidate topology nodes is equal to a preset third threshold.

[0024] Based on the skeleton graph, construct a fully connected graph;

[0025] The topological edge set is constructed based on multiple candidate connection paths that meet the second screening condition selected from multiple candidate connection paths in the fully connected graph; the second screening condition indicates that the connection distance of the candidate connection path is less than a preset fourth threshold and the candidate connection path does not cross an obstacle area.

[0026] The second aspect of this application discloses a target retrieval method, comprising:

[0027] Obtain a semantic environment database; the semantic environment database is generated based on the target scene graph described in any of the first aspects; the semantic environment database includes feature vectors corresponding to the multimodal data of each topological node in the target scene graph, and feature vectors corresponding to the multimodal data of each object node;

[0028] The object to be queried is input into the visual language feature extraction model to obtain the target feature vector; the object to be queried includes the image to be queried and / or the text to be queried.

[0029] The object node with the highest similarity to the target feature vector is determined from the semantic environment database and is taken as the target object.

[0030] In one optional implementation, determining the object node with the highest similarity to the target feature vector from the semantic environment database as the target object includes:

[0031] Calculate the similarity between the feature vector corresponding to each topological node in the semantic environment database and the target feature vector, and obtain the top M1 topological nodes with the highest similarity to the target feature vector; where M1 is an integer greater than or equal to 1.

[0032] For each of the M1 topology nodes, determine the multiple topology nodes adjacent to that topology node, and use them as the target topology node set corresponding to that topology node;

[0033] For each of the M1 topological nodes, calculate the similarity between the feature vector of the topological node and the feature vector of each target topological node in the target topological node set corresponding to the topological node, and obtain the top M2 topological nodes with the highest similarity to the target feature vector; where M2 is an integer greater than or equal to 1.

[0034] Determine the target region covering the M1 topological nodes and the M2 topological nodes corresponding to each of the M1 topological nodes, and determine the M3 object nodes covered by the target region; where M3 is an integer greater than or equal to 1.

[0035] Calculate the similarity between the feature vector corresponding to each of the M3 object nodes and the target feature vector, and obtain the object node with the highest similarity to the target feature vector;

[0036] The object node with the highest similarity to the target feature vector is taken as the target object.

[0037] In one alternative implementation, obtaining the semantic environment database includes:

[0038] The multimodal data corresponding to each topological node in the target scene graph is input into the visual language feature extraction model to obtain the feature vector corresponding to each topological node;

[0039] The multimodal data corresponding to each object node in the target scene graph is input into the visual language feature extraction model to obtain the feature vector corresponding to each object node;

[0040] The set constructed from the feature vectors corresponding to each topological node and the feature vectors corresponding to each object node is used as the semantic environment database.

[0041] A third aspect of this application provides a scene graph construction apparatus, comprising:

[0042] The data set acquisition module is used to construct a set of topological nodes and a set of topological edges based on the scene structure data;

[0043] The first multimodal data acquisition module is used to, for each topological node in the set of topological nodes, take the set of visual information, text information and spatial location information of the topological node as the multimodal data of the topological node;

[0044] The second multimodal data acquisition module is used to determine, for each topological node in the set of topological nodes, the multimodal data of the object node corresponding to the topological node and the set of connecting edges corresponding to the topological node based on the visual information of the topological node; the multimodal data of the object node includes the visual information, text information and spatial location information of the object node; the set of connecting edges is a set composed of the connecting edges between the topological node and the corresponding object node.

[0045] The target scene graph acquisition module is used to construct a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the set of topological edges, and the set of connecting edges corresponding to each topological node.

[0046] A fourth aspect of this application provides a target retrieval device, comprising:

[0047] A database acquisition module is used to acquire a semantic environment database; the semantic environment database is generated based on the target scene graph as described in any of the first aspects; the semantic environment database includes feature vectors corresponding to the multimodal data of each topological node in the target scene graph, and feature vectors corresponding to the multimodal data of each object node;

[0048] The target feature vector determination module is used to input the query object into the visual language feature extraction model to obtain the target feature vector; the query object includes the query image and / or the query text.

[0049] The target object query module is used to determine the object node with the highest similarity to the target feature vector from the semantic environment database, and use it as the target object.

[0050] The fifth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first and second aspects.

[0051] Compared with the prior art, this application has the following beneficial effects:

[0052] This application discloses a scene graph construction method, comprising: constructing a topological node set and a topological edge set based on scene structure data; for each topological node in the topological node set, using the set of visual information, text information, and spatial location information of the topological node as the multimodal data of the topological node; for each topological node in the topological node set, determining the multimodal data of the object node corresponding to the topological node and the set of connecting edges corresponding to the topological node based on the visual information of the topological node; and constructing a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set, and the set of connecting edges corresponding to each topological node.

[0053] Thus, the target scene graph of this application includes multimodal data such as visual information, text information, and spatial location information of topological nodes, as well as multimodal data such as visual information, text information, and spatial location information of object nodes; at the same time, it establishes a spatial continuity relationship between object nodes and topological nodes; making the target scene graph semantically consistent and spatially continuous, which can effectively solve the problem of semantics and spatial structure being disconnected in traditional scene graphs, significantly improve the expressive power of scene graphs in complex environments and their generalization performance across scenes, and provide a more complete and coherent environmental understanding foundation for downstream tasks such as scene understanding, object localization, and target navigation. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart illustrating a scene graph construction method provided in an embodiment of this application;

[0056] Figure 2 A flowchart of a target retrieval method provided in an embodiment of this application;

[0057] Figure 3 This is a schematic diagram of the structure of a scene graph construction device provided in an embodiment of this application;

[0058] Figure 4 This is a schematic diagram of the structure of a target retrieval device provided in an embodiment of this application. Detailed Implementation

[0059] In the specific field of intelligence, scene graphs serve as information carriers representing the structure of indoor environments, providing effective support for downstream tasks such as object localization, embodied question answering, and target navigation. This enables intelligent agents to make reasonable decisions based on multimodal sensory inputs and contextual cues when they lack explicit task experience.

[0060] However, traditional scene graphs commonly used in solutions have several problems. For example, room nodes in tree-structured scene graphs with a "building-room-object" structure are usually assigned an abstract semantic label and aggregated with a large number of object nodes, resulting in complex and chaotic semantic features of the room hierarchy, weakening salient features and expressive power; the edges between object nodes usually lack explicit modeling of spatial relationships such as proximity and visibility; and individual object nodes lack effective contextual interaction mechanisms, causing tree-structured scene graphs to suffer from semantic confusion and hierarchical disconnect.

[0061] For example, scene graphs in traditional solutions typically use independently constructed topological graphs that lack semantic connections to represent the spatial reachability of the environment. This results in navigable pathpoints in the scene graph not providing complete semantics of the surrounding environment, limiting the agent's ability to understand local details. If the observation information only contains semantic information but lacks topological structure, the agent cannot capture complete spatial connectivity, which reduces the agent's spatial reasoning and planning capabilities, causing the agent to get stuck in problems such as regional closure or action failure, making it difficult to smoothly advance the task process.

[0062] How to construct a scene graph that can collaboratively express the semantics and topological structure of complex scenes has become one of the urgent technical problems to be solved in the field of embodied intelligent environmental perception.

[0063] To address the aforementioned issues, this application discloses a scene graph construction method, comprising: constructing a topological node set and a topological edge set based on scene structure data; for each topological node in the topological node set, using the set of visual information, textual information, and spatial location information of the topological node as the multimodal data of the topological node; for each topological node in the topological node set, determining the multimodal data of the object node corresponding to the topological node and the set of connecting edges corresponding to the topological node based on the visual information of the topological node; and constructing a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set, and the set of connecting edges corresponding to each topological node.

[0064] Thus, the target scene graph of this application includes multimodal data such as visual information, text information, and spatial location information of topological nodes, as well as multimodal data such as visual information, text information, and spatial location information of object nodes; at the same time, it establishes a spatial continuity relationship between object nodes and topological nodes; making the target scene graph semantically consistent and spatially continuous, which can effectively solve the problem of semantics and spatial structure being disconnected in traditional scene graphs, significantly improve the expressive power of scene graphs in complex environments and their generalization performance across scenes, and provide a more complete and coherent environmental understanding foundation for downstream tasks such as scene understanding, object localization, and target navigation.

[0065] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0066] Figure 1 This is a flowchart illustrating a scene graph construction method provided in an embodiment of this application. (Combined with...) Figure 1 As shown, the scene graph construction method disclosed in this application includes:

[0067] S101, based on scene structure data, construct a set of topological nodes and a set of topological edges.

[0068] This application obtains scene structure data of a given target scene. The given target scene can be a living room in a home, a ward in a hospital, or a classroom in a school, etc. This application does not limit the specific scope of the given target scene, nor does it limit the specific method of obtaining the scene structure data.

[0069] Scene structure data can be 3D point cloud data or geometric structure representation data.

[0070] This application uses three-dimensional point cloud data as an example to illustrate the scene graph construction method disclosed in this application.

[0071] After acquiring 3D point cloud data, a pose region map is generated using the camera pose information collected during the 3D point cloud data reconstruction process; a ground region map is obtained by projecting the 3D point cloud data onto a bird's-eye view plane; and an obstacle region map is generated by projecting the 3D point cloud data within a specified height range.

[0072] After obtaining the pose region map, ground region map, and obstacle region map, the free region map can be obtained using formula (1). The specific expression of formula (1) is as follows:

[0073] (1)

[0074] In formula (1), M is the free region map, M P It is a pose region map, M f It is a ground area map and M o This is a map of the obstacle area.

[0075] After obtaining the free region map M, a thinning algorithm can be used to extract the skeleton of the free region M, thereby obtaining the skeleton map M. s The refinement algorithm emphasizes the preservation of the central portion of the free region, which can effectively suppress the skeleton point redundancy problem caused by edge alignment, thereby accurately preserving the core skeleton structure of the scene.

[0076] After obtaining the skeleton map M s Then, for the skeleton diagram M s Sampling is performed to obtain all candidate topological nodes on the skeleton path; then, multiple candidate topological nodes that meet the first screening condition, i.e., the node spacing between two candidate topological nodes is equal to the preset third threshold, are selected from all candidate topological nodes; the set of multiple candidate topological nodes that meet the first screening condition is taken as the topological node set. The topological nodes in the topological node set have spatial sparse distribution; the topological node set is expressed as shown in formula (2):

[0077] (2)

[0078] V in formula (2) c V represents the set of topological nodes; i c Dist(V) represents the i-th topological node in the set of topological nodes; i c V j c ) represents the node spacing between the i-th and j-th topological nodes in the topological node set; τ is a preset third threshold; M s This is the skeleton diagram mentioned in the foregoing embodiments of this application.

[0079] In the skeleton diagram M s Sampling was performed to obtain the skeleton map M. s After identifying all candidate topology nodes, each pair of distinct candidate topology nodes is connected, ensuring that there is exactly one edge connecting each pair, resulting in a fully connected graph. In this application, the edge connecting each pair of distinct candidate topology nodes in the fully connected graph is called a candidate connection path.

[0080] After obtaining the fully connected graph, multiple candidate connection paths that meet the second screening condition can be selected from the multiple candidate connection paths in the fully connected graph. That is, multiple candidate connection paths whose length (also known as connection distance) is less than a preset fourth threshold and do not pass through obstacle areas can be selected from the fully connected graph. The set of multiple candidate connection paths that meet the second screening condition is called the topological edge set. The sparse topological edge set can represent the effective and traversable spatial connection relationship between topological nodes.

[0081] It is understood that the second screening condition of this application indicates that the connection distance of the candidate connection path is less than the preset fourth threshold and the candidate connection path does not cross the obstacle area.

[0082] It is understandable that each topological edge in the set of topological edges has two topological nodes as its endpoints.

[0083] The set of topological nodes and edges obtained in this step together constitute the topology layer. As a fundamental component of the two-layer multimodal topology-object scene graph, the topology layer can effectively characterize the navigable spatial topology within the free region of the given target scene, providing structured support for downstream tasks.

[0084] S102, for each topological node in the set of topological nodes, the set of visual information, text information and spatial location information of the topological node is used as the multimodal data of the topological node.

[0085] The visual information of a topological node can be the image corresponding to the topological node; the text information can be text used to describe the characteristics of the topological node (such as text describing the image corresponding to the topological node); the spatial location information can be the coordinates of the topological node in three-dimensional space.

[0086] In one alternative implementation, after obtaining the visual information of the topological node, such as multiple visual images of the topological node, the multiple visual images of the topological node can be input into the visual language generation model to obtain the text information corresponding to each visual image.

[0087] The visual language generation model in this step is a model that can convert images into text information, such as Tongyi Qianwen or LLaVA.

[0088] Below, we will reiterate the process of this step, using visual information as an example of visual images.

[0089] For example, for a topological node 1 in the set of topological nodes, first obtain P1 view-aware images (P1≥1) corresponding to the spatial location of topological node 1; then input each of the P1 view-aware images into the visual language generation model to generate text information corresponding to each view-aware image, that is, obtain P1 text information; then use the set of P1 text information as the text information corresponding to topological node 1; finally, use the set of P1 view-aware images corresponding to topological node 1 and the set of text information corresponding to topological node 1 as the multimodal data of topological node 1.

[0090] The multimodal data of the topological nodes in this application are expressed in the form of formula (3):

[0091] (3)

[0092] AttrCon in formula (3) i V represents the i-th topological node in the set of topological nodes. i c Multimodal data; Ic i n V represents the i-th topological node. i c The nth visual information; T Ci V represents the i-th topological node. i c The text information, where N is the captured i-th topological node V. i c The total amount of visual information.

[0093] S103, for each topological node in the set of topological nodes, based on the visual information corresponding to the topological node, determine the multimodal data of the object node corresponding to the topological node and the set of connecting edges corresponding to the topological node.

[0094] It is understood that one topology node can correspond to multiple object nodes. In this application, the multimodal data of each object node includes: the visual information corresponding to the object node, the spatial location information of the object node, and the text information of the object node.

[0095] In one alternative implementation, the process of obtaining multimodal data for each object node corresponding to each topological node in the topological node set includes the following steps:

[0096] A1, obtain multiple visual information corresponding to each topology node in the topology node set.

[0097] For example, obtain P1 visual images of topological node 1 in the set of topological nodes.

[0098] A2. For each topological node in the set of topological nodes, based on the visual information of the topological node and the instance segmentation model, determine multiple sub-visual information of the topological node.

[0099] Among them, the open vocabulary detection and segmentation model is used to process visual information, generate bounding boxes and segmentation masks for each object contained in the visual information, and segment the visual information into multiple sub-visual information; the sub-visual information corresponds one-to-one with the object node, that is, there is one object instance in one sub-visual information, corresponding to one object node.

[0100] It is understandable that if a visual information of a topological node contains two object instances, then that visual information corresponds to two sub-visual information; each sub-visual information corresponds to one object instance.

[0101] Assuming the visual information is visual images, each viewpoint image in the P1 viewpoint perception images of topological node 1 is input into the instance segmentation model to obtain multiple sub-viewpoint images corresponding to each visual image.

[0102] A3 inputs each sub-visual information corresponding to each topological node in the topological node set into the visual language generation model to obtain the text description corresponding to each sub-visual information of each topological node.

[0103] A4. For each sub-visual information corresponding to each topological node in the topological node set, the set constructed by the sub-visual information, the text information corresponding to the sub-visual information, and the spatial location information of the object node corresponding to the sub-visual information is used as the multimodal data of the object node corresponding to the topological node.

[0104] Assuming the visual information is a visual image, if the two sub-visual information corresponding to topological node 1 in the topological node set are sub-visual image 1 and sub-visual image 2 respectively; the set consisting of sub-visual image 1, the text information corresponding to sub-visual image 1, and the spatial location information of the object node corresponding to sub-visual image 1, as well as sub-visual image 2, the text information corresponding to sub-visual image 2, and the spatial location information of the object node corresponding to sub-visual image 2, is taken as the multimodal data of the object node corresponding to topological node 1.

[0105] The multimodal data of object nodes in this application is expressed in the form of formula (4):

[0106] (4)

[0107] In formula (4), AttrObj k Multimodal data representing object node k; o k Image representation (visual information) of object node k; To kA text description representing object node k.

[0108] Among them, the three-dimensional coordinates of object node k in the world coordinate system are integrated in To k In addition, when the multimodal data of an object node includes its spatial location information in the world coordinate system, the multimodal data of the object node will further enhance the spatial semantic information of the object instance.

[0109] As mentioned above, the multiple visual information of a topological node are multiple visual information captured from the spatial location of the topological node. One possibility is that some of the multiple visual information corresponding to a topological node describes the same object; in this case, the object nodes corresponding to a topological node obtained using the methods in A1-A4 will contain duplicate object nodes.

[0110] To address the issue of duplicate object instances from different perspectives under the same topological node, after obtaining the multimodal data of the object nodes corresponding to each topological node, deduplication can be performed on multiple object nodes corresponding to the same topological node using the methods described in B1-B2. Specifically:

[0111] B1. For each topological node in the set of topological nodes, calculate the visual similarity and spatial distance between any two object nodes in the set of topological nodes.

[0112] Visual similarity is generated based on the two sub-visual information corresponding to two object nodes. For example, two sub-visual information can be input into the SuperGlue model to obtain the visual similarity between the two sub-visual information. It should be understood that the SuperGlue model is just an example.

[0113] The spatial distance is generated based on the spatial position information of two object nodes. For example, the spatial distance between corresponding object instances can be calculated using the position data in the three-dimensional coordinates of the object nodes.

[0114] B2. For every two object nodes in all object nodes, if the visual similarity between the two object nodes is greater than a preset first threshold and the spatial distance is less than a preset second threshold, then delete one of the two object nodes to obtain multiple object nodes corresponding to the topology node.

[0115] Alternatively, for any two object nodes among all object nodes, if the visual similarity between the two object nodes is greater than a preset first threshold and the spatial distance is less than a preset second threshold, then the two object nodes are determined to represent an object instance, and the description with the highest semantic category confidence is selected as the unified identifier of the object instance.

[0116] Connect the topological node to each corresponding object node to obtain multiple connecting edges; the set of these multiple connecting edges is taken as the connecting edge set corresponding to the topological node.

[0117] S104. Construct a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the set of topological edges, and the set of connection edges corresponding to each topological node.

[0118] Understandably, since the target scene graph is composed of a set of topological nodes, object nodes, and topological edges, and incorporates multimodal data, it can reflect the structure, semantics, and dynamic changes of the scene. It has rich connectivity, data correlation, semantic expressiveness, and scalability, providing a comprehensive and flexible framework for modeling and analyzing complex scenes.

[0119] In summary, the scene graph construction method disclosed in this application effectively combines the spatial connectivity embodied by topological information with the scene semantics expressed by the semantic attributes of nodes by jointly modeling topological nodes and object instances, constructing multimodal semantic representations of topological nodes and multimodal semantic tensors of object nodes. The scene graph representation generated by this method not only possesses semantic consistency and spatial continuity but also effectively solves the problem of semantic and spatial structure disconnect in traditional scene graphs. Through deep fusion of the semantic and topological layers, the expressive power and cross-scene generalization performance of the scene graph in complex environments are significantly improved, providing a more complete and coherent environmental understanding foundation for downstream tasks such as scene understanding, object localization, and target navigation.

[0120] Furthermore, this application also proposes a target retrieval method. Figure 2 This is a flowchart illustrating a target retrieval method provided in an embodiment of this application. (In conjunction with...) Figure 2 As shown, the target retrieval method disclosed in this application includes:

[0121] S201, Obtain the semantic environment database.

[0122] The semantic environment database in this application is generated based on the target scene graph obtained in the foregoing embodiments. The semantic environment database includes the feature vectors corresponding to the multimodal data of each topological node in the target scene graph, and the feature vectors corresponding to the multimodal data of each object node; that is, the essence of the semantic environment database in this application is a feature vector pool corresponding to the target scene graph.

[0123] In one alternative implementation, the steps to obtain the semantic environment database are:

[0124] First, the multimodal data corresponding to each topological node in the target scene graph is input into the visual feature extraction language model to obtain the feature vector corresponding to each topological node. Then, the multimodal data corresponding to each object node in the target scene graph is input into the visual language feature extraction model to obtain the feature vector corresponding to each object node. Finally, the set constructed by the feature vectors corresponding to each topological node and the feature vectors corresponding to each object node is used as the semantic environment database.

[0125] It can be understood that the visual language feature extraction model in this step is a model that encodes visual information and text descriptions and converts them into corresponding feature vectors; such as CLIP (Contrastive Language-Image Pre-training) or SigLIP (Sigmoid Loss for Language Image Pre-Training).

[0126] The eigenvectors corresponding to the topological nodes in this application are expressed as shown in formula (5):

[0127] (5)

[0128] EmbCon in formula (5) i It is the i-th topological node V i c The corresponding feature vector; Ec i n It is the i-th topological node V i c The feature vector corresponding to the nth visual information; F Ci V represents the i-th topological node. i c The feature vector corresponding to the text information; N is the captured i-th topological node V i c The total amount of visual information.

[0129] The expression for the feature vector corresponding to the object node in this application is shown in formula (6):

[0130] (6)

[0131] EmbObj in formula (6) k It is the feature vector corresponding to the k-th object node; Eo k It is the feature vector corresponding to the sub-visual information of the k-th object node; Fo k It is the feature vector corresponding to the text information of the k-th object node.

[0132] S202, input the object to be queried into the visual language feature extraction model to obtain the target feature vector.

[0133] The query object in this application includes the query image and / or the query text; that is, the query object can be an image, text, or a combination of both.

[0134] The object to be queried is input into the visual language feature extraction model, which processes the object to be queried to obtain the target feature vector.

[0135] S203, determine the object node with the highest similarity to the target feature vector from the semantic environment database, and use it as the target object.

[0136] The process includes the following steps:

[0137] D1 calculates the similarity between the feature vector corresponding to each topological node in the semantic environment database and the target feature vector, and obtains the top M1 topological nodes with the highest similarity to the target feature vector.

[0138] Where M1 is an integer greater than or equal to 1; for example, M1 can be 3.

[0139] For example, the similarity between the feature vector corresponding to each topological node in the language environment database and the target feature vector can be calculated using formula (7). The specific expression of formula (7) is as follows:

[0140] (7)

[0141] In formula (7) sim(I) represents the cosine similarity between vectors. q EmbCon i ) represents the i-th topological node V i c The similarity between the corresponding feature vector and the target feature vector; I q Represents the target feature vector; EmbCon i It is the i-th topological node V i c The corresponding feature vector; Ec i n It is the i-th topological node V i c The feature vector corresponding to the nth visual information; F Ci V represents the i-th topological node. i c The feature vector corresponding to the text description; N is the captured topological node V. ic The total amount of visual information.

[0142] D2, for each of the M1 topology nodes, determine multiple topology nodes adjacent to that topology node as the target topology node set corresponding to that topology node.

[0143] Specifically, for topology node 2 among M1 topology nodes, multiple topology nodes corresponding to topology node 2 can be determined based on the topology edges in the target scene graph that include topology node 2; then, the set of multiple topology nodes corresponding to topology node 2 is taken as the target topology node set corresponding to topology node 2.

[0144] D3. For each of the M1 topological nodes, calculate the similarity between the feature vector of the topological node and the feature vector of each target topological node in the target topological node set corresponding to the topological node, and obtain the top M2 topological nodes with the highest similarity to the target feature vector.

[0145] Wherein, M2 is an integer greater than or equal to 1; for example, M2 can be 3.

[0146] D4, determine the target region covering the M1 topological nodes and the M2 topological nodes corresponding to each of the M1 topological nodes, and determine the M3 object nodes covered by the target region; M3 is an integer greater than or equal to 1.

[0147] D5. Calculate the similarity between the feature vector corresponding to each of the M3 object nodes and the target feature vector, and obtain the object node with the highest similarity to the target feature vector.

[0148] For example, the similarity between the feature vector corresponding to the object node and the target feature vector can be calculated using formula (8). The specific expression of formula (8) is as follows:

[0149] (8)

[0150] In formula (8), cosine similarity between vectors is represented; sim(I q EmbObj k ) represents the similarity between the feature vector corresponding to the k-th object node and the target feature vector; I q Represents the target feature vector; Eo k It is the feature vector corresponding to the sub-visual information of the k-th object node; Fo k It is the feature vector corresponding to the text information of the k-th object node.

[0151] D6, the object node with the highest similarity to the target feature vector is taken as the target object.

[0152] The target retrieval method of this application supports a variety of downstream tasks in the field of embodied intelligent environmental perception, including target navigation and specific question answering.

[0153] Taking target navigation as an example, after identifying the target object node, the system performs path planning at the topology layer based on the coordinates of the corresponding topology nodes. A path planning algorithm is used to calculate the shortest path, generating a series of topology coordinates, and thus defining an ideal trajectory. This ideal trajectory minimizes navigation time and distance while considering spatial constraints and inherent obstacles in the scene.

[0154] Taking embodied question-answering as an example, after locating the target object node, the system first invokes the aforementioned target navigation path to guide the agent to autonomously move to the vicinity of the target object. After navigation is completed, the system further retrieves the multimodal semantic information of the target node in the scene knowledge graph (target scene graph), including visual features, spatial location, and associated objects. Using a visual language feature extraction model, the system transforms the user's natural language question into a semantic query, and combines this with environmental perception results to complete multimodal fusion reasoning, ultimately generating a question-answering result that conforms to the semantic context.

[0155] In summary, the target retrieval method disclosed in this application is a robust target retrieval strategy designed based on the target scene graph (also known as a multimodal two-layer topological semantic scene graph) in the aforementioned embodiments. By spatially expanding the topological nodes, the target retrieval performance of the agent in downstream tasks is significantly improved. This strategy fully utilizes multimodal semantic representations, supports user commands in both visual and linguistic modalities, and enables it to complete tasks efficiently and accurately in diverse environments.

[0156] It is understandable that the scene graph construction method and target retrieval method disclosed in this application solve the problem of separating semantic information from spatial structure in traditional scene graphs, significantly enhance the comprehensive cognitive ability of embodied intelligence to the environment, effectively alleviate the local optimum dilemma in the downstream task execution process, and provide solid technical support for the development of embodied intelligence.

[0157] Figure 3 This is a schematic diagram of a scene graph construction device provided in an embodiment of this application. (Combined with...) Figure 3 As shown, the scene graph construction apparatus 300 disclosed in this application includes:

[0158] The data set acquisition module 301 is used to construct a topological node set and a topological edge set based on the scene structure data;

[0159] The first multimodal data acquisition module 302 is used to, for each topological node in the set of topological nodes, take the set of visual information, text information and spatial location information of the topological node as the multimodal data of the topological node;

[0160] The second multimodal data acquisition module 303 is used to determine, for each topological node in the set of topological nodes, the multimodal data of the object node corresponding to the topological node and the set of connecting edges corresponding to the topological node based on the visual information of the topological node; the multimodal data of the object node includes the visual information, text information and spatial location information of the object node; the set of connecting edges is a set composed of the connecting edges between the topological node and the corresponding object node.

[0161] The target scene graph acquisition module 304 is used to construct a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set, and the connection edge set corresponding to each topological node.

[0162] In one alternative implementation, the second multimodal data acquisition module 303 includes:

[0163] The sub-image acquisition unit is used to determine multiple sub-visual information of the topological node based on the visual information of the topological node and the instance segmentation model; the sub-visual information corresponds one-to-one with the object node; each visual information corresponds to multiple sub-visual information.

[0164] The sub-text acquisition unit is used to input the sub-visual information of the topological node into the visual language generation model to determine the text information corresponding to the sub-visual information;

[0165] The object node modal data acquisition unit is used to construct a set of the sub-visual information, the text information corresponding to the sub-visual information, and the spatial position information of the object node corresponding to the sub-visual information for each sub-visual information corresponding to the topological node, and use it as the multimodal data of the object node corresponding to the topological node.

[0166] The edge set acquisition unit is used to connect the topological node with each corresponding object node to obtain multiple edge connections; the set of the multiple edge connections is used as the edge set corresponding to the topological node.

[0167] In one alternative implementation, the second multimodal data acquisition module 303 further includes:

[0168] The first filtering unit is used to calculate, for each topological node and all object nodes corresponding to that topological node, the visual similarity and spatial distance between every two object nodes in turn; the visual similarity is generated based on two sub-visual information corresponding to two object nodes; the spatial distance is generated based on two spatial location information corresponding to two object nodes.

[0169] The second filtering unit is used to, for every two object nodes among all object nodes, if the visual similarity between the two object nodes is greater than a preset first threshold and the spatial distance is less than a preset second threshold, delete one of the two object nodes to obtain multiple object nodes corresponding to the topological node.

[0170] In one alternative implementation, the data set acquisition module includes:

[0171] The skeleton diagram determination unit is used to construct a skeleton diagram based on the scene structure data;

[0172] The topology node set acquisition unit is used to construct the topology node set based on multiple candidate topology nodes that meet the first screening condition selected from multiple candidate topology nodes corresponding to the skeleton graph; the first screening condition indicates that the node spacing between two candidate topology nodes is equal to a preset third threshold.

[0173] A fully connected graph determination unit is used to construct a fully connected graph based on the skeleton graph;

[0174] The topology edge set acquisition unit is used to construct the topology edge set based on multiple candidate connection paths that meet the second screening condition selected from multiple candidate connection paths in the fully connected graph; the second screening condition indicates that the connection distance of the candidate connection path is less than a preset fourth threshold and the candidate connection path does not cross an obstacle area.

[0175] Figure 4 This is a schematic diagram of a target retrieval device provided in an embodiment of this application. (In conjunction with...) Figure 4 As shown, the target retrieval device 400 disclosed in this application includes:

[0176] The database acquisition module 401 is used to acquire a semantic environment database; the semantic environment database includes feature vectors corresponding to the multimodal data of each topological node in the target scene graph, and feature vectors corresponding to the multimodal data of each object node.

[0177] The target feature vector determination module 402 is used to input the query object into the visual language feature extraction model to obtain the target feature vector; the query object includes the query image and / or the query text;

[0178] The target object query module 403 is used to determine the object node with the highest similarity to the target feature vector from the semantic environment database, and use it as the target object.

[0179] In one alternative implementation, the target object query module 403 includes:

[0180] The first topology node query unit is used to calculate the similarity between the feature vector corresponding to each topology node in the semantic environment database and the target feature vector, and to obtain the top M1 topology nodes with the highest similarity to the target feature vector; where M1 is an integer greater than or equal to 1.

[0181] The target topology node set determination unit is used to determine, for each of the M1 topology nodes, multiple topology nodes adjacent to that topology node as the target topology node set corresponding to that topology node.

[0182] The second topology node query unit is used to calculate the similarity between the feature vector of each topology node and the feature vector of each target topology node in the target topology node set corresponding to the topology node for each topology node among the M1 topology nodes, and obtain the top M2 topology nodes with the highest similarity to the target feature vector; where M2 is an integer greater than or equal to 1.

[0183] The target region determination unit is used to determine a target region covering the M1 topological nodes and the M2 topological nodes corresponding to each of the M1 topological nodes, and to determine the M3 object nodes covered by the target region; where M3 is an integer greater than or equal to 1.

[0184] The object node query unit is used to calculate the similarity between the feature vector corresponding to each of the M3 object nodes and the target feature vector, and to obtain the object node with the highest similarity to the target feature vector.

[0185] The target object determination unit is used to identify the object node with the highest similarity to the target feature vector as the target object.

[0186] In one alternative implementation, the database retrieval module 401 includes:

[0187] The first feature vector acquisition unit is used to input the multimodal data corresponding to each topological node in the target scene graph into the visual language feature extraction model to obtain the feature vector corresponding to each topological node;

[0188] The second feature vector acquisition unit is used to input the multimodal data corresponding to each object node in the target scene graph into the visual language feature extraction model to obtain the feature vector corresponding to each object node.

[0189] The feature vector pool construction unit is used to construct a set of feature vectors corresponding to each topological node and feature vectors corresponding to each object node, which serves as the semantic environment database.

[0190] Based on the scene graph construction method, target retrieval method, and apparatus provided in the foregoing embodiments, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the scene graph construction method and target retrieval method mentioned above.

[0191] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0192] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing a scene graph, characterized in that, The method includes: Based on the scene structure data, construct a set of topological nodes and a set of topological edges; For each topological node in the set of topological nodes, the set of visual information, text information and spatial location information of the topological node is used as the multimodal data of the topological node; Based on the visual information and instance segmentation model of the topological node, multiple sub-visual information of the topological node are determined; each sub-visual information corresponds one-to-one with the object node; each visual information corresponds to multiple sub-visual information. The sub-visual information of the topological node is input into the visual language generation model to determine the text information corresponding to the sub-visual information; For each sub-visual information corresponding to the topological node, a set is constructed from the sub-visual information, the text information corresponding to the sub-visual information, and the spatial location information of the object node corresponding to the sub-visual information, which is used as the multimodal data of the object node corresponding to the topological node; the multimodal data of the object node includes the visual information, text information, and spatial location information of the object node. Connect the topological node to each corresponding object node to obtain multiple connecting edges; the set of these multiple connecting edges is taken as the connecting edge set corresponding to the topological node; the connecting edge set is the set of connecting edges between the topological node and its corresponding object node. A target scene graph is constructed based on the multimodal data of each topological node, the multimodal data of each object node, the set of topological edges, and the set of connection edges corresponding to each topological node.

2. The method according to claim 1, characterized in that, The method further includes: For each topological node, the visual similarity and spatial distance between any two object nodes are calculated sequentially. The visual similarity is generated based on the two sub-visual information corresponding to the two object nodes. The spatial distance is generated based on the two spatial position information corresponding to the two object nodes. For every two object nodes among all the object nodes, if the visual similarity between the two object nodes is greater than a preset first threshold and the spatial distance is less than a preset second threshold, then one of the two object nodes is deleted, resulting in multiple object nodes corresponding to the topology node.

3. The method according to claim 1, characterized in that, The construction of the topology node set and topology edge set based on scene structure data includes: Based on the scene structure data, a skeleton diagram is constructed; Based on multiple candidate topology nodes that meet the first screening condition selected from multiple candidate topology nodes corresponding to the skeleton graph, the topology node set is constructed; the first screening condition indicates that the node spacing between two candidate topology nodes is equal to a preset third threshold. Based on the skeleton graph, construct a fully connected graph; The topological edge set is constructed based on multiple candidate connection paths that meet the second screening condition selected from multiple candidate connection paths in the fully connected graph; the second screening condition indicates that the connection distance of the candidate connection path is less than a preset fourth threshold and the candidate connection path does not cross an obstacle area.

4. A target retrieval method, characterized in that, The method includes: A semantic environment database is obtained; the semantic environment database is generated based on the target scene graph according to any one of claims 1-3; the semantic environment database includes the feature vector corresponding to the multimodal data of each topological node in the target scene graph, and the feature vector corresponding to the multimodal data of each object node; The object to be queried is input into the visual language feature extraction model to obtain the target feature vector; the object to be queried includes the image to be queried and / or the text to be queried. The object node with the highest similarity to the target feature vector is determined from the semantic environment database and is taken as the target object.

5. The method according to claim 4, characterized in that, The step of determining the object node with the highest similarity to the target feature vector from the semantic environment database as the target object includes: Calculate the similarity between the feature vector corresponding to each topological node in the semantic environment database and the target feature vector, and obtain the top M1 topological nodes with the highest similarity to the target feature vector; where M1 is an integer greater than or equal to 1. For each of the M1 topology nodes, determine the multiple topology nodes adjacent to that topology node, and use them as the target topology node set corresponding to that topology node; For each of the M1 topological nodes, calculate the similarity between the feature vector of the topological node and the feature vector of each target topological node in the target topological node set corresponding to the topological node, and obtain the top M2 topological nodes with the highest similarity to the target feature vector; where M2 is an integer greater than or equal to 1. Determine the target region covering the M1 topological nodes and the M2 topological nodes corresponding to each of the M1 topological nodes, and determine the M3 object nodes covered by the target region; where M3 is an integer greater than or equal to 1. Calculate the similarity between the feature vector corresponding to each of the M3 object nodes and the target feature vector, and obtain the object node with the highest similarity to the target feature vector; The object node with the highest similarity to the target feature vector is taken as the target object.

6. The method according to claim 4, characterized in that, The acquisition of the semantic environment database includes: The multimodal data corresponding to each topological node in the target scene graph is input into the visual language feature extraction model to obtain the feature vector corresponding to each topological node; The multimodal data corresponding to each object node in the target scene graph is input into the visual language feature extraction model to obtain the feature vector corresponding to each object node; The set constructed from the feature vectors corresponding to each topological node and the feature vectors corresponding to each object node is used as the semantic environment database.

7. A scene graph construction device, characterized in that, The scene graph construction apparatus is applied to the method of claim 1, and the apparatus includes: The data set acquisition module is used to construct a set of topological nodes and a set of topological edges based on the scene structure data; The first multimodal data acquisition module is used to, for each topological node in the set of topological nodes, take the set of visual information, text information and spatial location information of the topological node as the multimodal data of the topological node; The second multimodal data acquisition module is used to determine multiple sub-visual information of the topological node based on the visual information and instance segmentation model of the topological node; the sub-visual information corresponds one-to-one with the object node; each visual information corresponds to multiple sub-visual information; the sub-visual information of the topological node is input into a visual language generation model to determine the text information corresponding to the sub-visual information; for each sub-visual information corresponding to the topological node, a set constructed from the sub-visual information, the corresponding text information, and the spatial location information of the object node corresponding to the sub-visual information is used as the multimodal data of the object node corresponding to the topological node; the topological node is connected to each corresponding object node to obtain multiple connection edges; the set of the multiple connection edges is used as the connection edge set corresponding to the topological node; the multimodal data of the object node includes the visual information, text information, and spatial location information of the object node; the connection edge set is the set composed of the connection edges between the topological node and the corresponding object node. The target scene graph acquisition module is used to construct a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the set of topological edges, and the set of connecting edges corresponding to each topological node.

8. A target retrieval device, characterized in that, The target detection device is applied to the method of claim 4, and the device comprises: A database acquisition module is used to acquire a semantic environment database; the semantic environment database is generated based on the target scene graph according to any one of claims 1-3; the semantic environment database includes feature vectors corresponding to the multimodal data of each topological node in the target scene graph, and feature vectors corresponding to the multimodal data of each object node; The target feature vector determination module is used to input the query object into the visual language feature extraction model to obtain the target feature vector; the query object includes the query image and / or the query text. The target object query module is used to determine the object node with the highest similarity to the target feature vector from the semantic environment database, and use it as the target object.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the scene graph construction method according to any one of claims 1-3 and the target retrieval method according to any one of claims 4-6.

Images