Scene graph construction method, target retrieval method and related devices

Abstract

The invention discloses a scene graph construction method, a target retrieval method and a related device. The scene graph construction method comprises the following steps: constructing a topological node set and a topological edge set; for each topological node in the topological node set, taking a set of visual information, text information and spatial position information of the topological node as multi-modal data of the topological node; for each topological node in the topological node set, determining multi-modal data of an object node corresponding to the topological node and a connection edge set corresponding to the topological node based on the visual information of the topological node; and constructing a target scene graph based on the multi-modal data of each topological node, the multi-modal data of each object node, the topological edge set and the connection edge set corresponding to each topological node. The target scene graph has semantic consistency and spatial continuity, and the expression ability and cross-scene generalization performance of the scene graph in a complex environment are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of embodied intelligent environmental perception, and in particular to a scene graph construction method, a target retrieval method, and related devices. Background Art

[0002] In the specific intelligent field, scene graphs serve as information carriers for expressing the structure of indoor environments, providing effective support for downstream tasks such as object positioning, embodied question answering, and target navigation, enabling intelligent agents to make reasonable decisions based on multimodal perceptual input and contextual clues in the absence of clear task experience.

[0003] However, the commonly used scene graphs in traditional solutions suffer from various problems. For example, room nodes in a tree-structured scene graph with a "building-room-object" structure are often assigned abstract semantic labels and aggregated with a large number of object nodes. This results in complex and confusing semantic features at the room level, weakening salient features and expressive power. Edges between object nodes often lack explicit modeling of spatial relationships such as proximity and visibility. Individual object nodes lack effective contextual interaction mechanisms, resulting in semantic confusion and hierarchical disconnection in the tree-structured scene graph.

[0004] For example, scene graphs in traditional solutions usually use independently constructed topological graphs that lack semantic associations to represent the spatial accessibility of the environment. As a result, the navigable path points in the scene graph cannot provide the complete semantics of the surrounding environment, limiting the agent's ability to understand local details. If the observation information only contains semantic information but lacks topological structure, the agent cannot capture the complete spatial connectivity, which reduces the agent's spatial reasoning and planning capabilities, causing the agent to fall into problems such as regional closure or action failure, making it difficult to smoothly advance the task process.

[0005] How to construct a scene graph that can collaboratively express the semantics and topological structures in complex scenes has become one of the technical problems that need to be solved urgently in the field of embodied intelligent environmental perception. Summary of the Invention

[0006] Based on the above problems, this application provides a scene graph construction method, which constructs a scene graph that can realize the collaborative expression of semantics and topology in complex scenes by integrating the semantic context relationship and spatial accessibility information of the environment into the graph structure.

[0007] The embodiments of this application disclose the following technical solutions: The first aspect of the present application discloses a scene graph construction method, comprising: Based on the scene structure data, construct a topological node set and a topological edge set; For each topological node in the topological node set, a set of visual information, text information and spatial position information of the topological node is used as multimodal data of the topological node; For each topological node in the topological node set, based on the visual information of the topological node, determining the multimodal data of the object node corresponding to the topological node and the connection edge set corresponding to the topological node; the multimodal data of the object node includes the visual information, text information and spatial position information of the object node; the connection edge set is a set consisting of the connection edges between the topological node and the corresponding object node; A target scene graph is constructed based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set, and the connection edge set corresponding to each topological node.

[0008] In an optional implementation, for each topological node in the topological node set, determining, based on visual information of the topological node, multimodal data of an object node corresponding to the topological node and a set of connecting edges corresponding to the topological node includes: Based on the visual information of the topological node and the instance segmentation model, a plurality of sub-visual information of the topological node is determined; the sub-visual information corresponds to the object node one by one; each visual information corresponds to a plurality of sub-visual information; Inputting the sub-visual information of the topological node into a visual language generation model to determine text information corresponding to the sub-visual information; For each sub-visual information corresponding to the topological node, a set of the sub-visual information, the text information corresponding to the sub-visual information, and the spatial position information of the object node corresponding to the sub-visual information is constructed as the multimodal data of the object node corresponding to the topological node; The topological node is connected to each corresponding object node to obtain a plurality of connection edges; and a set consisting of the plurality of connection edges is used as a connection edge set corresponding to the topological node.

[0009] In an optional implementation, the method further includes: For all object nodes corresponding to each topological node, sequentially calculating the visual similarity and spatial distance between every two object nodes in all object nodes corresponding to the topological node; the visual similarity is generated based on the two sub-visual information corresponding to the two object nodes; the spatial distance is generated based on the two spatial position information corresponding to the two object nodes; For every two object nodes among all the object nodes, if the visual similarity between the two object nodes is greater than a preset first threshold and the spatial distance is less than a preset second threshold, one of the two object nodes is deleted to obtain multiple object nodes corresponding to the topological node.

[0010] In an optional implementation, constructing a topological node set and a topological edge set based on scene structure data includes: Constructing a skeleton graph based on the scene structure data; Constructing the topological node set based on a plurality of candidate topological nodes that meet a first screening condition selected from a plurality of candidate topological nodes corresponding to the skeleton graph; wherein the first screening condition indicates that a node distance between two candidate topological nodes is equal to a preset third threshold; Based on the skeleton graph, construct a fully connected graph; The topological edge set is constructed based on multiple candidate connection paths that meet a second filtering condition and are screened out from multiple candidate connection paths in the fully connected graph; the second filtering condition indicates that the connection distance of the candidate connection path is less than a preset fourth threshold and the candidate connection path does not pass through an obstacle area.

[0011] The second aspect of the present application discloses a target retrieval method, comprising: Obtaining a semantic environment database; the semantic environment database is generated based on the target scene graph according to any one of the first aspects; the semantic environment database includes a feature vector corresponding to the multimodal data corresponding to each topological node in the target scene graph, and a feature vector corresponding to the multimodal data of each object node; Inputting the object to be queried into the visual language feature extraction model to obtain a target feature vector; the object to be queried includes an image to be queried and / or a text to be queried; An object node having the greatest similarity to the target feature vector is determined from the semantic environment database as the target object.

[0012] In an optional implementation, determining, from the semantic environment database, an object node having the greatest similarity to the target feature vector as the target object includes: Calculating the similarity between the feature vector corresponding to each topological node in the semantic environment database and the target feature vector, and obtaining the top M1 topological nodes with the highest similarity to the target feature vector; wherein M1 is an integer greater than or equal to 1; For each topological node among the M1 topological nodes, determining a plurality of topological nodes adjacent to the topological node as a target topological node set corresponding to the topological node; For each of the M1 topological nodes, calculate the similarity between the feature vector of the topological node and the feature vector of each target topological node in the target topological node set corresponding to the topological node, and obtain the top M2 topological nodes with the highest similarity to the target feature vector; M2 is an integer greater than or equal to 1; Determine a target area covering the M1 topological nodes and the M2 topological nodes corresponding to each topological node in the M1 topological nodes, and determine M3 object nodes covered by the target area; wherein M3 is an integer greater than or equal to 1; Calculating the similarity between the feature vector corresponding to each object node in the M3 object nodes and the target feature vector, and obtaining the object node with the highest similarity to the target feature vector; The object node with the highest similarity to the target feature vector is used as the target object.

[0013] In an optional implementation, obtaining the semantic environment database includes: Inputting the multimodal data corresponding to each topological node in the target scene graph into the visual language feature extraction model to obtain a feature vector corresponding to each topological node; Inputting the multimodal data corresponding to each object node in the target scene graph into the visual language feature extraction model to obtain a feature vector corresponding to each object node; A set constructed by the feature vector corresponding to each topological node and the feature vector corresponding to each object node is used as the semantic environment database.

[0014] A third aspect of the present application provides a scene graph construction device, comprising: The data set acquisition module is used to construct a topological node set and a topological edge set based on the scene structure data; a first multimodal data acquisition module, configured to, for each topological node in the set of topological nodes, take a set of visual information, textual information, and spatial position information of the topological node as multimodal data of the topological node; a second multimodal data acquisition module configured to determine, for each topological node in the topological node set, multimodal data of an object node corresponding to the topological node and a set of connection edges corresponding to the topological node based on visual information of the topological node; the multimodal data of the object node including visual information, textual information, and spatial location information of the object node; and the connection edge set consisting of connection edges between the topological node and the corresponding object node; The target scene graph acquisition module is used to construct a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set and the connection edge set corresponding to each topological node.

[0015] A fourth aspect of the present application provides a target retrieval device, comprising: A database acquisition module, configured to acquire a semantic environment database; the semantic environment database is generated based on the target scene graph according to any one of the first aspects; the semantic environment database includes a feature vector corresponding to the multimodal data corresponding to each topological node in the target scene graph, and a feature vector corresponding to the multimodal data of each object node; A target feature vector determination module is configured to input the object to be queried into the visual language feature extraction model to obtain a target feature vector; the object to be queried includes an image to be queried and / or a text to be queried; The target object query module is used to determine the object node with the greatest similarity to the target feature vector from the semantic environment database as the target object.

[0016] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspect and any one of the second aspect.

[0017] Compared with the existing technology, this application has the following beneficial effects: The present application discloses a scene graph construction method, comprising: constructing a topological node set and a topological edge set based on scene structure data; for each topological node in the topological node set, using the set of visual information, textual information and spatial position information of the topological node as the multimodal data of the topological node; for each topological node in the topological node set, determining the multimodal data of the object node corresponding to the topological node and the connection edge set corresponding to the topological node based on the visual information of the topological node; and constructing a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set and the connection edge set corresponding to each topological node.

[0018] In this way, the target scene graph of the present application includes multimodal data such as visual information, text information and spatial position information of topological nodes, as well as multimodal data such as visual information, text information and spatial position information of object nodes; at the same time, a spatial continuity relationship between object nodes and topological nodes is established; the target scene graph has semantic consistency and spatial continuity, which can effectively solve the problem of disconnection between semantics and spatial structure in traditional scene graphs, significantly improve the expressive ability of scene graphs in complex environments and cross-scene generalization performance, and provide a more complete and coherent environmental understanding basis for downstream tasks such as scene understanding, object positioning, and target navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A flowchart of a scene graph construction method provided in an embodiment of the present application; Figure 2 A flowchart of a target retrieval method provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a scene graph construction device provided in an embodiment of the present application; Figure 4 A schematic structural diagram of a target retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In the specific intelligent field, scene graphs serve as information carriers for expressing the structure of indoor environments, providing effective support for downstream tasks such as object positioning, embodied question answering, and target navigation, enabling intelligent agents to make reasonable decisions based on multimodal perceptual input and contextual clues in the absence of clear task experience.

[0022] However, the commonly used scene graphs in traditional solutions suffer from various problems. For example, room nodes in a tree-structured scene graph with a "building-room-object" structure are often assigned abstract semantic labels and aggregated with a large number of object nodes. This results in complex and confusing semantic features at the room level, weakening salient features and expressive power. Edges between object nodes often lack explicit modeling of spatial relationships such as proximity and visibility. Individual object nodes lack effective contextual interaction mechanisms, resulting in semantic confusion and hierarchical disconnection in the tree-structured scene graph.

[0023] For example, scene graphs in traditional solutions usually use independently constructed topological graphs that lack semantic associations to represent the spatial accessibility of the environment. As a result, the navigable path points in the scene graph cannot provide the complete semantics of the surrounding environment, limiting the agent's ability to understand local details. If the observation information only contains semantic information but lacks topological structure, the agent cannot capture the complete spatial connectivity, which reduces the agent's spatial reasoning and planning capabilities, causing the agent to fall into problems such as regional closure or action failure, making it difficult to smoothly advance the task process.

[0024] How to construct a scene graph that can collaboratively express the semantics and topological structures in complex scenes has become one of the technical problems that need to be solved urgently in the field of embodied intelligent environmental perception.

[0025] Based on the above problems, the present application discloses a scene graph construction method, including: constructing a topological node set and a topological edge set based on scene structure data; for each topological node in the topological node set, using the set of visual information, text information and spatial position information of the topological node as the multimodal data of the topological node; for each topological node in the topological node set, determining the multimodal data of the object node corresponding to the topological node and the connection edge set corresponding to the topological node based on the visual information of the topological node; constructing a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set and the connection edge set corresponding to each topological node.

[0026] In this way, the target scene graph of the present application includes multimodal data such as visual information, text information and spatial position information of topological nodes, as well as multimodal data such as visual information, text information and spatial position information of object nodes; at the same time, a spatial continuity relationship between object nodes and topological nodes is established; the target scene graph has semantic consistency and spatial continuity, which can effectively solve the problem of disconnection between semantics and spatial structure in traditional scene graphs, significantly improve the expressive ability of scene graphs in complex environments and cross-scene generalization performance, and provide a more complete and coherent environmental understanding basis for downstream tasks such as scene understanding, object positioning, and target navigation.

[0027] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0028] Figure 1 A flowchart of a scene graph construction method provided in an embodiment of the present application. Figure 1 As shown, the scene graph construction method disclosed in this application includes: S101, constructing a topological node set and a topological edge set based on scene structure data.

[0029] Obtain scene structure data for a target given scene. The target given scene may be a home living room, a hospital ward, or a school classroom, among other scenes. This application does not limit the specific scope of the target given scene, nor does it limit the specific method for obtaining the scene structure data.

[0030] The scene structure data can be three-dimensional point cloud data or geometric structure representation data.

[0031] In the embodiment of the present application, the scene structure data is three-dimensional point cloud data as an example to introduce the scene graph construction method disclosed in the present application.

[0032] After acquiring the 3D point cloud data, the camera pose information collected during the 3D point cloud data reconstruction process is used to generate a pose area map. The ground area map is obtained by projecting the 3D point cloud data onto a bird's-eye view plane. The obstacle area map is generated by projecting the 3D point cloud data within a specified height range.

[0033] After obtaining the pose area map, ground area map, and obstacle area map, the free area map can be obtained by formula (1). The specific expression of formula (1) is: (1) In formula (1), M is the free area graph, M P is the pose area map, M f is the ground area map and M o It is the obstacle area map.

[0034] After obtaining the free area map M, the skeleton of the free area M can be extracted using the thinning algorithm to obtain the skeleton map M. s The refinement algorithm emphasizes the preservation of the center of the free area, which can effectively suppress the skeleton point redundancy problem caused by edge alignment, thereby accurately preserving the core skeleton structure of the scene.

[0035] After obtaining the skeleton graph M s Then, the skeleton graph M s Sampling is performed to obtain all candidate topological nodes on the skeleton path; multiple candidate topological nodes that meet the first screening condition, that is, the node distance between two candidate topological nodes is equal to the preset third threshold, are screened from all candidate topological nodes; the set consisting of multiple candidate topological nodes that meet the first screening condition is used as the topological node set. The topological nodes in the topological node set have sparse spatial distribution; the expression of the topological node set is shown in formula (2): (2) V in formula (2) c Represents a set of topological nodes; V i c Represents the i-th topological node in the topological node set; Dist(V i c , V j c ) represents the node distance between the i-th topological node and the j-th topological node in the topological node set; τ is the preset third threshold; M s It is the skeleton diagram mentioned in the aforementioned embodiment of this application.

[0036] In the skeleton graph M s Sampling is performed to obtain the skeleton graph M s After finding all candidate topological nodes, connect each pair of different candidate topological nodes in all candidate topological nodes so that there is exactly one edge connecting each pair of different candidate topological nodes, thus obtaining a completely connected graph. In this application, the edge connecting each pair of different candidate topological nodes in the completely connected graph is called a candidate connection path.

[0037] After obtaining the fully connected graph, multiple candidate connection paths in the fully connected graph can be screened to select multiple candidate connection paths that meet the second screening condition. Specifically, candidate connection paths whose lengths (also known as connection distances) are less than a preset fourth threshold and do not pass through any obstacle areas can be screened from the fully connected graph. The set of candidate connection paths that meet the second screening condition is then called a topological edge set. A sparsely structured topological edge set can represent valid and traversable spatial connectivity between topological nodes.

[0038] It can be understood that the second screening condition of the present application indicates that the connection distance of the candidate connection path is less than a preset fourth threshold, and the candidate connection path does not pass through an obstacle area.

[0039] It can be understood that each topological edge in the topological edge set has two topological nodes as endpoints.

[0040] The set of topological nodes and edges obtained in this step together constitute the topological layer. As the foundational component of the two-layer multimodal topological-object scene graph, the topological layer effectively depicts the navigable spatial topology within the free area of ​​a given target scene, providing structural support for downstream tasks.

[0041] S102 : For each topological node in the topological node set, a set of visual information, text information, and spatial position information of the topological node is used as multimodal data of the topological node.

[0042] The visual information of a topological node may be an image corresponding to the topological node; the text information may be text used to describe the characteristics of the topological node (such as text describing the image corresponding to the topological node); and the spatial position information may be the coordinates of the topological node in three-dimensional space.

[0043] In an optional implementation, after obtaining visual information of a topological node, such as multiple visual images of the topological node, the multiple visual images of the topological node can be input into a visual language generation model to obtain text information corresponding to each visual image.

[0044] The visual language generation model in this step is a model that can convert images into text information, such as Tongyi Qianwen or LLaVA.

[0045] Next, the process of this step will be introduced again by taking the case where the visual information is a visual image as an example.

[0046] Exemplarily, for a topological node 1 in a topological node set, first obtain P1 perspective perception images (P1≥1) corresponding to the spatial position of the topological node 1; then input each of the P1 perspective perception images into the visual language generation model to generate text information corresponding to each perspective perception image, that is, obtain P1 text information; then take the set consisting of the P1 text information as the text information corresponding to the topological node 1; finally, take the set consisting of the P1 perspective perception images corresponding to the topological node 1 and the text information corresponding to the topological node 1 as the multimodal data of the topological node 1.

[0047] The expression of multimodal data of topological nodes in this application is shown in formula (3): (3) AttrCon in formula (3) i Represents the i-th topological node V in the topological node set i c Multimodal data of Ic i n Represents the i-th topological node V i c The nth visual information of T Ci Represents the i-th topological node V i c Text information, N is the captured i-th topological node V i c The total amount of visual information.

[0048] S103 : For each topological node in the topological node set, based on visual information corresponding to the topological node, determine multimodal data of the object node corresponding to the topological node and a connection edge set corresponding to the topological node.

[0049] It is understandable that a topological node may correspond to multiple object nodes. In this application, the multimodal data of each object node includes: the visual information corresponding to the object node, the spatial position information of the object node, and the text information of the object node.

[0050] In an optional implementation, the process of obtaining multimodal data of each object node corresponding to each topological node in the topological node set includes the following steps: A1, obtain multiple visual information corresponding to each topological node in the topological node set.

[0051] For example, obtain P1 visual images of topological node 1 in the topological node set.

[0052] A2: for each topological node in the topological node set, based on the visual information of the topological node and the instance segmentation model, determine multiple sub-visual information of the topological node.

[0053] Among them, the open vocabulary detection and segmentation model is used to process visual information, generate bounding boxes and segmentation masks for each object contained in the visual information, and segment the visual information into multiple sub-visual information models; the sub-visual information corresponds one-to-one to the object nodes, that is, there is an object instance in one sub-visual information, corresponding to one object node.

[0054] It can be understood that if there are two object instances in one visual information of a topological node, then the visual information corresponds to two sub-visual information; each sub-visual information corresponds to one object instance.

[0055] Assuming that the visual information is a visual image, each perspective image in the P1 perspective perception image of topological node 1 is input into the instance segmentation model to obtain multiple sub-perspective images corresponding to each visual image.

[0056] A3: Input each sub-visual information corresponding to each topological node in the topological node set into the visual language generation model to obtain a text description corresponding to each sub-visual information corresponding to each topological node.

[0057] A4, for each sub-visual information corresponding to each topological node in the topological node set, a set constructed by the sub-visual information, the text information corresponding to the sub-visual information, and the spatial position information of the object node corresponding to the sub-visual information is used as the multimodal data of the object node corresponding to the topological node.

[0058] Assuming that the visual information is a visual image, if the two sub-visual information corresponding to the topological node 1 in the topological node set are sub-visual image 1 and sub-visual image 2; the set consisting of sub-visual image 1, the text information corresponding to sub-visual image 1, and the spatial position information of the object node corresponding to sub-visual image 1, as well as sub-visual image 2, the text information corresponding to sub-visual image 2, and the spatial position information of the object node corresponding to sub-visual information 2, is used as the multimodal data of the object node corresponding to the topological node 1.

[0059] The expression of multimodal data of object nodes in this application is shown in formula (4): (4) In formula (4), AttrObjk Represents the multimodal data of object node k; o k Image representation (visual information) representing object node k; To k Represents the text description of object node k.

[0060] Among them, the three-dimensional coordinates of the object node k in the world coordinate system are integrated in To k When the multimodal data of an object node includes the spatial position information of the object node in the world coordinate system, the multimodal data of the object node will further enhance the spatial semantic information of the object instance.

[0061] As mentioned above, the multiple visual information of a topological node is captured from the spatial location of the topological node. Sometimes, some of the multiple visual information corresponding to a topological node describes the same object. In this case, the object nodes corresponding to a topological node obtained using methods A1-A4 may contain duplicate object nodes.

[0062] In order to solve the problem of duplicate object instances in different perspectives under the same topological node, after obtaining the multimodal data of the object node corresponding to each topological node, the method described in B1-B2 can be used to deduplicate multiple object nodes corresponding to the same topological node. Specifically: B1, for all object nodes corresponding to each topological node in the topological node set, sequentially calculate the visual similarity and spatial distance between every two object nodes in all object nodes corresponding to the topological node.

[0063] The visual similarity is generated based on the two sub-visual information corresponding to the two object nodes. For example, the two sub-visual information can be input into the SuperGlue model to obtain the visual similarity of the two sub-visual information. It is understandable that the SuperGlue model is just an example.

[0064] The spatial distance is generated based on two spatial position information corresponding to two object nodes. For example, the spatial distance of the corresponding object instances can be calculated using the position data in the three-dimensional coordinates of the object nodes.

[0065] B2: For every two object nodes among all object nodes, if the visual similarity between the two object nodes is greater than a preset first threshold and the spatial distance is less than a preset second threshold, delete one of the two object nodes to obtain multiple object nodes corresponding to the topological node.

[0066] Alternatively, for two object nodes among all object nodes, when the visual similarity between the two object nodes is greater than a preset first threshold and the spatial distance is less than a preset second threshold, it is determined that the two object nodes represent an object instance, and the description with the highest semantic category confidence is selected as the unified identifier of the object instance.

[0067] The topological node is connected to each corresponding object node to obtain a plurality of connection edges; and a set consisting of the plurality of connection edges is used as a connection edge set corresponding to the topological node.

[0068] S104: Construct a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set, and the connection edge set corresponding to each topological node.

[0069] It is understandable that since the target scene graph is composed of topological nodes, object nodes and topological edge sets, and integrates multimodal data, it can reflect the structure, semantics and dynamic changes of the scene, and has rich connectivity, data correlation, semantic expression capabilities and scalability, providing a comprehensive and flexible framework for the modeling and analysis of complex scenes.

[0070] In summary, the scene graph construction method disclosed in the present application, by jointly modeling topological nodes and object instances, constructs a multimodal semantic representation of topological nodes and a multimodal semantic representation of object nodes, thereby achieving an effective combination of the spatial connectivity reflected by topological information and the scene semantics expressed by the semantic attributes of the nodes. The scene graph representation generated by this method not only has semantic consistency and spatial continuity, but also effectively solves the problem of disconnection between semantics and spatial structure in traditional scene graphs. Through the deep fusion of the semantic layer and the topological layer, the expressive power of the scene graph in complex environments and the generalization performance across scenes are significantly improved, providing a more complete and coherent environmental understanding foundation for downstream tasks such as scene understanding, object positioning, and target navigation.

[0071] Furthermore, this application also proposes a target retrieval method. Figure 2 A flow chart of a target retrieval method provided in an embodiment of the present application. Figure 2 As shown, the target retrieval method disclosed in this application includes: S201, obtaining a semantic environment database.

[0072] The semantic environment database in this application is a database generated based on the target scene graph obtained in the aforementioned embodiments. The semantic environment database includes feature vectors corresponding to the multimodal data corresponding to each topological node in the target scene graph, as well as feature vectors corresponding to the multimodal data of each object node; that is, the essence of the semantic environment database in this application is a pool of feature vectors corresponding to the target scene graph.

[0073] In an optional implementation, the steps of obtaining the semantic environment database are: First, the multimodal data corresponding to each topological node in the target scene graph is input into the visual feature extraction language model to obtain the feature vector corresponding to each topological node; then the multimodal data corresponding to each object node in the target scene graph is input into the visual language feature extraction model to obtain the feature vector corresponding to each object node; finally, the feature vector corresponding to each topological node and the feature vector corresponding to each object node are constructed as a set as the semantic environment database.

[0074] It can be understood that the visual language feature extraction model in this step encodes visual information and text descriptions and converts them into corresponding feature vectors; such as CLIP (Contrastive Language-Image Pre-training) or SigLIP (Sigmoid Loss for Language Image Pre-Training).

[0075] The expression of the feature vector corresponding to the topological node in this application is shown in formula (5): (5) EmbCon in formula (5) i is the i-th topological node V i c The corresponding eigenvector; Ec i n is the i-th topological node V i c The feature vector corresponding to the nth visual information; F Ci Represents the i-th topological node V i c The feature vector corresponding to the text information; N is the captured i-th topological node V i c The total amount of visual information.

[0076] The expression of the feature vector corresponding to the object node in this application is shown in formula (6): (6) EmbObj in formula (6) k is the eigenvector corresponding to the kth object node; Eo k is the feature vector corresponding to the sub-visual information of the k-th object node; Fo k It is the feature vector corresponding to the text information of the k-th object node.

[0077] S202: Input the object to be queried into the visual language feature extraction model to obtain a target feature vector.

[0078] The query in this application corresponds to an image to be queried and / or a text to be queried; that is, the object to be queried can be an image, a text, or a combination of the two.

[0079] The object to be queried is input into the visual language feature extraction model, and the object to be queried is processed by the visual language feature extraction model to obtain a target feature vector.

[0080] S203 : Determine, from the semantic environment database, an object node having the greatest similarity to the target feature vector as a target object.

[0081] The process includes the following steps: D1, calculate the similarity between the feature vector corresponding to each topological node in the semantic environment database and the target feature vector, and obtain the top M1 topological nodes with the highest similarity to the target feature vector.

[0082] Wherein, M1 is an integer greater than or equal to 1; illustratively, M1 may be 3.

[0083] For example, the similarity between the feature vector corresponding to each topological node in the language environment database and the target feature vector can be calculated by formula (7). The specific expression of formula (7) is: (7) In formula (7) Represents the cosine similarity between vectors, sim(I q ,EmbCon i ) represents the i-th topological node V i c The similarity between the corresponding feature vector and the target feature vector; I q Represents the target feature vector; EmbCon i is the i-th topological node V i c The corresponding eigenvector; Ec i n is the i-th topological node V i c The feature vector corresponding to the nth visual information; F Ci Represents the i-th topological node V i c The feature vector corresponding to the text description; N is the captured i-th topological node V i c The total amount of visual information.

[0084] D2: For each topological node in the M1 topological nodes, determine a plurality of topological nodes adjacent to the topological node as a target topological node set corresponding to the topological node.

[0085] Specifically, for topological node 2 among M1 topological nodes, multiple topological nodes corresponding to topological node 2 can be determined based on the topological edges including topological node 2 in the target scene graph; then the set consisting of multiple topological nodes corresponding to topological node 2 is used as the target topological node set corresponding to topological node 2.

[0086] D3, for each topological node in the M1 topological nodes, calculate the similarity between the feature vector of the topological node and the feature vector of each target topological node in the target topological node set corresponding to the topological node, and obtain the top M2 topological nodes with the highest similarity to the target feature vector.

[0087] Wherein, M2 is an integer greater than or equal to 1; illustratively, M2 may be 3.

[0088] D4, determining a target area covering the M1 topological nodes and the M2 topological nodes corresponding to each topological node in the M1 topological nodes, and determining M3 object nodes covered by the target area; M3 is an integer greater than or equal to 1.

[0089] D5, calculating the similarity between the feature vector corresponding to each object node in the M3 object nodes and the target feature vector, and obtaining the object node with the highest similarity to the target feature vector.

[0090] For example, the similarity between the feature vector corresponding to the object node and the target feature vector can be calculated by formula (8). The specific expression of formula (8) is: (8) Formula (8) represents the cosine similarity between vectors; sim(I q , EmbObj k ) is the similarity between the feature vector corresponding to the k-th object node and the target feature vector; I q represents the target feature vector; Eo k is the feature vector corresponding to the sub-visual information of the k-th object node; Fo k It is the feature vector corresponding to the text information of the k-th object node.

[0091] D6: The object node with the highest similarity to the target feature vector is used as the target object.

[0092] The target retrieval method of this application supports a variety of downstream tasks in the field of embodied intelligent environment perception, including target navigation and specific question answering.

[0093] Taking the target navigation task as an example, after the system identifies the target object node, it performs path planning at the topological level based on the coordinates of the topological nodes corresponding to the target object node. A path planning algorithm is used to calculate the shortest path, generating a series of topological coordinates that define an ideal trajectory. This ideal trajectory minimizes navigation time and distance while accounting for spatial constraints and inherent obstacles in the scene.

[0094] Taking the embodied question-and-answer task as an example, after locating the target object node, the system first invokes the aforementioned target navigation path to guide the agent to autonomously move to the vicinity of the target object. After navigation is complete, the system further retrieves the target node's multimodal semantic information from the scene knowledge graph (target scene graph), including visual features, spatial location, and associated objects. Leveraging a visual language feature extraction model, the system converts the user's natural language question into a semantic query. Combined with environmental perception results, multimodal fusion reasoning is performed to ultimately generate a question-and-answer result that aligns with the semantic context.

[0095] In summary, the object retrieval method disclosed in this application is a robust object retrieval strategy designed based on the object scene graph (also known as the multimodal two-layer topological semantic scene graph) described in the aforementioned embodiments. By spatially expanding the topological nodes, the object retrieval performance of the intelligent agent in downstream tasks is significantly improved. This strategy fully leverages multimodal semantic representations and supports user instructions in both visual and verbal modalities, enabling efficient and accurate task completion in diverse environments.

[0096] It can be understood that the scene graph construction method and target retrieval method disclosed in this application solve the problem of separation of semantic information and spatial structure in traditional scene graphs, significantly enhance the comprehensive cognitive ability of embodied intelligence to the environment, effectively alleviate the local optimal dilemma in the execution of downstream tasks, and provide solid technical support for the development of embodied intelligence.

[0097] Figure 3 A schematic diagram of the structure of a scene graph construction device provided in an embodiment of the present application. Figure 3 As shown, the scene graph construction device 300 disclosed in this application includes: The data set acquisition module 301 is used to construct a topological node set and a topological edge set based on the scene structure data; a first multimodal data acquisition module 302 for, for each topological node in the topological node set, generating a set of visual information, textual information, and spatial position information of the topological node as multimodal data of the topological node; A second multimodal data acquisition module 303 is configured to determine, for each topological node in the topological node set, multimodal data of an object node corresponding to the topological node and a set of connection edges corresponding to the topological node based on the visual information of the topological node; the multimodal data of the object node includes visual information, text information, and spatial position information of the object node; and the connection edge set is a set consisting of connection edges between the topological node and the corresponding object node. The target scene graph acquisition module 304 is configured to construct a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set, and the connection edge set corresponding to each topological node.

[0098] In an optional implementation, the second multimodal data acquisition module 303 includes: A sub-image acquisition unit is configured to determine a plurality of sub-visual information of the topological node based on the visual information of the topological node and the instance segmentation model; the sub-visual information corresponds to the object node in a one-to-one manner; and each visual information corresponds to a plurality of sub-visual information; a sub-text acquisition unit, configured to input the sub-visual information of the topological node into a visual language generation model to determine text information corresponding to the sub-visual information; an object node modal data acquisition unit configured to, for each sub-visual information corresponding to the topological node, construct a set of the sub-visual information, the text information corresponding to the sub-visual information, and the spatial position information of the object node corresponding to the sub-visual information as multimodal data of the object node corresponding to the topological node; The connection edge set acquisition unit is used to connect the topological node with each corresponding object node to obtain multiple connection edges; and use the set composed of the multiple connection edges as the connection edge set corresponding to the topological node.

[0099] In an optional implementation, the second multimodal data acquisition module 303 further includes: A first screening unit is configured to sequentially calculate, for all object nodes corresponding to each topological node, a visual similarity and a spatial distance between every two object nodes among all object nodes corresponding to the topological node; the visual similarity is generated based on two sub-visual information corresponding to the two object nodes; and the spatial distance is generated based on two spatial position information corresponding to the two object nodes; The second screening unit is used to delete one of every two object nodes among all the object nodes if the visual similarity between the two object nodes is greater than a preset first threshold and the spatial distance between the two object nodes is less than a preset second threshold, so as to obtain multiple object nodes corresponding to the topological node.

[0100] In an optional implementation, the data set acquisition module includes: a skeleton graph determining unit, configured to construct a skeleton graph based on the scene structure data; a topological node set acquisition unit, configured to construct the topological node set based on a plurality of candidate topological nodes that are screened out from the plurality of candidate topological nodes corresponding to the skeleton graph and that meet a first screening condition; the first screening condition indicating that a node distance between two candidate topological nodes is equal to a preset third threshold; a fully connected graph determining unit, configured to construct a fully connected graph based on the skeleton graph; A topological edge set acquisition unit is used to construct the topological edge set based on multiple candidate connection paths that meet a second filtering condition and are screened out from multiple candidate connection paths in the fully connected graph; the second filtering condition indicates that the connection distance of the candidate connection path is less than a preset fourth threshold and the candidate connection path does not pass through an obstacle area.

[0101] Figure 4 A schematic diagram of the structure of a target retrieval device provided in an embodiment of the present application. Figure 4 As shown, the target search device 400 disclosed in this application includes: The database acquisition module 401 is used to acquire a semantic environment database; the semantic environment database includes a feature vector corresponding to the multimodal data corresponding to each topological node in the target scene graph, and a feature vector corresponding to the multimodal data of each object node; The target feature vector determination module 402 is configured to input the object to be queried into a visual language feature extraction model to obtain a target feature vector; the object to be queried includes an image to be queried and / or a text to be queried; The target object query module 403 is configured to determine, from the semantic environment database, an object node having the greatest similarity to the target feature vector as the target object.

[0102] In an optional implementation, the target object query module 403 includes: A first topological node query unit is configured to calculate the similarity between a feature vector corresponding to each topological node in the semantic environment database and the target feature vector, and obtain the top M1 topological nodes having the highest similarity to the target feature vector; M1 is an integer greater than or equal to 1; a target topological node set determining unit, configured to determine, for each topological node in the M1 topological nodes, a plurality of topological nodes adjacent to the topological node as a target topological node set corresponding to the topological node; A second topological node query unit is configured to calculate, for each of the M1 topological nodes, a similarity between a feature vector of the topological node and a feature vector of each target topological node in a target topological node set corresponding to the topological node, and obtain top M2 topological nodes having the highest similarity to the target feature vector; M2 is an integer greater than or equal to 1; a target area determination unit, configured to determine a target area covering the M1 topological nodes and the M2 topological nodes corresponding to each topological node in the M1 topological nodes, and determine M3 object nodes covered by the target area; wherein M3 is an integer greater than or equal to 1; an object node query unit, configured to calculate a similarity between a feature vector corresponding to each of the M3 object nodes and the target feature vector, and obtain an object node having the highest similarity to the target feature vector; The target object determination unit is configured to select the object node having the highest similarity to the target feature vector as the target object.

[0103] In an optional implementation, the database acquisition module 401 includes: A first feature vector acquisition unit is configured to input the multimodal data corresponding to each topological node in the target scene graph into the visual language feature extraction model to obtain a feature vector corresponding to each topological node; A second feature vector acquisition unit is used to input the multimodal data corresponding to each object node in the target scene graph into the visual language feature extraction model to obtain a feature vector corresponding to each object node; The feature vector pool construction unit is used to construct a set of feature vectors corresponding to each topological node and feature vectors corresponding to each object node as the semantic environment database.

[0104] Based on the scene graph construction method, target retrieval method and device provided in the aforementioned embodiments, the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements some or all of the steps in the scene graph construction method and target retrieval method mentioned above.

[0105] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0106] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A scene graph construction method, characterized in that: The method comprises: Based on the scene structure data, construct a topological node set and a topological edge set; For each topological node in the topological node set, a set of visual information, text information and spatial position information of the topological node is used as multimodal data of the topological node; For each topological node in the topological node set, based on the visual information of the topological node, determining the multimodal data of the object node corresponding to the topological node and the connection edge set corresponding to the topological node; the multimodal data of the object node includes the visual information, text information and spatial position information of the object node; the connection edge set is a set consisting of the connection edges between the topological node and the corresponding object node; A target scene graph is constructed based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set, and the connection edge set corresponding to each topological node.

2. The method according to claim 1, characterized in that The method of determining, for each topological node in the topological node set, multimodal data of an object node corresponding to the topological node and a set of connection edges corresponding to the topological node based on visual information of the topological node, includes: Based on the visual information of the topological node and the instance segmentation model, a plurality of sub-visual information of the topological node is determined; the sub-visual information corresponds to the object node one by one; each visual information corresponds to a plurality of sub-visual information; Inputting the sub-visual information of the topological node into a visual language generation model to determine text information corresponding to the sub-visual information; For each sub-visual information corresponding to the topological node, a set of the sub-visual information, the text information corresponding to the sub-visual information, and the spatial position information of the object node corresponding to the sub-visual information is constructed as the multimodal data of the object node corresponding to the topological node; The topological node is connected to each corresponding object node to obtain a plurality of connection edges; and a set consisting of the plurality of connection edges is used as a connection edge set corresponding to the topological node.

3. The method according to claim 2, characterized in that The method further comprises: For all object nodes corresponding to each topological node, sequentially calculating the visual similarity and spatial distance between every two object nodes in all object nodes corresponding to the topological node; the visual similarity is generated based on the two sub-visual information corresponding to the two object nodes; the spatial distance is generated based on the two spatial position information corresponding to the two object nodes; For every two object nodes among all the object nodes, if the visual similarity between the two object nodes is greater than a preset first threshold and the spatial distance is less than a preset second threshold, one of the two object nodes is deleted to obtain multiple object nodes corresponding to the topological node.

4. The method according to claim 1, wherein The constructing of a topological node set and a topological edge set based on the scene structure data includes: Constructing a skeleton graph based on the scene structure data; Constructing the topological node set based on a plurality of candidate topological nodes that meet a first screening condition selected from a plurality of candidate topological nodes corresponding to the skeleton graph; wherein the first screening condition indicates that a node distance between two candidate topological nodes is equal to a preset third threshold; Based on the skeleton graph, construct a fully connected graph; The topological edge set is constructed based on multiple candidate connection paths that meet a second filtering condition and are screened out from multiple candidate connection paths in the fully connected graph; the second filtering condition indicates that the connection distance of the candidate connection path is less than a preset fourth threshold and the candidate connection path does not pass through an obstacle area.

5. A target retrieval method, characterized in that: The method comprises: Obtaining a semantic environment database; the semantic environment database is generated based on the target scene graph according to any one of claims 1 to 4; the semantic environment database includes a feature vector corresponding to the multimodal data corresponding to each topological node in the target scene graph, and a feature vector corresponding to the multimodal data of each object node; Inputting the object to be queried into the visual language feature extraction model to obtain a target feature vector; the object to be queried includes the image to be queried and / or the text to be queried; An object node having the greatest similarity to the target feature vector is determined from the semantic environment database as the target object.

6. The method according to claim 5, characterized in that The determining, from the semantic environment database, an object node having the greatest similarity to the target feature vector as the target object includes: Calculating the similarity between the feature vector corresponding to each topological node in the semantic environment database and the target feature vector, and obtaining the top M1 topological nodes with the highest similarity to the target feature vector; wherein M1 is an integer greater than or equal to 1; For each topological node among the M1 topological nodes, determining a plurality of topological nodes adjacent to the topological node as a target topological node set corresponding to the topological node; For each of the M1 topological nodes, calculate the similarity between the feature vector of the topological node and the feature vector of each target topological node in the target topological node set corresponding to the topological node, and obtain the top M2 topological nodes with the highest similarity to the target feature vector; M2 is an integer greater than or equal to 1; Determine a target area covering the M1 topological nodes and the M2 topological nodes corresponding to each topological node in the M1 topological nodes, and determine M3 object nodes covered by the target area; wherein M3 is an integer greater than or equal to 1; Calculating the similarity between the feature vector corresponding to each object node in the M3 object nodes and the target feature vector, and obtaining the object node with the highest similarity to the target feature vector; The object node with the highest similarity to the target feature vector is used as the target object.

7. The method according to claim 5, characterized in that The obtaining of the semantic environment database includes: Inputting the multimodal data corresponding to each topological node in the target scene graph into the visual language feature extraction model to obtain a feature vector corresponding to each topological node; Inputting the multimodal data corresponding to each object node in the target scene graph into the visual language feature extraction model to obtain a feature vector corresponding to each object node; A set constructed by the feature vector corresponding to each topological node and the feature vector corresponding to each object node is used as the semantic environment database.

8. A scene graph construction device, characterized in that: The device comprises: The data set acquisition module is used to construct a topological node set and a topological edge set based on the scene structure data; a first multimodal data acquisition module, configured to, for each topological node in the set of topological nodes, take a set of visual information, textual information, and spatial position information of the topological node as multimodal data of the topological node; a second multimodal data acquisition module configured to determine, for each topological node in the topological node set, multimodal data of an object node corresponding to the topological node and a set of connection edges corresponding to the topological node based on visual information of the topological node; the multimodal data of the object node including visual information, textual information, and spatial location information of the object node; and the connection edge set consisting of connection edges between the topological node and the corresponding object node; The target scene graph acquisition module is used to construct a target scene graph based on the multimodal data of each topological node, the multimodal data of each object node, the topological edge set and the connection edge set corresponding to each topological node.

9. A target retrieval device, characterized in that: The device comprises: A database acquisition module for acquiring a semantic environment database; the semantic environment database is generated based on the target scene graph according to any one of claims 1 to 4; the semantic environment database includes a feature vector corresponding to the multimodal data corresponding to each topological node in the target scene graph, and a feature vector corresponding to the multimodal data of each object node; A target feature vector determination module is configured to input the object to be queried into the visual language feature extraction model to obtain a target feature vector; the object to be queried includes an image to be queried and / or a text to be queried; The target object query module is used to determine the object node with the greatest similarity to the target feature vector from the semantic environment database as the target object.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the scene graph construction method according to any one of claims 1 to 4 and the target retrieval method according to any one of claims 5 to 7 are implemented.

Images