A scene graph generation method and system

Abstract

The present invention belongs to the field of robot scene perception and interaction, and provides a scene graph generation method and system, which can: extract the robot's ontological information and environmental information in real time; calculate the robot's posture transformation distance in real time; if the distance is greater than a threshold, form a first node based on the currently extracted robot's ontological information and environmental information, and use the robot's posture under the first node as the starting posture to calculate the next posture transformation distance until the robot drives out of the target scene; starting from the second formation of a new node, after each node is formed, add edges based on posture distance and semantic features between the currently formed node and the last formed node and add attribute values ​​to them; in real time, all the nodes generated above and all the edges between all nodes are used as the result of scene perception to generate a scene graph of the target scene. The present invention is used to break the limitations of scene graphs in complex scene modeling.

Description

Technical Field

[0001] The present invention relates to the field of robot scene perception and interaction, and in particular to a scene graph generation method and system. Background Art

[0002] In the field of robot scene perception and interaction, scene graph is a graph structure used to describe entities and their relationships in a specific scene. It is generated through the perception, extraction and organization of scene information, and can realize tasks such as scene information mining and scene information question and answer.

[0003] Currently, in the field of robotic scene perception and interaction, scene graphs primarily include object-based scene graphs, topological graphs, and hierarchical graphs. The object-based scene graph centers around the object entities in the scene, recording their category labels and spatial location information (such as 3D coordinates and bounding boxes). The topological graph builds upon the object-based scene graph by adding spatial relationships between objects, constructing topological connections between them and forming a relationship network. The hierarchical graph further abstracts the hierarchical structure of the scene, aggregating from the object layer upwards into higher-level nodes such as rooms, floors, and buildings.

[0004] However, in practical applications, object scene graphs and topology graphs focus more on object-level semantics and relationships, lacking other higher-level information. While hierarchical graphs introduce a hierarchical structure, node information is overly simplistic, typically retaining only node names and rough location information, lacking semantic attributes. Clearly, these three scene graphs have limitations when modeling complex scenes. Summary of the Invention

[0005] In response to the above shortcomings, the present invention provides a scene graph generation method and system to solve at least one of the above technical problems, so as to break the limitations of scene graphs in complex scene modeling to a certain extent.

[0006] In a first aspect, the present invention provides a method for generating a scene graph, which is applied to a robot driving in a target scene, and specifically includes:

[0007] Step S1: extracting the robot's body information and environment information in real time; the body information includes the robot's current posture;

[0008] Step S2: Calculate the robot's posture transformation distance in real time ;

[0009] Step S3: Real-time determination of the calculated posture transformation distance Is it greater than the preset distance threshold? If it is determined Greater than , a new node is formed based on the currently extracted robot's body information and environment information, recorded as the first node, and the robot's posture at the first node is used as the starting posture to calculate the next posture transformation distance , until the robot drives out of the target scene; starting from the second formation of the first node based on the robot's body information and environment information, each time the first node is formed, an edge based on posture distance and an edge based on semantic features are added between the current first node and the first node formed last time, and an attribute value is added to each added edge; the attribute value of the edge based on posture distance is the posture distance of the two nodes connected by the edge, and the attribute value of the edge based on semantic features is the semantic distance of the two nodes connected by the edge;

[0010] Step S4: All nodes generated in the above steps and all edges between all nodes are used as the result of scene perception in real time to generate a scene graph of the target scene.

[0011] In an optional embodiment, an edge based on a semantic feature is added between the first node currently formed and the first node previously formed, and the implementation method includes:

[0012] Calculating the similarity between the semantic feature of the first node currently formed and the semantic feature of the first node formed last time;

[0013] Determine whether the calculated similarity is greater than or equal to the preset similarity threshold :

[0014] If so, add an edge based on the semantic feature between the first node currently formed and the first node formed last time;

[0015] If not, no edge based on the semantic feature is added between the first node currently formed and the first node formed last time.

[0016] In an optional embodiment, the environmental information includes a color image captured by a camera on the robot. and depth maps;

[0017] A node is formed based on the currently extracted robot's body information and environment information. The implementation method includes:

[0018] Extract the color image from the currently extracted robot's environmental information Semantic features of ;

[0019] Extract the spatial features of the depth map from the currently extracted robot's environmental information ;

[0020] Aggregate the semantic features , the spatial characteristics and the currently extracted robot body information and environment information to obtain aggregated information;

[0021] The nodes are formed by gathering information and the color image of the robot's environment information currently extracted is used. Semantic features of is the semantic feature of the node.

[0022] In an optional embodiment, the color image of the robot's environmental information currently extracted is extracted Semantic features of ,include:

[0023] For color images Extract text semantic information and obtain color images The semantic information of the text;

[0024] For color images Perform feature extraction to obtain a color image The eigenvector of

[0025] Performing feature extraction on the text semantic information to obtain a feature vector of the text semantic information;

[0026] Color image The feature vector of the word is concatenated with the feature vector of the word semantic information to obtain the semantic feature .

[0027] In an optional embodiment, the color image of the robot's environmental information currently extracted is extracted Semantic features of , specifically including:

[0028] Color image And large model prompt statement As input, we use a large model of visual language Extract color image Semantic information of text in ;

[0029] Color image and text semantic information , respectively input into the multimodal large model Perform feature extraction and obtain two feature vectors and ;

[0030] The two eigenvectors and Splice to get semantic features .

[0031] In an optional embodiment, between step S3 and step S4, the method further includes:

[0032] Step L: Count the number n of newly added first nodes in the graph in real time, and update the graph every time the counted n reaches a preset number N;

[0033] The updating of the graph includes: updating the node information of each first node in the graph, and updating each edge based on semantic features in the graph.

[0034] In an optional embodiment, updating the node information of each first node in the graph includes:

[0035] For each first node in the graph, find the closest edge to the node in the graph based on its pose distance. nodes; ; ;

[0036] The The semantic features of the nodes and the semantic features of the current first node are fused to obtain new semantic features;

[0037] Using the new semantic feature, updating the semantic feature of the current first node;

[0038] Update the edges in the graph based on semantic features, including:

[0039] For each semantic feature-based edge in the graph, the semantic features of the two updated first nodes connected at both ends of the edge are used to calculate the similarity C of the semantic features of the two updated nodes connected at both ends of the semantic feature-based edge;

[0040] Determine whether the calculated similarity C is greater than or equal to the similarity threshold :

[0041] If so, update the attribute value of the edge based on the semantic feature to the similarity C;

[0042] If not, delete the edge based on semantic features.

[0043] In an optional embodiment, between step L and step S4, the method further includes:

[0044] Step H: After each update of the graph, a node connected domain with similar semantics is formed for all updated first nodes in the graph based on all updated first nodes and all updated edges based on semantic features in the graph; then, for each node connected domain formed: the first nodes in the connected domain and the edges between the first nodes are formed as a whole into a regional node and attribute information is added to it, and the attribute information is the posture of the regional node and the semantic features of the regional node; the posture of the regional node is the average of the postures of all nodes under the regional node, and the semantic features of the regional node are the average of the semantic features of all nodes under the regional node; then, according to the order of generation of the first nodes in the graph, the regional nodes formed in the graph are sorted to obtain a regional node sequence, and then edges based on posture distance and edges based on semantic features are added to each adjacent regional node in the regional node sequence in the graph, and attribute values ​​are added to each added edge.

[0045] In an optional embodiment, step H further includes:

[0046] Starting from the second time that a node connected domain with similar semantics is formed for all updated first nodes in the graph based on all updated first nodes in the graph and all updated edges based on semantic features, before each time a node connected domain with similar semantics is formed for all updated first nodes in the graph based on all updated first nodes in the graph and all updated edges based on semantic features, it is first determined whether there is a regional node in the graph: if so, all current regional nodes in the graph are dissolved, and then a node connected domain with similar semantics is formed for all updated first nodes in the graph; if not, a node connected domain with similar semantics is directly formed for all updated first nodes in the graph.

[0047] In a second aspect, the present invention provides a scene graph generation system, which is applied to a robot driving in a target scene, and specifically includes:

[0048] The information extraction module is used to extract the robot's body information and environment information in real time; the body information includes the robot's current position and posture;

[0049] Distance calculation module, used to calculate the robot's posture transformation distance in real time ;

[0050] The first generation module is used to judge the calculated pose transformation distance in real time Is it greater than the preset distance threshold? If it is determined Greater than , a new node is formed based on the currently extracted robot's body information and environment information, recorded as the first node, and the robot's posture at the first node is used as the starting posture to calculate the next posture transformation distance , until the robot drives out of the target scene; starting from the second formation of the first node based on the robot's body information and environment information, each time the first node is formed, an edge based on posture distance and an edge based on semantic features are added between the current first node and the first node formed last time, and an attribute value is added to each added edge; the attribute value of the edge based on posture distance is the posture distance of the two nodes connected by the edge, and the attribute value of the edge based on semantic features is the semantic distance of the two nodes connected by the edge;

[0051] The second generation module is used to generate a scene graph of the target scene in real time using all the nodes generated in the above steps and all the edges between all the nodes as the result of scene perception.

[0052] It can be seen from the above technical solutions that the present invention has the following advantages:

[0053] The present invention can extract the robot's body information and environmental information in real time, and calculate the robot's posture transformation distance in real time. , real-time judgment of the calculated pose transformation distance Is it greater than the preset distance threshold? , and can be used to determine Greater than A new node is formed based on the currently extracted robot's ontology information and environment information, and since the second time the first node is formed, each time the first node is formed, an edge based on posture distance and an edge based on semantic features are added between the currently formed first node and the first node formed last time, and the attribute value of the edge based on posture distance is set to the posture distance of the two nodes connected by the edge, and the attribute value of the edge based on semantic features is set to the semantic distance of the two nodes connected by the edge. It can be seen that the present invention can introduce the robot's ontology information and environment information into the node information, which helps to overcome the deficiency that the node information of the hierarchical graph is too simple. On the other hand, it can introduce edges based on semantic features between nodes and introduce semantic attributes for the edges based on semantic features, which makes the expression of node information more perfect to a certain extent, helps to overcome the problem that the hierarchical graph nodes only retain the node name and rough location information and lack semantic attributes. It can be seen that it helps to break the limitations of scene graphs in complex scene modeling to a certain extent.

[0054] The present invention can form a node connected domain with similar semantics for all updated nodes in the graph based on all updated nodes and all updated edges in the graph after each update of the graph. Then, for each node connected domain formed, all nodes and edges between nodes in the connected domain are formed as a regional node and attribute information is added to it. The attribute information is the posture of the regional node and the semantic feature of the regional node; the posture of the regional node is the average of the postures of all nodes under the regional node, and the semantic feature of the regional node is the average of the semantic features of all nodes under the regional node. It can also add edges based on posture distance and edges based on semantic features between adjacent regional nodes in the graph and add attribute values ​​to these edges in the order of generation of the first node in the graph. It can be seen that compared with the object scene graph and the topology graph, the present invention introduces a higher level, and can add edge nodes (i.e., edges with attribute values) to higher-level nodes (region nodes) to express node relationship information, which further helps to break the limitations of scene graphs in complex scene modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0057] Figure 2 is a schematic flow chart of a method according to another embodiment of the present invention.

[0058] Figure 3 is a schematic flow chart of a method according to another embodiment of the present invention.

[0059] Figure 4 FIG. 4 is a schematic block diagram of a system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The scene graph generation method provided by the present invention draws on the flexibility of knowledge graphs, can fully express the information collected by the robot in the scene, and can freely associate the relationship between each information node, thereby supporting robot perception and interaction in complex scenarios.

[0061] The following describes in detail the specific execution steps of the scene graph generation method. For the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention can also be implemented in other embodiments without these specific details.

[0062] It should be understood that when used in the present specification, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their collections. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0063] When describing the present invention, phrases such as "one embodiment" or "some embodiments" mean that the particular features, structures, or characteristics described in the embodiment are included in one or more embodiments of the present invention. Therefore, phrases such as "in one embodiment" or "in other embodiments" that appear in different places in the present invention do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0065] Figure 1 A schematic flow chart of a scene graph generation method provided in an embodiment of the present invention. Figure 1 The execution subject may be a scene graph generation system. The scene graph generation method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the scene graph generation system runs in the computer device.

[0066] As an embodiment of the present invention, the scene graph generation method is applied to the robot's driving process in the target scene, specifically including the following steps S1 to S4, please refer to Figure 1 .

[0067] The target scene can be a single scene or a collection of several scenes.

[0068] Step S1: Extract the robot's body information and environment information in real time.

[0069] The present invention extracts the robot's body information and environmental information in real time during the robot's driving process. It can be understood that the above body information and environmental information are all information collected during the robot's driving process.

[0070] The body information includes the current posture of the robot.

[0071] In specific implementation, the body information may also include the robot's current linear velocity, angular velocity and other information.

[0072] Step S2: Calculate the robot's posture transformation distance in real time .

[0073] Step S3: Real-time determination of the calculated pose transformation distance Is it greater than the preset distance threshold? If it is determined Greater than , a new node is formed based on the currently extracted robot's body information and environment information, recorded as the first node, and the robot's posture at the first node is used as the starting posture to calculate the next posture transformation distance , until the robot drives out of the target scene; starting from the second time the first node is formed based on the robot's body information and environment information, each time the first node is formed, an edge based on posture distance and an edge based on semantic features are added between the current first node and the last first node, and an attribute value is added to each added edge.

[0074] The attribute value of the edge based on pose distance is the pose distance between the two nodes connected by the edge, and the attribute value of the edge based on semantic features is the semantic distance between the two nodes connected by the edge.

[0075] In specific implementation, if it is determined No more than , no new node is formed.

[0076] In one embodiment, the robot's posture transformation distance is calculated in real time ,include:

[0077] First, the robot's starting position in the target scene is used as the starting position, and the robot's position transformation distance is calculated in real time. , until the calculated pose transformation distance Greater than the pre-set distance threshold ;

[0078] After each formation of the first node, the robot's posture at the first node is used as the starting posture to calculate the next posture transformation distance. , until the robot drives out of the target scene.

[0079] For example, the above The value range of .

[0080] In one embodiment, the pose transformation distance The calculation formula is:

[0081] ,

[0082] Where, represents the translation distance, represents the rotation distance, is the weight factor of the translation distance, is the weight factor of the rotation distance, , , Represents the horizontal and vertical coordinates of the starting position, The horizontal and vertical coordinates representing the extracted real-time pose of the robot, represents the normalization factor of the translation distance, represent The transpose operation. Represents the trace of the matrix.

[0083] It can be understood that the translation distance refers to the translation part of the two postures The normalized Euclidean distance of .

[0084] 、 as well as The specific value of can be set by those skilled in the art based on experience.

[0085] For example, the above The value can be 0.8. The acceptable value is 0.2.

[0086] For example, the above The possible value is 0.5.

[0087] The above rotation distance refers to the rotation part of the two postures The distance between :

[0088] .

[0089] Step S4: All nodes generated in the above steps and all edges between all nodes are used as the result of scene perception in real time to generate a scene graph of the target scene.

[0090] In some embodiments of the present invention, a method for adding an edge based on a semantic feature between a currently formed first node and a previously formed first node includes:

[0091] Calculating the similarity between the semantic feature of the first node currently formed and the semantic feature of the first node formed last time;

[0092] Determine whether the calculated similarity is greater than or equal to the preset similarity threshold :

[0093] If so, add an edge based on the semantic feature between the first node currently formed and the first node formed last time;

[0094] If not, no edge based on the semantic feature is added between the first node currently formed and the first node formed last time.

[0095] Optionally, a formula for calculating the similarity between the semantic feature of the first node currently formed and the semantic feature of the first node formed last time is:

[0096] ,

[0097] in, Represents the calculated similarity, represents the cosine distance, represents the semantic features of the first node currently formed, Represents the semantic features of the first node formed last time.

[0098] In this specification, the semantic features of a node refer to the semantic features contained in the node information of the node.

[0099] In this specification, the semantic distance between two nodes is the similarity between the semantic features of the two nodes, and the similarity between the semantic features of two nodes is the cosine distance between the semantic features of the two nodes.

[0100] In some embodiments of the present invention, the environmental information includes a color image captured by a camera on the robot. and depth map.

[0101] It can be understood that the color image captured by the camera on the robot in this embodiment and depth maps, i.e., color images and depth maps collected by the robot.

[0102] In specific implementation, environmental information may also include other environmental information collected during the robot's driving process, such as radar point cloud information, infrared temperature measurement information, environmental sound information, etc.

[0103] In this embodiment, a node is formed based on the currently extracted robot body information and environment information, and its implementation method includes:

[0104] Extract the color image from the currently extracted robot's environmental information Semantic features of ;

[0105] Extract the spatial features of the depth map from the currently extracted robot's environmental information ;

[0106] Aggregate the semantic features , the spatial characteristics and the currently extracted robot body information and environment information to obtain aggregated information;

[0107] The nodes are formed by gathering information and the color image of the robot's environment information currently extracted is used. Semantic features of is the semantic feature of the node.

[0108] It can be understood that the node information of the first node in this specification includes all the information of the robot collected at the corresponding position of the first node (robot body information and environment information) and the relevant semantic features obtained through information mining. and spatial characteristics .

[0109] In some embodiments of the present invention, the color image of the robot's environmental information currently extracted is extracted. Semantic features of ,include:

[0110] For color images Extract text semantic information and obtain color images The semantic information of the text;

[0111] For color images Perform feature extraction to obtain a color image The eigenvector of

[0112] Performing feature extraction on the text semantic information to obtain a feature vector of the text semantic information;

[0113] Color image The feature vector of the word is concatenated with the feature vector of the word semantic information to obtain the semantic feature .

[0114] In some other specific embodiments, the color image in the currently extracted robot environment information is Semantic features of , specifically including:

[0115] Color image And large model prompt statement As input, we use a large model of visual language Extract color image Semantic information of text in ;

[0116] Color image and text semantic information , respectively input into the multimodal large model Perform feature extraction and obtain two feature vectors and ;

[0117] The two eigenvectors and Splice to get semantic features .

[0118] In some embodiments, .

[0119] In this embodiment, the spatial features of the extracted depth map Includes the minimum value of the value in the depth map and maximum value .

[0120] In some embodiments, between step S3 and step S4, the method further includes step L.

[0121] Specifically, please refer to Figure 2 , step L includes: real-time counting of the number n of newly added first nodes in the graph, and updating the graph each time the counted n reaches a preset number N.

[0122] It is understandable that when the counted number n does not reach the preset number N, the atlas is not updated.

[0123] The updating of the graph includes: updating the node information of each first node in the graph, and updating each edge based on semantic features in the graph.

[0124] In some embodiments, updating the node information of each first node in the graph includes:

[0125] For each first node in the graph, find the closest edge to the node in the graph based on its pose distance. nodes; ; ;

[0126] The The semantic features of the nodes and the semantic features of the current first node are fused to obtain new semantic features;

[0127] The semantic feature of the current first node is updated using the new semantic feature.

[0128] Understandably, and All are integers.

[0129] The above will be The semantic features of the nodes and the semantic features of the current first node are fused to obtain new semantic features, that is, The new semantic feature is obtained by averaging the semantic features of the nodes and the semantic features of the current first node, specifically including:

[0130] The The semantic features of the nodes and the semantic features of the current first node are added together to obtain the summed semantic features;

[0131] Divide the summed semantic features by , and obtain the new semantic feature.

[0132] Optionally, update each semantic feature-based edge in the graph, including:

[0133] For each semantic feature-based edge in the graph, the semantic features of the two updated first nodes connected at both ends of the edge are used to calculate the similarity C of the semantic features of the two updated nodes connected at both ends of the semantic feature-based edge;

[0134] Determine whether the calculated similarity C is greater than or equal to the similarity threshold :

[0135] If so, update the attribute value of the edge based on the semantic feature to the similarity C;

[0136] If not, delete the edge based on semantic features.

[0137] In this embodiment, the similarity threshold The value is 0.8.

[0138] Each time the present invention updates the graph, the nodes are updated first, and then the edges based on the semantic features are updated based on the updated nodes.

[0139] In some embodiments, between step L and step S4, the method further includes step H.

[0140] Specifically, if Figure 3As shown, step H includes: each time the atlas is updated, based on all the updated first nodes in the atlas and all the updated edges based on semantic features, a node connected domain with similar semantics is formed for all the updated first nodes in the atlas; then for each node connected domain formed: the first nodes in the connected domain and the edges between the first nodes are formed as a whole into a regional node and attribute information is added thereto, the attribute information being the posture of the regional node and the semantic features of the regional node; the posture of the regional node is the average of the postures of all nodes under the regional node, and the semantic features of the regional node is the average of the semantic features of all nodes under the regional node; then, according to the order in which the first nodes in the atlas are generated, the regional nodes formed in the atlas are sorted to obtain a regional node sequence, and then edges based on posture distance and edges based on semantic features are added to each adjacent regional node in the regional node sequence in the atlas, and attribute values ​​are added to each added edge.

[0141] In some embodiments, based on all updated first nodes in the graph and all updated edges based on semantic features, forming a node connected domain with similar semantics for all updated first nodes in the graph includes:

[0142] Gather the updated first nodes in the graph into a node sequence V according to the order in which the nodes are formed;

[0143] Calculate the similarity between the semantic features of each adjacent node in the node sequence V to obtain the semantic feature similarity between each adjacent node in the node sequence V; when calculating the similarity between the semantic features of each adjacent node in the node sequence V: for two adjacent nodes between which there is an edge based on the semantic feature, the updated attribute value of the edge based on the semantic feature is used as the semantic feature similarity between the two nodes; for two adjacent nodes between which there is no edge based on the semantic feature, calculate the cosine distance between the semantic features of the two nodes as the semantic feature similarity between the two nodes;

[0144] Based on the calculated semantic feature similarity, all nodes in the node sequence V whose absolute value of the semantic feature similarity difference does not exceed a preset error value are divided into the same group; the nodes in the same group have similar semantics;

[0145] For each divided group, the node connectivity domain under each group is obtained.

[0146] The above-mentioned preset error value can be set by those skilled in the art based on experience, for example, it can be set to 0.01.

[0147] It can be understood that each node connected domain includes the first nodes in the connected domain and all edges between the first nodes.

[0148] In some other embodiments, step H further comprises:

[0149] Starting from the second time that a node connected domain with similar semantics is formed for all updated first nodes in the graph based on all updated first nodes in the graph and all updated edges based on semantic features, before each time a node connected domain with similar semantics is formed for all updated first nodes in the graph based on all updated first nodes in the graph and all updated edges based on semantic features, it is first determined whether there is a regional node in the graph: if so, all current regional nodes in the graph are dissolved, and then a node connected domain with similar semantics is formed for all updated first nodes in the graph; if not, a node connected domain with similar semantics is directly formed for all updated first nodes in the graph.

[0150] It can be understood that disbanding all the current regional nodes in the graph means deleting the attribute information of each of the current regional nodes in the graph and undoing the formation of each of the current regional nodes in the graph.

[0151] If edges based on posture distance and edges based on semantic features are added to adjacent region nodes in the graph, then the above-mentioned dissolution of all current region nodes in the graph also includes: deleting all edges and edge attribute information added between adjacent region nodes in the graph.

[0152] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0153] Figure 4 The scene graph generation system provided by the present invention is applied to the robot's driving process in the target scene, specifically including:

[0154] The information extraction module 201 is used to extract the robot's body information and environment information in real time; the body information includes the robot's current position and posture;

[0155] Distance calculation module 202, used to calculate the robot's posture transformation distance in real time ;

[0156] The first generation module 203 is used to determine the calculated posture transformation distance in real time Is it greater than the preset distance threshold? If it is determined Greater than , a new node is formed based on the currently extracted robot's body information and environment information, recorded as the first node, and the robot's posture at the first node is used as the starting posture to calculate the next posture transformation distance , until the robot drives out of the target scene; starting from the second formation of the first node based on the robot's body information and environment information, each time the first node is formed, an edge based on posture distance and an edge based on semantic features are added between the current first node and the first node formed last time, and an attribute value is added to each added edge; the attribute value of the edge based on posture distance is the posture distance of the two nodes connected by the edge, and the attribute value of the edge based on semantic features is the semantic distance of the two nodes connected by the edge;

[0157] The second generation module 204 is used to generate a scene graph of the target scene in real time using all nodes generated in the above steps and all edges between all nodes as the result of scene perception.

[0158] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the same or similar parts can be referred to the description in the method embodiment.

[0159] The scene graph generated by the present invention can be used to implement tasks such as scene information mining, question answering, task planning, and navigation planning. For example, if there are several regions and categories in the target scene, the number of merged nodes (i.e., region nodes) and semantic information (i.e., semantic features in the node information of the merged nodes) can be extracted. For example, if it is necessary to describe the scene, the semantic information of the nodes can be extracted for summary. For another example, when task planning and navigation planning are required, the relevant first node can be searched based on the scene graph node, and planning can be performed based on the node pose (i.e., based on the pose contained in the node information in the first node).

[0160] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A scene graph generation method, characterized in that: The method is applied to the robot's driving process in the target scene, specifically including: Step S1: extracting the robot's body information and environment information in real time; the body information includes the robot's current posture; Step S2: Calculate the robot's posture transformation distance in real time ; Step S3: Real-time determination of the calculated posture transformation distance Is it greater than the preset distance threshold? If it is determined Greater than , a new node is formed based on the currently extracted robot's body information and environment information, recorded as the first node, and the robot's posture at the first node is used as the starting posture to calculate the next posture transformation distance , until the robot drives out of the target scene; starting from the second formation of the first node based on the robot's body information and environment information, each time the first node is formed, an edge based on posture distance and an edge based on semantic features are added between the current first node and the first node formed last time, and an attribute value is added to each added edge; the attribute value of the edge based on posture distance is the posture distance of the two nodes connected by the edge, and the attribute value of the edge based on semantic features is the semantic distance of the two nodes connected by the edge; Step S4: All nodes generated in the above steps and all edges between all nodes are used as the result of scene perception in real time to generate a scene graph of the target scene.

2. The scene graph generation method according to claim 1, characterized in that: Adding an edge based on semantic features between the first node currently formed and the first node previously formed, wherein the implementation method includes: Calculating the similarity between the semantic feature of the first node currently formed and the semantic feature of the first node formed last time; Determine whether the calculated similarity is greater than or equal to the preset similarity threshold : If so, add an edge based on the semantic feature between the first node currently formed and the first node formed last time; If not, no edge based on the semantic feature is added between the first node currently formed and the first node formed last time.

3. The scene graph generation method according to claim 1, characterized in that: Environmental information includes color images captured by the camera on the robot and depth maps; A node is formed based on the currently extracted robot's body information and environment information. The implementation method includes: Extract the color image from the currently extracted robot's environmental information Semantic features of ; Extract the spatial features of the depth map from the currently extracted robot's environmental information ; Aggregate the semantic features , the spatial characteristics and the currently extracted robot body information and environment information to obtain aggregated information; The nodes are formed by gathering information and the color image of the robot's environment information currently extracted is used. Semantic features of is the semantic feature of the node.

4. The scene graph generation method according to claim 3, characterized in that: Extract the color image from the currently extracted robot's environmental information Semantic features of ,include: For color images Extract text semantic information and obtain color images The semantic information of the text; For color images Perform feature extraction to obtain a color image The eigenvector of Performing feature extraction on the text semantic information to obtain a feature vector of the text semantic information; Color image The feature vector of the word is concatenated with the feature vector of the word semantic information to obtain the semantic feature .

5. The scene graph generation method according to claim 4, characterized in that: Extract the color image from the currently extracted robot's environmental information Semantic features of , specifically including: Color image And large model prompt statement As input, we use a large model of visual language Extract color image Semantic information of text in ; Color image and text semantic information , respectively input into the multimodal large model Perform feature extraction and obtain two feature vectors and ; The two eigenvectors and Splice to get semantic features .

6. The scene graph generation method according to claim 2, characterized in that: Between step S3 and step S4, the method further comprises: Step L: Count the number n of newly added first nodes in the graph in real time, and update the graph every time the counted n reaches a preset number N; The updating of the graph includes: updating the node information of each first node in the graph, and updating each edge based on semantic features in the graph.

7. The scene graph generation method according to claim 6, characterized in that: Update the node information of each first node in the graph, including: For each first node in the graph, find the closest edge to the node in the graph based on its pose distance. nodes; ; ; The The semantic features of the nodes and the semantic features of the current first node are fused to obtain new semantic features; Using the new semantic feature, updating the semantic feature of the current first node; Update the edges in the graph based on semantic features, including: For each semantic feature-based edge in the graph, the semantic features of the two updated first nodes connected at both ends of the edge are used to calculate the similarity C of the semantic features of the two updated nodes connected at both ends of the semantic feature-based edge; Determine whether the calculated similarity C is greater than or equal to the similarity threshold : If so, update the attribute value of the edge based on the semantic feature to the similarity C; If not, delete the edge based on semantic features.

8. The scene graph generation method according to claim 6, characterized in that: Between step L and step S4, the method further comprises: Step H: After each update of the graph, a node connected domain with similar semantics is formed for all updated first nodes in the graph based on all updated first nodes and all updated edges based on semantic features in the graph; then, for each node connected domain formed: the first nodes in the connected domain and the edges between the first nodes are formed as a whole into a regional node and attribute information is added to it, the attribute information being the posture of the regional node and the semantic features of the regional node; the posture of the regional node is the average of the postures of all nodes under the regional node, and the semantic features of the regional node is the average of the semantic features of all nodes under the regional node; then, according to the order in which the first nodes in the graph are generated, the regional nodes formed in the graph are sorted to obtain a regional node sequence, and then edges based on posture distance and edges based on semantic features are added to each adjacent regional node in the regional node sequence in the graph, and attribute values ​​are added to each added edge.

9. The scene graph generation method according to claim 8, characterized in that: Step H also includes: Starting from the second time that a node connected domain with similar semantics is formed for all updated first nodes in the graph based on all updated first nodes in the graph and all updated edges based on semantic features, before each time a node connected domain with similar semantics is formed for all updated first nodes in the graph based on all updated first nodes in the graph and all updated edges based on semantic features, it is first determined whether there is a regional node in the graph: if so, all current regional nodes in the graph are dissolved, and then a node connected domain with similar semantics is formed for all updated first nodes in the graph; if not, a node connected domain with similar semantics is directly formed for all updated first nodes in the graph.

10. A scene graph generation system, characterized in that: The system is applied to the robot's driving process in the target scene, specifically including: The information extraction module is used to extract the robot's body information and environment information in real time; the body information includes the robot's current position and posture; Distance calculation module, used to calculate the robot's posture transformation distance in real time ; The first generation module is used to judge the calculated pose transformation distance in real time Is it greater than the preset distance threshold? If it is determined Greater than , a new node is formed based on the currently extracted robot's body information and environment information, recorded as the first node, and the robot's posture at the first node is used as the starting posture to calculate the next posture transformation distance , until the robot drives out of the target scene; starting from the second formation of the first node based on the robot's body information and environment information, each time the first node is formed, an edge based on posture distance and an edge based on semantic features are added between the current first node and the first node formed last time, and an attribute value is added to each added edge; the attribute value of the edge based on posture distance is the posture distance of the two nodes connected by the edge, and the attribute value of the edge based on semantic features is the semantic distance of the two nodes connected by the edge; The second generation module is used to generate a scene graph of the target scene in real time using all the nodes generated in the above steps and all the edges between all the nodes as the result of scene perception.

Images