Scene map generation method and system

Abstract

The invention belongs to the field of robot scene perception and interaction, and provides a scene map generation method and system, and the method comprises the steps: extracting the ontology information and environment information of a robot in real time; the pose transformation distance of the robot is calculated in real time; if the distance is greater than a threshold value, forming a first node based on currently extracted body information and environment information of the robot, and calculating a next pose transformation distance by taking the pose of the robot under the first node as an initial pose until the robot drives out of the target scene; from the second time of formation of the new node, after each time of formation of the node, respectively adding an edge based on the pose distance and an edge based on the semantic feature between the currently formed node and the last formed node, and adding an attribute value for the currently formed node and the last formed node; and taking all the generated nodes and all the edges among all the nodes as a scene perception result in real time, and generating a scene map of the target scene. The method is used for breaking the limitation of a scene map in complex scene modeling.

Description

Technical Field

[0001] The present invention relates to the field of robot scene perception and interaction, and particularly relates to a method and system for generating a scene graph. Background Art

[0002] In the field of robot scene perception and interaction, a scene graph is a graph structure used to describe entities and their mutual relationships in a specific scene. It can be generated by perceiving, extracting, and organizing scene information, and can be used to perform tasks such as mining scene information and answering questions about scene information.

[0003] Currently, in the field of robot scene perception and interaction, scene graphs mainly include object scene graphs, topological graphs, and hierarchical graphs. Among them, the object scene graph takes the object entities in the scene as the core and records their category labels and spatial position information (such as three-dimensional coordinates, bounding boxes). Based on the object scene graph, the topological graph adds the spatial relationships between objects, constructs the topological connections between objects, and forms a relationship network. The hierarchical graph further abstracts the hierarchical structure of the scene, aggregating from the object layer upwards to high-level nodes such as rooms, floors, and buildings.

[0004] However, in practical applications, the object scene graph and the topological graph focus more on object-level semantics and relationships, without other higher-level information. Although the hierarchical graph introduces a hierarchical structure, the node information is too brief, usually only retaining the name and approximate position information of the nodes, lacking semantic attributes. Obviously, the above three types of scene graphs have limitations in complex scene modeling. Summary of the Invention

[0005] In view of the above deficiencies, the present invention provides a method and system for generating a scene graph to solve at least one of the above technical problems and to break the limitations of scene graphs in complex scene modeling to a certain extent.

[0006] In a first aspect, the present invention provides a method for generating a scene graph, which is applied during the driving process of a robot in a target scene. Specifically, it includes: Step S1: Extract the robot's body information and environmental information in real time; the body information includes the current pose of the robot. Step S2: Calculate the pose transformation distance of the robot in real time ; Step S3: Judge in real time whether the calculated pose transformation distance is greater than a pre-set distance threshold . If it is determined to be greater than , then form a new node based on the current robot body information and environmental information extracted, denoted as the first node, and use the pose of the robot under the first node as the starting pose to calculate the next pose transformation distance until the robot drives out of the target scenario; starting from the second time to form the first node based on the robot's body information and environmental information, after each formation of the first node, add an edge based on the pose distance and an edge based on the semantic feature between the currently formed first node and the previously formed first node, and add an attribute value to each added edge; the attribute value of the edge based on the pose distance is the pose distance between the two nodes connected by the edge, and the attribute value of the edge based on the semantic feature is the semantic distance between the two nodes connected by the edge; Step S4. Real-time use all the nodes generated in the above steps and all the edges between all the nodes as the result of scene perception to generate a scene graph of the target scenario.

[0007] In an alternative embodiment, the method for adding an edge based on the semantic feature between the currently formed first node and the previously formed first node includes: Calculate the similarity between the semantic feature of the currently formed first node and the semantic feature of the previously formed first node; Determine whether the calculated similarity is greater than or equal to a preset similarity threshold : If so, add an edge based on the semantic feature between the currently formed first node and the previously formed first node; If not, do not add an edge based on the semantic feature between the currently formed first node and the previously formed first node.

[0008] In an alternative embodiment, the environmental information includes the color image collected by the camera on the robot and the depth map; The method for forming a node based on the currently extracted robot body information and environmental information includes: Extract the semantic feature of the color image in the currently extracted robot environmental information ; Extract the spatial feature of the depth map in the currently extracted robot environmental information ; Pool the semantic feature , the spatial feature and the currently extracted robot body information and environmental information to obtain pooled information; Form a node with the pooled information as the node information, and use the semantic feature of the color image in the currently extracted robot environmental information as the semantic feature of this node. ;

[0009] ​In an alternative embodiment, the semantic features of the color image in the currently extracted environmental information of the robot are extracted as follows: including: Performing text semantic information extraction on the color image to obtain the text semantic information of the color image ; Performing feature extraction on the color image to obtain the feature vector of the color image ; Performing feature extraction on the text semantic information to obtain the feature vector of the text semantic information; Concatenating the feature vector of the color image with the feature vector of the text semantic information to obtain the semantic features .

[0010] In an alternative embodiment, the semantic features of the color image in the currently extracted environmental information of the robot are extracted as follows: Specifically including: Using the color image and the large model prompt statement as inputs, and using the vision-language large model to extract the text semantic information in the color image ; Inputting the color image and the text semantic information into the multi-modal large model respectively for feature extraction to obtain two feature vectors and ; Concatenating the two feature vectors and to obtain the semantic features .

[0011] In an alternative embodiment, between step S3 and step S4, the method further includes: Step L: Real-time statistics of the number n of newly added first nodes in the graph, and each time the counted n reaches the preset number N, the graph is updated; The updating of the graph includes: updating the node information of each first node in the graph and updating each edge based on semantic features in the graph.

[0012] In an alternative embodiment, updating the node information of each first node in the graph includes: For each first node in the graph, find the node closest to this node in the graph according to its edges based on pose distance nodes; ; ; Fuse the semantic features of the said nodes and the semantic features of the current first node to obtain new semantic features; Use the said new semantic features to update the semantic features of the current first node; Update each edge based on semantic features in the graph, including: For each edge based on semantic features in the graph, calculate the similarity C of the semantic features of the two updated nodes connected at both ends of the edge using the semantic features of the two updated first nodes connected at both ends of the edge; Judge whether the calculated similarity C is greater than or equal to the said similarity threshold : If so, update the attribute value of this edge based on semantic features to the similarity C; If not, delete this edge based on semantic features.

[0013] In an alternative embodiment, between step L and step S4, the method further includes: Step H. After each update of the graph, based on all updated first nodes and all updated edges based on semantic features in the graph, form node connected domains with similar semantics for all updated first nodes in the graph; then for each formed node connected domain: Integrate all the first nodes and the edges between the first nodes in the connected domain to form a regional node and add attribute information to it, and the attribute information is the pose and semantic features of the regional node; the pose of the regional node is the average of the poses of all nodes under the regional node, and the semantic features of the regional node are the average of the semantic features of all nodes under the regional node; then, in the order of the generation of the first nodes in the graph, sort the formed regional nodes in the graph to obtain a regional node sequence, and then add edges based on pose distance and edges based on semantic features to adjacent regional nodes in the regional node sequence in the graph, and add attribute values to each added edge.

[0014] In an alternative embodiment, step H further includes: Since the second time when the node connectivity domains with similar semantics are formed for all the updated first nodes in the graph based on all the updated first nodes in the graph and all the updated edges based on semantic features, before each time when the node connectivity domains with similar semantics are formed for all the updated first nodes in the graph based on all the updated first nodes in the graph and all the updated edges based on semantic features, it is first determined whether there are regional nodes in the current graph: if there are, all the current regional nodes in the graph are dissolved, and then the node connectivity domains with similar semantics are formed for all the updated first nodes in the graph; if not, the node connectivity domains with similar semantics are directly formed for all the updated first nodes in the graph.

[0015] In a second aspect, the present invention provides a scenario graph generation system, which is applied during the driving process of a robot in a target scenario, and specifically includes: An information extraction module, configured to extract the ontology information and environmental information of the robot in real time; the ontology information includes the current pose of the robot. A distance calculation module, configured to calculate the pose transformation distance of the robot in real time ; A first generation module, configured to determine in real time the calculated pose transformation distance whether it is greater than a preset distance threshold , if it is determined to be greater than , a new node, denoted as the first node, is formed based on the current extracted ontology information and environmental information of the robot, and the pose of the robot under the first node is used as the starting pose to calculate the next pose transformation distance , until the robot drives out of the target scenario; since the second time when the first node is formed based on the ontology information and environmental information of the robot, after each formation of the first node, an edge based on the pose distance and an edge based on semantic features are respectively added between the currently formed first node and the previously formed first node, and an attribute value is added to each added edge; the attribute value of the edge based on the pose distance is the pose distance between the two nodes connected by the edge, and the attribute value of the edge based on semantic features is the semantic distance between the two nodes connected by the edge. A second generation module, configured to generate a scenario graph of the target scenario in real time with all the nodes generated in the above steps and all the edges between all the nodes as the result of scenario perception.

[0016] It can be seen from the above technical solutions that the present invention has the following advantages: The present invention can extract the ontology information and environmental information of the robot in real time, calculate the pose transformation distance of the robot in real time , and determine in real time the calculated pose transformation distance whether it is greater than a preset distance threshold , and be able to determine greater than When it is, a new node is formed based on the currently extracted body information and environmental information of the robot, and since the second formation of the first node, each time the first node is formed, an edge based on pose distance and an edge based on semantic features are added between the currently formed first node and the previously formed first node. The attribute value of the edge based on pose distance is set to the pose distance between the two nodes connected by the edge, and the attribute value of the edge based on semantic features is set to the semantic distance between the two nodes connected by the edge. It can be seen that the present invention can introduce the body information and environmental information of the robot into the node information, which helps to overcome the deficiency that the node information of the hierarchical graph is too brief. On the other hand, an edge based on semantic features can be introduced between nodes and semantic attributes can be introduced for the edge based on semantic features, which to a certain extent makes the expression of node information more perfect and helps to overcome the problem that the hierarchical graph nodes only retain the name and rough position information of the nodes and lack semantic attributes. It can be seen that it helps to break the limitations of the scene graph in complex scene modeling to a certain extent.

[0017] After each update of the graph, the present invention can form node connectivity domains with similar semantics for all updated nodes in the graph based on all updated nodes and all updated edges in the graph. Then, for each formed node connectivity domain, all nodes and the edges between the nodes within the connectivity domain are integrally formed into a regional node and attribute information is added to it. The attribute information is the pose of the regional node and the semantic features of the regional node; the pose of the regional node is the average value of the poses of all nodes under the regional node, and the semantic features of the regional node are the average value of the semantic features of all nodes under the regional node. And it can add an edge based on pose distance and an edge based on semantic features between adjacent regional nodes in the graph according to the order of generation of the first node in the graph and add attribute values to these edges. It can be seen that compared with the object scene graph and the topological graph, the present invention introduces a higher level and can add edge nodes (i.e., edges with attribute values) to higher-level nodes (regional nodes) to express node relationship information, which further helps to break the limitations of the scene graph in complex scene modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the present invention, the drawings required for description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 is a schematic flowchart of the method according to an embodiment of the present invention.

[0020] Figure 2It is a schematic flowchart of the method according to another embodiment of the present invention.

[0021] Figure 3 It is a schematic flowchart of the method according to another embodiment of the present invention.

[0022] Figure 4 It is a schematic block diagram of the system according to an embodiment of the present invention. Detailed implementation manners

[0023] The method for generating a scene graph provided by the present invention draws on the flexibility of the knowledge graph, can fully express the information collected by the robot in the scene, and can freely associate the relationships between various information nodes, thereby supporting the perception and interaction of the robot in complex scenarios.

[0024] The following will describe in detail the specific execution steps of the method for generating a scene graph. For the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are proposed to thoroughly understand the embodiments of the present invention. However, those skilled in the art should clearly understand that the present invention can also be implemented in other embodiments without these specific details.

[0025] It should be understood that when used in the specification of the present invention, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0026] The statements such as "in an embodiment" or "in some embodiments" described in the present invention mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present invention. Thus, the statements such as "in an embodiment" and "in some other embodiments" that appear in different parts of the present invention do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0027] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0028] Figure 1 It is a schematic flowchart of a method for generating a scene graph provided by an embodiment of the present invention. Figure 1The executing entity can be a scenario graph generation system. The scenario graph generation method provided by the embodiments of the present invention is executed by a computer device. Correspondingly, the scenario graph generation system runs in the computer device.

[0029] As an embodiment of the present invention, the scenario graph generation method is applied to the driving process of a robot in a target scenario, and specifically includes the following steps S1 to S4. Please refer to Figure 1 .

[0030] The target scenario can be a single scenario or a collection of several scenarios.

[0031] Step S1: Extract the robot's body information and environmental information in real time.

[0032] In the driving process of the robot in the present invention, the robot's body information and environmental information are extracted in real time. It can be understood that the above-mentioned body information and environmental information are both information collected during the robot's driving process.

[0033] The body information includes the current pose of the robot.

[0034] Specifically, the body information may also include information such as the current linear velocity and angular velocity of the robot.

[0035] Step S2: Calculate the pose transformation distance of the robot in real time .

[0036] Step S3: Judge in real time whether the calculated pose transformation distance is greater than a pre-set distance threshold . If it is determined to be greater than , then a new node is formed based on the currently extracted body information and environmental information of the robot, denoted as the first node, and the pose of the robot under the first node is used as the starting pose to calculate the next pose transformation distance , until the robot drives out of the target scenario; starting from the second time a first node is formed based on the robot's body information and environmental information, each time a first node is formed, an edge based on the pose distance and an edge based on the semantic feature are added between the currently formed first node and the previously formed first node, and an attribute value is added to each added edge.

[0037] The attribute value of the edge based on the pose distance is the pose distance between the two nodes connected by the edge, and the attribute value of the edge based on the semantic feature is the semantic distance between the two nodes connected by the edge.

[0038] Specifically, if it is determined not to be greater than , then no new node is formed.

[0039] In one embodiment, the pose transformation distance of the robot is calculated in real time , including: First, taking the pose of the starting position where the robot travels in the target scene as the starting pose, calculate the pose transformation distance of the robot in real time , until the calculated pose transformation distance is greater than a pre-set distance threshold ; After that, every time a first node is formed, take the pose of the robot under the formed first node as the starting pose, and calculate the next pose transformation distance , until the robot drives out of the target scene

[0040] Exemplarily, the above value range is .

[0041] In one embodiment, the calculation formula of the pose transformation distance is: , In the formula, represents the translation distance, represents the rotation distance, is the weight factor of the translation distance, is the weight factor of the rotation distance, , , represents the abscissa and ordinate of the starting pose, represents the abscissa and ordinate of the extracted real-time pose of the robot, represents the translation distance normalization factor, represents transpose operation of represents the trace of the matrix

[0042] It can be understood that the translation distance refers to the translation part of two poses normalized Euclidean distance of .

[0043] , and specific values of can be set by those skilled in the art according to empirical values

[0044] Exemplarily, the above can take the value of 0.8, and the above can take the value of 0.2

[0045] Exemplarily, the above can take the value of 0.5

[0046] The above rotation distance refers to the rotational part of two poses between them : .

[0047] Step S4: In real time, all the nodes generated in the above steps and all the edges between all the nodes are used as the result of scene perception to generate a scene graph of the target scene.

[0048] In some embodiments of the present invention, an edge based on semantic features is added between the currently formed first node and the previously formed first node, and its implementation method includes: Calculate the similarity between the semantic features of the currently formed first node and the semantic features of the previously formed first node; Judge whether the calculated similarity is greater than or equal to a preset similarity threshold : If so, add an edge based on semantic features between the currently formed first node and the previously formed first node; If not, do not add an edge based on semantic features between the currently formed first node and the previously formed first node.

[0049] Optionally, the calculation formula for the similarity between the semantic features of the currently formed first node and the semantic features of the previously formed first node is: , where represents the calculated similarity, represents the cosine distance, represents the semantic features of the currently formed first node, represents the semantic features of the previously formed first node.

[0050] In this specification, the semantic features of a node are the semantic features included in the node information of the node.

[0051] In this specification, the semantic distance between two nodes is the similarity between the semantic features of the two nodes, and the similarity between the semantic features of the two nodes is the cosine distance between the semantic features of the two nodes.

[0052] In some embodiments of the present invention, the environmental information includes a color image collected by a camera on the robot and a depth map.

[0053] It can be understood that the color image collected by the camera on the robot in this embodiment and the depth map are the color image and the depth map collected by the robot.

[0054] In specific implementation, the environmental information may further include other environmental information collected during the robot's travel, such as radar point cloud information, infrared temperature measurement information, environmental sound information, etc.

[0055] In this embodiment, nodes are formed based on the currently extracted body information and environmental information of the robot, and the implementation method includes: Extract the semantic features of the color image in the currently extracted environmental information of the robot ; Extract the spatial features of the depth map in the currently extracted environmental information of the robot ; Collect the semantic features , the spatial features , and the currently extracted body information and environmental information of the robot to obtain collected information; Form nodes with the collected information as node information, and use the semantic features of the color image in the currently extracted environmental information of the robot as the semantic features of this node. ; ;

[0056] It can be understood that the node information of the first node in this specification includes all information (robot body information and environmental information) of the robot collected in the corresponding pose of the first node, as well as the relevant semantic features and spatial features obtained through information mining.

[0057] In some embodiments of the present invention, extracting the semantic features of the color image in the currently extracted environmental information of the robot includes: Performing text semantic information extraction on the color image to obtain the text semantic information of the color image ; Performing feature extraction on the color image to obtain the feature vector of the color image ; Performing feature extraction on the text semantic information to obtain the feature vector of the text semantic information; Concatenating the feature vector of the color image with the feature vector of the text semantic information to obtain semantic features . ;

[0058] In some other specific embodiments, the above-mentioned extraction of the semantic features of the color image in the currently extracted environmental information of the robot includes: , specifically including: Using a color image and large model prompt statements as inputs, use a vision - language large model to extract the semantic information of the text in the color image ; ; Input the color image and the semantic information of the text into a multi - modal large model respectively for feature extraction to obtain two feature vectors and ; Concatenate the two feature vectors and to obtain a semantic feature .

[0059] In some embodiments, .

[0060] In this embodiment, the spatial features of the depth map extracted include the minimum value and the maximum value of the values in the depth map.

[0061] In some embodiments, between step S3 and step S4, the method further includes step L.

[0062] Specifically, referring to Figure 2 , step L includes: real - time statistics of the number n of newly added first nodes in the graph, and each time the counted n reaches a preset number N, the graph is updated.

[0063] It can be understood that when the counted n does not reach the preset number N, the graph is not updated.

[0064] The update of the graph includes: updating the node information of each first node in the graph and updating each edge based on semantic features in the graph.

[0065] In some embodiments, updating the node information of each first node in the graph includes: For each first node in the graph, according to its edge based on pose distance, find the nodes closest to this node in the graph; ; ; Fuse the semantic features of the nodes and the semantic feature of the current first node to obtain a new semantic feature; Update the semantic features of the current first node with the new semantic features.

[0066] Understandably, and are both integers.

[0067] The above-mentioned feature fusion of the semantic features of the nodes and the semantic features of the current first node to obtain new semantic features, that is, taking the average of the semantic features of the nodes and the semantic features of the current first node to obtain the new semantic features, specifically including: Perform an addition operation on the semantic features of the nodes and the semantic features of the current first node to obtain the sum semantic features; Divide the sum semantic features by to obtain the new semantic features.

[0068] Optionally, update each edge based on semantic features in the graph, including: For each edge based on semantic features in the graph, calculate the similarity C of the semantic features of the two updated nodes connected at both ends of the edge by using the semantic features of the two updated first nodes connected at both ends of the edge; Judge whether the calculated similarity C is greater than or equal to the similarity threshold : If so, update the attribute value of this edge based on semantic features to the similarity C; If not, delete this edge based on semantic features.

[0069] In this embodiment, the similarity threshold takes a value of 0.8.

[0070] Each time the graph is updated in the present invention, the nodes are updated first, and then the edges based on semantic features are updated based on the updated nodes.

[0071] In some embodiments, between step L and step S4, the method further includes step H.

[0072] Specifically, as Figure 3As shown, step H includes: after each update of the graph, based on all updated first nodes in the graph and all updated edges based on semantic features, forming node connection domains with similar semantics for all updated first nodes in the graph; then for each formed node connection domain: forming a regional node by integrating all the first nodes and the edges between all the first nodes within the connection domain and adding attribute information to it, where the attribute information is the pose of the regional node and the semantic features of the regional node; the pose of the regional node is the average of the poses of all nodes under the regional node, and the semantic features of the regional node are the average of the semantic features of all nodes under the regional node; then, in the order of generation of the first nodes in the graph, sorting the formed regional nodes in the graph to obtain a regional node sequence, and then adding an edge based on pose distance and an edge based on semantic features between adjacent regional nodes in the regional node sequence in the graph, and adding an attribute value to each added edge.

[0073] In some embodiments, forming node connection domains with similar semantics for all updated first nodes in the graph based on all updated first nodes in the graph and all updated edges based on semantic features includes: Collecting all updated first nodes in the graph into a node sequence V in the order of node formation; Calculating the similarity between the semantic features of adjacent nodes in the node sequence V to obtain the semantic feature similarity between adjacent nodes in the node sequence V; when calculating the similarity between the semantic features of adjacent nodes in the node sequence V: for two adjacent nodes with an edge based on semantic features between them, using the updated attribute value of the edge based on semantic features as the semantic feature similarity between them; for two adjacent nodes without an edge based on semantic features between them, calculating the cosine distance between the semantic features of the two nodes as the semantic feature similarity between them; According to the calculated semantic feature similarity, dividing all nodes in the node sequence V whose absolute value of the difference in semantic feature similarity does not exceed a preset error value into the same group; the nodes in the same group have similar semantics; For each divided group, obtaining the node connection domain under each group.

[0074] Among them, the above-mentioned preset error value can be set by those skilled in the art according to experience, for example, it can be set to 0.01.

[0075] It can be understood that each node connection domain includes all the first nodes and all the edges between the first nodes under the connection domain.

[0076] In some other embodiments, step H further includes: Since the second time when node connection domains with similar semantics are formed for all updated first nodes in the graph based on all updated first nodes in the graph and all updated edges based on semantic features, before each time when node connection domains with similar semantics are formed for all updated first nodes in the graph based on all updated first nodes in the graph and all updated edges based on semantic features, it is first determined whether there are regional nodes in the current graph: if so, all current regional nodes in the graph are dissolved, and then node connection domains with similar semantics are formed for all updated first nodes in the graph; if not, node connection domains with similar semantics are directly formed for all updated first nodes in the graph.

[0077] Understandably, dissolving all current regional nodes in the graph means deleting the attribute information of each current regional node in the graph and revoking the formation of each current regional node in the graph.

[0078] If edges based on pose distance and edges based on semantic features are added to adjacent regional nodes in the graph, then the above-mentioned dissolution of all current regional nodes in the graph further includes: deleting all added edges and the attribute information of the edges between adjacent regional nodes in the graph.

[0079] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0080] Figure 4 This is a scene graph generation system provided by the present invention. This system is applied during the driving process of a robot in a target scene and specifically includes: An information extraction module 201, configured to extract the ontology information and environmental information of the robot in real time; the ontology information includes the current pose of the robot. A distance calculation module 202, configured to calculate the pose transformation distance of the robot in real time ; A first generation module 203, configured to determine in real time whether the calculated pose transformation distance is greater than a preset distance threshold , if it is determined to be greater than , then a new node is formed based on the current ontology information and environmental information of the robot, denoted as the first node, and with the pose of the robot under the first node as the starting pose, the next pose transformation distance is calculated until the robot drives out of the target scenario; starting from the second time when the first node is formed based on the robot's body information and environmental information, after each formation of the first node, an edge based on pose distance and an edge based on semantic features are added between the currently formed first node and the previously formed first node, and an attribute value is added to each added edge; the attribute value of the edge based on pose distance is the pose distance between the two nodes connected by the edge, and the attribute value of the edge based on semantic features is the semantic distance between the two nodes connected by the edge; The second generation module 204 is configured to, in real time, use all the nodes generated in the above steps and all the edges between all the nodes as the result of scene perception to generate a scene graph of the target scenario.

[0081] For the same or similar parts among the various embodiments in this specification, reference can be made to each other. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for the same or similar parts, reference can be made to the description in the method embodiments.

[0082] The scene graph generated by the present invention can be used to implement tasks such as mining of scene information, question answering, task planning, and navigation planning. For example, if there are several regions and categories in the target scenario, the number and semantic information of the merged nodes (i.e., region nodes) can be extracted (i.e., the semantic features in the node information of the merged nodes). For example, if it is necessary to describe the scene, the semantic information of the nodes can be extracted for summarization. Also, for example, when task planning and navigation planning are required, relevant first nodes can be searched based on the scene graph nodes, and planning can be performed based on the node poses (i.e., based on the poses included in the node information of the first nodes).

[0083] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a scene graph, characterized in that, The method is applied to the driving process of the robot in the target scenario, specifically including: Step S1: Extract the body information and environmental information of the robot in real time; the body information includes the current pose of the robot. Step S2: Real-time calculation of the pose transformation distance of the robot ; Step S3: Continuously determine the calculated pose transformation distance Is it greater than a pre-set distance threshold , if it is determined Greater than , then form a new node based on the current robot's body information and environmental information, denoted as the first node, and use the robot's pose under the first node as the starting pose to calculate the next pose transformation distance , until the robot exits the target scene; starting from the second time the first node is formed based on the robot's body information and environmental information, each time the first node is formed, add an edge based on pose distance and an edge based on semantic features between the currently formed first node and the previously formed first node, and add an attribute value to each added edge; the attribute value of the edge based on pose distance is the pose distance between the two nodes connected by the edge, and the attribute value of the edge based on semantic features is the semantic distance between the two nodes connected by the edge; Step S4: In real time, take all the nodes generated in the above steps and all the edges between all the nodes as the result of scene perception, and generate a scene graph of the target scenario.

2. The method for generating a scenario graph according to claim 1, wherein Add an edge based on semantic features between the currently formed first node and the previously formed first node. The implementation method includes: Calculate the similarity between the semantic features of the currently formed first node and the semantic features of the previously formed first node. Determine whether the calculated similarity is greater than or equal to a preset similarity threshold : If so, add an edge based on semantic features between the currently formed first node and the previously formed first node. If not, do not add an edge based on semantic features between the currently formed first node and the previously formed first node.

3. The method for generating a scenario map according to claim 1, wherein The environmental information includes the color images and depth maps collected by the cameras on the robot. and depth maps; Form nodes based on the currently extracted body information and environmental information of the robot. The implementation method includes: Extract the color image in the environmental information of the currently extracted robot semantic features of ; Extract the spatial features of the depth map in the currently extracted environmental information of the robot ; Collect the semantic features and the spatial features as well as the current body information and environmental information of the robot that have been extracted, to obtain aggregated information; Nodes are formed by gathering information as node information, and the semantic features of the color image in the currently extracted environmental information of the robot are used as the semantic features of the node.

4. The method for generating a scenario graph according to claim 3, wherein Extract the color image in the environmental information of the currently extracted robot semantic features of , including: Extract the literal semantic information from the color image to obtain the literal semantic information of the color image ; Extract features from a color image to obtain the feature vector of the color image ; Extract features from the text semantic information to obtain the feature vector of the text semantic information. Concatenate the feature vector of the color image with the feature vector of the literal semantic information to obtain a semantic feature .

5. The method for generating a scenario map according to claim 4, wherein Extract the color image in the environmental information of the currently extracted robot semantic features of , specifically including: With a color image and large model prompt statements as inputs, use a vision-language large model to extract the semantic information of the text in the color image ; Input the color image and the text semantic information into the multimodal large model respectively for feature extraction to obtain two feature vectors and ; Concatenate two feature vectors and to obtain a semantic feature .

6. The method for generating a scenario map according to claim 2, wherein Between step S3 and step S4, the method further includes: Step L: Statistically count the number n of newly added first nodes in the graph in real time. Each time the counted n reaches the preset number N, update the graph. The update of the graph includes: updating the node information of each first node in the graph and updating each edge based on semantic features in the graph.

7. The method for generating a scenario graph according to claim 6, wherein Updating the node information of each first node in the graph includes: For each first node in the graph, find the nearest node to this node in the graph according to its edges based on pose distance; ; ; Fuse the semantic features of the nodes and the semantic features of the current first node to obtain new semantic features; Using the new semantic features to update the semantic features of the current first node. Updating each edge based on semantic features in the graph includes: For each edge based on semantic features in the graph, calculate the similarity C between the semantic features of the two updated nodes connected at both ends of the edge by using the semantic features of the two updated first nodes connected at both ends of the edge. Determine whether the calculated similarity C is greater than or equal to the similarity threshold : If so, update the attribute value of the edge based on semantic features to the similarity C. If not, delete the edge based on semantic features.

8. The method for generating a scenario graph according to claim 6, wherein Between step L and step S4, the method further includes: Step H: After each update of the graph, based on all the updated first nodes and all the updated edges based on semantic features in the graph, form node connectivity domains with similar semantics for all the updated first nodes in the graph; then for each formed node connectivity domain: form a regional node by taking all the first nodes and the edges between the first nodes in the connectivity domain as a whole and add attribute information to it. The attribute information is the pose and semantic features of the regional node; the pose of the regional node is the average value of the poses of all the nodes under the regional node, and the semantic features of the regional node are the average value of the semantic features of all the nodes under the regional node; then, according to the order of generation of the first nodes in the graph, sort the formed regional nodes in the graph to obtain a regional node sequence, and then add edges based on pose distance and edges based on semantic features between adjacent regional nodes in the regional node sequence in the graph, and add an attribute value to each added edge.

9. The method for generating a scenario map according to claim 8, wherein Step H further includes: Since the second time when the node connection domains with similar semantics are formed for all the updated first nodes in the graph based on all the updated first nodes and all the updated edges based on semantic features in the graph, before each time when the node connection domains with similar semantics are formed for all the updated first nodes in the graph based on all the updated first nodes and all the updated edges based on semantic features in the graph, it is first determined whether there are regional nodes in the graph currently: if so, all the current regional nodes in the graph are dissolved, and then the node connection domains with similar semantics are formed for all the updated first nodes in the graph; if not, the node connection domains with similar semantics are directly formed for all the updated first nodes in the graph.

10. A scene graph generation system, characterized in that, The system is applied to the driving process of the robot in the target scenario, specifically including: An information extraction module, configured to extract the ontology information and environmental information of the robot in real time; the ontology information includes the current pose of the robot. A distance calculation module, configured to calculate the pose transformation distance of the robot in real time ; The first generation module is used to determine in real time the calculated pose transformation distance whether it is greater than a pre-set distance threshold , if it is determined to be greater than , then a new node is formed based on the current robot body information and environmental information extracted, denoted as the first node, and the pose of the robot under the first node is used as the starting pose to calculate the next pose transformation distance , until the robot exits the target scene; starting from the second time the first node is formed based on the robot body information and environmental information, each time after the first node is formed, an edge based on the pose distance and an edge based on the semantic feature are added between the currently formed first node and the previously formed first node, and an attribute value is added to each added edge; the attribute value of the edge based on the pose distance is the pose distance between the two nodes connected by the edge, and the attribute value of the edge based on the semantic feature is the semantic distance between the two nodes connected by the edge; A second generation module, configured to generate a scene graph of the target scenario by using all the nodes generated in the above steps and all the edges between all the nodes as the result of scene perception in real time.

Images