Method and system for generating heterogeneous common sense scene graph for man-machine mutual assistance

Abstract

The invention discloses a method and a system for generating a heterogeneous common sense scene graph for man-machine mutual assistance, and belongs to the technical field of robots. Comprising the steps of obtaining an environment image sequence and extracting key frames, and extracting entity information based on the key frames; the geometric position relation and the feature similarity of the entities are considered, the same entity under different key frames is fused, and a map is constructed in an incremental mode; and modeling the task relationship and the common sense relationship between the entities on the map, and generating a heterogeneous common sense scene graph to assist the robot in executing a man-machine mutual assistance task. Strong semantic understanding and reasoning capabilities brought by a large model can be reserved, and meanwhile, the operation cost and response time of the system are greatly reduced; the problem that an existing semantic map for man-machine mutual assistance cannot meet the real demand of man-machine mutual assistance is solved.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a method and system for generating heterogeneous common-sense scene graphs for human-machine cooperation. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] With the vigorous development of robotics and the advancement of computer vision technology, robots can perform more and more tasks, such as navigation, grasping, palletizing, etc.

[0004] With the continuous improvement of robots' "human-like characteristics" and their own algorithmic capabilities, as well as the emergence of intelligent products such as AI robots, the possibility and necessity of coexistence and cooperation between AI and humans are also increasing. In other words, it is difficult for humans to avoid contact with AI. Therefore, in the new context, robots' understanding of humans' non-specific intentions and the construction of human-machine mutual assistance systems are becoming increasingly important.

[0005] The development of deep learning technology, especially the emergence of VLM (visual language model) and LLM (large language model) has provided a solution to this task. In order to complete the task of human-machine cooperation, the robot first needs to understand the environmental information in which it is located. The existing methods generally use SLAM technology combined with target detection technology (YOLO series) or semantic segmentation technology (U-NET) to build a semantic map, but the map is limited by the closed data set and can only recognize objects that have been annotated in the data set. It has poor scalability and lacks common sense understanding of objects in the environment. As for the understanding of human intentions, the existing methods use recognition technology based on neural networks, which also use closed data sets. The closed data set means that the robot will make a fixed response to a fixed action of a human.

[0006] Research on human-robot cooperation methods is limited to closed data sets and specific intentions. However, most scenes in real life and production environments are unstructured, with complex environments and changeable human actions. As a result, existing semantic maps used for human-robot cooperation cannot meet actual needs.

[0007] In addition, current human-machine cooperative systems or robot task planning systems usually rely on large-scale pre-trained models (such as large language models or deep learning models) for task planning and reasoning during the execution phase. This reliance brings significant computing costs and time consumption, especially in scenarios with high-frequency or complex tasks, running these models will incur high costs or great time delays. Summary of the invention

[0008] In order to address the deficiencies of the prior art, the present invention provides a method, system, electronic device, computer-readable storage medium and computer program product for generating heterogeneous common-sense scene graphs for human-computer collaboration, breaking the limitations of closed data sets, constructing heterogeneous common-sense scene graphs and using them for reasoning and decision-making, thus getting rid of the reliance on large models in the reasoning and decision-making process.

[0009] In a first aspect, the present invention provides a method for generating heterogeneous common sense scene graphs for human-machine collaboration;

[0010] A method for generating heterogeneous common-sense scene graphs for human-machine collaboration, comprising:

[0011] Obtain an environment image sequence and extract key frames, and extract entity information based on the key frames;

[0012] Considering the geometric position relationship and feature similarity of entities, the same entity under different key frames is fused to incrementally construct the map; the task relationship and common-sense relationship between entities on the map are modeled to generate heterogeneous common-sense scene graphs to assist robots in performing human-machine cooperative tasks.

[0013] In some implementations, extracting entity information based on key frames specifically includes:

[0014] Perform entity extraction on the RGB image in each keyframe, obtain the entity mask, and convert the entity mask into a point cloud in world coordinates using the camera intrinsic parameters and keyframe pose;

[0015] The RGB image corresponding to the entity mask is encoded through the visual language model to obtain visual features aligned with the text features.

[0016] In some implementations, the considering the geometric position relationship and feature similarity of the entities and fusing the same entity in different key frames specifically includes:

[0017] Calculate the feature similarity and node distance between the node in the key frame and the existing node in the map, and merge the node and the existing node when the feature similarity is greater than a preset feature threshold and the node distance is less than a preset distance threshold;

[0018] The point clouds of the same entity in different key frames are combined into different point cloud combinations, and the inclusion rate of the point cloud combination is calculated; when the inclusion rate is greater than the preset inclusion rate threshold, the point clouds in the point cloud combination are merged.

[0019] In some embodiments, the inclusion rate is expressed as:

[0020]

[0021] In the formula, point A Represents point cloud A, pointB represents point cloud B, represents the volume of point cloud A, Represents the volume of point cloud B.

[0022] In some implementations, modeling the task relationships and common sense relationships between entities on the map to generate a heterogeneous common sense scene graph includes:

[0023] Generate all executable tasks for entities through a large language model and build a task-level graph; predict common-sense relationships between entities through a large language model and build an entity-level graph;

[0024] The task-level graph and the entity-level graph are fused to generate a heterogeneous common-sense scene graph.

[0025] In some implementations, before generating the heterogeneous common sense scene graph, the method further includes:

[0026] Based on the visual features corresponding to the entity, the similarity score is calculated and the explicit category label of the entity is determined;

[0027] The large language model is used to semantically classify entities whose maximum similarity scores are less than a preset score threshold.

[0028] In a second aspect, the present invention provides a human-machine mutual assistance system based on heterogeneous common sense scene graphs;

[0029] A human-machine mutual assistance system based on heterogeneous common sense scene graphs, comprising:

[0030] The entity extraction module is configured to: obtain the environment image sequence and extract key frames, extract entity information based on the key frames; consider the geometric position relationship and feature similarity of the entities, fuse the same entity under different key frames, and incrementally construct the map;

[0031] The heterogeneous common-sense scene graph construction module is configured to: model the task relationships and common-sense relationships between entities on the map, and generate heterogeneous common-sense scene graphs to assist robots in performing human-machine cooperative tasks.

[0032] In a third aspect, the present invention provides an electronic device;

[0033] An electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned method for generating heterogeneous common sense scene graphs for human-computer cooperation.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium;

[0035] A computer-readable storage medium stores a computer program / instruction thereon, which, when executed by a processor, implements the steps of the above-mentioned method for generating a heterogeneous common-sense scene graph for human-computer cooperation.

[0036] In a fifth aspect, the present invention provides a computer program product;

[0037] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned method for generating heterogeneous common sense scene graphs for human-computer cooperation.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The technical solution provided by the present invention provides a method for incrementally constructing maps, which can break through the limitations of closed data sets, accurately identify the locations and categories of entities in the environment, and incrementally fuse the same entity in multiple frames; at the same time, in order to address the problem of over-segmentation of the entity segmentation model, the point cloud geometric relationship and feature similarity are used to perform point cloud post-processing, and the fragmented point clouds are merged to construct a complete entity.

[0040] 2. The technical solution provided by the present invention uses big model technology to model common-sense relationships between entities, expresses task-oriented relationships between entities and more delicate common-sense relationships between two entities; it can break the limitations of closed data sets, construct heterogeneous common-sense scene graphs, and understand the common-sense relationships between entities in the map.

[0041] 3. The technical solution provided by the present invention uses a large model only in the stage of building a heterogeneous common sense scene graph. In the actual task planning and reasoning process, the system completely gets rid of its dependence on the large model and instead relies on the previously built heterogeneous common sense scene graph for reasoning and decision-making; it not only retains the powerful semantic understanding and reasoning capabilities brought by the large model, but also greatly reduces the system's operating costs and response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0043] Figure 1 A schematic diagram of a flow chart of a method for generating a heterogeneous common sense scene graph for human-machine cooperation provided by an embodiment of the present invention;

[0044] Figure 2An example diagram of entities in a scene provided by an embodiment of the present invention, wherein (a) is an example diagram of all entities in the scene before executing the cross-frame point cloud merging method (one color represents one entity), and (b) is an example diagram of all entities in the scene after executing the cross-frame point cloud merging method;

[0045] Figure 3 An example diagram of a task level diagram provided by an embodiment of the present invention;

[0046] Figure 4 An example diagram of an entity level diagram provided by an embodiment of the present invention;

[0047] Figure 5 An example diagram of a heterogeneous common-sense scene graph provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0049] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0050] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0051] Embodiment 1

[0052] Existing human-computer interaction methods lack common-sense understanding of objects in the environment, have limited understanding of human intentions, and rely on large models; therefore, the present invention provides a method for generating heterogeneous common-sense scene graphs for human-computer interaction, which breaks the limitations of closed data sets, constructs heterogeneous common-sense scene graphs, understands the common-sense relationships between entities in the map, identifies non-specific intentions from human behavior, and then performs corresponding actions.

[0053] Next, combine Figure 1-Figure 5, a method for generating a heterogeneous common sense scene graph for human-machine cooperation disclosed in this embodiment is described in detail. The method for generating a heterogeneous common sense scene graph for human-machine cooperation comprises the following steps:

[0054] S1. Obtain an environment image sequence, wherein the environment image sequence includes an RGB image, a depth image, and corresponding degrees of freedom posture.

[0055] In this embodiment, the environmental image sequence is collected by the robot during the process of pre-exploring the environment, and is expressed as:

[0056] I={I1,I2,I3,I4,…,I n};

[0057] I i ={rgb i ,depth i ,T i};

[0058] In the formula, rgb i Represents the RGB image of the i-th frame, depth i represents the depth image of the i-th frame, T i Represents the six-degree-of-freedom pose of the i-th frame.

[0059] S2. Extract key frames from the environment image sequence.

[0060] If the frequency of processing RGB images is too high, the information redundancy will increase rapidly, but this redundant information has very limited impact on the accuracy of the system, or even does not improve it, but consumes more computing resources. If simple sampling is adopted, some frames containing important information may be missed.

[0061] Therefore, in this embodiment, key frames are selected by calculating the overlap rate between frames, and only the key frames are processed subsequently.

[0062] Exemplarily, the frame overlap function is defined as:

[0063] p′(u′,v′)=K·T i,j ·D(p)·K -1 p(u,v);

[0064]

[0065] Where K is the camera internal parameter, T i,jis the transformation matrix between the i-th frame and the j-th frame, D is the depth of the pixel, p(u,v) is the pixel of the new incoming frame, p′(u′,v′) is the pixel after projecting p(u,v) to the target frame, (u′,v′) is the pixel coordinate of p(u,v), δ is the indicator function, which takes the value of 1 when the condition is true and 0 otherwise.

[0066] When the overlap-ratio between two frames is less than a preset threshold, the newly input frame is changed into a new key frame, and when a new frame is input, it is compared with the new key frame. This process is repeated to determine the key frame in the input sequence.

[0067] S3. Extract entity information based on key frames.

[0068] Traditional segmentation networks can only segment objects in closed data sets. Even open-vocabulary segmentation networks require a pre-defined semantic list. These two factors limit the robot's mapping capabilities and make it easy to miss some objects.

[0069] Therefore, in this embodiment, first, the RGB image rgb is segmented by a class-independent entity segmentation network. i Processing is performed to extract the class-independent entity-level mask M i ={m i,1 ,m i,2 ,...,m i,n} to ensure that the final map contains all objects in the environment, rather than being limited to objects in a closed dataset or a predefined semantic list.

[0070] Here, the class-independent entity segmentation network may be a SAM (Segment Anything Model) model or a CropFormer.

[0071] Then, for each m i,n The corresponding image area is processed using VLM (Visual Language Model) to extract its visual features F aligned with the text i ={f i,1 ,f i,2 ,...,f i,n}.

[0072] At the same time, in order to capture more extensive semantic information, the visual features of the same entity extracted at different scales are fused to obtain different levels of language embedding at the same 3D position (e.g., "tool" vs. "screwdriver"). The fused visual features are represented as:

[0073]

[0074] In the formula, represents the features extracted from the k-th scale image corresponding to the entity.

[0075] Finally, for m i,n The corresponding pixel point is based on the camera internal parameter K and the six-degree-of-freedom pose T of this frame i , get its point cloud p in the world coordinate system i,n , expressed as:

[0076] p i,n =T i ·K -1 ·m i,n ;

[0077] That is, for each m i,n A node will be generated i,n ={f i,n ,p i,n}.

[0078] S4. Considering the geometric position relationship and feature similarity of entities, the same entity under different key frames is fused to incrementally construct the map.

[0079] Before executing S4, set the mode in the first key frame to 1,n Add all to the map and complete the map initialization.

[0080] For example, when the i-th frame generates n corresponding nodes {node i,1 ,node i,2 ,...,node i,n}, assuming that the number of nodes in the map is m, it can be expressed as:

[0081] Map = {node map,1, node map,2, ...node map,m,}.

[0082] Then, S4 specifically includes:

[0083] S401, calculate the feature similarity matrix between the nodes not added to the key frame in the map and the nodes in the map, expressed as:

[0084]

[0085] S402, calculate the node distance matrix between the nodes not added to the key frame in the map and the nodes in the map, expressed as:

[0086] D∈R n×m ,D[n,m]=distacne(p i,n ,p map,m );

[0087] In the formula, distacne(p i,n ,p map,m ) indicates p i,n ,p map,m The distance between the centroids.

[0088] S403: When the feature similarity is greater than a preset feature threshold and the node distance is less than a preset distance threshold, merge the node and the existing node.

[0089] That is, when node i,n With node map,m satisfy:

[0090]

[0091] Then the node i,n With node map,m Merge, the node after the merger map,m for:

[0092] node map,m =(nf map,m +f i,n ) / (n+1);

[0093] p map,m ={p mao,m +p i,n}.

[0094] If for node i,n If no entity object in the map is found that satisfies the feature similarity greater than the preset feature threshold and the node distance less than the preset distance threshold, the node i,n Added to the map as a new entity.

[0095] S404, forming different point cloud combinations from point clouds of the same entity in different key frames, and calculating the inclusion rate of the point cloud combination; when the inclusion rate is greater than a preset inclusion rate threshold, merging the point clouds in the point cloud combination.

[0096] In the segmentation task based on RGB images, the generated mask is not always ideal, and often divides a complete entity into multiple fragments. This fragmentation phenomenon enables the visual language model (VLM) to capture only local features, resulting in different fragments of the same object presenting differentiated semantic features. In addition, even for spatially adjacent fragments, the distance between their centers of mass may be large, which increases the difficulty of merging fragments of the same entity. The inventors have observed that the fragment point clouds generated by the same frame usually do not overlap, while the fragment point clouds generated by different frames often have a relationship of inclusion or overlap. Based on this observation, in this embodiment, a method for merging cross-frame point clouds for the same entity is proposed.

[0097] Specifically, the ratio of point cloud A to point cloud B is expressed by the inclusion ratio. The inclusion ratio is expressed as

[0098]

[0099] In the formula, point A Represents point cloud A, point B represents point cloud B, represents the volume of point cloud A, Represents the volume of point cloud B.

[0100] For each possible pair of point clouds (point A ,point B ), calculate CR(point A ,point B ). In order to avoid mis-merging, a inclusion rate threshold CR is set threshold If CR(point A ,point B )>CR threshold , then it is considered that the entity point cloud A is sufficiently contained by the entity point cloud B, and the fragments belonging to the same object are merged and the boundaries are updated.

[0101] This strategy is executed repeatedly until there is no CR (point A ,point B )>CR threshold .

[0102] In this embodiment, the proposed cross-frame point cloud merging method shows significant effects in reducing fragmentation noise in the scene, especially for the reconstruction and merging of larger entities. This method effectively compensates for the fragmentation problem caused by incomplete local features in the single-frame segmentation process by capturing the inclusion relationship and overlapping characteristics of the fragmented point clouds between different frames. Figure 2 As shown in Figure 2. Cross-frame merging not only improves the overall coherence of the point cloud, but also reduces the generation of isolated fragments and noise points, thereby significantly improving the integrity and geometric accuracy of target objects in complex scenes. This strategy is particularly effective for the reconstruction of large entities and can better preserve the global structure of the object.

[0103] S5. Model the task relationships and common sense relationships between entities on the map to generate a heterogeneous common sense scene graph. Specifically, it includes:

[0104] S501. Calculate similarity scores and determine explicit category labels of entities based on visual features corresponding to the entities; and semantically classify entities whose maximum similarity scores are less than a preset score threshold using a large language model.

[0105] In order to cover all objects in the room during the mapping stage, in this embodiment, a class-independent entity segmentation network is used. Therefore, the entities in the map obtained through the above steps have no clear semantic information, only implicit semantic features aligned with text features.

[0106] In order to construct a heterogeneous common-sense scene graph, the LLM needs to be prompted, and the explicit semantic information of the node still needs to be known. Therefore, in this embodiment, a large semantic label set is selected to cover as many common objects in indoor environments as possible. Specifically, first, labelSet∈R of the lvis dataset is used. 1203 , which contains 1203 classes; then, the text encoder in VLM is used to pre-encode the 1203 texts to obtain the text feature matrix, which is expressed as:

[0107] F text ∈R 1023×1024 .

[0108] Then calculate the similarity between the visual features and text features of the entities in the map. If the maximum similarity score of the 1203 classes is greater than SimScore threshod , then labelSet[argmax] corresponding to the maximum value is the category of this entity.

[0109] If the maximum value is less than SimScore threshod There are three possibilities. The first possibility is noise (such as fragmented point clouds that are not correctly merged into the corresponding entity, textureless walls, etc.), the second possibility is a class that is not included in the predefined semantic label set, and the third possibility is SimScore threshod If the SimScore threshod If the value is set too small, the possibility of entities being misclassified increases. A predefined semantic label set cannot contain all objects that may appear indoors, even if the set is very large. If all entities with a maximum similarity score less than the threshold are filtered out, it is no different from directly using an open vocabulary detection model with a pre-set semantic list, and it is likely to miss some entities with clear semantic meanings, such as screws. This will have some negative impact on the subsequent construction of common sense scene graphs.

[0110] Now LLVM has shown powerful functions in image understanding, such as GPT4-V, llava, and BLIP. Therefore, in this embodiment, if the maximum value is less than SimScore threshod, then LLVM is used to classify entities (unsceneNode) whose maximum similarity score is less than the threshold. The topk views of unsceneNode, i.e., the k frames with the highest entity visibility, are selected and semantic classification is performed again to assign them clear categories.

[0111] unsceneNodeLabel=LLVM(SelectTopK(unsceneNode).

[0112] Finally, unsceneNodes for which LLVM returns NULL are considered noise.

[0113] If all nodes are classified using LLVM, poor results will be achieved. This is because LLVM tends to classify nodes into broader categories, such as classifying screwdrivers and wrenches as tools. Therefore, in order to identify all entities in the scene and avoid the drawbacks of LLVM classification as much as possible, the similarity score is less than SimScore. threshod The nodes are classified using LLVM.

[0114] In summary, all nodes in the map are divided into two types: entities with clear semantic types and entities without clear semantic categories. For the entity set with clear semantic types, a label attribute is added to it, that is, node map,n ={f map,n ,p map,n ,label map,n}.

[0115] S502: Generate all executable tasks for the entity through the large language model and construct a task level graph.

[0116] For example, the prompt "You are a robot with common sense knowledge. I will give you the objects in a scene, and you can tell me which tasks can be completed using a subset of them." is input into the large language model LLM, and several prompt examples are given to obtain the task level graph G Task .

[0117] Combination Figure 3 , G TaskIt is a graph containing three types of heterogeneous nodes. The first type of heterogeneous node is Room Node, which is unique and represented by R = {r1}, indicating a specific room. The second type of heterogeneous node is Task Node, which has multiple nodes and represented by T = {t1, t2, ..., t n ,}, indicating different tasks that can be performed in the scene; the third type of node is Entity Node, which also has multiple nodes, represented by E = {e1, e2, ..., e n ,} represents various entities in the scene.

[0118] Room Node is only connected to Task Node, indicating the association between room and task, defined as ε RT ={(r,t i )|t i ∈T}, representing the tasks that can be completed in the room. In addition to being connected to the Room Node, the Task Node is also connected to the EntityNode, which represents the set of entities required to complete the task, defined as ε TE ={(t i ,e j )|t i ∈T,e j ∈E}. That is, G Task It is expressed as:

[0119] G Task =({R,T,E,},{ε RT ,ε TE}).

[0120] Through this structure, LLM can systematically generate the relationship between tasks and entities, facilitating the optimization of task planning and human-computer interaction.

[0121] S503: Predict the common sense relationship between entities through a large language model and construct an entity level graph G Entity .

[0122] G Task It cannot fully express the common sense connection between objects in the scene. For example, there is obviously a common sense connection between tape and scissors, but G Task They were not classified under the same Task Node, and the connection between the two was ignored. The inventor tried many different prompts, but due to G Task The limitations of task orientation and the lack of common sense in constructing multidimensionality, G TaskOrganizing objects under specific tasks ignores the commonalities or implicit associations of objects outside of tasks, and ignores some common sense relationships between entities. For example, although tape and scissors do not directly belong to a specific task, they have a relationship of "being used for cutting or fixing" under higher-level common sense semantics. Ignoring some common sense relationships between entities focuses on the association between entities and tasks, and fails to capture these more subtle semantic connections.

[0123] Therefore, in this step, we continue to build the entity level graph G in the Entity Node collection. Entity To more fully express the common sense connections in the scene.

[0124] Specifically, the set of entities in the scene is input to the LLM to predict whether there is a commonsense relationship between every two entities; consistent with the previous step, the LLM will receive a brief prompt "You are a robot that understands the commonsense interaction relationships between every pair of ordered objects in the environment. Here is an ordered pair [headnode, tail node] for you. If there is a head that can act on the tail, please return 1 and tell me the reason. If not, please return 0 without telling me the reason. (You are a robot that understands the commonsense interaction relationships between every pair of ordered objects in the environment. Here is an ordered pair [head, tail] for you. If there is a head that can act on the tail, please return 1 and tell me the reason. If not, please return 0 without telling me the reason.)."

[0125] If there is a common sense relationship, an edge is used to connect the two entities. If there is no common sense relationship, no connection is made. Therefore, the output in this step is an undirected graph G Entity =(E,ε EE )=({e1,e2,…,e n ,},{{e i ,e j ,}|e i ,e j ∈E}).

[0126] S504: Fusing the task level graph and the entity level graph to generate a heterogeneous common sense scene graph G CommonScene , expressed as:

[0127] G CommonScene =G Task ∪G Entity =({R,T,E,},{ε RT ,ε TE ,ε EE}).

[0128] G Task and G Entity There is no substitution relationship. They complement each other in task planning and common-sense relationship expression and cannot replace each other alone. Combining the two can more comprehensively optimize task planning and human-computer interaction in the scene.

[0129] The fusion of the two retains the task-oriented characteristics while also having the ability to capture common-sense connections, forming a more robust and flexible heterogeneous common-sense scene graph.

[0130] S6. Use heterogeneous common-sense scene graphs to assist robots in performing human-robot cooperative tasks.

[0131] Next, taking the human-machine cooperative tasks of open vocabulary query and navigation, robot task planning and specific intention recognition as examples, the specific process of using heterogeneous common-sense scene graphs to assist robots in performing human-machine cooperative tasks is explained in detail.

[0132] As an implementation method, the specific process of using heterogeneous common sense scene graphs to assist robots in performing open vocabulary query and navigation tasks is as follows:

[0133] The constructed heterogeneous common sense scene graph contains N entity nodes node map,n ={f map,n ,p map,n ,label map,n Humans can enter any word as a query, not limited to the explicit labels obtained by the node classification module. The text encoder of VLM is used to encode the human input to obtain the feature feature. quary ∈R 1×1024 .

[0134] Feature quary With each entity node f map,n Calculate the cosine similarity. If it is greater than the threshold, the query node is considered to have been found. map,n The position of the node in the world coordinate system is calculated and used as the navigation end point. The system then plans a path based on the current position and the target position to guide the user or robot to the node.

[0135] As an implementation method, the specific process of using heterogeneous common sense scene graphs to assist robots in performing robot task planning is as follows:

[0136] Previous methods often rely on large models in the robot planning stage, which undoubtedly increases the cost of money and system computing. However, in this embodiment, the task planning can be effectively completed by simply executing the graph search method in the heterogeneous common sense scene graph constructed above.

[0137] G Task For planning high-level task structure and tool selection, G Entity It is used to refine the common sense operation steps between tools. When a human provides a task description (such as "assemble furniture"), it is treated as a query. The system then traverses the task nodes to find a node that matches the task. Once the corresponding task node is found, the system will activate all connected EntityNodes and treat them as necessary nodes required to complete the task.

[0138] In G Entity In the task planning, it is considered that the second-order neighbors may interfere with the task planning. For example, in the task of "building a computer workstation", the second-order neighbor node of the chair is "artifact", which obviously does not help the task substantially. Therefore, it is considered that only the first-order neighbors directly connected to the Entity Node have strong relevance, and these nodes will be regarded as optional nodes.

[0139] Taking the task of "assembling furniture" as an example, {furniture, screwdriver, screw, wrench} are considered as necessary nodes to complete the task, while {cart, broom} are considered as optional nodes. This method effectively separates the core requirements of the task from the additional auxiliary tools, improving the accuracy and flexibility of task planning.

[0140] Finally, the robot runs the navigation and grasping algorithms to obtain the corresponding nodes in the map.

[0141] As an implementation method, the specific process of using heterogeneous common sense scene graphs to assist robots in performing specific intent recognition tasks is as follows:

[0142] In the past, human-machine cooperative algorithms often focused on identifying specific human intentions, that is, training a classifier to classify the human intentions trained in the dataset and then perform the corresponding actions. Based on heterogeneous common sense scene graphs, non-specific intentions can be identified to better perform human-machine cooperative tasks.

[0143] The robot places the human at the center of the camera’s view and observes the human’s work process. The above entity classification method is used to process the robot’s first-person perspective observation. Next, the entity node e currently observed by the robot is activated in the heterogeneous common sense scene graph. i ∈Eobs In G CommonScene , activate task node t k The condition is that the proportion of activated nodes in the entity nodes related to the task reaches the threshold θ, that is:

[0144]

[0145] in, Represents task node t k The associated entity node set. If the condition is met, the task node t k is activated.

[0146] For each activated task node t k , if there are unactivated entity nodes and These nodes are regarded as nodes that humans may need, and are recorded as the node set to be obtained E req , expressed as:

[0147]

[0148] G Task Suitable for task-oriented human-machine cooperation, but weak in representing common sense relationships between entities. Task It is not possible to complete the task of human-machine cooperation well. Entity The execution graph search algorithm ensures more accurate reasoning of the person's work intention. We believe that two directly connected nodes have the closest common sense connection, and indirectly adjacent nodes have weak common sense connections. Therefore, in G Entity When searching on , only the first-order neighbors of the activated Entity-level nodes are searched. i ∈E obs , and its first-order neighbor set is: N(e i )={e j |{e i ,e j}∈ε EE}, then the alternative set E cand for:

[0149]

[0150] The final human intention node set E cand for:

[0151] E intent =E req ∪E cand .

[0152] For example, when the robot observes that a human is using {computer monitor, power strip, computer, mouse}, it activates the "build computer workstation" Task Node, and then searches for {keyboard, headphones} as a node that humans may need, and then takes {keyboard, headphones} as the human intention node set E intent ; When the robot observes that humans are using tape, no Task Node is activated, but in G Enttty We search on {scissors} and find that {scissors} is a node that humans may need, so we take {scissors} as the human intention node set E intent .

[0153] After determining the set of candidate nodes, wait for human feedback to determine the nodes that humans really need, and then perform navigation, grabbing, and returning actions.

[0154] Next, we evaluate the robot's scene graph-based application performance in a real indoor environment.

[0155] First, a simulated factory environment was built, which contains 21 objects. The number of wrong entities and missing entities was calculated to calculate the precision and recall. The results are shown in Table 1.

[0156] Table 1

[0157]

[0158]

[0159] Five volunteers were recruited to evaluate the accuracy of non-specific intent recognition in the human-machine cooperative method described in this embodiment. When the node that humans really need appears in the robot's candidate node set, the robot's non-specific intent recognition is considered successful.

[0160] The average evaluation result of 5 people is: 84.25%.

[0161] Embodiment 2

[0162] This embodiment discloses a human-machine mutual assistance system based on a heterogeneous common sense scene graph, including:

[0163] The entity extraction module is configured to: obtain the environment image sequence and extract key frames, extract entity information based on the key frames; consider the geometric position relationship and feature similarity of the entities, fuse the same entity under different key frames, and incrementally construct the map;

[0164] The heterogeneous common-sense scene graph construction module is configured to: model the task relationships and common-sense relationships between entities on the map, and generate heterogeneous common-sense scene graphs to assist robots in performing human-machine cooperative tasks.

[0165] It should be noted here that the above-mentioned entity extraction module and heterogeneous common sense scene graph construction module correspond to the steps in Example 1, and the examples and application scenarios implemented by the above-mentioned modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0166] Embodiment 3

[0167] Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned method for generating heterogeneous common sense scene graphs for human-computer cooperation are completed.

[0168] Embodiment 4

[0169] Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned method for generating heterogeneous common sense scene graphs for human-computer cooperation are completed.

[0170] Embodiment 5

[0171] Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned method for generating heterogeneous common sense scene graphs for human-computer cooperation.

[0172] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0175] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating heterogeneous common sense scene graphs for human-machine collaboration, characterized in that: include: Obtain the environment image sequence and extract key frames, and extract entity information based on the key frames; Considering the geometric position relationship and feature similarity of entities, the same entity in different keyframes is merged to incrementally construct the map; The task relationships and common-sense relationships between entities on the map are modeled to generate heterogeneous common-sense scene graphs to assist robots in performing human-robot cooperative tasks.

2. The method for generating heterogeneous common sense scene graphs for human-machine collaboration according to claim 1, characterized in that: The extracting entity information based on key frames specifically includes: Perform entity extraction on the RGB image in each keyframe, obtain the entity mask, and convert the entity mask into a point cloud in world coordinates using the camera intrinsic parameters and keyframe pose; The RGB image corresponding to the entity mask is encoded through the visual language model to obtain visual features aligned with the text features.

3. The method for generating heterogeneous common sense scene graphs for human-machine collaboration according to claim 1, characterized in that: The method of considering the geometric position relationship and feature similarity of entities and fusing the same entity in different key frames specifically includes: Calculate the feature similarity and node distance between the node in the key frame and the existing node in the map, and merge the node and the existing node when the feature similarity is greater than a preset feature threshold and the node distance is less than a preset distance threshold; The point clouds of the same entity in different key frames are combined into different point cloud combinations, and the inclusion rate of the point cloud combination is calculated; when the inclusion rate is greater than the preset inclusion rate threshold, the point clouds in the point cloud combination are merged.

4. The method for generating heterogeneous common sense scene graphs for human-machine collaboration according to claim 3, characterized in that: The inclusion rate is expressed as: In the formula, point A Represents point cloud A, point B represents point cloud B, represents the volume of point cloud A, Represents the volume of point cloud B.

5. The method for generating heterogeneous common sense scene graphs for human-machine collaboration according to claim 1, characterized in that: The step of modeling the task relationship and common sense relationship between entities on the map to generate a heterogeneous common sense scene graph includes: Generate all executable tasks for entities through a large language model and build a task-level graph; predict common-sense relationships between entities through a large language model and build an entity-level graph; The task-level graph and the entity-level graph are fused to generate a heterogeneous common-sense scene graph.

6. The method for generating heterogeneous common sense scene graphs for human-machine collaboration according to claim 1, characterized in that: Before generating heterogeneous common-sense scene graphs, it also includes: Based on the visual features corresponding to the entity, the similarity score is calculated and the explicit category label of the entity is determined; The large language model is used to semantically classify entities whose maximum similarity scores are less than a preset score threshold.

7. A human-machine mutual assistance system based on heterogeneous common sense scene graphs, characterized in that: include: The entity extraction module is configured to: obtain an environment image sequence and extract key frames, and extract entity information based on the key frames; Considering the geometric position relationship and feature similarity of entities, the same entity in different keyframes is merged to incrementally construct the map; The heterogeneous common-sense scene graph construction module is configured to: model the task relationships and common-sense relationships between entities on the map, and generate heterogeneous common-sense scene graphs to assist robots in performing human-machine cooperative tasks.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method for generating a heterogeneous common sense scene graph for human-computer cooperation as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the method for generating a heterogeneous common sense scene graph for human-computer cooperation as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method for generating a heterogeneous common sense scene graph for human-computer cooperation as described in any one of claims 1 to 6 are implemented.

Images