A method and system for generating heterogeneous common-sense scenario graphs for human-computer interaction

Abstract

This invention discloses a method and system for generating heterogeneous common-sense scene graphs for human-robot collaboration, belonging to the field of robotics technology. It includes: acquiring environmental image sequences and extracting keyframes; extracting entity information based on the keyframes; considering the geometric positional relationships and feature similarities of entities, fusing the same entity from different keyframes, and incrementally constructing a map; modeling the task relationships and common-sense relationships between entities on the map, and generating a heterogeneous common-sense scene graph to assist robots in performing human-robot collaborative tasks. It retains the powerful semantic understanding and reasoning capabilities of large models while significantly reducing system operating costs and response time; it solves the problem that existing semantic maps for human-robot collaboration cannot meet the real-world needs of human-robot collaboration.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method and system for generating heterogeneous common-sense scene graphs for human-machine collaboration. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] With the rapid development of robotics technology and the advancement of computer vision technology, robots are capable of performing an increasing number of tasks, such as navigation, grasping, and palletizing.

[0004] As robots develop human-like characteristics and their algorithmic capabilities continue to improve, along with the emergence of intelligent products such as AI robots, the possibility and necessity for AI and humans to coexist and collaborate are constantly increasing. In other words, it is difficult for humans to avoid contact with AI. Therefore, in this new context, the understanding of non-specific human intentions by robots and the construction of human-machine collaborative systems are becoming increasingly important.

[0005] The development of deep learning technology, especially the emergence of VLM (Visual Language Model) and LLM (Large Language Model), has provided solutions for this task. To accomplish human-robot collaborative tasks, robots first need to understand their environment. Existing methods generally use SLAM technology combined with object detection techniques (YOLO series) or semantic segmentation techniques (U-NET) to construct semantic maps. However, these maps are limited by closed datasets, only recognizing labeled objects in the dataset, exhibiting poor scalability and lacking common-sense understanding of objects in the environment. For understanding human intentions, existing methods employ neural network-based recognition technology, which also utilizes closed datasets—that is, a fixed human action to which the robot will make a fixed response.

[0006] Research on human-robot collaboration methods is limited to closed datasets and specific intentions. However, most real-life and production environments are unstructured, complex, and human actions are varied, making existing semantic maps for human-robot collaboration unable to meet real-world needs.

[0007] Furthermore, current human-machine collaborative systems or robot task planning systems typically rely on large-scale pre-trained models (such as large language models or deep learning models) for task planning and reasoning during the execution phase. This reliance leads to significant computational costs and time consumption, especially in scenarios involving high-frequency or complex tasks, where running these models can result in high expenses or significant time delays. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a method, system, electronic device, computer-readable storage medium, and computer program product for generating heterogeneous common-sense scenario graphs for human-computer interaction. It breaks the limitations of closed datasets, constructs heterogeneous common-sense scenario graphs, and utilizes them for reasoning and decision-making, thus eliminating the dependence on large models in the reasoning and decision-making process.

[0009] In a first aspect, the present invention provides a method for generating heterogeneous common-sense scene graphs for human-computer interaction;

[0010] A method for generating heterogeneous commonsense scene graphs for human-computer interaction includes:

[0011] Acquire environmental image sequences and extract keyframes, then extract entity information based on the keyframes;

[0012] Considering the geometric positional relationships and feature similarity of entities, the same entity under different keyframes is merged to incrementally build the map; the task relationships and common-sense relationships between entities on the map are modeled to generate heterogeneous common-sense scene maps to assist the robot in performing human-robot collaborative tasks.

[0013] In some implementations, the extraction of entity information based on keyframes specifically includes:

[0014] Entity extraction is performed on the RGB image in each keyframe to obtain the entity mask, and the entity mask is converted into a point cloud in world coordinates using camera intrinsics and keyframe pose.

[0015] The RGB image corresponding to the entity mask is encoded using a visual language model to obtain visual features aligned with text features.

[0016] In some implementations, the process of considering the geometrical positional relationships and feature similarities of entities and fusing the same entity across different keyframes specifically includes:

[0017] Calculate the feature similarity and node distance between nodes in keyframes and existing nodes in the map. When the feature similarity is greater than a preset feature threshold and the node distance is less than a preset distance threshold, merge the node with the existing node.

[0018] Point clouds of the same entity in different keyframes are combined into different point cloud combinations, and the inclusion rate of the point cloud combination is calculated. When the inclusion rate is greater than the preset inclusion rate threshold, the point clouds in the point cloud combination are merged.

[0019] In some implementations, the coverage rate is expressed as:

[0020]

[0021] In the formula, point A Represents point cloud A, pointB Representing point cloud B, This represents the volume of point cloud A. This represents the volume of point cloud B.

[0022] In some implementations, modeling the task relationships and common-sense relationships between entities on the map to generate a heterogeneous common-sense scene graph includes:

[0023] Generate all executable tasks for entities using a large language model and construct a task-level graph; predict common-sense relationships between entities using a large language model and construct an entity-level graph.

[0024] By merging task-level and entity-level graphs, a heterogeneous common-sense scenario graph is generated.

[0025] In some implementations, the process of generating a heterogeneous commonsense scene graph also includes:

[0026] Calculate similarity scores and determine explicit category labels for entities based on their visual features.

[0027] Semantic classification is performed on entities whose maximum similarity score is less than a preset score threshold using a large language model.

[0028] Secondly, the present invention provides a human-computer interaction system based on heterogeneous common-sense scene graphs;

[0029] A human-computer interaction system based on heterogeneous commonsense scene graphs includes:

[0030] The entity extraction module is configured to: acquire environmental image sequences and extract keyframes, extract entity information based on keyframes; consider the geometric positional relationship and feature similarity of entities, fuse the same entity under different keyframes, and incrementally construct a map;

[0031] The heterogeneous common-sense scene graph construction module is configured to model the task relationships and common-sense relationships between entities on the map and generate a heterogeneous common-sense scene graph to assist the robot in performing human-robot collaborative tasks.

[0032] Thirdly, the present invention provides an electronic device;

[0033] An electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method for generating heterogeneous common-sense scene graphs for human-computer interaction.

[0034] Fourthly, the present invention provides a computer-readable storage medium;

[0035] A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the above-described method for generating heterogeneous common-sense scene graphs for human-computer interaction.

[0036] Fifthly, the present invention provides a computer program product;

[0037] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the above-described method for generating heterogeneous common-sense scene graphs for human-computer interaction.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] 1. The technical solution provided by this invention provides an incremental map construction method that can break through the limitations of closed datasets, accurately identify the location and category of entities in the environment, and incrementally fuse the same entity in multiple frames. At the same time, in response to the problem of over-segmentation in entity segmentation models, point cloud post-processing is performed using point cloud geometric relationships and feature similarity to merge fragmented point clouds and construct complete entities.

[0040] 2. The technical solution provided by this invention uses large model technology to model the common-sense relationships between entities, expressing the task-oriented relationships between entities and the more nuanced common-sense relationships between pairs; it can break the limitations of closed datasets, construct heterogeneous common-sense scene graphs, and understand the common-sense relationships between entities in the map.

[0041] 3. The technical solution provided by this invention uses a large model only in the stage of constructing a heterogeneous common-sense scenario graph. In the actual task planning and reasoning process, the system completely gets rid of its dependence on the large model and instead relies on the previously constructed heterogeneous common-sense scenario graph for reasoning and decision-making; this not only retains the powerful semantic understanding and reasoning capabilities brought by the large model, but also significantly reduces the system's operating costs and response time. Attached Figure Description

[0042] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0043] Figure 1 This is a flowchart illustrating a method for generating heterogeneous common-sense scenario diagrams for human-computer interaction provided in an embodiment of the present invention.

[0044] Figure 2The following is an example diagram of entities in a scene provided in an embodiment of the present invention, wherein (a) is a diagram of all entities in the scene before the cross-frame point cloud merging method is executed (one color represents one entity), and (b) is a diagram of all entities in the scene after the cross-frame point cloud merging method is executed.

[0045] Figure 3 Example diagram of a task level diagram provided in an embodiment of the present invention;

[0046] Figure 4 Example diagram of entity-level diagram provided in embodiments of the present invention;

[0047] Figure 5 This is an example diagram of a heterogeneous common-sense scenario provided for an embodiment of the present invention. Detailed Implementation

[0048] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0049] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0050] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0051] Example 1

[0052] Existing human-computer interaction methods lack common-sense understanding of objects in the environment, have limited understanding of human intentions, and rely on large models. Therefore, this invention provides a heterogeneous common-sense scene graph generation method for human-computer interaction, which breaks the limitation of closed datasets, constructs heterogeneous common-sense scene graphs, understands the common-sense relationships of entities in the map, identifies non-specific intentions from human behavior, and then executes corresponding actions.

[0053] Next, combined Figures 1-5This embodiment provides a detailed description of a method for generating heterogeneous common-sense scenario graphs for human-computer interaction. The method includes the following steps:

[0054] S1. Obtain the environmental image sequence, which includes RGB images, depth images, and corresponding degrees of freedom poses.

[0055] In this embodiment, the environmental image sequence is collected by the robot during its pre-exploration of the environment, and is represented as follows:

[0056] I = {I1, I2, I3, I4, ..., I} n};

[0057] I i ={rgb i depth i ,T i};

[0058] In the formula, rgb i Represents the i-th frame of the RGB image, depth i T represents the depth image of the i-th frame. i This represents the six-degree-of-freedom pose of the i-th frame.

[0059] S2. Extract keyframes from the environmental image sequence.

[0060] Processing RGB images too frequently leads to a rapid increase in information redundancy. This redundancy has little to no effect on the system's accuracy, and may even fail to improve it, while consuming more computational resources. Conversely, simple sampling may miss frames containing important information.

[0061] Therefore, in this embodiment, key frames are selected by calculating the overlap rate between frames, and only the key frames are processed subsequently.

[0062] For example, the frame overlap function is defined as follows:

[0063] p′(u′,v′)=K·T i,j ·D(p)·K -1 ·p(u,v);

[0064]

[0065] In the formula, K is the camera intrinsic parameter, and T is... i,jLet be the transformation matrix between the i-th frame and the j-th frame, D be the depth of the pixel, p(u,v) be the pixel of the newly input frame, p′(u′,v′) be the pixel of the target frame after p(u,v), (u′,v′) be the pixel coordinates of p(u,v), and δ be the indicator function, which takes the value 1 when the condition is true and 0 otherwise.

[0066] When the overlap ratio between two frames is less than a preset threshold, the newly received frame becomes a new keyframe. When another new frame is received, it is compared with the new keyframe. This process is repeated to determine the keyframes in the incoming sequence.

[0067] S3. Extract entity information based on keyframes.

[0068] Traditional segmentation networks can only segment objects in closed datasets. Even open-vocabulary segmentation networks require a predefined semantic list. Both of these limitations restrict the robot's mapping capabilities, making it easy for it to miss some objects.

[0069] Therefore, in this embodiment, firstly, the RGB image is processed by a class-independent entity segmentation network. i Process the code to extract the class-independent entity-level mask M. i ={m i,1 ,m i,2 ,...,m i,n This ensures that the final map includes all objects in the environment, rather than being limited to objects in a closed dataset or a predefined semantic list.

[0070] Here, the class-independent entity segmentation network can be either the SAM (Segment Anything Model) model or CropFormer.

[0071] Then, for each m i,n The corresponding image region is processed using a Visual Language Model (VLM) to extract its visual features F that align with the text. i ={f i,1 ,f i,2 ,...,f i,n}

[0072] Simultaneously, to capture broader semantic information, visual features of the same entity extracted at different scales are fused to obtain language embeddings at different levels at the same 3D location (e.g., "tool" and "screwdriver"). The fused visual features are represented as follows:

[0073]

[0074] In the formula, This represents the features extracted from the k-th scale image corresponding to the entity.

[0075] Finally, for m i,n The corresponding pixel is determined by the camera intrinsic parameter K and the six-DOF pose T of this frame. i The point cloud p in the world coordinate system is obtained. i,n , represented as:

[0076] p i,n =T i ·K -1 ·m i,n ;

[0077] That is, for each m i,n A node will be generated i,n ={f i,n ,p i,n}

[0078] S4. Considering the geometric positional relationship and feature similarity of entities, merge the same entity under different keyframes and incrementally build the map.

[0079] Before executing S4, set the mode in the first keyframe. 1,n Add everything to the map to complete map initialization.

[0080] For example, when the i-th frame generates n corresponding nodes {node i,1 ,node i,2 ,...,node i,n}, assuming the number of nodes in the map is m, it can be represented as:

[0081] Map = {node map,1, node map,2, ...node map,m,}

[0082] Therefore, S4 specifically includes:

[0083] S401. Calculate the feature similarity matrix between nodes not included in the keyframes of the map and nodes in the map, as follows:

[0084]

[0085] S402. Calculate the node distance matrix between nodes not included in the keyframes of the map and nodes in the map, expressed as:

[0086] D∈R n×m ,D[n,m]=distacne(p i,n ,p map,m );

[0087] In the formula, distacne(p i,n ,p map,m ) represents p i,n ,p map,m The distance between the centers of mass.

[0088] S403. When the feature similarity is greater than the preset feature threshold and the node distance is less than the preset distance threshold, merge the node with the existing node.

[0089] That is, when node i,n With node map,m satisfy:

[0090]

[0091] Then node i,n With node map,m Merge the nodes, and the merged nodes will be named as follows: map,m for:

[0092] node map,m =(nf map,m +f i,n ) / (n+1);

[0093] p map,m ={p mao,m +p i,n}

[0094] If for node i,n If no entity object is found in the map that satisfies both a feature similarity greater than a preset feature threshold and a node distance less than a preset distance threshold, then the node will be... i,n It is added to the map as a new entity.

[0095] S404. Combine the point clouds of the same entity in different keyframes into different point cloud combinations and calculate the inclusion rate of the point cloud combination. When the inclusion rate is greater than the preset inclusion rate threshold, merge the point clouds in the point cloud combination.

[0096] In RGB image-based segmentation tasks, the generated masks are not always ideal, often dividing a complete entity into multiple fragments. This fragmentation phenomenon allows visual language models (VLMs) to capture only local features, resulting in different fragments of the same object exhibiting differentiated semantic features. Furthermore, even spatially adjacent fragments may have significant distances between their centroids, increasing the difficulty of merging fragments of the same entity. The inventors observed that fragment point clouds generated within the same frame typically do not overlap, while fragment point clouds generated across different frames often exhibit inclusion or overlap relationships. Based on this observation, this embodiment proposes a cross-frame point cloud merging method for the same entity.

[0097] Specifically, the containment ratio represents the proportion of point cloud A contained within point cloud B. The containment ratio is expressed as...

[0098]

[0099] In the formula, point A Represents point cloud A, point B Representing point cloud B, This represents the volume of point cloud A. This represents the volume of point cloud B.

[0100] For each possible pair of merged point cloud combinations (point A ,point B ), calculate CR (point) A ,point B To avoid erroneous merging, a coverage rate threshold CR is set. threshold If CR(point) A ,point B )>CR threshold If entity point cloud A is sufficiently contained by entity point cloud B, and belongs to fragments of the same object, then it is considered to be merged and its boundary updated.

[0101] This strategy is executed repeatedly until there are no more CRs (points). A ,point B )>CR threshold .

[0102] In this embodiment, the proposed cross-frame point cloud merging method demonstrates a significant effect in reducing fragmentation noise in the scene, especially showing a clear advantage for the reconstruction and merging of larger entities. This method effectively compensates for the fragmentation problem caused by incomplete local features during single-frame segmentation by capturing the inclusion relationship and overlap characteristics of fragmented point clouds between different frames, achieving the desired effect. Figure 2 As shown, cross-frame merging not only improves the overall coherence of the point cloud but also reduces the generation of isolated fragments and noise points, thereby significantly improving the integrity and geometric accuracy of target objects in complex scenes. This strategy is particularly effective for reconstructing large entities, better preserving the global structure of the object.

[0103] S5. Model the task relationships and common-sense relationships between entities on the map to generate heterogeneous common-sense scene graphs. Specifically, this includes:

[0104] S501. Calculate the similarity score and determine the explicit category label of the entity based on the visual features corresponding to the entity; perform semantic classification on entities with a maximum similarity score less than a preset score threshold using a large language model.

[0105] In order to cover all objects in the room during the mapping phase, this embodiment uses a class-independent entity segmentation network. Therefore, the entities in the map obtained through the above steps do not have explicit semantic information, but only implicit semantic features aligned with text features.

[0106] To construct a heterogeneous commonsense scene graph, prompts are needed for the LLM, requiring knowledge of the explicit semantic information of the nodes. Therefore, in this embodiment, a large semantic label set is chosen to cover as many common objects in the indoor environment as possible. Specifically, firstly, the labelSet∈R of the lvis dataset is used. 1203 This contains 1203 classes; then, using the text encoder in VLM, the 1203 texts are pre-encoded to obtain the text feature matrix, represented as:

[0107] F text ∈R 1023×1024 .

[0108] Next, the visual features and text features of entities in the map are compared to calculate their similarity. If the maximum similarity score with any of the 1203 classes is greater than the SimScore, then the similarity is calculated. threshod If the maximum value corresponds to the labelSet[argmax], then the labelSet[argmax] is the category of this entity.

[0109] If the maximum value is less than SimScore threshod There are three possibilities: first, it could be noise (such as fragmented point clouds that failed to be correctly merged into the corresponding entity, or textureless walls); second, it could be a class not present in the predefined semantic label set; and third, it could be SimScore. threshod The setting is too high. If SimScore is set too high... threshod Setting the threshold too low increases the likelihood of entities being misclassified. A predefined set of semantic labels cannot possibly encompass all objects that might appear indoors, even if the set is very large. Filtering out all entities with a maximum similarity score below the threshold is no different from using an open-vocabulary detection model with a pre-defined semantic list, and it's likely to miss some entities with clear semantic meaning, such as screws. This will negatively impact the subsequent construction of common-sense scene graphs.

[0110] LLVM now demonstrates powerful capabilities in image understanding, as seen with GPT4-V, llava, and BLIP. Therefore, in this embodiment, if the maximum value is less than SimScore... threshodThen, LLVM is used to classify entities (unsceneNodes) whose maximum similarity score is less than a threshold. The topk views of the unsceneNodes, i.e., the k frames with the highest entity visibility, are selected for semantic classification again, and they are assigned a clear category.

[0111] unsceneNodeLabel=LLVM(SelectTopK(unsceneNode).

[0112] Ultimately, unsceneNodes that LLVM returns NULL are considered noise.

[0113] Using LLVM to classify all nodes will yield poor results because LLVM tends to classify nodes into broader categories, such as classifying both screwdrivers and wrenches as tools. Therefore, to identify all entities in the scene and minimize the drawbacks of LLVM classification, nodes with similarity scores lower than SimScore should be classified. threshod The nodes are classified using LLVM.

[0114] In summary, all nodes in the map are divided into two types: entities with explicit semantic types and entities without explicit semantic categories. For the set of entities with explicit semantic types, a label attribute, i.e., node, is added. map,n ={f map,n ,p map,n ,label map,n}

[0115] S502. Generate all executable tasks for entities using a large language model and construct a task-level graph.

[0116] For example, inputting the prompt "You are a robot with common sense knowledge. I will give you the objects in a scene, and you can tell me which tasks can be completed using a subset of them" into the large language model LLM, along with several example prompts, yields the task-level graph G. Task .

[0117] Combination Figure 3 G TaskConsider a graph containing three types of heterogeneous nodes. The first type is the Room Node, which is unique and denoted by R = {r1}, representing a specific room. The second type is the Task Node, which can have multiple nodes, denoted by T = {t1, t2, ..., t3}. n The first type of node is represented by ,}, which indicates different tasks that can be performed in this scenario; the third type of node is the Entity Node, which also has multiple nodes, denoted by E = {e1, e2, ..., e}. n ,} represents various entities in the scene.

[0118] A Room Node is connected only to a Task Node, representing the association between a room and a task, and is defined as ε. RT ={(r,t i )|t i ∈T} represents the tasks that can be completed in that room. In addition to being connected to the Room Node, the Task Node is also connected to the Entity Node, representing the set of entities required to complete the task, defined as ε. TE ={(t i ,e j )|t i ∈T,e j ∈E}. That is, G Task Represented as:

[0119] G Task =({R,T,E,},{ε RT ,ε TE}).

[0120] Through this structure, LLM can systematically generate relationships between tasks and entities, facilitating the optimization of task planning and human-computer interaction.

[0121] S503. Predict common-sense relationships between entities using a large language model and construct an entity-level graph G. Entity .

[0122] G Task It cannot fully express the common-sense connections between objects in the scene. For example, there is an obvious common-sense connection between tape and scissors, but G... Task They weren't grouped under the same Task Node, ignoring the connection between them. The inventors tried various prompts, but due to G... Task The limitations of task-oriented approach and the lack of common-sense connections in the multi-dimensional nature of G during construction TaskOrganizing objects under specific tasks ignores their commonalities or implicit connections outside of the task, and overlooks some common-sense relationships between entities. For example, although tape and scissors do not directly belong to a specific task, they share a relationship of "being used together for cutting or fixing" at a higher level of common-sense semantics. Ignoring some common-sense relationships between entities focuses on the association between entities and tasks, failing to capture these more subtle semantic connections.

[0123] Therefore, in this step, we continue to build the entity-level graph G in the Entity Node collection. Entity To more comprehensively express the common-sense connections in the scenario.

[0124] Specifically, the set of entities in the scene is input into the LLM to predict whether a commonsense relationship exists between any two entities. Consistent with the previous step, the LLM receives a brief prompt: "You are a robot that understands the commonsense interaction relationships between every pair of ordered objects in the environment. Here is an ordered pair [head, tail] for you. If there is a head that can act on the tail, please return 1 and tell me the reason. If not, please return 0 without explaining the reason."

[0125] If a common-sense relationship exists, an edge is used to connect the two entities; otherwise, no connection is made. Therefore, the output of this step is an undirected graph G. Entity =(E,ε EE )=({e1,e2,…,e n ,},{{e i ,e j ,}|e i ,e j ∈E}).

[0126] S504. Merge the task-level graph and the entity-level graph to generate a heterogeneous common-sense scenario graph G. CommonScene , represented as:

[0127] G CommonScene =G Task ∪G Entity =({R,T,E,},{ε RT ,ε TE ,ε EE}).

[0128] G Task and G Entity There is no substitution relationship between them; they complement each other in task planning and common-sense relationship expression, and neither can replace the other alone. Combining the two can more comprehensively optimize task planning and human-computer interaction in the scenario.

[0129] By combining the two, the task-oriented characteristics are preserved, while the ability to capture common-sense connections is also gained, forming a more robust and flexible heterogeneous common-sense scenario diagram.

[0130] S6. Utilize heterogeneous common-sense scene diagrams to assist robots in performing human-robot collaborative tasks.

[0131] Next, taking human-machine collaborative tasks as examples of open vocabulary query and navigation, robot task planning, and specific intent recognition, we will explain in detail the specific process of using heterogeneous common-sense scene graphs to assist robots in performing human-machine collaborative tasks.

[0132] As one implementation method, the specific process of using heterogeneous common-sense scene graphs to assist robots in performing open vocabulary lookup and navigation tasks is as follows:

[0133] The constructed heterogeneous common-sense scenario graph contains N entity nodes. map,n ={f map,n ,p map,n ,label map,n Humans can input any words as queries, not limited to the explicit labels obtained from the node classification module. The VLM text encoder encodes the human input to obtain features. quary ∈R 1×1024 .

[0134] Feature quary f with each entity node map,n Calculate the cosine similarity; if it exceeds a threshold, the query node is considered to have been found. Then, from its p... map,n The system calculates the node's position in the world coordinate system and uses it as the navigation endpoint. The system then plans a path based on the current and target positions to guide the user or robot to that node.

[0135] As one implementation method, the specific process of using heterogeneous common-sense scene graphs to assist robots in performing robot task planning is as follows:

[0136] Traditional methods often rely on large models during the robot planning phase, which undoubtedly increases financial and computational costs. However, in this embodiment, task planning can be effectively completed simply by executing a graph search method on the heterogeneous commonsense scene graph constructed above.

[0137] G Task Used for planning high-level task structures and tool selection, while G Entity This is used to refine the common-sense operational steps between tools. When a human provides a task description (such as "assemble furniture"), it is treated as a query. The system then traverses the task nodes, searching for nodes that match the task. Once a corresponding task node is found, the system activates all connected EntityNodes, treating them as necessary nodes to complete the task.

[0138] In G Entity In this context, we consider that second-order neighbors may interfere with task planning. For example, in the task of "building a computer workstation," the second-order adjacent node of the chair is the "workpiece," which obviously does not provide any substantial help to the task. Therefore, we assume that only first-order neighbors directly connected to the Entity Node have strong relevance, and these nodes will be considered optional nodes.

[0139] Taking the task of "assembling furniture" as an example, {furniture, screwdriver, screw, wrench} are considered necessary nodes for completing the task, while {trolley, broom} are considered optional nodes. This method effectively distinguishes between the core requirements of the task and the additional auxiliary tools, improving the accuracy and flexibility of task planning.

[0140] Finally, the robot runs navigation and grasping algorithms to obtain the corresponding nodes on the map.

[0141] As one implementation method, the specific process of using heterogeneous common-sense scene graphs to assist robots in performing specific intent recognition tasks is as follows:

[0142] Traditional human-computer interaction algorithms often focus on recognizing specific human intentions, i.e., training a classifier to categorize human intentions trained in the dataset and then executing the corresponding actions. Heterogeneous commonsense scene graphs can be used to recognize non-specific intentions, thus enabling better performance of human-computer interaction tasks.

[0143] The robot positions a human at the center of the camera's viewpoint to observe the human's work process. The entity classification method described above is used to process the robot's first-person perspective. Next, the entity node e currently observed by the robot is activated in the heterogeneous commonsense scene graph. i ∈Eobs In G CommonScene In the middle, activate task node t k The condition is that the proportion of active nodes among the entity nodes related to the task reaches a threshold θ, that is:

[0144]

[0145] in, Represents task node t k The set of associated entity nodes. If the condition is met, task node t k Activated.

[0146] For each active task node t k If there are any inactive entity nodes among them and These nodes are then considered as nodes that humans might need, denoted as the set E of nodes to be acquired. req , represented as:

[0147]

[0148] G Task Suitable for task-oriented human-computer interaction, but weak in representing common-sense relationships between pairs of entities, using only G. Task It cannot perform human-machine collaborative tasks well. Therefore, it is also in G Entity The execution graph search algorithm is used to ensure more accurate inferences about the user's intentions. We consider directly connected nodes to have the strongest common-sense connection, while indirectly adjacent nodes have a weaker common-sense connection. Therefore, in G... Entity When searching, only first-order neighbors of active entity-level nodes are sought. For each e i ∈E obs Let its first-order neighbor set be: N(e i )={e j |{e i ,e j}∈ε EE}, then the candidate set E cand for:

[0149]

[0150] The final set of human intention nodes E cand for:

[0151] E intent =E req ∪E cand .

[0152] For example, when the robot observes that a human is using {computer monitor, power strip, computer, mouse}, it activates the "Build a computer workstation" Task Node. It then searches for {keyboard, headphones} as nodes that the human might need, and adds {keyboard, headphones} as the human intent node set E. intent When the robot observes that a human is using tape, no Task Nodes are activated, but in G... Enttty The search revealed that {scissors} is a node that humans might need, so {scissors} is included in the human intent node set E. intent .

[0153] After determining the set of candidate nodes, wait for human feedback to determine the node that the human actually needs, and then execute navigation, grabbing, and returning actions.

[0154] Next, we will evaluate the robot's performance based on scene maps in a real indoor environment.

[0155] First, a simulated factory environment was built, containing 21 objects. The number of erroneous and missing entities was calculated to determine accuracy and recall, and the results are shown in Table 1.

[0156] Table 1

[0157]

[0158]

[0159] Five volunteers were recruited to evaluate the accuracy of non-specific intent recognition in the human-machine interaction method described in this embodiment. When the node that the human actually needs appears in the robot's candidate node set, the robot's non-specific intent recognition is considered successful.

[0160] With an average of 5 reviews, the final success rate was 84.25%.

[0161] Example 2

[0162] This embodiment discloses a human-computer interaction system based on heterogeneous common-sense scenario graphs, including:

[0163] The entity extraction module is configured to: acquire environmental image sequences and extract keyframes, extract entity information based on keyframes; consider the geometric positional relationship and feature similarity of entities, fuse the same entity under different keyframes, and incrementally construct a map;

[0164] The heterogeneous common-sense scene graph construction module is configured to model the task relationships and common-sense relationships between entities on the map and generate a heterogeneous common-sense scene graph to assist the robot in performing human-robot collaborative tasks.

[0165] It should be noted that the entity extraction module and the heterogeneous commonsense scenario graph construction module described above correspond to the steps in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0166] Example 3

[0167] Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps of the above-described method for generating heterogeneous common sense scene graphs for human-computer interaction.

[0168] Example 4

[0169] Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above-described method for generating heterogeneous common-sense scenario diagrams for human-computer interaction.

[0170] Example 5

[0171] Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described method for generating heterogeneous common sense scenario graphs for human-computer interaction.

[0172] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0173] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0175] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0176] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating heterogeneous common-sense scenario graphs for human-robot collaboration, characterized in that, The method comprises the following steps: obtaining an environment image sequence and extracting key frames, and extracting entity information based on the key frames; considering the geometric position relationship and feature similarity of the entity, fusing the same entity under different key frames, and incrementally constructing a map; wherein the point clouds of the same entity in different key frames form different point cloud combinations, and the inclusion rate of the point cloud combination is calculated; when the inclusion rate is greater than a preset inclusion rate threshold, the point clouds in the point cloud combination are merged; the inclusion rate is represented as ； In the formula, denotes a point cloud A, denotes a point cloud B, denotes a volume of the point cloud A, denotes a volume of the point cloud B; For each pair of possibly merged point clouds combination , compute ; to avoid false merging, set a containment ratio threshold ; if , consider entity point cloud A is sufficiently contained by entity point cloud B, belongs to the same object's fragment, merge and update the boundary; This strategy is executed in a loop until there are ; modeling the task relationship and common sense relationship between entities on the map to generate a heterogeneous common sense scene graph, comprising: generating all executable tasks for the entity through a large language model to construct a task level graph; predicting the common sense relationship between entities through a large language model to construct an entity level graph; fusing the task level graph and the entity level graph to generate a heterogeneous common sense scene graph; using the heterogeneous common sense scene graph to assist a robot in performing a human-robot interaction task.

2. The heterogeneous commonsense scenario graph generation method for human-machine co-operation of claim 1, wherein, The entity information extraction based on the key frames specifically comprises: performing entity extraction on the RGB image in each key frame to obtain an entity mask, and converting the entity mask into a point cloud in the world coordinate system using the camera intrinsic parameter and the key frame pose; encoding the RGB image corresponding to the entity mask through a visual language model to obtain visual features aligned with text features.

3. The heterogeneous commonsense scenario graph generation method for human-machine co-operation of claim 1, wherein, The consideration of the geometric position relationship and feature similarity of the entity, and the fusion of the same entity under different key frames specifically comprises: calculating the feature similarity and node distance of the node in the key frame and the existing node in the map, and merging the node and the existing node when the feature similarity is greater than a preset feature threshold and the node distance is less than a preset distance threshold.

4. The heterogeneous commonsense scenario graph generation method for human-machine symbiosis of claim 1, wherein, Before generating the heterogeneous common sense scene graph, it further comprises: calculating a similarity score and determining an explicit category label of the entity according to the visual features corresponding to the entity; performing semantic classification on the entity with the maximum similarity score less than a preset score threshold through a large language model.

5. A human-machine assist system based on heterogeneous common-sense scenario graphs, characterized in that, The method comprises the following steps: an entity extraction module configured to obtain an environment image sequence and extract key frames, and extract entity information based on the key frames; considering the geometric position relationship and feature similarity of the entity, fusing the same entity under different key frames, and incrementally constructing a map; wherein the point clouds of the same entity in different key frames form different point cloud combinations, and the inclusion rate of the point cloud combination is calculated; when the inclusion rate is greater than a preset inclusion rate threshold, the point clouds in the point cloud combination are merged; the inclusion rate is represented as ； wherein represents a point cloud A, represents a point cloud B, represents a volume of the point cloud A, represents a volume of the point cloud B; For each pair of possibly merged point clouds combination , compute ; to avoid false merging, set a containment ratio threshold ; if , consider entity point cloud A is sufficiently contained by entity point cloud B, belongs to the same object's fragment, merge and update the boundary; This strategy is executed in a loop until there are ; a heterogeneous common sense scene graph construction module configured to model the task relationship and common sense relationship between entities on the map to generate a heterogeneous common sense scene graph, comprising: generating all executable tasks for the entity through a large language model to construct a task level graph; predicting the common sense relationship between entities through a large language model to construct an entity level graph; fusing the task level graph and the entity level graph to generate a heterogeneous common sense scene graph; using the heterogeneous common sense scene graph to assist a robot in performing a human-robot interaction task. The processor executes the computer program to realize the steps of the method for generating a heterogeneous common sense scene graph for human-robot interaction according to any one of claims 1-4.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-5. ​ 7. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the steps of the method for generating a heterogeneous common-sense scenario graph for human-machine collaboration according to any one of claims 1-4.

8. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the steps of the method for generating a heterogeneous common-sense scenario graph for human-machine collaboration according to any one of claims 1-4.

Images