A scene graph-based visual scene understanding system and method for robots

Abstract

The application discloses a kind of scene graph-based visual scene understanding system and method for robot, system includes: color picture acquisition and calling module, scene graph relationship identification module, crowd relationship prediction module;Method includes: through color picture acquisition and calling module, the real-time image in application scene is obtained;Object classification module calculates the class to which object belongs according to the feature of object, scene graph relationship identification module is associated with the combination of all objects and relationship, generates scene graph;Through relationship classification module, the reasonable relationship class between two objects is calculated;Crowd division module carries out reasonable crowd division according to the relationship class between character and character in entire scene;According to the result of crowd division and the relationship class between character, VIP prediction module carries out relevant relationship fusion to each person in crowd, and according to fusion result, judge the importance of each person in crowd, finally obtain VIP in this crowd.

Description

Technical Field

[0001] This invention relates to the field of visual scene understanding, and in particular to a scene graph-based visual scene understanding system and method for robot exhibition hall scenarios. Background Technology

[0002] A robot's effective human-computer interaction relies heavily on a correct understanding of the scene, and crowd relationship analysis is a crucial part of scene understanding. To interact with crowds, the robot needs to obtain visual characteristics and determine semantic relationships based on visual information. For example, if multiple people are standing in front of the robot, it needs to determine whether they know each other and belong to the same group. Additionally, within a group, there may be important figures that the robot needs to identify and focus on interacting with. However, due to the randomness, diversity, unstructured nature, and dynamic changes over time of crowds, predicting crowd relationships is a very challenging task.

[0003] Previous work has utilized vision-based scene graph methods to detect and associate objects within a scene. While these methods can identify objects in images and establish reasonable relationships, limitations in datasets and the lack of defined usage environments prevent them from effectively recognizing objects in specific scenarios or identifying relationships between people. Furthermore, the exploration and application of scene graphs in understanding social relationships are also lacking. In addition, while traditional social relationship models provide various relationships between people, they cannot achieve online object detection, proactively segment crowds, or differentiate the importance of each individual within the overall group. Therefore, achieving real-time, flexible, and reliable crowd relationship analysis is a crucial problem that urgently needs to be solved to improve the robot's environmental perception capabilities in human-computer interaction. Summary of the Invention

[0004] To address the shortcomings of existing technologies and achieve the goals of online object detection, identification of interpersonal relationships, and assessment of the importance of tasks within a population, this invention adopts the following technical solution:

[0005] A scene graph-based visual scene understanding system for robots includes: a color image acquisition and retrieval module (10), a scene graph relationship recognition module (11), and a crowd relationship prediction module (12). The color image acquisition and retrieval module (10) acquires color images in real time and allows other parts of the system to retrieve them. The scene graph relationship recognition module (11) retrieves images from the color image acquisition and retrieval module (10) and inputs them into a trained deep learning network. It outputs the detected object positions, predicted object categories, and relationships between objects in each scene image, generating a scene graph of objects and relationships for scene understanding. The crowd relationship prediction module (12) classifies objects into multiple crowds based on object and relationship classification required by the application scenario. The relationships between people within a crowd are strong, while the relationships between crowds are weak. Within a single crowd, the importance of each person in the crowd is predicted by fusing the relationships to which each person belongs, and the most important person in the crowd is identified as the most important person in the crowd.

[0006] The scene graph relationship recognition module (11) includes an object detection module (110), an object classification module (111), a relationship classification module (112), an object category library (113) for the exhibition hall scene, and a relationship category library (114) for the exhibition hall scene. The object detection module (110) can use deep learning methods to input the images called by the color image acquisition and calling module (10) into the trained deep learning network to extract the image object features and object positions. Each detected object has independent object features and object positions. The object classification module (111) can use deep learning methods to pass the object features through the trained deep learning network to obtain the predicted object category. The relationship classification module (112) can use deep learning methods to use the color images, object features, object positions, and object classifications obtained from the color image acquisition and calling module (10), the object detection module (110), and the object classification module (111) to predict and output the relationship between pairs of objects through the trained deep learning network.

[0007] The object features extracted by the scene graph relationship recognition module (11) are represented by a vector, and the object position is represented by the upper left and lower right pixel coordinates of the object's target box in the image.

[0008] The object classification module (111) generates object categories from the object category library (113) in the exhibition hall scene. The object categories in the object category library (113) in the exhibition hall scene include: person, screen, board, plant, camera, table, and exhibit.

[0009] The object relationships generated by the relationship classification module (112) come from the relationship category library (114) in the exhibition hall scene. The relationship categories in the relationship category library (114) in the exhibition hall scene include: looking at, next to, talk to, align with, back to, pass by, hold hands with, sit together with, taking a photo of, pointing at, and shake hands with. The categories of relationships between people and other objects include: looking at, next to, back to, pass by, taking a photo of, pointing at, etc.; and the categories of relationships between people include: looking at, next to, talk to, align with, back to, pass by, hold hands with, sit together with, pointing at, shake hands with, etc.

[0010] The crowd relationship prediction module (12) includes a crowd segmentation module (120) and a VIP prediction module (121). The crowd segmentation module (120) distinguishes the crowds by analyzing and comparing the closeness of relationships between people obtained by the scene graph relationship recognition module (11). The VIP prediction module (121) integrates the relationships of each person in each crowd segmented by the crowd segmentation module (120), and judges the importance of each person in the crowd based on the integration results, and finally determines the important people in the crowd.

[0011] A scene graph-based visual scene understanding method for robots includes the following steps:

[0012] Step S1: Obtain real-time images in the application scenario through the color image acquisition and calling module (10), and ensure that other modules can call these images in a reasonable time order. Use the object detection module (110) to detect the object features and their positions in the images.

[0013] In step S2, the object classification module (111) calculates the category to which the object belongs based on the object characteristics through the trained deep learning network, and the scene graph relationship recognition module (11) associates and combines all objects with relationships to generate a scene graph.

[0014] Step S3: Based on the scene graph generated by the above method, calculate the relationship category between each pair of objects through the relationship classification module (112).

[0015] Step S4, the crowd segmentation module (120) performs reasonable crowd segmentation based on the objects in the entire scene, namely, people and the relationship categories between people, including the following steps:

[0016] Step S400: There are K people in the preset scene;

[0017] Step S401: Randomly select one individual i0 from each group as the initial state. In this case, the individuals selected in the scene diagram cannot have any direct relationship with each other.

[0018] Step S402: Based on the scene diagram, place individuals who are directly related to individuals in the K groups into their corresponding groups;

[0019] Step S403: Check the categorized individuals. If an individual belongs to multiple groups, classify the individual into the group with higher importance based on the individual's importance I in each group, ignoring the individual's relationship with other groups. If the individual's importance I reaches a certain threshold t in multiple groups, then the individual is important in both groups, and the two groups are merged into one group.

[0020] Step S404: If there are still unclassified individuals, start from that individual and find related individuals and classify them into the group to which the related individuals belong; if they are classified repeatedly, proceed to step S403; if the individual has no relation to other individuals, classify them into a separate group.

[0021] Step S405, repeat steps S402, S403, and S404 until all people are categorized into reasonable groups.

[0022] Step S5: Based on the results of the population segmentation and the relationship categories between individuals, the VIP prediction module (121) performs relevant relationship fusion for each individual in the population, and determines the importance of each individual in the population based on the fusion results, ultimately identifying the important individuals in the population, including the following steps:

[0023] Step S500: Select a population group and calculate the importance I of each person in the population group.

[0024] Step S501: Based on the importance calculation results, the person with the highest importance is designated as the VIP in this group.

[0025] Importance I = Σw i c i w i c i w represents the contribution value of relation i to the importance of a certain person, w represents the weight assigned to the relation, and c represents the confidence level.

[0026] The advantages and beneficial effects of this invention are as follows:

[0027] This invention utilizes the ability of scene graphs to detect and associate objects in images, thereby improving the robot's understanding of the scene; it identifies common objects and relationships in the scene, helping the robot focus on the main objects and issues in the exhibition hall scene; based on K-Cluster, it proposes a crowd classification method using scene graphs; and based on scene graphs, it proposes a way to assign weights to relationship classifications to determine the importance of people, and then sequentially identifies important people in the crowd. Attached Figure Description

[0028] Figure 1 This is a system architecture diagram of the present invention.

[0029] Figure 2 This is a flowchart of the method of the present invention.

[0030] Figure 3 This is a schematic diagram of a scene in this invention. Detailed Implementation

[0031] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0032] like Figure 1 As shown, a scene graph-based visual scene understanding system for robot exhibition hall scenarios includes a color image acquisition and retrieval module 10, a scene graph relationship recognition module 11, and a crowd relationship prediction module 12.

[0033] The color image acquisition and retrieval module 10 can capture color images in real time using a camera and allow other parts of the system to retrieve them.

[0034] The scene graph relationship recognition module 11, based on deep learning, calls the color image acquisition and call module 10 to retrieve the images and inputs them into a deep learning network that has been trained on the dataset. According to the pre-established model, it outputs the detected object positions, predicted object categories and predicted relationships between the objects in each scene image, and generates a scene graph composed of objects and relationships as an understanding of the scene.

[0035] In a specific embodiment of the present invention, the scene graph relationship recognition module 11 specifically includes an object detection module 110, an object classification module 111, a relationship classification module 112, an object category library 113 in the exhibition hall scene, and a relationship category library 114 in the exhibition hall scene;

[0036] The object detection module 110 uses deep learning methods to input images from the color image acquisition and retrieval module 10 into the network, extracting object features and object locations from the images. Each detected object has independent object features and object location. The object features are represented by a vector, and the object location is represented by the top-left and bottom-right pixel coordinates of its bounding box in the image.

[0037] The object classification module 111 uses deep learning methods to input object features obtained by the object detection module into a deep learning network, which then outputs the object category to which the object belongs. The object categories generated by the object classification module are all derived from an object category library within the exhibition hall scene.

[0038] The relationship classification module 112 uses deep learning methods to input color images, object features, object locations, and object classifications obtained from the color image acquisition and retrieval module 10, the object detection module 110, and the object classification module 111 into a deep learning network. Based on this input information, the network predicts and outputs the relationships between pairs of objects. The object relationships generated by the relationship classification module 112 are all derived from a relationship category library within the exhibition hall scene.

[0039] The object category library 113 in the exhibition hall scene includes, but is not limited to, the following object categories: person, screen, board, plant, camera, table, exhibit, etc.

[0040] The exhibition hall scene contains a relationship category library of 114, including but not limited to the following: looking at, next to, talk to, align with, back to, pass by, hold hands with, sit together with, take a photo of, point at, and shake hands with. Among these, the relationship categories between people and other objects include: looking at, next to, back to, pass by, take a photo of, and point at; and the relationship categories between people include: looking at, next to, talk to, align with, back to, pass by, hold hands with, sit together with, point at, and shake hands with.

[0041] When a relationship category connects people, the relationship is assigned a weight w and a confidence level c. i c i Let Σw represent the contribution of relation i to the importance of a certain person. Therefore, the importance I of each person can be expressed as: Σw i c i

[0042] In a specific embodiment of the present invention, the crowd relationship prediction module 12, based on the object classification and relationship classification required by the application scenario, reasonably divides people into multiple crowds, with strong relationships within each crowd and weak relationships between crowds. Within each crowd, the importance of each person in that crowd is predicted by integrating the relationships to which each person belongs, and the most important person in that crowd is designated as the VIP.

[0043] Furthermore, the crowd relationship prediction module 12 includes a crowd segmentation module 120 and a VIP prediction module 121;

[0044] The crowd segmentation module 120 distinguishes crowds based on the relationships between people obtained by the scene graph relationship recognition module 11 and by analyzing and comparing the closeness of these relationships.

[0045] The VIP prediction module 121 can integrate all the relationships of each person in each group divided by the group segmentation module 120, and judge the importance of each person in the group based on the integration result, and finally determine the VIP in the group.

[0046] like Figure 2 As shown, a scene graph-based visual scene understanding method for robot exhibition hall scenarios includes the following steps:

[0047] Step S1: The color image acquisition and retrieval module 10 acquires real-time images of the application scenario, ensuring that other modules can retrieve these images in a reasonable time sequence. The object detection module 110 detects the object features and their locations in the images.

[0048] Specifically, the color image at the current moment refers to the color image acquired in real time during actual operation. Preferably, in a robot system using Robot Operating System (ROS), monocular camera color images are published as ROS topics, and color images can be acquired in real time by subscribing to image topics.

[0049] The object detection in this embodiment uses computer vision technology, which can detect objects in an image that belong to an object category library.

[0050] In a preferred example, object detection is performed using deep learning-based computer vision techniques, such as YOLO, SSD, Mask RCNN, RetinaNet, etc.

[0051] In one exemplary embodiment, such as Figure 3 As shown, each circle represents an object detected that belongs to the object category library.

[0052] In step S2, the object classification module 111 calculates the category to which the object belongs based on the object's characteristics, and the scene graph relationship recognition module 11 associates and combines all objects with relationships to generate a scene graph.

[0053] In a preferred example, deep learning-based computer vision technology is used to calculate the category to which the object belongs, and the scene graph relationship recognition module 11 associates and combines all objects with relationships to generate a scene graph.

[0054] Step S3: Based on the scene graph generated by the above method, the reasonable relationship category between each pair of objects is calculated by the relationship classification module 112.

[0055] In one exemplary embodiment, such as Figure 3 As shown, based on the detected positions of objects and the relationships between pairs of objects, the lines between the circles represent the relationships between the objects.

[0056] Step S4: The crowd segmentation module 120 performs reasonable crowd segmentation based on the characters in the entire scene and the relationship between them.

[0057] Specifically, step S4 further includes:

[0058] S400, with K clusters in the preset scenario.

[0059] S401, each cluster randomly selects one individual i0 as the initial state, where, in the scenario graph, the selected individuals cannot have any direct relationship with each other.

[0060] S402, based on the scenario diagram, place individuals that are directly related to individuals in the K clusters into the corresponding clusters.

[0061] S403, check the categorized individuals. If an individual belongs to multiple clusters, classify the individual into the cluster with higher importance based on the individual's importance I in each cluster, ignoring the individual's relationship with other clusters. If the individual's importance I reaches a certain threshold t in multiple clusters, then the individual is important in both clusters, and the two clusters are merged into one cluster.

[0062] S404. If there are still unclassified individuals, start from that individual and find the individuals related to him / her, and classify them into the cluster to which the related individuals belong; if they are classified repeatedly, proceed to step S403; if the individual is not related to other individuals, classify him / her into a separate cluster.

[0063] Repeat steps S402, S403, and S404 until all characters are categorized into appropriate clusters.

[0064] In one exemplary embodiment, such as Figure 3 As shown, the default number of clusters is 3. Three individuals (10, 20, and 30) with no mutual relationship are randomly selected as the initial individuals. Individuals directly related to them are assigned to their corresponding clusters. When an individual belongs to both clusters 2 and 3, its importance in each cluster is determined based on the weight of its relationship in each cluster. If the individual is more important in cluster 2 and its importance in cluster 3 is not higher than a threshold t, it is assigned to cluster 2; otherwise, it is assigned to cluster 3. If the individual's importance in both clusters 2 and 3 is higher than the threshold t, then clusters 2 and 3 are merged.

[0065] Afterwards, there are still blank individuals that have not been classified. Starting from these individuals, we search for the related individuals and classify them into the cluster to which the related objects belong.

[0066] Step S5: Based on the results of the population segmentation and the relationship categories between individuals, the VIP prediction module 121 performs relevant relationship fusion on each person in the population, and judges the importance of each person in the population based on the fusion results, and finally obtains the VIPs in the population.

[0067] Specifically, step S5 further includes:

[0068] S500, select a cluster, and calculate the importance I of each person in the cluster.

[0069] S501: Based on the importance calculation results, the person with the highest importance is designated as the VIP in this cluster.

[0070] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A scene graph-based visual scene understanding system for a robot, comprising: The picture acquisition and calling module, the scene graph relationship identification module (11), and the crowd relationship prediction module (12) are characterized in that: The picture acquisition and calling module acquires pictures. The scene graph relationship identification module (11) calls the pictures in the picture acquisition and calling module, inputs the pictures into a trained deep learning network, outputs the object positions detected in the pictures, the object categories predicted, and the relationships between the objects, and generates a scene graph of the objects and the relationships. The crowd relationship prediction module (12) divides the objects of the characters into multiple crowds based on the object classification and the relationship classification, the relationships between the characters in a crowd are strong, the relationships between the characters in different crowds are weak, the importance of the characters in a single crowd is predicted by fusing the relationships of the characters in the crowd, and the most important character in the crowd is regarded as an important character in the crowd. The crowd relationship prediction module (12) includes a crowd division module (120) and a VIP prediction module (121). The crowd division module (120) divides the crowds according to the relationships between the characters obtained by the scene graph relationship identification module (11) by analyzing and comparing the relationships, and specifically, K clusters are preset in a scene, one individual i0 is randomly selected as an initial state in each cluster, the selected individuals in the scene graph cannot have direct relationships with each other, individuals having direct relationships with the individuals in the K clusters are put into the corresponding clusters according to the scene graph, the classified character individuals are checked, if a character belongs to multiple clusters at the same time, the character is classified into a cluster with higher importance according to the importance of the character in different clusters, and the relationship classification between the character and other clusters is ignored, if the importance of the character in multiple clusters reaches a certain threshold, the character is important in both clusters, and the two clusters are merged into one cluster, if there are still unclassified character individuals, the individuals are found from the character individuals, and are classified into the clusters to which the related character individuals belong, if the individuals are repeatedly classified, the checking of the classified character individuals is returned and the execution is continued, if the character individuals have no relationship with other character individuals, the individuals are classified into a cluster alone, the individuals are put into the corresponding clusters according to the scene graph, and the execution is continued until all characters are classified into reasonable clusters. The VIP prediction module (121) fuses the relationships of the characters in the crowds divided by the crowd division module (120), judges the importance of the characters in the crowds according to the fusion results, and finally obtains important characters in the crowds. 2.The robot-oriented scene graph based visual scene understanding system according to claim 1, wherein The scene graph relationship identification module (11) includes an object detection module (110), an object classification module (111), a relationship classification module (112), an object category library under an exhibition hall scene (113), and a relationship category library under an exhibition hall scene (114). The object detection module (110) inputs the pictures called from the picture acquisition and calling module into a trained deep learning network, extracts object features and object positions of the pictures, and outputs the extracted object features and object positions. The object classification module (111) obtains the predicted object category through the trained deep learning network based on the object features. The relationship classification module (112) predicts and outputs the relationship between two objects by using the trained deep learning network based on the picture, the object features, the object position, and the classification of the object obtained from the picture acquisition and calling module, the object detection module (110), and the object classification module (111). 3.The robot-oriented scene graph based visual scene understanding system according to claim 2, characterized in that The object features extracted by the scene graph relationship identification module (11) are represented by a vector, and the object position is represented by the upper left pixel coordinate and the lower right pixel coordinate of the object in the target frame of the picture. 4.The robot-oriented scene graph based visual scene understanding system according to claim 2, wherein The object categories generated by the object classification module (111) are from the object category library (113) in the exhibition hall scene, and the object categories in the object category library (113) in the exhibition hall scene include person, screen, board, plant, camera, table, and exhibit.

5. The robot-oriented scene graph based visual scene understanding system according to claim 3, characterized in that The object relationships generated by the relationship classification module (112) are from the relationship category library (114) in the exhibition hall scene, and the relationship categories in the relationship category library (114) in the exhibition hall scene include looking at, next to, talk to, align with, back to, pass by, taking photo of, pointing at, and shake hands with; the relationship categories between the person and other objects include looking at, next to, back to, pass by, taking photo of, and pointing at; and the relationship categories between the person and the person include looking at, next to, talk to, align with, back to, pass by, hold hands with, sit together with, pointing at, and shake hands with.

6. A scene graph-based visual scene understanding method for robots, characterized by The method comprises the following steps: Step S1: acquiring a picture in an application scene, and detecting the object features and the positions of the objects in the picture; Step S2: calculating the categories to which the objects belong based on the object features, associating and combining the objects and the relationships, and generating a scene graph; Step S3: calculating the relationship categories between two objects; Step S4: performing crowd division according to the person and the relationship categories between the persons; the step comprises the following steps: Step S400, there are K groups of people in the preset scene; Step S401, randomly select one individual i0 in each group as the initial state, wherein the selected individuals in the scene graph cannot have direct relationships with each other; Step S402, according to the scene graph, put individuals having direct relationships with individuals in the K groups of people into the corresponding groups; Step S403, check the classified individual, if the individual belongs to multiple groups, according to the importance I of the individual in different groups, put the individual into the group with higher importance, ignore the relationship classification of the individual with other groups; if the importance I of the individual in multiple groups reaches the threshold t, combine the two groups into one group; Step S404, if there are still unclassified individual, start from the individual, find the individual having relationships with the individual, and put the individual into the group to which the individual having relationships belongs; if the individual is repeatedly classified, execute step S403; if the individual has no relationship with other individuals, put the individual into a group alone; Step S405, repeat steps S402, S403, and S404 until all individuals are classified into groups; Step S5, according to the relationship classification between the results of the group division and the individuals, perform relationship fusion on the individuals in the group, and according to the fusion results, judge the importance of the individuals in the group, and finally obtain important individuals in the group.

7. The robot-oriented scene graph based visual scene understanding method according to claim 6, characterized in that The S5 includes the following steps: Step S500, select a group, and calculate the importance I of each individual in the group; Step S501, according to the importance calculation result, take the individual with the maximum importance as the VIP in the group.

8. The robot-oriented scene graph based visual scene understanding method according to claim 6 or 7, characterized in that The importance I =∑w i c i , w i c i represents the contribution value of the relationship i to the importance of the certain person, w represents that the relationship is given weight, and c represents the confidence degree.

Images