Scene graph-based skill planning method and apparatus

Abstract

This application relates to a scene graph-based skill planning method, including: generating a first scene graph based at least partially on an input image; determining, based at least partially on the first scene graph, a first condition that is unmet and that is associated with a first skill; obtaining a first question corresponding to the first condition; obtaining a first VQA result for the first question by performing visual question answering (VQA) by using the first question; updating, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain a second scene graph; and planning the first skill as an executable action based at least partially on the second scene graph.

Description

[0001] SCENE GRAPH-BASED SKILL PLANNING METHOD AND

[0002] APPARATUS

[0003] TECHNICAL FIELD

[0004] This application relates to artificial intelligence technologies, and more specifically, to a scene graph-based skill planning method and apparatus.

[0005] BACKGROUND

[0006] An automatic control apparatus may generate a control signal based on a collected image, to control an action of the apparatus. For example, the automatic control apparatus may be an autonomous robot having a skill set. The autonomous robot determines, based on an input image about an observed object or a surrounding environment, that a condition for performing a skill is met. The autonomous robot may plan the skill meeting the condition as an executable action, and may execute the skill planned as the executable action.

[0007] To improve efficiency of analyzing information in the input image, a scene graph (SG) may be constructed to represent objects and a relationship between the objects in the input image. However, using the scene graph to represent the input image may cause omission of some information in the input image. For example, when the scene graph is generated based on the input image, the scene graph may not completely reflect the information included in the input image. When the autonomous robot plans the skill based on the scene graph, absence of the information may cause the autonomous robot to determine that the condition for performing the skill is unmet, and consequently, the autonomous robot cannot perform the skill that is to be planned as the executable action. This reduces use efficiency of the autonomous robot.

[0008] In addition, if a more accurate input image-based scene graph can be provided, and reasonableness of skill planning can be improved, an apparatus can be optimized more favorably to be more efficient and cost-effective in executing a task.

[0009] SUMMARY

[0010] The following brief introduction is provided to introduce some selected concepts in a simple form, and these concepts are further described in the following detailed description. The brief introduction is not intended to highlight key features or necessary features of a theme that claims protection, nor is intended to limit a scope of the theme that claims protection.

[0011] For the foregoing problem, this application provides a novel scene graph-based skill planning method implemented by a computer, and correspondingly, provides a novel architecture of a neural network (NN) model for scene graph-based skill planning. By using the method in embodiments of this application, a possibility that an apparatus properly completes skill planning based on an input image can be increased, and improved task execution efficiency can be provided.

[0012] According to an aspect of this application, a scene graph-based skill planning method is provided and includes: generating a first scene graph based at least partially on an input image; determining, based at least partially on the first scene graph, a first condition that is unmet and that is associated with a first skill; obtaining a first question corresponding to the first condition; obtaining a first VQA result for the first question by performing visual question answering (VQA) by using the first question; updating, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain a second scene graph; and planning the first skill as an executable action based at least partially on the second scene graph.

[0013] According to an aspect of this application, a scene graph-based skill planning method is provided and includes: generating a first scene graph based at least partially on an input image; determining, based at least partially on the first scene graph, a first condition that is met and that is associated with a first skill; obtaining a first question corresponding to the first condition; obtaining a first VQA result for the first question by performing visual question answering (VQA) by using the first question; updating, in response to the first VQA result indicating that the first condition is unmet, the first scene graph to obtain a second scene graph; and planning the first skill as an unexecutable action based at least partially on the second scene graph.

[0014] According to an aspect of this application, a scene graph-based skill planning apparatus is provided and includes: a scene graph generation module, configured to generate a first scene graph based at least partially on an input image; a skill planning module, configured to determine, based at least partially on the first scene graph, a first condition that is unmet and that is associated with a first skill; a question obtaining module, configured to obtain a first question corresponding to the first condition; a VQA module, configured to obtain a first VQA result for the first question by performing VQA by using the first question; and a scene graph update module, configured to update, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain a second scene graph; the skill planning module planning the first skill as an executable action based at least partially on the second scene graph.

[0015] According to an aspect of this application, a scene graph-based skill planning apparatus is provided and includes: a scene graph generation module, configured to generate a first scene graph based at least partially on an input image; a skill planning module, configured to determine, based at least partially on the first scene graph, a first condition that is met and that is associated with a first skill; a question obtaining module, configured to obtain a first question corresponding to the first condition; a visual question answering (VQA) module, configured to obtain a first VQA result for the first question by performing VQA by using the first question; and a scene graph update module, configured to update, in response to the first VQA result indicating that the first condition is unmet, the first scene graph to obtain a second scene graph; the skill planning module planning the first skill as an unexecutable action based at least partially on the second scene graph.

[0016] According to an aspect of this application, a scene graph-based skill planning apparatus is provided and includes: a plurality of sensors; one or more processors; and one or more memories, storing computer-executable instructions. When the instructions are executed by the one or more processors, an operation of scene graph-based skill planning according to the embodiments of this application is performed.

[0017] According to an aspect of this application, a processing apparatus is provided and includes: one or more processors; and one or more memories, storing computer-executable instructions. When the instructions are executed by the one or more processors, an operation of scene graphbased skill planning according to the embodiments of this application is performed.

[0018] According to an aspect of this application, a machine-readable storage medium is provided. The machine-readable storage medium stores executable instructions. When the instructions are executed, one or more processors are caused to perform an operation of scene graph-based skill planning according to the embodiments of this application.

[0019] According to an aspect of this application, a computer program product is provided. The computer program product includes executable instructions. When the instructions are executed, one or more processors are caused to perform an operation of scene graph-based skill planning according to the embodiments of this application.

[0020] According to various aspects of the present disclosure, a technical solution of scene graphbased skill planning provided in the present disclosure includes a question obtaining module and a VQA module. The question obtaining module obtains a question corresponding to a first condition that is associated with a first skill and that is unmet currently, so that the VQA module can determine again, by using the question, whether an input image indeed does not meet the first condition. By using this technical means, it is ensured that skill planning is based on accurate understanding of information conveyed by the input image. Subsequently, the technical solution of scene graph-based skill planning provided in the present disclosure further includes a novel scene graph update module, which correspondingly updates the scene graph for subsequent skill planning in response to the input image actually meeting the first condition. An updated scene graph can more accurately reflect an actual situation related to the first condition in the input image, thereby improving accuracy of the scene graph in an aspect of reflecting a condition of a skill while retaining benefits brought by the scene graph, and therefore making skill planning more proper. Other advantages of the various aspects of the present disclosure are described below.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The essence and advantages of this application may be further understood by referring to the following accompanying drawings. In the accompanying drawings, similar components or features may have same reference signs.

[0023] FIG. l is a block diagram of an automatic control apparatus according to an embodiment;

[0024] FIG. 2 is a block diagram of a scene graph-based skill planning apparatus according to an embodiment;

[0025] FIG. 3 A and FIG. 3B are respectively schematic diagrams of a question obtaining module according to an embodiment;

[0026] FIG. 4A to FIG. 4C are respectively schematic diagrams of a VQA module according to an embodiment;

[0027] FIG. 5 A to FIG. 5C are respectively schematic diagrams of a scene graph update module according to an embodiment;

[0028] FIG. 6 is a block diagram of a scene graph-based skill planning apparatus according to another embodiment;

[0029] FIG. 7 and FIG. 8 are respectively flowcharts of a scene graph-based skill planning method according to an embodiment;

[0030] FIG. 9 is a block diagram of a scene graph-based skill planning apparatus according to an embodiment; and

[0031] FIG. 10 is a block diagram of a processing apparatus according to an embodiment.

[0032] DETAILED DESCRIPTION

[0033] A subject described in this specification will now be discussed with reference to exemplary implementations. It is to be understood that, discussion of the embodiments is merely intended to make a person skilled in the art better understand and implement the subject described in this specification, and is not intended to limit the protection scope of the claims, the applicability, or examples. Changes may be made to the functions and arrangements of the discussed elements without departing from the protection scope of the content of this application. Various processes or components may be omitted, replaced, or added in each example according to requirements. For example, the described method may be performed according to a sequence different from the sequence described herein, and steps may be added, omitted, or combined. In addition, features described with respect to some examples may also be combined in another example.

[0034] As used in this specification, a term "include" and its variants represent open terms, and mean "including but not limited to". The term "based on" means "based at least partially on". Terms "one embodiment" and "an embodiment" mean "at least one embodiment". A term "another embodiment" means "at least one other embodiment". Terms "first", "second", and the like may represent different objects or the same object. Other definitions may be included explicitly or implicitly in the following. Unless otherwise clearly specified in the context, a definition of a term is consistent throughout the specification.

[0035] FIG. 1 is a block diagram of an automatic control apparatus 100 according to an embodiment. For example, the automatic control apparatus 100 may be a robot, a vehicle, a household appliance, an electric tool, a machine for manufacturing, a personal digital assistant, an access control system, a surveillance system, a medical (imaging) system, or the like that has an automatic control function. The automatic control apparatus 100 includes a sensor system 110, a processing system 120, and an execution system 130. The sensor system 110 is configured to sense a sensor signal of an ambient environment or an observed object. For example, the sensor system may include a sensor such as a video sensor, a camera sensor, a radar sensor, a laser radar sensor, an ultrasonic sensor, a motion sensor, a thermal imaging camera, or a sonar, and is configured to obtain corresponding sensing data. The processing system 120 performs processing based on the sensing data, to control an operation of the apparatus 100. For example, the processing system 120 may include a perception module, a planning module, and an action module. The perception module extracts or generates, based on the sensing data, information that can reflect a perceived object. The planning module plans a next action based on the information obtained by the perception module. The action module controls an operation of the execution system 130 based on the planned action. The execution system 130 may include a mechanical apparatus to implement a specific action.

[0036] It may be understood that the automatic control apparatus 100 may include other components. Only components related to the technical solutions of the present disclosure are shown in the embodiment of FIG. 1. For example, the automatic control apparatus 100 may include a communication system. One or more of the perception module, the planning module, and the action module may also be deployed on another processing apparatus other than the automatic control apparatus 100, for example, a processing apparatus such as a server or a cloud. Data is exchanged between the automatic control apparatus 100 and the another processing apparatus by using the communication system.

[0037] FIG. 2 is a block diagram of a scene graph-based skill planning apparatus 200 according to an embodiment. The scene graph-based skill planning apparatus 200 includes a scene graph generation module 210, a skill planning module 220, a question obtaining module 230, a VQA module 240, a scene graph update module 250, and an action module 260. In an embodiment, the apparatus 200 is implemented in the processing system 120 shown in FIG. 1. The scene graph generation module 210, the skill planning module 220, and the action module 260 may correspond to the perception module, the planning module, and the action module described above with reference to FIG. 1. It may be understood that the apparatus 200 may include other modules, and only modules related to this embodiment of the present disclosure are shown in FIG. 2. In the example in FIG. 2, the apparatus 200 receives an input image 205 based on data collected by the sensor system 110. Although the input image 205 collected by a camera is used as an example for description in FIG. 2, it may be understood that the input image may alternatively be sensing data, for example, radar data, laser radar data, or ultrasonic data, collected by the foregoing another type of sensor device, or may be sensing data obtained by combining a plurality of types of sensing data, and these different types of sensing data may be collectively referred to as an image. The scene graph generation module 210 generates a first scene graph 215 based on the input image 205. It may be understood that, for ease of description, the scene graph 215 shown in FIG. 1 shows only some nodes corresponding to some objects in the image 205.

[0038] The scene graph generation module 210 may include a scene graph generation model (for example, a neural network model), to generate the scene graph based on the input image. An entire training process of the scene graph generation model is usually supervised, and a data set with an annotated image is needed. The annotated image includes information about objects, attributes of the objects, and a relationship between the objects. The scene graph generation model learns from training data by using an optimization technology, and adjusts a model parameter of the scene graph generation model, to minimize a prediction error and improve a capability of generating an accurate scene graph from a new image.

[0039] The scene graph generation module 210 may construct the scene graph 215 by recognizing objects in the input image 205, attributes of the objects, and a relationship between the objects. The scene graph usually includes a node set and an edge set. Nodes in the node set respectively correspond to different detected objects in the input image. An edge in the edge set is between two nodes in the node set and represents a relationship between the two nodes. For example, the scene graph generation module 210 first recognizes or detects the objects in the input image 205. In the example in FIG. 2, in the input image 205, the recognized objects are included in boundary boxes. A person skilled in the art may understand that FIG. 2 is only shown as an example, and any other object may be recognized in the input image 205. An object may be associated with an object type name. For example, the objects recognized in the input image 205 may be associated with object type names such as "dish washer", "dish", and "bowl". The scene graph generation module 210 may recognize the objects in the input image by using any object detection model. For example, the object detection model may be Faster R-CNN, You Only Look Once (YOLO), or Single Shot Detector (SSD). Each recognized object is represented as a node in the scene graph. For example, nodes of different colors in the generated scene graph 215 correspond to the objects in corresponding boundary boxes in the input image 205. The apparatus 200 can more efficiently perform, by using the scene graph to structurally represent the information about the objects in the input image, a subsequent operation, for example, a skill planning operation, performed based on analysis on the input image.

[0040] Optionally, for each recognized object, the scene graph generation module 210 may include an additional model to predict an attribute used for describing a characteristic of the object. For example, the attribute may include a type, a color, a shape, a size (for example, a diameter), and the like of the object. The attribute may enrich information that can be included in the scene graph and provide a more detailed description of the object. For example, in the example in FIG. 2, an attribute of a node corresponding to a "dish" in the scene graph 215 may include "tableware", "white", "circle", "8 cm", and the like.

[0041] After the objects and the attributes of the objects are recognized, the scene graph generation module 210 determines a relationship between the recognized objects, including predicting a verb or a preposition, such as "on", "hold", or "next to", that describes how the objects interact with each other. For example, the neural network model in the scene graph generation module 210 may use images of a pair of cropped objects and category names of the cropped objects as an input, and predict a relationship between the pair of objects. The relationship is represented in the scene graph as an edge used for connecting related object nodes.

[0042] As described above, after the objects in the image and the attributes of the objects are recognized and the relationship between the objects is established, the scene graph can be constructed. For example, a node in the scene graph represents an object, a node label represents an object type and an attribute, an edge represents a relationship between objects, and an edge label represents an attribute of the relationship.

[0043] It may be understood that in the implementations of the present disclosure, the scene graph generation module 210 may be implemented by using any suitable neural network model used for generating a scene graph based on an image.

[0044] The apparatus 100 can implement a series of skills. For a task target, the skill planning module 220 may select, based on a skill reasoning algorithm, one or more skills to be executed from the skills that can be implemented by the apparatus 100. Each of the one or more skills is associated with one or more conditions. For example, the skill reasoning algorithm includes algorithms such as random selection, a planning domain description language (PDDL) planner, graph-based planning, and language-based planning. Selection of a skill by the skill planning module 220 may alternatively be based on the task target. In the example in FIG. 2, the apparatus 100 may be a robot. For a target of the apparatus 100 for emptying a dish washer, task planning, and / or based on the skill reasoning algorithm, the skill planning module 220 may select a first skill for taking out tableware from the dish washer.

[0045] A skill may be associated with one or more conditions. The skill planning module 220 may determine, based on evaluation on the first scene graph 215, whether a condition associated with the selected first skill is met. For example, the skill planning module 220 may evaluate the first scene graph 215 by using the neural network model or a function logic unit, to obtain a Boolean return value, to determine whether the first scene graph 215 meets the condition associated with the first skill. For example, meeting the condition may include that the scene graph 215 is required to meet at least one of the following: a particular object exists or does not exist, a particular object has or does not have a particular attribute, or a particular edge exists or does not exist between particular objects.

[0046] In some embodiments, in response to determining that the condition associated with the selected skill is met, the skill planning module 220 may plan the skill as an executable action A 255, and input information used for indicating the executable action A 255 to the action module 260, for the action module 260 to control execution of the executable action A 255.

[0047] In some embodiments, the scene graph may not meet the condition associated with the selected skill. The skill planning module 220 may determine an unmet first condition set based at least partially on the first scene graph 215. The first condition set includes an unmet condition associated with an unmet skill in the selected one or more skills. In the example in FIG. 2, a first condition C 225 associated with the first skill of "take out tableware from a dish washer" may be "the dish washer includes the tableware". As shown in FIG. 2, in the scene graph 215 initially generated by the scene graph generation module 210, objects used as instances of the tableware, such as bowls shown by purple and green nodes and a dish shown by a blue node, are detected, but an "include" relationship between the dish washer shown by a red node and the illustrated instances of the tableware is not detected. Therefore, a constructed scene graph 215 includes only the foregoing nodes but does not include edges between the foregoing nodes. The skill planning module 220 may determine, based on the first scene graph 215, that: the "dish washer" node (red) in the first scene graph 215 does not have an "include" relationship to any node having a tableware attribute (for example, there is no edge used for representing the "include" relationship). Therefore, the first condition C 225 associated with the first skill is unmet. In this example, the first skill of "take out tableware from a dish washer" is the only skill in this scene, and therefore the unmet first condition set includes the determined unmet first condition C 225 of "the dish washer includes the tableware".

[0048] As another example, in a scene of the dish washer shown in FIG. 2, a plurality of skills may alternatively be included. For example, the plurality of skills include a skill 1 of "take out a dish whose diameter is greater than 20 cm from the dish washer", a skill 2 of "take out a dish whose diameter is less than 20 cm from the dish washer", a skill 3 of "take out a bowl whose diameter is greater than 10 cm from the dish washer", a skill 4 of "take out a bowl whose diameter is less than 10 cm from the dish washer", and a skill 5 of "take out the tableware from the dish washer". Conditions of the skill 1 to the skill 5 are respectively a condition 1 of "the dish washer includes a dish whose diameter is greater than 20 cm", a condition 2 of "the dish washer includes a dish whose diameter is less than 20 cm", a condition 3 of "the dish washer includes a bowl whose diameter is greater than 10 cm", a condition 4 of "the dish washer includes a bowl whose diameter is less than 10 cm", and a condition 5 of "the dish washer includes the tableware". By planning and executing the foregoing skills in sequence, instances of different sizes of the tableware can be taken out according to a more proper sequence. It may be understood that the condition 1 corresponding to the skill 1 may alternatively be set as a condition set 1, which includes a condition 1(1) of "a dish is included in the dish washer" and a condition 1(2) of "a diameter of the dish is greater than 20 cm". Similarly, the condition 2 to the condition 4 may alternatively be set as corresponding condition sets.

[0049] In this example, the skill planning module 220 may first select the skill 1 and determine, based on the scene graph 215, whether the corresponding condition 1 is met. In the example shown in FIG. 2, the skill planning module 220 may determine, based on the scene graph 215, that: the "dish washer" node (red) in the first scene graph 215 does not have an "include" relationship to any node having a dish attribute (for example, there is no edge used for representing the "include" relationship). Therefore, the condition 1 associated with the skill 1 is unmet. In this example, assuming that a diameter of a large-sized dish in the image 205 is greater than 20 cm, and the scene graph generation model 210 detects the dish and a size attribute of the dish, but does not detect an "include" relationship between the dish washer and the dish, the skill planning module 220 may determine, based on the scene graph 215, the foregoing unmet condition 1 of "the dish washer includes a dish whose diameter is greater than 20 cm", to add the unmet condition 1 to the unmet first condition set. In addition, when the foregoing condition 1 is set to the foregoing condition set, the skill planning module 220 may determine the met condition 1(2) and the unmet condition 1(1) based on the scene graph 215, to add the unmet condition 1(1) to the unmet first condition set. In this example, assuming that a diameter of a large-sized dish in the image 205 is greater than 20 cm, and the scene graph generation model 210 detects the dish, but detects a size of the dish as a specific value such as "less than 20 cm" or "18 cm", and does not detect an "include" relationship between the dish washer and the dish, the skill planning module 220 may determine, based on the scene graph 215, the foregoing unmet condition 1 of "the dish washer includes a dish whose diameter is greater than 20 cm", to add the unmet condition 1 to the unmet first condition set. In addition, when the foregoing condition 1 is set to the foregoing condition set, the skill planning module 220 may determine the unmet condition 1(1) and the unmet condition 1(2) based on the scene graph 215, to add the unmet condition 1(1) and the unmet condition 1(2) to the unmet first condition set. Similarly, the skill planning module 220 may then select the skill 2 to the skill 5 in sequence, determine, based on the scene graph 215, whether a corresponding condition in the corresponding condition 2 (or a condition set 2) to the corresponding condition 5 (or a condition set 5) is met, plan the corresponding skill as the executable action A 255 when the corresponding condition is met, and determine that the corresponding skill is an unmet skill and add the corresponding unmet condition to the unmet first condition set when the corresponding condition is unmet.

[0050] After the unmet first condition set is determined, the skill planning module 220 may determine the first condition in the first condition set, the first condition being associated with the first skill. In the example in FIG. 2, in an example in which the first skill of "take out tableware from a dish washer" is the only skill in this scene, the skill planning module 220 may determine the condition of "the dish washer includes the tableware" in the first condition set as the unmet first condition C 225. In the foregoing example including the skill 1 to the skill 5, the skill planning module 220 may randomly select a condition from the first condition set as the unmet first condition C 225. For example, the skill planning module 220 may determine the condition 1 of "the dish washer includes a dish whose diameter is greater than 20 cm" in the first condition set as the unmet first condition C 225. For another example, the skill planning module 220 may select the condition 1(2) of "a diameter of the dish is greater than 20 cm" from the first condition set as the unmet first condition C 225. For another example, the skill planning module 220 may select the condition 5 of "the dish washer includes the tableware" from the first condition set as the unmet first condition C 225. For another example, the skill planning module 220 may select the condition 1(1) of "the dish washer includes the dish" and the condition 1(2) of "a diameter of the dish is greater than 20 cm" from the first condition set as the unmet first condition C 225.

[0051] In an embodiment, the skill planning module 220 may randomly select the first condition C 225 from the first condition set. In another embodiment, the skill planning module 220 may determine, in the unmet skill, the first skill that is associated with a minimum condition set, and determine the first condition in the minimum condition set. For example, the determined first skill may be a skill that is easiest to implement, a skill that is fastest to implement, a skill that has a minimum quantity of unmet conditions, or the like among unmet skills, and the condition set associated with such a first skill is referred to as the minimum condition set. Determining the first condition in the minimum condition set may help the apparatus 200 to plan and implement the first skill in priority, thereby improving efficiency of the apparatus 200 in implementing the one or more skills selected by the apparatus 200 or completing the target task. In another embodiment, the skill planning module 220 may determine, in the unmet skill, the first skill having a highest priority, and determine the first condition in one or more conditions associated with the first skill. For example, a skill may be associated with a skill priority, and the skill priority may indicate severity, importance, urgence, basality, or the like associated with the skill. Determining the first condition associated with the first skill having a highest skill priority may help the apparatus 200 to plan and implement the first skill in priority, thereby improving reasonableness when the apparatus 200 implements one or more skills.

[0052] The question obtaining module 230 obtains a first question Q 235 corresponding to the unmet first condition C 225. For example, the first question Q 235 may be in a form of a natural language. For example, the question obtaining module 230 may include a large language model (LLM). For example, the skill planning module 220 determines that the first condition C 225 is "the dish washer includes the tableware", and provides the first condition C 225 to the question obtaining module 230. The question obtaining module 230 may obtain the first question Q 235 corresponding to the first condition C 225: "Does a dish washer include tableware?", "Does a dish washer include a dish?", "What is included in a dish washer?", or the like. As another example, the skill planning module 220 determines that the first condition C 225 is "a diameter of the dish is greater than 20 cm", and provides the first condition C 225 to the question obtaining module 230. The question obtaining module 230 may obtain the first question Q 235 corresponding to the first condition C 225: "Is the diameter of the dish greater than 10 cm?" or "What is the diameter of the dish?". The question obtaining module 230 may be implemented according to various embodiments described in this specification with reference to FIG. 3 A or FIG. 3B.

[0053] The question obtaining module 230 may provide the obtained first question Q 235 to the VQA module 240. The VQA module 240 obtains a first VQA result R 245 for the first question Q 235 by performing VQA by using the first question Q 235. In an example, the first VQA result R 245 may be generated by the neural network model based on at least one of the first scene graph 215 and the input image 205 and the first question Q 235. In the example in FIG. 2, for the first question Q 235 of "Does a dish washer include tableware?" or "Does a dish washer include a dish?", the VQA module 240 may obtain, based on the first question Q 235 and at least one of the scene graph 215 and the image 205, for example, the first VQA result R 245 indicating "include". For example, the first VQA result R 245 may be a positive answer such as "the dish washer includes the tableware", "the dish washer includes the dish", or "include". Alternatively, for the first question Q 235 of "What is included in a dish washer?", the VQA module 240 may obtain, for example, the first VQA result R 245 indicating "tableware" or "bowls and dishes", and the first VOA result R 245 is, for example, a positive answer such as "the dish washer includes the tableware", "the dish washer includes bowls and dishes", "tableware is included", or "bowls and dishes are included". As another example, for the first question Q 235 of "What is the diameter of the dish?" or "Is the diameter of the dish greater than 20 cm?", the VQA module 240 may obtain, for example, the first VQA result R 245 indicating "21 cm" or "yes", and the first VQA result R 245 is, for example, a positive or negative answer such as "the diameter of the dish is 21 cm", "the diameter of the dish is 10 cm", "the diameter of the dish is greater than 20 cm", or "the diameter of the dish is not greater than 20 cm". The VQA module 240 may be implemented according to various embodiments described in this specification with reference to FIG. 4A to FIG. 4C.

[0054] The VQA module 240 may provide the obtained first VQA result R 245 to the scene graph update module 250. The scene graph update module 250 determines whether the first VQA result R 245 indicates that the first condition C 225 is met, and in response to the first VQA result R 245 indicating that the first condition C 225 is met, updates the first scene graph 215 to obtain a second scene graph 215'. When the scene graph update module 250 determines that the first VQA result R 245 indicates that the first condition C 225 is unmet, the skill planning module 220 is directly returned to. The skill planning module 220 selects a next first condition C 225 from the unmet first condition set, and then operations of the foregoing modules 230, 240, and 250 are repeatedly performed based on the first condition C 225. It may be understood that, the operation of determining whether the first VQA result R 245 indicates that the first condition C 225 is met may alternatively be performed by another module other than the scene graph update module 250, and the module may be referred to as a determining module. As described above, the determining module may be in or out of the scene graph update module 250. In an embodiment, the determining module may determine, based on the first condition C 225 and the first VQA result R 245, whether the first VQA result R 245 indicates that the first condition C 225 is met. For example, when the first condition C 225 is a positive condition and the first VQA result R 245 is a positive result, it may be determined that the first condition C 225 is met; when the first condition C 225 and the first VQA result R 245 are both negative, it may be determined that the first condition C 225 is met; and when one of the first condition C 225 and the first VQA result R 245 is negative and the other is positive, it may be determined that the first condition C 225 is unmet.

[0055] That the first VQA result R 245 indicates that the first condition C is met may indicate at least one of the following: an edge representing a particular relationship is to exist or is not to exist in the scene graph 215, a particular node is to exist or is not to exist, or a particular node in the scenario graph is to or is not to be associated with a particular attribute. In the example in FIG. 2, that the first condition C 225 of "the dish washer includes the tableware" is met may indicate that: in the input image 205-based scene graph 215, an edge representing an "include" relationship from the "dish washer" node (red) to other nodes (for example, the other three nodes shown) is to exist. The scene graph update module 250 may update, in response to meeting the first condition C 225, the first scene graph 215, to obtain the second scene graph 215'. In the second scene graph 215', edges representing an "include" relationship from the "dish washer" node (red) to other three nodes are added.

[0056] In another example, although not shown in FIG. 2, in the foregoing example of the unmet condition 1(2) of "a diameter of the dish is greater than 20 cm" corresponding to the skill 1 of "take out the dish whose diameter is greater than 20 cm from the dish washer", that the first VQA result R 245 indicates that the first condition C 225 of "a diameter of the dish is greater than 20 cm" is met may indicate that: in the input image 205-based first scene graph 215, the "dish" node is to have a size attribute of "the diameter is greater than 20 cm". The scene graph update module 250 may update, in response to meeting the first condition C 225, the size attribute of the object in the first scene graph 215, to obtain an updated second scene graph. The scene graph update module 250 may be implemented according to, for example, various embodiments described in this specification with reference to FIG. 5A to FIG. 5C.

[0057] The second scene graph 215' may be provided to the skill planning module 220. The skill planning module 220 may evaluate the second scene graph 215', determine, based on the evaluation, that the condition associated with the selected first skill is met, plan the first skill as the executable action A 255, and input the information used for indicating the executable action A 255 to the action module 260, for the action module 260 to execute the executable action A 255.

[0058] In some embodiments, the first VQA result R 245 may indicate that the first condition C 225 is unmet. In this case, the scene graph update module 250 may skip updating the first scene graph 215 in response to the first VQA result R 245 indicating that the first condition C 225 is unmet, and provide information indicating that the first condition C 225 is unmet to the skill planning module 220, and the skill planning module 220 correspondingly plans the first skill as the unexecutable action. In another example, in this case, the scene graph update module 250 may skip updating the first scene graph 215 in response to the first VQA result R 245 indicating that the first condition C 225 is unmet, and input the first scene graph 215 to the skill planning module 220. In this embodiment, the skill planning module 220 may evaluate the first scene graph 215 again, determine, based on the evaluation, that the condition associated with the first skill is unmet, and plan the first skill as the unexecutable action. The unexecutable action is not performed by the action module 260.

[0059] In the embodiments of this specification, VQA is used to perform task-centric detection on a possibly missing scene graph element, rather than simply relying on elements such as an object, an attribute, and a relationship in the original scene graph. By performing extension on classic scene graph detection in a background of skill reasoning, in a case that an independent skill or task target is considered, the scene graph may be specifically enhanced by using VQA, to additionally include or explicitly exclude a scene graph element related to subsequent skill planning, thereby efficiently improving accuracy of skill planning.

[0060] FIG. 3A is a schematic diagram of a question obtaining module 230 according to an embodiment.

[0061] In an embodiment, the question obtaining module 230 may obtain the first question Q 235 corresponding to the first condition C 225 based at least partially on a predetermined rule. In the embodiment of FIG. 3A, the question obtaining module 230 may include a condition analysis module 310 and a question generation module 320.

[0062] For example, according to the unmet first condition C 225 inputted from the skill planning module 220, the condition analysis module 310 may recognize a scene graph element E 315 associated with the first condition C 225. The scene graph element E 315 includes at least one of an object, an attribute of the object, or an edge used for representing a particular relationship in the scene graph. For example, for the first condition C 225 of "the dish washer includes the tableware", the condition analysis module 310 may recognize that the first condition C 225 is associated with the scene graph element E 315 that includes the following: a "dish washer" object, an "include" relationship, and an object having a "tableware" attribute (or an object corresponding to a tableware instance such as a bowl or a dish, for example, an object classified as "bowl" or an object classified as "dish"). The condition analysis module 310 may input the recognized scene graph element E 315 to the question generation module 320.

[0063] In an embodiment, the question generation module 320 may obtain the first question Q 235 corresponding to the first condition C 225 based at least partially on the scene graph element E 315. For example, the question generation module 320 may generate the first question Q 235 corresponding to the recognized scene graph element E 315 according to the predetermined rule (for example, according to a predetermined format). Following the foregoing example, based on the scene graph element E 315 including the "dish washer" object, the "include" relationship, and the object having the "tableware" attribute (or the object corresponding to the tableware instance), the question generation module 320 may generate the first question Q 235 according to the predetermined rule (for example, the predetermined format) of "whether the object is related to the object having a particular attribute (or an object corresponding to an instance of a particular attribute): "Does a dish washer include tableware?", "Does a dish washer include a bowl?", "Does a dish washer include a dish?", or the like. A person skilled in the art may understand that the foregoing predetermined rule is only an example, and the first question Q 235 corresponding to the first condition C 225 may be obtained by using a different rule.

[0064] In another embodiment, the question generation module 320 may obtain the first question Q 235 corresponding to the first condition C 225 based on the scene graph element E 315 and the first scene graph 215. For example, the question generation module 320 may determine, based on the scene graph 215, that the current scene includes a plurality of objects, such as a bowl and a dish, represented by a plurality of nodes. The question generation module 320 may generate the first question Q 235 based on the scene graph element E 315 and an object represented by one of the plurality of nodes. For example, the question generation module 320 may randomly select a node "bowl" from the plurality of nodes, and generate the first question Q 235 of "Does a dish washer include a bowl?" based on the randomly selected node "bowl" and the scene graph element E 315.

[0065] FIG. 3B is a schematic diagram of a question obtaining module 230 according to an embodiment.

[0066] The question obtaining module 230 may include a neural network model 330. The neural network model 330 may generate the first question Q 235 corresponding to the first condition C 225 based on at least one of the first condition C 225 and a first skill S 225. For example, the neural network model 330 may be implemented through the LLM. For the unmet first condition C 225, at least one of the unmet condition and the corresponding skill may be converted into a corresponding question as an input of the neural network model 330, to generate the corresponding first question Q 235. For example, for the unmet first condition C 225 of "the dish washer includes the tableware", the first condition may be converted into a question of "the condition of 'the dish washer includes the tableware' is unmet, please generate a VQA question" as the input of the neural network model 330. Correspondingly, the neural network model 330 may generate the first question Q 235, such as "Does a dish washer include tableware?". For another example, for the unmet first condition C 225 of "the dish washer includes the tableware", the first condition may be converted into a question of "the skill of 'take out tableware from a dish washer' does not meet the condition, please generate a VQA question", or converted into a question of "the skill of 'take out tableware from a dish washer' does not meet the condition and the condition of 'the dish washer includes the tableware' is unmet, please generate a VQA question". Correspondingly, the neural network model 330 may generate the first question Q 235, such as "Does a dish washer include tableware?".

[0067] In another embodiment, the neural network model 330 may generate the first question Q 235 corresponding to the first condition C 225 based on at least one of the input image 205 and the first scene graph 215, and at least one of the first condition C 225 and the first skill S 225. For example, for the unmet first condition C 225 of "the dish washer includes the tableware", the first condition may be converted into the question described above as an example, and at least one of the input image 205 and the first scene graph 215 is used as the input of the neural network model 330. Correspondingly, the neural network model 330 generates the first question Q 235 associated with the unmet condition, for example, "Does a dish washer include tableware?", "Does a dish washer include a bowl?", and "Does a dish washer include a dish?".

[0068] FIG. 4A is a schematic diagram of a VQA module 240 according to an embodiment.

[0069] The VQA module 240 may include a neural network model 410. The neural network model 410 may generate the first VQA result R 245 based on the first scene graph 215 and the first question Q 235 obtained by using the question obtaining module 230. For example, the neural network model 410 may perform the VQA method on the first scene graph 215 by using the first question Q 235 (for example, "Does a dish washer include tableware?"), to generate the first VQA result R 245 (for example, "yes").

[0070] FIG. 4B is a schematic diagram of a VQA module 240 according to an embodiment.

[0071] The VQA module 240 may include a neural network model 420. The neural network model 420 may generate the first VQA result R 245 based on the input image 205 and the first question Q 235 obtained by using the question obtaining module 230. For example, the neural network model 420 may perform the VQA method on the input image 205 by using the first question Q 235 (for example, "What is included in the dish washer?"), to generate the first VQA result R 245 (for example, "a dish and a bowl"). FIG. 4C is a schematic diagram of a VQA module 240 according to an embodiment.

[0072] The VQA module 240 may include a neural network model 430. The neural network model 430 may generate the first VQA result R 245 based on the input image 205, the first scene graph 215, and the first question Q 235 obtained by using the question obtaining module 230.

[0073] For example, the neural network model 430 may generate the first VQA result R 245 by using the input image 205, the first scene graph 215, and the first question Q 235 as an input.

[0074] For another example, the neural network model 430 may separately perform the VQA method on the input image 205 and the first scene graph 215 by using the first question Q 235, to separately obtain a first intermediate VQA result and a second intermediate VQA result. The neural network model 430 may generate the first VQA result R 245 based on the first intermediate VQA result and the second intermediate VQA result. For example, the first intermediate VQA result and the second intermediate VQA result may each have a confidence about the VQA result, and a confidence of the first VQA result R 245 is calculated based on a weighted first intermediate VQA result and a weighted second intermediate VQA result.

[0075] It may be understood that the neural network model 430 shown in FIG. 4A to FIG. 4C may be implemented by using any suitable VQA model.

[0076] FIG. 5A is a schematic diagram of a scene graph update module 250 according to an embodiment.

[0077] The scene graph update module 250 includes an update indication module 510 and an update logic unit 520. The update indication module 510 may include function logic and / or a neural network model. The update logic unit 520 may include function logic used for updating the scene graph, and is used for adding a node to or deleting a node from a node set of the first scene graph, adding an edge to or deleting an edge from an edge set of the first scene graph, and / or adding an attribute to or deleting an attribute from an attribute set of a particular node set of the first scene graph. Although not shown in the figure, the scene graph update module 250 may further include the foregoing determining module, or the foregoing determining module may be outside the scene graph update module 250.

[0078] The update indication module 510 may generate, based on the first VQA result R 245 indicating that the first condition is met, an indication 1 515 about a scene graph element that is or is not to be included in the first scene graph 215. For example, in response to the first VQA result R 245 indicating that the first condition is met, the update indication module 510 may generate the indication 1 515, which indicates a node that is to or is not to be included in the first scene graph, an attribute of the node, and / or an edge representing a particular relationship.

[0079] In the example in FIG. 5A, in response to the first VQA result R 245 indicating that the first condition is met, the indication 1 515 generated by the update indication module 510 may indicate that: in the first scene graph, edges representing a particular relationship are to exist between a node 1 and nodes 2, 3, and 4. The update indication module 510 may provide the generated indication I 515 to the update logic unit 520. In response to the indication 1 515, the update logic unit 520 may add a corresponding edge to the edge set of the first scene graph 215, to obtain a second scene graph 525. For example, based on the unmet first condition C 225 of "the dish washer includes the tableware" and the positive first VQA result R 245, such as "yes", "include", or "the dish washer includes the tableware", the update indication module 510 may determine that the elements, that is, the "dish washer" object, the "include" relationship, and the "tableware" object in the first condition, are all met, and further determine an update indication based on the scene graph 215: an edge indicating an "include" relationship from the node 1 to the node 2 is added between the node 1 (corresponding to the dish washer) and the node 2 (corresponding to the tableware instance "bowl"), an edge indicating an "include" relationship from the node 1 to the node 3 is added between the node 1 (corresponding to the dish washer) and the node 3 (corresponding to the tableware instance "bowl"), and an edge indicating an "include" relationship from the node 1 to the node 4 is added between the node 1 (corresponding to the dish washer) and the node 4 (corresponding to the tableware instance "dish"). The update logic unit 520 correspondingly updates the first scene graph 215 according to the indication I 515 to obtain the updated second scene graph 525, to which an edge indicating the "include" relationship between the dish washer node and each tableware instance is added.

[0080] FIG. 5B is a schematic diagram of a scene graph update module 250 according to an embodiment. The update indication module 510 and the update logic unit 520 shown in FIG. 5B are the same as or similar to the corresponding modules or units shown in FIG. 5A, and details of the update indication module 510 and the update logic unit 520 are not described again. The same or similar component in the figure is represented by the same or similar reference sign. In the example in FIG. 5B, in response to the first VQA result R 245 indicating that the first condition is met, the indication 1 515 generated by the update indication module 510 may indicate that: the node 4 in the first scene graph is to be associated with a first attribute. Correspondingly, the update logic unit 520 updates attribute information of the node 4, to obtain the second scene graph 525. For example, based on the unmet first condition C 225 of "a diameter of the dish is greater than 20 cm" and the positive first VQA result R 245, such as "yes", "greater than", or "a diameter of the dish is greater than 20 cm", the update indication module 510 may determine that the elements, that is, the "dish" object and an attribute of "a diameter is greater than 20 cm" in the first condition, are all met, and further determine an update indication based on the scene graph 215: a size attribute of the node 4 is updated to "a diameter is greater than 20 cm". The update logic unit 520 correspondingly updates the first scene graph 215 based on the indication I 515 to obtain the updated second scene graph 525, in which the size attribute of the node 4 is updated.

[0081] FIG. 5C is a schematic diagram of a scene graph update module 250 according to an embodiment.

[0082] The scene graph update module 250 includes the update indication module 510 and a scene graph regeneration module 530. The update indication module 510 may include function logic and / or a neural network model. The scene graph regeneration module 530 may include a neural network model configured to regenerate scene graph data.

[0083] The update indication module 510 may generate, based on the first VQA result R 245 indicating that the first condition is met, the indication 1 515 about a scene graph element that is or is not to be included in the first scene graph 215.

[0084] In the example in FIG. 5C, in response to the first VQA result R 245 indicating that the first condition is met, the indication 1 515 generated by the update indication module 510 may indicate that: a node 5 is to exist in the first scene graph. Optionally, the indication 1 515 may further indicate a particular edge or node attribute that is to or is not to be included in the first scene graph. The update indication module 510 may provide the generated indication I 515 to the scene graph regeneration module 530. The indication 1 515 can improve accuracy of object detection, accuracy of relationship prediction, or disambiguation between similar objects and relationships. For example, the indication 1 515 can help the scene graph regeneration module 530 to more accurately generate, based on the input image 205, a second scene graph 535 that includes or does not include an indicated scene graph element.

[0085] Based on the indication I 515 and the input image 205, the scene graph regeneration module 530 may generate the second scene graph 535. For example, based on the unmet first condition C 225 of "the dish washer includes a dish whose diameter is greater than 20 cm" and the positive first VQA result R 245, such as "yes", "include", or "the dish washer includes a dish whose diameter is greater than 20 cm", the update indication module 510 may determine that the elements, that is, the "dish washer" object, the "include" relationship, the "dish" object, and the attribute of "a diameter is greater than 20 cm" in the first condition, are all met, and further determine an update indication I 515 based on the scene graph 215: the dish washer includes a dish whose diameter is greater than 20 cm. The scene graph regeneration module 530 regenerates the second scene graph 535 based on the indication 1 515 (or referred to as a prompt) and the input image 205, where the second scene graph 535 includes an additional node, namely, the node 5. The node 5 represents the "dish" object, and includes attribute information of "a diameter is greater than 20 cm", and the second scene graph 535 includes an edge indicating an "include" relationship from the node 1 to the node 5.

[0086] Although in the examples shown in FIG. 5 A to FIG. 5C, the update indication module 510 generates the update indication 1515 based on the first condition C 225 and the first VQA result R 245, in other examples, the update indication module 510 may alternatively generate the update indication 1 515 based on the first question Q 235 and the first VQA result R 245, or the update indication module 510 may alternatively generate the update indication 1 515 based on the first condition C 225, the first question Q 235, and the first VQA result R 245.

[0087] FIG. 6 is a block diagram of a scene graph-based skill planning apparatus 600 according to another embodiment. Modules in the apparatus 600 shown in FIG. 6 are the same as or similar to corresponding modules in the apparatus 200 shown in FIG. 2, and details of the modules are not described again. The same or similar component in the figure is represented by the same or similar reference sign.

[0088] In the example shown in FIG. 6, the apparatus 600 may be a part of an automobile, and the apparatus 600 receives an input image 605 from, for example, a camera sensor. A scene graph generation module 610 generates a first scene graph 615 based on the input image 605. In the example in FIG. 6, the scene graph 615 includes a node 1 (red) associated with a "zebra crossing" object and a node 2 (pink) associated with an "automobile" object. The apparatus 600 may include a task target of autonomous driving. For the task target and / or based on a skill reasoning algorithm, a skill planning module 620 may select a first skill for running through an intersection. For ease of description, in this example, the first skill is associated with only a first condition of "no pedestrian exists on the zebra crossing". The skill planning module 620 may determine, based on the first scene graph 615, that a first condition C 625 associated with the selected skill is met.

[0089] A condition associated with a skill may be associated with a corresponding condition priority. The condition priority may indicate severity, importance, urgence, basality, or the like associated with the condition. For example, for a condition with a low condition priority, in a case that the skill planning module determines that the condition is met, the skill planning module may plan a skill associated with the condition as an executable action, for an action module to execute the executable action. However, for a condition with a high condition priority, even if the skill planning module determines, based on the first scene graph for the first time, that the condition is met, the skill planning module determines again whether the condition is indeed met, and plans a skill associated with the condition as an executable action only if the condition is indeed met. Otherwise, if it is determined that the condition is actually unmet, the first scene graph is updated, and the skill associated with the condition is planned as an unexecutable action. Repeated confirmation on a met condition can help improve certainty of skill planning and robustness of task execution.

[0090] In the example in FIG. 6, the first condition C 625 may be associated with a high condition priority, and therefore, the skill planning module 620 does not directly plan the first skill associated with the first condition C 625 as the executable action, but inputs the first condition C 625 of "no pedestrian exists on the zebra crossing" to a question obtaining module 630, to determine again whether the first condition C 625 is met.

[0091] According to the embodiments in this specification, the question obtaining module 630 may obtain a corresponding first question Q 635 based on the first condition C 625. In the example in FIG. 6, the first question Q 635 may include: "Does a pedestrian exist on the zebra crossing?", "What is on the zebra crossing?", or the like.

[0092] The question obtaining module 630 may provide the obtained first question Q 635 to a VQA module 640. The VQA module 640 obtains a first VQA result R 645 for the first question Q 635 by performing VQA by using the first question Q 635. In the example in FIG. 6, for the first question Q 635 of "Does a pedestrian exist on the zebra crossing?" or "What is on the zebra crossing?", the VQA module 640 may obtain, for example, the first VQA result R 645 indicating "yes" or "pedestrian", and the first VQA result R 645 may be, for example, a positive VQA result such as "yes", "pedestrian", "a pedestrian exists on the zebra crossing", or "a pedestrian is on the zebra crossing".

[0093] The VQA module 640 may input the obtained first VQA result R 645 to a scene graph update module 650. A condition determining module in the scene graph update module 650 determines, based on the first VQA result R 645 and the first condition C 625, that the first condition C 625 of "no pedestrian exists on the zebra crossing" is unmet. The scene graph update module 650 updates, in response to the first VQA result R 645 indicating that the first condition C 625 is unmet, the first scene graph 615 to obtain a second scene graph 615'. For example, any implementation shown in FIG. 5A to FIG. 5C may be used to update the first scene graph 615 to obtain the second scene graph 615 ', which additionally includes an object "pedestrian" (a blue node) and an edge indicating a relationship of "including" or "overlapping" between an object "zebra crossing" (a red node) and the object "pedestrian" (the blue node).

[0094] The second scene graph 615' may be provided to the skill planning module 620. The skill planning module 620 may evaluate the second scene graph 615', determine, based on the evaluation, that the first condition C 625 associated with the selected first skill is unmet, and plan the first skill as the unexecutable action. The unexecutable action is not performed by the action module 560.

[0095] In some embodiments, the first VQA result R 645 may indicate that the first condition C 225 is met. In this case, the scene graph update module 650 may skip updating the first scene graph 615 in response to the first VQA result R 645 indicating that the first condition C 225 is met, and provide information indicating that the first condition C 225 is met to the skill planning module 620. The skill planning module 620 correspondingly plans the first skill as the executable action. In another example, the scene graph update module 650 may skip updating the first scene graph 615 in response to the first VQA result R 645 indicating that the first condition C 225 is met, and input the first scene graph 615 to the skill planning module 620. In this example, the skill planning module 620 may evaluate the first scene graph 615 again, determine, based on the evaluation, that the condition associated with the first skill is met, plan the first skill as the executable action, and input the information used for indicating the executable action to the action module 560, for the action module 560 to control execution of the executable action.

[0096] FIG. 7 is a flowchart of a scene graph-based skill planning method 700 according to an embodiment.

[0097] Step 710: Generate a first scene graph based at least partially on an input image.

[0098] Step 720: Determine, based at least partially on the first scene graph, a first condition that is unmet and that is associated with a first skill.

[0099] Step 730: Obtain a first question corresponding to the first condition.

[0100] Step 740: Obtain a first VQA result for the first question by performing VQA by using the first question.

[0101] Step 750: Update, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain a second scene graph.

[0102] Step 760: Plan the first skill as an executable action based at least partially on the second scene graph.

[0103] According to an embodiment, in step 720, one or more skills to be executed are selected, each of the one or more skills being associated with one or more conditions; an unmet first condition set is determined based at least partially on the first scene graph, the first condition set including an unmet condition associated with an unmet skill in the one or more skills; and the first condition in the first condition set is determined, the first condition being associated with the first skill in the one or more skills.

[0104] According to an embodiment, that the first condition in the first condition set is determined further includes at least one of the following: the first condition is randomly determined in the first condition set; the first skill that is associated with a minimum condition set is determined in the unmet skill, and the first condition is determined in the minimum condition set; or the first skill having a highest priority is determined in the unmet skill, and the first condition is determined in one or more conditions associated with the first skill.

[0105] According to an embodiment, in step 730, the first question corresponding to the first condition is obtained based at least partially on a predetermined rule; or the first question corresponding to the first condition is obtained by using a first neural network model. That the first question corresponding to the first condition is obtained based at least partially on a predetermined rule further includes: at least one of a relationship, an object, or an attribute associated with the first condition is recognized; and the first question corresponding to the first condition is obtained based at least partially on at least one of the recognized relationship, the recognized object, or the recognized attribute.

[0106] According to an embodiment, that the first question corresponding to the first condition is obtained by using a first neural network model further includes: the first neural network model generates, based on at least one of the first condition and the first skill, the first question corresponding to the first condition; or the first neural network model generates, based on at least one of the input image and the first scene graph and at least one of the first condition and the first skill, the first question corresponding to the first condition.

[0107] According to an embodiment, in step 740, a second neural network model generates the first VQA result based on at least one of the first scene graph and the input image and the first question.

[0108] According to an embodiment, the first scene graph includes a node set and an edge set, nodes in the node set respectively correspond to different objects in the input image, and an edge in the edge set is between two nodes in the node set and represents a relationship between the two nodes.

[0109] According to an embodiment, in step 750, a first edge is added to or deleted from the edge set in response to meeting the first condition indicating that the first edge is to exist or not to exist in the edge set, to obtain the second scene graph; or the second scene graph including or not including a second node is obtained in response to meeting the first condition indicating that the second node is to exist or not to exist in the node set.

[0110] According to an embodiment, that the second scene graph including or not including the second node is obtained in response to meeting the first condition indicating that a second node is to exist or not to exist in the node set includes at least one of the following: in response to meeting the first condition indicating that the second node is not to exist in the node set, the second node is deleted from the node set, and an edge associated with the second node is deleted from the edge set, to obtain the second scene graph; in response to meeting the first condition indicating that the second node is to exist in the node set, the second node is added to the node set, to obtain the second scene graph; or an indication about existence or nonexistence of the second node is generated based at least partially on meeting the first condition indicating that the second node is to exist or not to exist in the node set, and the second scene graph is generated based at least partially on the indication and the input image.

[0111] According to an embodiment, the nodes in the node set are associated with an attribute set of a corresponding object in the input image.

[0112] According to an embodiment, in step 750, in response to meeting the first condition indicating that a first node in the node set is to be or not to be associated with a first attribute, the first attribute is added to or deleted from the attribute set associated with the first node, to obtain the second scene graph.

[0113] According to an embodiment, the method 700 further includes: determining, based at least partially on the second scene graph, a second condition that is met, the second condition being associated with a second skill; obtaining a second question corresponding to the second condition; obtaining a second VQA result for the second question by performing VQA by using the second question; updating, in response to the second VQA result indicating that the second condition is unmet, the second scene graph to obtain a third scene graph; and planning the second skill as an unexecutable action based at least partially on the third scene graph.

[0114] According to an embodiment, the second condition is associated with a high condition priority.

[0115] FIG. 8 is a flowchart of a scene graph-based skill planning method 800 according to an embodiment.

[0116] Step 810: Generate a first scene graph based at least partially on an input image.

[0117] Step 820: Determine, based at least partially on the first scene graph, a first condition that is met and that is associated with a first skill.

[0118] Step 830: Obtain a first question corresponding to the first condition.

[0119] Step 840: Obtain a first VQA result for the first question by performing visual question answering (VQA) by using the first question.

[0120] Step 850: Update, in response to the first VQA result indicating that the first condition is unmet, the first scene graph to obtain a second scene graph.

[0121] Step 860: Plan the first skill as an unexecutable action based at least partially on the second scene graph.

[0122] According to an embodiment, the first condition is associated with a high condition priority.

[0123] According to an embodiment, in step 830, the first question corresponding to the first condition is obtained based at least partially on a predetermined rule; or the first question corresponding to the first condition is obtained by using a first neural network model. That the first question corresponding to the first condition is obtained based at least partially on a predetermined rule further includes: at least one of a relationship, an object, or an attribute associated with the first condition is recognized; and the first question corresponding to the first condition is obtained based at least partially on at least one of the recognized relationship, the recognized object, or the recognized attribute.

[0124] According to an embodiment, that the first question corresponding to the first condition is obtained by using a first neural network model further includes: the first neural network model generates, based on at least one of the first condition and the first skill, the first question corresponding to the first condition; or the first neural network model generates, based on the input image or the first scene graph and the first condition or the first skill, the first question corresponding to the first condition.

[0125] According to an embodiment, in step 840, a second neural network model generates the first VQA result based on at least one of the first scene graph and the input image and the first question.

[0126] According to an embodiment, the first scene graph includes a node set and an edge set, nodes in the node set respectively correspond to different objects in the input image, and an edge in the edge set is between two nodes in the node set and represents a relationship between the two nodes.

[0127] According to an embodiment, in step 850, a first edge is added to or deleted from the edge set in response to not meeting the first condition indicating that the first edge is to exist or not to exist in the edge set, to obtain the second scene graph; or the second scene graph including or not including a second node is obtained in response to not meeting the first condition indicating that the second node is to exist or not to exist in the node set.

[0128] According to an embodiment, that the second scene graph including or not including a second node is obtained in response to not meeting the first condition indicating that the second node is to exist or not to exist in the node set includes at least one of the following: in response to not meeting the first condition indicating that the second node is not to exist in the node set, the second node is deleted from the node set, and an edge associated with the second node is deleted from the edge set, to obtain the second scene graph; in response to not meeting the first condition indicating that the second node is to exist in the node set, the second node is added to the node set, to obtain the second scene graph; or an indication about existence or nonexistence of the second node is generated based at least partially on not meeting the first condition indicating that the second node is to exist or not to exist in the node set, and the second scene graph is generated based at least partially on the indication and the input image.

[0129] According to an embodiment, the nodes in the node set are associated with an attribute set of a corresponding object in the input image.

[0130] According to an embodiment, in step 850, in response to not meeting the first condition indicating that a first node in the node set is to be or not to be associated with a first attribute, the first attribute is added to or deleted from the attribute set associated with the first node, to obtain the second scene graph.

[0131] FIG. 9 is a block diagram of a scene graph-based skill planning apparatus according to an embodiment.

[0132] An apparatus 900 includes: a scene graph generation module 910, a skill planning module 920, a question obtaining module 930, a VQA module 940, and a scene graph update module 950. The scene graph generation module 910 generates a first scene graph based at least partially on an input image. The skill planning module 920 determines, based at least partially on the first scene graph, a first condition that is unmet and that is associated with a first skill. The question obtaining module 930 obtains a first question corresponding to the first condition. The VQA module 940 obtains a first VQA result for the first question by performing VQA by using the first question. The scene graph update module 950 updates, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain a second scene graph. The skill planning module 920 plans the first skill as an executable action based at least partially on the second scene graph.

[0133] According to an embodiment, the skill planning module 920 selects one or more skills to be executed, each of the one or more skills being associated with one or more conditions; determines an unmet first condition set based at least partially on the first scene graph, the first condition set including an unmet condition associated with an unmet skill in the one or more skills; and determines the first condition in the first condition set, the first condition being associated with the first skill in the one or more skills.

[0134] According to an embodiment, the skill planning module 920 determines the first condition in the first condition set through at least one of the following: randomly determining the first condition in the first condition set; determining, in the unmet skill, the first skill that is associated with a minimum condition set, and determining the first condition in the minimum condition set; or determining, in the unmet skill, the first skill having a highest priority, and determining the first condition in one or more conditions associated with the first skill.

[0135] According to an embodiment, the question obtaining module 930 obtains the first question corresponding to the first condition through the following: obtaining, based at least partially on a predetermined rule, the first question corresponding to the first condition; or obtaining, by using a first neural network model, the first question corresponding to the first condition.

[0136] According to an embodiment, the question obtaining module 930 obtains, based at least partially on the predetermined rule, the first question corresponding to the first condition through the following: recognizing at least one of a relationship, an object, or an attribute associated with the first condition; and obtaining, based at least partially on at least one of the recognized relationship, the recognized object, or the recognized attribute, the first question corresponding to the first condition.

[0137] According to an embodiment, the question obtaining module 930 obtains the first question corresponding to the first condition by using the first neural network model through the following: generating, by the first neural network model based on at least one of the first condition and the first skill, the first question corresponding to the first condition; or generating, by the first neural network model based on at least one of the input image and the first scene graph and at least one of the first condition and the first skill, the first question corresponding to the first condition.

[0138] According to an embodiment, the VQA module 940 obtains the first VQA result for the first question by performing VQA by using the first question through the following: generating, by a second neural network model, the first VQA result based on at least one of the first scene graph and the input image and the first question.

[0139] According to an embodiment, the first scene graph includes a node set and an edge set, nodes in the node set respectively correspond to different objects in the input image, and an edge in the edge set is between two nodes in the node set and represents a relationship between the two nodes.

[0140] According to an embodiment, the scene graph update module 950 updates, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain the second scene graph through at least one of the following: adding a first edge to or deleting a first edge from the edge set in response to meeting the first condition indicating that the first edge is to exist or not to exist in the edge set, to obtain the second scene graph; or obtaining, in response to meeting the first condition indicating that a second node is to exist or not to exist in the node set, the second scene graph including or not including the second node.

[0141] According to an embodiment, the nodes in the node set are associated with an attribute set of a corresponding object in the input image.

[0142] According to an embodiment, the scene graph update module 950 updates, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain the second scene graph through at least one of the following: in response to meeting the first condition indicating that a first node in the node set is to be or not to be associated with a first attribute, adding the first attribute to or deleting the first attribute from the attribute set associated with the first node, to obtain the second scene graph.

[0143] According to an embodiment, the scene graph update module 950 obtains, in response to meeting the first condition indicating that the second node is to exist or not to exist in the node set, the second scene graph including or not including the second node through at least one of the following: in response to meeting the first condition indicating that the second node is not to exist in the node set, deleting the second node from the node set, and deleting an edge associated with the second node from the edge set, to obtain the second scene graph; in response to meeting the first condition indicating that the second node is to exist in the node set, adding the second node to the node set, to obtain the second scene graph; or generating an indication about existence or nonexistence of the second node based at least partially on meeting the first condition indicating that the second node is to exist or not to exist in the node set, and generating the second scene graph based at least partially on the indication and the input image.

[0144] According to an embodiment, the scene graph generation module 910 generates a third scene graph based at least partially on the second scene graph. The skill planning module 920 determines, based at least partially on the third scene graph, a second condition that is met, the second condition being associated with a second skill. The question obtaining module 930 obtains a second question corresponding to the second condition. The VQA module 940 obtains a second VQA result for the second question by performing VQA by using the second question. The scene graph update module 950 updates, in response to the second VQA result indicating that the second condition is unmet, the third scene graph to obtain a fourth scene graph. The skill planning module 920 plans the second skill as an unexecutable action based at least partially on the fourth scene graph. In further various implementations of this embodiment, the modules 910 to 950 may separately perform corresponding operations described above with reference to FIG. 8, and details are not described herein again.

[0145] According to an embodiment, the second condition is associated with a high condition priority.

[0146] FIG. 10 is a block diagram of a processing apparatus according to an embodiment.

[0147] A processing apparatus or processing system 1000 includes one or more control units or processing units 1010, which execute one or more machine-readable instructions stored or encoded in a machine-readable storage medium (namely, a memory 1020). In an embodiment, when executing program instructions, the processing unit 1010 is configured to perform various operations and functions described above with reference to FIG. 1 to FIG. 9.

[0148] Although not shown in FIG. 1, FIG. 6, and FIG. 9, a person skilled in the art can understand that the apparatus 100, 200, 600, or 800 may further include various other components, such as various communication modules, bus modules, and possibly user interface modules.

[0149] According to an embodiment, a program product such as a non-transitory machine- readable medium is provided. The non-transitory machine-readable medium may have instructions. The instructions, when executed by the processing unit 1010, can perform various operations and functions described above with reference to FIG. 1 to FIG. 9 in the embodiments of this application.

[0150] According to an embodiment, a computer program product is provided. The computer program product includes machine-executable instructions. When the instructions are executed by the processing unit 1010, various operations and functions described above with reference to FIG. 1 to FIG. 9 in the embodiments of this application can be performed.

[0151] Exemplary embodiments are described above in combination with specific implementations illustrated in the accompanying drawings, but this does not represent all embodiments that may be implemented or fall within the protection scope of the claims. A term "exemplary" used in the entire specification means "used as an example, an instance, or an illustration", and does not mean "preferred" or "superior" over other embodiments. To provide an understanding of the described technologies, the specific implementations include specific details. However, these technologies may be implemented without these specific details. In some embodiments, to avoid confusing the concept of the described embodiments, a well- known structure and apparatus are shown in a block diagram form.

[0152] The above descriptions of the content of this application are provided to allow any person of ordinary skill in the art to implement or use the content of this application. Various modifications to the content of this application are obvious to a person of ordinary skill in the art, and the general principles defined in this specification may also be applied to other variations without departing from the protection scope of the content of this application. Therefore, the content of this application is not limited to the examples and designs described in this specification, but is consistent with the widest range conforming to the principle and novelty disclosed in this specification.

Claims

CLAIMSWhat is claimed is:

1. A scene graph-based skill planning method, comprising: generating a first scene graph based at least partially on an input image; determining, based at least partially on the first scene graph, a first condition that is unmet and that is associated with a first skill; obtaining a first question corresponding to the first condition; obtaining a first VQA result for the first question by performing visual question answering (VQA) by using the first question; updating, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain a second scene graph; and planning the first skill as an executable action based at least partially on the second scene graph.

2. The method according to claim 1, wherein the determining, based at least partially on the first scene graph, a first condition that is unmet and that is associated with a first skill further comprises: selecting one or more skills to be executed, each of the one or more skills being associated with one or more conditions; determining an unmet first condition set based at least partially on the first scene graph, the first condition set comprising an unmet condition associated with an unmet skill in the one or more skills; and determining the first condition in the first condition set, the first condition being associated with the first skill in the one or more skills.

3. The method according to claim 2, wherein the determining the first condition in the first condition set further comprises at least one of the following: randomly determining the first condition in the first condition set; determining, in the unmet skill, the first skill that is associated with a minimum condition set, and determining the first condition in the minimum condition set; ordetermining, in the unmet skill, the first skill having a highest priority, and determining the first condition in one or more conditions associated with the first skill.

4. The method according to claim 1, wherein the obtaining a first question corresponding to the first condition further comprises: obtaining, based at least partially on a predetermined rule, the first question corresponding to the first condition; or obtaining, by using a first neural network model, the first question corresponding to the first condition.

5. The method according to claim 4, wherein the obtaining, based at least partially on a predetermined rule, the first question corresponding to the first condition further comprises: recognizing at least one of a relationship, an object, or an attribute associated with the first condition; and obtaining, based at least partially on at least one of the recognized relationship, the recognized object, or the recognized attribute, the first question corresponding to the first condition.

6. The method according to claim 4, wherein the obtaining, by using a first neural network model, the first question corresponding to the first condition further comprises: generating, by the first neural network model based on at least one of the first condition and the first skill, the first question corresponding to the first condition; or generating, by the first neural network model based on at least one of the input image and the first scene graph and at least one of the first condition and the first skill, the first question corresponding to the first condition.

7. The method according to claim 1, wherein the obtaining a first VQA result for the first question by performing VQA by using the first question further comprises: generating, by a second neural network model, the first VQA result based on at least one of the first scene graph and the input image and the first question.

8. The method according to claim 1, wherein the first scene graph comprises a node set and an edge set, nodes in the node set respectively correspond to different objects in the input image, and an edge in the edge set is between two nodes in the node set and represents a relationship between the two nodes; and the updating, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain a second scene graph comprises at least one of the following: adding a first edge to or deleting a first edge from the edge set in response to meeting the first condition indicating that the first edge is to exist or not to exist in the edge set, to obtain the second scene graph; or obtaining, in response to meeting the first condition indicating that a second node is to exist or not to exist in the node set, the second scene graph comprising or not comprising the second node.

9. The method according to claim 8, wherein the nodes in the node set are associated with an attribute set of a corresponding object in the input image, and the updating, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain a second scene graph further comprises: in response to meeting the first condition indicating that a first node in the node set is to be or not to be associated with a first attribute, adding the first attribute to or deleting the first attribute from an attribute set associated with the first node, to obtain the second scene graph.

10. The method according to claim 8, wherein the obtaining, in response to meeting the first condition indicating that a second node is to exist or not to exist in the node set, the second scene graph comprising or not comprising the second node comprises at least one of the following: in response to meeting the first condition indicating that the second node is not to exist in the node set, deleting the second node from the node set, and deleting an edge associated with the second node from the edge set, to obtain the second scene graph; in response to meeting the first condition indicating that the second node is to exist in the node set, adding the second node to the node set, to obtain the second scene graph; or generating an indication about existence or nonexistence of the second node based at leastpartially on meeting the first condition indicating that the second node is to exist or not to exist in the node set, and generating the second scene graph based at least partially on the indication and the input image.

11. The method according to claim 1, further comprising: determining, based at least partially on the second scene graph, a second condition that is met, the second condition being associated with a second skill; obtaining a second question corresponding to the second condition; obtaining a second VQA result for the second question by performing VQA by using the second question; updating, in response to the second VQA result indicating that the second condition is unmet, the second scene graph to obtain a third scene graph; and planning the second skill as an unexecutable action based at least partially on the third scene graph.

12. The method according to claim 11, wherein the second condition is associated with a high condition priority.

13. A scene graph-based skill planning method, comprising: generating a first scene graph based at least partially on an input image; determining, based at least partially on the first scene graph, a first condition that is met and that is associated with a first skill; obtaining a first question corresponding to the first condition; obtaining a first VQA result for the first question by performing visual question answering (VQA) by using the first question; updating, in response to the first VQA result indicating that the first condition is unmet, the first scene graph to obtain a second scene graph; and planning the first skill as an unexecutable action based at least partially on the second scene graph.

14. The method according to claim 13, wherein the first condition is associated with ahigh condition priority.

15. A scene graph-based skill planning apparatus, comprising: a scene graph generation module, configured to generate a first scene graph based at least partially on an input image; a skill planning module, configured to determine, based at least partially on the first scene graph, a first condition that is unmet and that is associated with a first skill; a question obtaining module, configured to obtain a first question corresponding to the first condition; a visual question answering (VQA) module, configured to obtain a first VQA result for the first question by performing VQA by using the first question; and a scene graph update module, configured to update, in response to the first VQA result indicating that the first condition is met, the first scene graph to obtain a second scene graph; the skill planning module planning the first skill as an executable action based at least partially on the second scene graph.

16. A scene graph-based skill planning apparatus, comprising: a scene graph generation module, configured to generate a first scene graph based at least partially on an input image; a skill planning module, configured to determine, based at least partially on the first scene graph, a first condition that is met and that is associated with a first skill; a question obtaining module, configured to obtain a first question corresponding to the first condition; a visual question answering (VQA) module, configured to obtain a first VQA result for the first question by performing VQA by using the first question; and a scene graph update module, configured to update, in response to the first VQA result indicating that the first condition is unmet, the first scene graph to obtain a second scene graph; the skill planning module planning the first skill as an unexecutable action based at least partially on the second scene graph.

17. A scene graph-based skill planning apparatus, comprising:one or more processors; and one or more memories, storing computer-executable instructions, the instructions, when executed by the one or more processors, performing the operations of any one of claims 1 to 14.

18. A machine-readable storage medium, storing executable instructions, the instructions, when executed, causing one or more processors to perform the method according to any one of claims 1 to 14.

19. A computer program product, comprising executable instructions, the instructions, when executed, causing one or more processors to perform the method according to any one of claims 1 to 14.

Images