Robot control method, device, control apparatus, and storage medium

Abstract

The application provides a robot control method and device, a control equipment and a storage medium, and relates to the technical field of intelligent control. The method comprises the following steps: obtaining an input task command for a target robot, and a static scene image of a task scene collected by the target robot based on the task command; generating a task action sequence corresponding to the task command and object space state information corresponding to each action in the task action sequence by using a preset visual space reasoning model according to the static scene image and the task command; generating an action sequence instruction corresponding to the task command by using a preset task generator according to the task action sequence and the object space state information corresponding to each action; and sequentially controlling the target robot to perform corresponding actions according to each action instruction in the action sequence instruction. The application improves the control efficiency and accuracy of the robot to meet the control requirements in complex scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent control, in particular to a robot control method and device, a control equipment and a storage medium. BACKGROUND

[0002] With the rapid development of robot technology, the application scenarios of robots are increasingly complex, gradually expanding from structured industrial environments to open dynamic environments. Single modal information is difficult to cope with environmental uncertainty and task diversity, so robot control technology based on multi-modal models has become a research hotspot.

[0003] Currently, multi-modal models focus on fusing multi-source perception data such as vision, touch, voice, and inertial measurement in the field of robot control, and using intelligent algorithms such as deep learning and reinforcement learning to achieve adaptive human-robot interaction and autonomous decision-making capabilities, thereby enabling robots to perform tasks more intelligently in complex environments.

[0004] However, when dealing with spatial reasoning tasks, there is a significant defect of over-reliance on implicit spatial knowledge, which often deviates from the geometric basis relationship between objects for reasoning, and thus leads to fragmentation of the reasoning chain. When faced with tasks that comprehensively consider object functions and topological constraints, the sequential decision-making ability of such models is easily broken down, ultimately resulting in low efficiency and accuracy of multi-robot collaborative control, which is difficult to meet the actual application requirements in complex scenarios. SUMMARY

[0005] The present application aims to overcome the deficiencies in the prior art and provide a robot control method, device, control equipment and storage medium to improve the control efficiency and accuracy of robots to meet the control requirements in complex scenarios.

[0006] To achieve the above-mentioned purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a robot control method, which comprises:

[0008] obtaining an input task command for a target robot, and a static scene image of a task scene collected by the target robot based on the task command;

[0009] using a preset visual spatial reasoning model to generate a task action sequence corresponding to the task command and object space state information corresponding to each action in the task action sequence according to the static scene image and the task command, wherein the object space state information corresponding to each action is used to represent the spatial state change information of each object in the static scene image under each action;

[0010] According to the task action sequence and the object space state information corresponding to each action, a preset task generator is used to generate an action sequence instruction corresponding to the task command;

[0011] According to each action instruction in the action sequence instruction, the target robot is sequentially controlled to perform a corresponding action.

[0012] Optionally, the preset visual space reasoning model includes a graph node extractor, a visual auxiliary extractor, a scene image generator, and a space reasoning module. According to the static scene image and the task command, a preset visual space reasoning model is used to generate a task action sequence corresponding to the task command and object space state information corresponding to each action in the task action sequence, which includes:

[0013] The graph node extractor is used to perform node extraction on the static scene image to obtain node recognition information in the static scene image;

[0014] The visual auxiliary extractor is used to perform feature extraction on the node recognition information to obtain scene visual features corresponding to the static scene image;

[0015] The scene image generator is used to generate a dynamic structured scene image corresponding to the static scene image according to the scene visual features, and the dynamic structured scene image includes attribute information of each object and a spatial position relationship between the objects;

[0016] The space reasoning module is used to perform space reasoning on the dynamic structured scene image according to the task command to obtain the task action sequence and object space state information corresponding to each action in the task action sequence.

[0017] Optionally, the visual auxiliary extractor includes an object detection network and a depth estimation network. The visual auxiliary extractor is used to perform feature extraction on the node recognition information to obtain scene visual features corresponding to the static scene image, which includes:

[0018] The object detection network is used to perform object detection according to the node recognition information to generate attribute information of each object in the static scene image;

[0019] The depth estimation network is used to perform depth estimation according to the node recognition information to obtain depth estimation information of the task scene, and the scene visual features include the attribute information of each object and the depth estimation information.

[0020] Optionally, the node extraction on the static scene image by the graph node extractor to obtain the node recognition information in the static scene image comprises:

[0021] If the task command is a reorganization task command, the node extraction on the static scene image by the graph node extractor to obtain the recognition information of each node in the static scene image, the recognition information of each node comprising: the spatial occupancy information of each node, the node recognition information comprising: the attribute information of each node and the spatial occupancy information of each node.

[0022] Optionally, the method further comprises:

[0023] acquiring an input first question for the target robot, and a first scene image collected by the target robot based on the first question;

[0024] generating, according to the first question and the first scene image, first visual space reasoning information of the first scene image under the first question by using the preset visual space reasoning model; the first visual space reasoning information comprising: spatial state information of each object in the first scene image;

[0025] generating, according to the first visual space reasoning information, first answer information corresponding to the first question by using the preset task generator;

[0026] controlling the target robot to play the first answer information.

[0027] Optionally, the method further comprises:

[0028] acquiring sample video data in a preset space data set and a preset question-answer pair corresponding to the sample video data;

[0029] generating, according to a second question in the preset question-answer pair and a second scene image corresponding to the second question in the sample video data, second visual space reasoning information of the second scene image under the second question by using the preset visual space reasoning model; the second visual space reasoning information comprising: spatial state information of each object in the second scene image;

[0030] generating, according to the second visual space reasoning information, second answer information corresponding to the second question by using the preset task generator;

[0031] testing the preset visual space reasoning model according to the second answer information and a preset answer corresponding to the second question in the preset question-answer pair.

[0032] Optionally, before obtaining the sample video data in the input preset spatial data set and the preset question-answer pairs corresponding to the sample video data, the method further includes:

[0033] Acquire the sample video data and the preset task category corresponding to the sample video data;

[0034] Based on the preset task category and the sample video data, a preset question-answer pair corresponding to the sample video data is generated.

[0035] In a second aspect, another embodiment of the present application provides a control device for a robot, the device comprising:

[0036] An acquisition module is used to acquire an input task command for a target robot and a static scene image of the task scene in which the target robot is located, which is collected based on the task command;

[0037] a generation module, configured to generate, based on the static scene image and the task command, a task action sequence corresponding to the task command and object spatial state information corresponding to each action in the task action sequence using a preset visual-spatial reasoning model; wherein the object spatial state information corresponding to each action is used to represent spatial state change information of each object in the static scene image under each action;

[0038] A generating module, configured to generate an action sequence instruction corresponding to the task command using a preset task generator according to the task action sequence and the object space state information corresponding to each action;

[0039] The control module is used to control the target robot to perform corresponding actions in sequence according to each action instruction in the action sequence instruction.

[0040] In the third aspect, another embodiment of the present application provides a robot control device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the robot control device is used, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the robot control method as described in any one of the first aspects above.

[0041] In a fourth aspect, another embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the robot control method as described in any one of the above-mentioned first aspects are executed.

[0042] The beneficial effects of this application are:

[0043] The application provides a robot control method and device, a control equipment and a storage medium. The application obtains an input task command for a target robot and a static scene image of a task scene collected by the target robot based on the task command. The application uses multiple forms of task commands and combines the static image to realize cross-modal alignment, ensure the acquisition of scene information related to the task, avoid redundant calculation of full-scene scanning, and improve processing efficiency. According to the static scene image and the task command, a preset visual space reasoning model is used to generate a task action sequence corresponding to the task command and object space state information corresponding to each action in the task action sequence. The object state and spatial relationship are explicitly coded through a dynamic scene image, zero-sample space reasoning can be realized, and the limitation of a traditional model depending on pre-training data is solved. According to the task action sequence and the object space state information corresponding to each action, a preset task generator is used to generate an action sequence instruction corresponding to the task command. The application converts an abstract action sequence into a specific instruction executable by the robot, and ensures that the instruction is consistent with the real physical environment. According to each action instruction in the action sequence instruction, the target robot is controlled to perform the corresponding action. The application uses the object space state information and the object space state information corresponding to each action to generate an action sequence instruction corresponding to the task command by using the preset task generator, solves the fragmentation problem of the reasoning chain, guarantees the physical consistency of multi-step decision-making, ensures the correction of early space constraint conflicts of subsequent actions, avoids error accumulation, and thus improves the control efficiency and control accuracy of the robot. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0045] Figure 1 A robot control scene graph provided for the embodiments of the application;

[0046] Figure 2 A flowchart of a robot control method provided for the embodiments of the application;

[0047] Figure 3 A flowchart of determining object space state information in a robot control method provided for the embodiments of the application;

[0048] Figure 4 A flowchart of determining scene visual features in a robot control method provided for the embodiments of the application;

[0049] Figure 5Another flowchart of a robot control method provided by an embodiment of the present application is shown in FIG. 6.

[0050] Figure 6 A flowchart of a preset space model test in a robot control method provided by an embodiment of the present application is shown in FIG. 7.

[0051] Figure 7 A flowchart of determining a preset space data set in a robot control method provided by an embodiment of the present application is shown in FIG. 8.

[0052] Figure 8 An architecture diagram of a robot control provided by an embodiment of the present application is shown in FIG. 9.

[0053] Figure 9 A structure diagram of a robot control device provided by an embodiment of the present application is shown in FIG. 10.

[0054] Figure 10 A structure diagram of a robot control device provided by an embodiment of the present application is shown in FIG. 10. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application are only used for the purpose of description and illustration, and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowchart shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowchart or removed from the flowchart under the guidance of the content of the present application by those skilled in the art.

[0056] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0057] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0058] Currently, a multimodal large language model is usually used to control a robot according to recognized multimodal information, so that the robot can establish language reasoning on the basis of visual perception to control the robot. Specifically, the multimodal model needs to interpret scene semantics or generate plans conditioned on object detection from visual data. However, although the existing multimodal model has visual coding capability, it will incorrectly present object relationships without explicit geometric basis, and sequential decision-making will usually fail when both functional attributes and topological constraints of objects need to be considered. To this end, the present application provides a robot control method, which acquires an input task command for a target robot, and a static scene image of a task scene collected by the target robot based on the task command, generates a task action sequence corresponding to the task command and object space state information corresponding to each action in the task action sequence according to the static scene image and the task command using a preset visual space reasoning model, generates an action sequence instruction corresponding to the task command using a preset task generator according to the task action sequence and the object space state information corresponding to each action, and controls the target robot to perform corresponding actions in sequence according to each action instruction in the action sequence instruction. The present application significantly improves the understanding ability of the robot through the object space state information, reduces decision-making errors in the robot control process through the preset task generator, and thus improves the accuracy of robot control.

[0059] As follows, the use scenario of the robot control method provided by the embodiments of the present application is described in combination with multiple drawings, Figure 1 A robot control scenario provided by the embodiments of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the robot can be a joint robot, including a base, a large arm, a small arm, and a small arm front end provided with a gripper that can be operated, and the large arm, the small arm, and the gripper are connected through joints to form a multi-degree-of-freedom structure similar to a human arm. The robot can also be a mobile robot with a mobile chassis that can move autonomously or semi-autonomously in an environment. Or any other structure of the robot, the embodiments of the present application do not limit this. When the target robot acquires an input task command for the target robot, a static scene image of a task scene is collected by a visual sensor based on the task command, and a task action sequence and object space state information are generated according to the task command and the static scene image, an action sequence instruction is generated according to the task action sequence and the object space state information corresponding to each action, and the target robot is controlled in sequence according to each action instruction in the action sequence instruction to perform corresponding actions. The object space state information is used to represent the spatial state change information of each object in the static scene image under each action, for example, Figure 1The objects in the table can include, for example, a table, a cup, a plate or a bowl on the table, and the spatial state change information of the objects can include attribute information, position information of the objects, and relationship information between the objects. For example, the object is a green cup, the coordinates of the cup are (3, 5, 6), the attribute of the cup can be green, the cup is on a red plate, and the like. When the task command is “put the green cup on the pink plate”, a corresponding action sequence instruction is generated to control the target robot to perform the corresponding action, so that the green cup is placed on the pink plate. It should be noted that the present application only takes the content in Figure 1 as an example, and the specific robot use scenario can also be a Lego assembly scenario or any scenario for moving objects, so that Figure 1 the tableware in

[0060] The robot control method provided by the embodiments of the present application is described as follows in combination with the drawings. The robot control method is applied to a control device of a robot, and the control device of the robot includes a processor and a memory. The processor can be a local controller or a cloud server of the robot, and the processor is configured to execute a robot motion control model training method and a robot control method. Figure 2 A flowchart of a robot control method provided by the embodiments of the present application is shown in Figure 2 , and the method includes the following steps.

[0061] In step 201, an input task command for a target robot is obtained, and a static scene image of a task scene in which the target robot is located is collected based on the task command.

[0062] The task command can be input by text or obtained by recognizing the input voice. The static scene image is the content in a fixed scene and cannot reflect the changes of entities in the scene. The static scene image can include an RGB (Red, Green, Blue) image and a depth image. The RGB image records visual information such as color and texture of the task scene, and the depth image records the depth value of each point in the task scene to the camera, forming spatial geometric information. Specifically, a color image and depth information (RGB-Depth, abbreviated as RGB-D) camera is mounted on the target robot, and the RGB image and the depth image are obtained by the RGB-D camera.

[0063] Optionally, the input text or voice information determines the task command for the target robot. When the task command is obtained, the target entity corresponding to the question or instruction in the task command is determined according to the task command, and the static scene image of the task scene in which the target entity is located is obtained.

[0064] For example, when the input voice is identified as "What is the tableware below the blue cup?", the task command is determined as "Identify the parameter below the blue cup", the task scene centered on the blue cup is determined from the current scene, and the static scene graph of the task scene is obtained by the RGB-D camera mounted on the robot.

[0065] In step 202, according to the static scene image and the task command, a preset visual space reasoning model is used to generate a task action sequence corresponding to the task command and object space state information corresponding to each action in the task action sequence.

[0066] The preset visual space reasoning model is a model obtained by pre-training for processing multi-modal information, which is used to convert the static scene image and the task command into an executable action sequence. The object space state information corresponding to each action is used to represent the spatial state change information of each object in the static scene image under each action. The task action sequence can be the steps of operating the objects in the task scene under the task command, for example, "grab the green cup, move above the pink plate, and put it down vertically". The object space state information is the spatial state change information of each object under each action, including the type, color, position, and position relationship with other objects of each object.

[0067] Optionally, according to the static scene image and the task command, the preset visual space reasoning model is used to determine the object space state information corresponding to each node, and based on the spatial state information, the task action sequence corresponding to the task command and the object space state information corresponding to each action in the task action sequence are generated.

[0068] In step 203, according to the task action sequence and the object space state information corresponding to each action, a preset task generator is used to generate an action sequence instruction corresponding to the task command.

[0069] The preset task generator is used to sort the task action sequence according to the task action sequence and the object space state information corresponding to each action, so that each action after sorting is performed smoothly, each step of reasoning is verified based on geometric rules, and illusory actions are avoided to improve the success rate of task execution. For example, if the preset task generator is directly executed to perform the task action sequence, an illusion will be formed at a certain node, for example, Figure 1 After the execution of a certain action in the middle, the green cup floats in the air without support.

[0070] Optionally, according to the task action sequence and the object space state information corresponding to each action, it is judged whether the object space state information corresponding to each action in the task action sequence is continuous state information. If it is continuous state information, the action sequence instruction is generated. If it is not continuous state information, the action sequence determined from the object space state information corresponding to each action is determined as the action sequence instruction according to the preset task planner.

[0071] Optionally, according to the task action sequence and the object space state information corresponding to each action, the preset task generator is used to infer the task action sequence and the object space state information corresponding to each action, so that the task action sequence is consistent with the logic, and the action sequence instruction corresponding to the task command is generated.

[0072] For example, when the task command is "move the green cup from the red plate to the pink plate", the task action sequence in the task scene is "grab the green cup, move above the pink plate, and put down vertically". The object space state information corresponding to grabbing the green cup is "red plate A (position (0, 0)), green cup B (position (0, 1), grab), and pink plate C (position (2, 0))". The object space state information corresponding to moving above the pink plate is "red plate A (position (0, 0)), green cup B (position (2, 1), grab), and pink plate C (position (2, 0))". The object space state information corresponding to putting down vertically is "red plate A (position (0, 0)), green cup B (position (2, 0)), and pink plate C (position (2, 0))".

[0073] In step 204, according to each action instruction in the action sequence instruction, the target robot is controlled to perform the corresponding action in sequence.

[0074] Optionally, according to each action instruction in the action sequence instruction, the target robot is controlled to perform the corresponding action in sequence, and the actual space state change information of each object after performing each action is fed back. It is judged whether the actual space state change information of each object after performing each action is consistent with the space state information corresponding to the action instruction. If it is not consistent, the task planner is re-planned. If it is consistent, each action instruction in the action sequence instruction is continued to be executed.

[0075] In the embodiment of the present application, the input task command for the target robot is obtained, and the static scene image of the task scene collected by the target robot based on the task command is obtained. The multi-form task command of the present application realizes cross-modal alignment combined with the static image, which can ensure the acquisition of scene information related to the task, avoid redundant calculation of full-scene scanning, and improve processing efficiency. According to the static scene image and the task command, a preset visual space reasoning model is used to generate a task action sequence corresponding to the task command and object space state information corresponding to each action in the task action sequence. The object state and spatial relationship are explicitly coded through the dynamic scene image, zero-shot spatial reasoning can be realized without task-specific fine-tuning, and the limitation of traditional models relying on pre-training data is solved. According to the task action sequence and the object space state information corresponding to each action, a preset task generator is used to generate an action sequence instruction corresponding to the task command. The present application converts the abstract action sequence into specific instructions executable by the robot, ensuring that the instructions are consistent with the real physical environment. According to each action instruction in the action sequence instruction, the target robot is controlled to perform the corresponding action in turn. Through the object space state information and the object space state information corresponding to each action, the preset task generator is used to generate the action sequence instruction corresponding to the task command, which solves the fragmentation problem of the reasoning chain, guarantees the physical consistency of multi-step decision-making, ensures the correction of early space constraints for subsequent actions, avoids error accumulation, and thus improves the control efficiency and control accuracy of the robot.

[0076] On the basis of the above-mentioned embodiment, the preset visual space reasoning model comprises a graph node extractor, a visual auxiliary extractor, a scene image generator and a space reasoning module. Therefore, the present application provides a flow of determining object space state information in a robot control method, Figure 3 A flowchart of determining object space state information in a robot control method provided by the embodiment of the present application is shown in Figure 3 As shown in the above-mentioned step 202, according to the static scene image and the task command, a preset visual space reasoning model is used to generate a task action sequence corresponding to the task command and object space state information corresponding to each action in the task action sequence, which comprises:

[0077] Step 301, using a graph node extractor to extract nodes from the static scene image to obtain node recognition information in the static scene image.

[0078] The node recognition information can include action, color, type, size, quantity and the like.

[0079] Optionally, a graphical node extractor is adopted to extract entities in the static scene image, take the entities in the static image as nodes, and identify information such as actions, colors, types, sizes, and quantities of the nodes to obtain node identification information. The entities are specific objects or regions that can be perceived in the static scene image and are digital representations of physical objects in the real world. For example, the objects such as the cup and the plate in the static scene image. Figure 1

[0080] Step 302: A visual auxiliary extractor is adopted to extract features of the node identification information to obtain scene visual features corresponding to the static scene image.

[0081] Optionally, a visual auxiliary extractor is adopted to extract features of the node identification information to obtain attribute information and depth information of the static scene image as scene visual features.

[0082] Step 303: A scene image generator is adopted to generate a dynamic structured scene image corresponding to the static scene image according to the scene visual features.

[0083] The dynamic structured scene image includes attribute information of each object and spatial position relationships between the objects.

[0084] Optionally, a scene image generator is adopted to construct a dynamic structured scene image according to visual information features and depth information features in the scene visual features, so that attribute information between multiple objects and spatial position relationships between the objects are determined based on the dynamic structured scene image.

[0085] Step 304: A spatial reasoning module is adopted to perform spatial reasoning on the dynamic structured scene image according to the task command to obtain a task action sequence and object spatial state information corresponding to each action in the task action sequence.

[0086] The task command is logically decomposed into multiple atomic reasoning steps, and the atomic reasoning steps include the task action sequence. Specifically, objects corresponding to the task command are determined from the dynamic structured scene according to the task command, and ordered atomic reasoning steps are generated according to object attributes and position relationships between the objects.

[0087] Optionally, a spatial reasoning module is adopted to determine target objects corresponding to the task command from the dynamic scene image according to the task command, perform spatial reasoning to obtain a task action sequence, and determine object spatial state information corresponding to each action in the dynamic scene image after each action is executed according to the task action sequence.

[0088] ​Optionally, in the process of spatial reasoning on the dynamic structured scene image according to the task command by adopting the spatial reasoning module, the coherence of the task execution needs to be ensured, and the action feasibility needs to be verified by physical rules.

[0089] In the embodiment of the application, the node recognition information in the static scene image is obtained by adopting a graph node extractor; the scene visual features corresponding to the static scene image are obtained by adopting a visual auxiliary extractor; the dynamic structured scene image corresponding to the static scene image is generated by adopting a scene image generator; and the task action sequence and the object spatial state information corresponding to each action in the task action sequence are obtained by adopting a spatial reasoning module. The application can accurately mine key node information from the static scene image, understand the scene as dynamic and related through the generation of the dynamic structured scene image, improve the accuracy of processing complex task processes, adapt to the needs of robots and other execution embodied tasks, better cope with complex situations in real environments, and improve the generality and practicality of robot control.

[0090] On the basis of the above embodiment, the visual auxiliary extractor comprises an object detection network and a depth estimation network. For this purpose, the application further provides a process for determining scene visual features in a robot control method, Figure 4 A schematic diagram of a process for determining scene visual features in a robot control method according to an embodiment of the application is shown in Figure 4 As shown in the above step 302, the node recognition information is feature-extracted by adopting the visual auxiliary extractor to obtain the scene visual features corresponding to the static scene image, which comprises:

[0091] Step 401, the object detection network is adopted to perform object detection according to the node recognition information to generate the attribute information of each object in the static scene image.

[0092] The attribute information can include visual attribute features such as appearance, color, shape, and texture, and the object detection network can be an open vocabulary detection model.

[0093] Optionally, the specific area corresponding to the node recognition information is extracted from the static scene image by the object detection network in the visual auxiliary extractor, so as to accurately obtain the visual attribute features such as appearance, color, shape, and texture of each node, thereby generating the attribute information of each object in the static scene image.

[0094] Step 402, the depth estimation network is adopted to perform depth estimation according to the node recognition information to obtain the depth estimation information of the task scene.

[0095] The scene visual features include the attribute information of each object and the depth estimation information. The depth estimation information is used to describe the distance relationship between objects and the camera in the scene, as well as the position and layout of the objects in the three-dimensional space.

[0096] Optionally, the depth value of each pixel point in the task scene is determined by a depth estimation network in the visual auxiliary extractor, the three-dimensional coordinates of the node are generated according to the node recognition information and the depth value of each pixel point in the task scene, the physical position of the node in the task scene is determined, and the depth estimation information of the task scene is obtained according to the three-dimensional coordinates of the node.

[0097] In the embodiment of the application, an object detection network is used to perform object detection according to the node recognition information, to generate attribute information of each object in the static scene image. The attribute information is used to accurately locate the node region, to split adjacent or overlapping objects of the same color into independent nodes, and to avoid identification errors of the node caused by similar color. A depth estimation network is used to perform depth estimation according to the node recognition information, to obtain the depth estimation information of the task scene, to avoid incorrect reasoning of the traditional model for unsupported objects, and to ensure that the node state conforms to the physical law, thereby avoiding collision in the control of the robot.

[0098] On the basis of the above-mentioned embodiment, the application further provides a process for determining node recognition information in a robot control method, wherein the static scene image is subjected to node extraction by using a graphic node extractor in step 302, to obtain the node recognition information in the static scene image, including:

[0099] If the task command is a reorganization task command, the graphic node extractor is used to extract the nodes in the static scene image, to obtain the recognition information of each node in the static scene image.

[0100] The attribute information of each node and the spatial occupation information of each node. The attribute information of each node can include the color, size, category and other information of each node. The spatial occupation information of each node can be the position information and the occupation space information of each node in the task scene. For example, Figure 1 The green cup in the static scene image is a node in the current task scene, the attribute information of the green cup is green, cylindrical and water cup, and the spatial occupation information is that the green cup is located at the second layer (5, 1) in the three-dimensional space and occupies a volume of 2×2×4.

[0101] Optionally, the reorganization task is a task of recombining the nodes in the task scene, in combination with the content in Figure 1 The reorganization task can be “picking up the green cup from the red plate and placing it on the pink plate”, or “placing the yellow cup next to the orange bowl”, or other tasks of recombining the nodes in the task scene.

[0102] Optionally, if the task command is a reorganization task command, adjustment of the nodes in the task scene is required, a graphic node extractor is used to perform node extraction on the static scene image to obtain identification information of each node in the static scene image. And based on the identification information of each node in the static scene image, the corresponding reorganization task command is executed.

[0103] In the embodiment of the application, by extracting the node attribute information and the node space occupation information in the static scene image, comprehensive description of the objects in the static scene image can be realized, the scattered information of the scene is integrated into structured knowledge, thereby enhancing the task reasoning efficiency and improving the execution accuracy.

[0104] On the basis of the above-mentioned embodiment, the application further provides another flow of a robot control method, Figure 5 Another flow of a robot control method provided by the embodiment of the application is shown in the flowchart of Figure 5 On the basis of the above-mentioned steps 201-204, the method further includes:

[0105] Step 501, obtaining an input first question for a target robot, and a first scene image collected by the target robot based on the first question.

[0106] The first question is a spatial question and answer, not a question for driving the robot to operate. For example, it can be a question such as "what is the closest object to the robot?" or "what is the yellow object?", and the embodiment of the application does not limit this. At this time, the target robot can be any robot, and the structure of the robot is not required.

[0107] Optionally, the first question can be a question converted from input speech to text, or a directly input text question. According to the input first question, the target robot performs image collection based on the first question to obtain a first scene image collected by the target robot based on the first question.

[0108] Step 502, generating first visual space reasoning information of the first scene image under the first question according to the first question and the first scene image, using a preset visual space reasoning model.

[0109] The first visual space reasoning information includes spatial state information of each object in the first scene image. The first scene image is a static scene image.

[0110] Optionally, according to the first question and the first scene image, a plurality of nodes are extracted from the first question and the first scene image to obtain node identification information of the plurality of nodes, feature extraction is performed on the node identification information to obtain scene visual features in the first scene image, a dynamic structured scene graph corresponding to the first scene image is generated based on the scene visual features in the first scene image, and first visual space reasoning information of the first scene image is determined from the dynamic structured scene graph corresponding to the first scene image.

[0111] In step 503, according to the first visual space reasoning information, a first answer information corresponding to the first question is generated by using a preset task generator.

[0112] Optionally, according to the first visual space reasoning information, a spatial state information of a corresponding node in the first visual space reasoning information is determined from the first question by using the preset task generator, and the first answer information corresponding to the first question is determined.

[0113] For example, in combination with Figure 1 For example, when the first question is “what is the closest object to the robot”, the position of the robot is determined from the first visual space reasoning information, and the spatial state information of the robot and the spatial state information of other nodes except the robot are determined from the first visual space reasoning information, so that the answer to the first question is “yellow cup”.

[0114] In step 504, the target robot is controlled to play the first answer information.

[0115] For example, at this time, the target robot plays the first answer information “yellow cup”.

[0116] In the embodiments of the present application, an input first question for a target robot is acquired, and a first scene image collected by the target robot based on the first question is acquired; according to the first question and the first scene image, a first visual space reasoning information of the first scene image under the first question is generated by using a preset visual space reasoning model; according to the first visual space reasoning information, a first answer information corresponding to the first question is generated by using a preset task generator, and the target robot is controlled to play the first answer information. The present application can collect the first question and the first scene image to determine the first visual space reasoning information in the current scene, avoid the illusion of spatial relationship, and thus improve the accuracy of the first answer information.

[0117] On the basis of the above-mentioned embodiments, the present application further provides a process for testing a preset space model in a robot control method, Figure 6 A process diagram for testing a preset space model in a robot control method provided by the embodiments of the present application is shown in Figure 6 On the basis of the above-mentioned steps 201-204, the method further comprises:

[0118] Step 601, obtaining sample video data in the input preset spatial data set and a preset question and answer pair corresponding to the sample video data.

[0119] Among them, the preset spatial data set includes a plurality of categories of preset question and answer pairs, specifically including: accessibility questions, adjacent questions, distance questions and the like. The accessibility question is used to judge whether a certain topic reaches a specific target, and the preset question and answer pair can be: "Can the robot get the blue cup?" "Yes". The adjacent question is used to judge the spatial adjacent relationship between objects, and the preset question and answer pair can be: "What object is next to the yellow cup?" "Pink plate". The distance question revolves around the distance between the object and the reference object, and the preset question and answer pair can be: "Which object is closest to the yellow plate?" "Purple bowl". The sample video data is a dynamic change of a real scene or a simulated scene. The preset spatial data set can be data collected in advance according to the current scene or a public data set.

[0120] Optionally, the sample video data in the input preset spatial data set and the preset question and answer pair corresponding to the sample video data are obtained. The sample video data and the preset question and answer pair corresponding to the sample video data are questions that can be answered or processed by the target robot.

[0121] Step 602, according to the second question in the preset question and answer pair and the second scene image corresponding to the second question in the sample video data, a preset visual spatial reasoning model is used to generate second visual spatial reasoning information of the second scene image under the second question.

[0122] Among them, the second visual spatial reasoning information includes the spatial state information of each object in the second scene image. The second scene image is a static scene image.

[0123] Optionally, according to the second question and the sample video data, the sample video data is identified to obtain a second scene image corresponding to the second question in the sample video data, a plurality of nodes are extracted from the second question and the second scene image to obtain node identification information of the plurality of nodes, scene visual features in the second scene image are obtained by performing feature extraction on the node identification information, a dynamic structured scene graph corresponding to the second scene image is generated based on the scene visual features in the second scene image, and the second visual spatial reasoning information of the second scene image is determined from the dynamic structured scene graph corresponding to the second scene image.

[0124] Step 603, according to the second visual spatial reasoning information, a preset task generator is used to generate second answer information corresponding to the second question.

[0125] Optionally, according to the first visual space reasoning information, a preset task generator is used to determine the spatial state information of the corresponding node in the first visual space reasoning information from the first question, and determine the first answer information corresponding to the first question.

[0126] For example, in combination with Figure 1 For example, when the second question is "which object is closest to the yellow plate", the position of the yellow plate is determined from the second visual space reasoning information, and the spatial state information of the yellow plate and the spatial state information of other nodes except the yellow plate are determined from the second visual space, so that the answer to the second question is "purple bowl".

[0127] Step 604, according to the second answer information and the preset answer corresponding to the second question in the preset question and answer pair, test the preset visual space reasoning model.

[0128] For example, when the second answer information is "purple bowl" and the preset answer corresponding to the second question in the preset question and answer pair is "purple bowl", the reasoning ability of the preset visual space reasoning model is strong; when the second answer information is "pink plate" and the preset answer corresponding to the second question in the preset question and answer pair is "purple bowl", the reasoning ability of the preset visual space reasoning model is poor, and then the preset visual space reasoning model is trained according to the preset space data set to improve the reasoning ability of the preset visual space reasoning model.

[0129] On the basis of the above embodiment, the sample video data in the input preset space data set and the preset question and answer pair corresponding to the sample video data are obtained; according to the second question in the preset question and answer pair and the second scene image corresponding to the second question in the sample video data, the preset visual space reasoning model is used to generate the second visual space reasoning information of the second scene image under the second question; according to the second visual space reasoning information, the preset task generator is used to generate the second answer information corresponding to the second question; according to the second answer information and the preset answer corresponding to the second question in the preset question and answer pair, the preset visual space reasoning model is tested. The application can strengthen the pertinence of the spatial relationship reasoning of the preset visual space reasoning model, and improve the scene generalization and robustness of the preset visual space reasoning model.

[0130] On the basis of the above embodiment, the application also provides a process for determining a preset space data set in a control method of a robot, Figure 7 A process for determining a preset space data set in a control method of a robot is provided in the embodiment of the application, as shown in Figure 7 Before the sample video data in the input preset space data set and the preset question and answer pair corresponding to the sample video data are obtained in the above step 601, the method further comprises:

[0131] Step 701, obtaining sample video data and a preset task category corresponding to the sample video data.

[0132] The preset task category can include a spatial reasoning problem and an assembly task. The spatial reasoning task is used to evaluate the dynamic interaction of the robot, for example, can be: quantity, color, direction and orientation, object overlap, accessibility, task success judgment, mechanical arm feasibility, distance or adjacency relationship, etc. The assembly task is used to evaluate the understanding ability of the static spatial attribute of the robot, focusing on the attributes and combination relationship of the object itself, for example, can be: quantity, color, relative relationship and size.

[0133] Optionally, the sample video data can be obtained by pre-constructing sample video data for the task of the target robot, or by obtaining relevant video data from a data set of an open source library, or by constructing sample video data according to the preset task category, etc. The present application does not limit this.

[0134] Optionally, when obtaining relevant video data from a data set of an open source library, interference data needs to be removed, and sample video data is obtained from continuous video, and scene fragments related to reasoning are retained. The interference data can be task interruption data, video or image fragmentation error data.

[0135] Step 702, generating a preset question and answer pair corresponding to the sample video data according to the preset task category and the sample video data.

[0136] Optionally, when the preset task category is a spatial reasoning problem, the sample video data is labeled with a multi-modal large model to generate a question and answer pair corresponding to the sample video data. When the preset task category is an assembly task, it is determined whether the assembly task is completed according to the sample video data.

[0137] In the embodiments of the present application, sample video data and a preset task category corresponding to the sample video data are obtained. A preset question and answer pair corresponding to the sample video data is generated according to the preset task category and the sample video data. The present application filters and classifies the sample video data through the preset task category, which can ensure that the generated question and answer pair is closely around a specific task target, and avoids irrelevant information interference. By obtaining sample data and a preset question and answer pair, the coverage ability of the preset visual spatial reasoning model data to complex scenes can be enhanced, thereby improving the accuracy of the target robot control.

[0138] The robot control method provided by the embodiments of the present application is described as follows in combination with the accompanying drawings, Figure 8 A schematic diagram of the robot control architecture provided by the embodiments of the present application is shown in Figure 8As shown, the preset visual space reasoning model includes a graph node extractor, a visual auxiliary extractor, a scene image generator, and a space reasoning module. The visual auxiliary extractor includes an object detection network and a depth estimation network.

[0139] Optionally, a task command for a target robot input is acquired, and a static scene image of a task scene where the target robot is based on the task command is acquired. The static scene image is subjected to node extraction by using a graph node extractor to obtain node recognition information in the static scene image; the node recognition information is subjected to feature extraction by using a visual auxiliary extractor to obtain scene visual features corresponding to the static scene image; a dynamic structured scene image corresponding to the static scene image is generated according to the scene visual features by using a scene image generator, and the dynamic structured scene image includes attribute information of each object and a spatial position relationship between the objects; a task action sequence and object space state information corresponding to each action in the task action sequence are obtained by performing space reasoning on the dynamic structured scene image according to the task command by using a space reasoning module. An action sequence instruction corresponding to the task command is generated by using a preset task generator according to the task action sequence and the object space state information corresponding to each action; and the target robot is controlled to perform corresponding actions in sequence according to each action instruction in the action sequence instruction.

[0140] Based on the same inventive concept, the embodiments of the present application also provide a robot control device corresponding to the robot control method. Since the principle of the device in the embodiments of the present application solves the problem similarly to the robot control method in the embodiments of the present application, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described herein.

[0141] Figure 9 A structural schematic diagram of a robot control device provided in the embodiments of the present application is shown in FIG. 9. The device includes an acquisition module 901, a generation module 902, and a control module 903. The acquisition module 901 is configured to acquire a task command for a target robot input and a static scene image of a task scene where the target robot is based on the task command.

[0142] The generation module 902 is configured to generate a task action sequence corresponding to the task command and object space state information corresponding to each action in the task action sequence by using a preset visual space reasoning model according to the static scene image and the task command; and the object space state information corresponding to each action is used to represent spatial state change information of each object in the static scene image under each action.

[0143] The generation module 902 is configured to generate an action sequence instruction corresponding to the task command by using a preset task generator according to the task action sequence and the object space state information corresponding to each action.

[0144] The control module 903 is configured to sequentially control the target robot to perform corresponding actions according to each action instruction in the action sequence instruction.

[0145] Optionally, the preset visual space reasoning model comprises a graph node extractor, a visual auxiliary extractor, a scene image generator, and a space reasoning module.

[0146] The visual auxiliary extractor is configured to perform feature extraction on the node recognition information to obtain scene visual features corresponding to the static scene image.

[0147] The scene image generator is configured to generate a dynamic structured scene image corresponding to the static scene image according to the scene visual features, the dynamic structured scene image comprising attribute information of each object and a spatial position relationship between the objects.

[0148] The space reasoning module is configured to perform space reasoning on the dynamic structured scene image according to the task command to obtain a task action sequence and object space state information corresponding to each action in the task action sequence.

[0149] Optionally, the visual auxiliary extractor comprises an object detection network and a depth estimation network.

[0150] The object detection network is configured to perform object detection according to the node recognition information to generate attribute information of each object in the static scene image.

[0151] Optionally, the generation module 902 is configured to, if the task command is a reorganization task command, use the graph node extractor to perform node extraction on the static scene image to obtain recognition information of each node in the static scene image, the recognition information of each node comprising attribute information of each node and spatial occupancy information of each node.

[0152] Optionally, the device further comprises a question and answer module, the question and answer module being configured to obtain an input first question for the target robot and a first scene image collected by the target robot based on the first question.

[0153] According to the first question and the first scene image, the preset visual space reasoning model is used to generate first visual space reasoning information of the first scene image under the first question, the first visual space reasoning information comprising spatial state information of each object in the first scene image.

[0154] According to the first visual space inference information, a first answer corresponding to the first question is generated by using a preset task generator.

[0155] The target robot is controlled to play the first answer information.

[0156] Optionally, the apparatus further includes a test module, which is specifically configured to: acquire sample video data in the input preset space data set and a preset question-answer pair corresponding to the sample video data.

[0157] According to a second question in the preset question-answer pair and a second scene image corresponding to the second question in the sample video data, second visual space inference information of the second scene image under the second question is generated by using a preset visual space inference model; the second visual space inference information includes spatial state information of each object in the second scene image.

[0158] According to the second visual space inference information, second answer information corresponding to the second question is generated by using a preset task generator.

[0159] According to the second answer information and a preset answer corresponding to the second question in the preset question-answer pair, the preset visual space inference model is tested.

[0160] Optionally, the test module is further configured to: acquire the sample video data and a preset task category corresponding to the sample video data.

[0161] According to the preset task category and the sample video data, the preset question-answer pair corresponding to the sample video data is generated.

[0162] The processing flow of each module in the apparatus and the interaction flow between the modules can refer to the related descriptions in the above method embodiments, and will not be described in detail here.

[0163] Embodiments of the present application also provide a robot, Figure 10 A structural schematic diagram of a control device of a robot provided by the embodiments of the present application is shown in Figure 10 The control device of the robot includes a processor 1001, a memory 1002, and optionally, a bus 1003. The memory 1002 stores machine-readable instructions executable by the processor 1001 (for example, execution instructions corresponding to the acquisition module 901, the generation module 902, and the control module 903 in the apparatus in Figure 9 When the control device is running, the processor 1001 and the memory 1002 communicate through the bus 1003, and the machine-readable instructions are executed by the processor 1001 to perform the steps of the above robot control method.

[0164] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program.

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and the present application will not be repeated herein. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other means. The device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some communication interface, device or module, which can be electrical, mechanical or other forms.

[0166] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. When the functions are realized in the form of software functional units and sold or used as independent products, the software functional units can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts of the technical solutions that make contributions to the prior art or the parts of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0167] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A robot control method, characterized in that: The method comprises: Obtaining an input task command for a target robot and a static scene image of a task scene captured by the target robot based on the task command; Based on the static scene image and the task command, a preset visual-spatial reasoning model is used to generate a task action sequence corresponding to the task command, and object spatial state information corresponding to each action in the task action sequence; wherein the object spatial state information corresponding to each action is used to represent spatial state change information of each object in the static scene image under each action; According to the task action sequence and the object space state information corresponding to each action, a preset task generator is used to generate an action sequence instruction corresponding to the task command; According to each action instruction in the action sequence instruction, the target robot is controlled to perform the corresponding action in sequence; The preset visual-spatial reasoning model includes: a graphic node extractor, a visual aid extractor, a scene image generator, and a spatial reasoning module; The method of generating a task action sequence corresponding to the task command and object space state information corresponding to each action in the task action sequence by using a preset visual-spatial reasoning model based on the static scene image and the task command includes: Using the graphic node extractor, extracting nodes from the static scene image to obtain node identification information in the static scene image; Using the visual aid extractor, extracting features from the node identification information to obtain scene visual features corresponding to the static scene image; The scene image generator is used to generate a dynamic structured scene image corresponding to the static scene image according to the scene visual features, wherein the dynamic structured scene image includes: attribute information of each object and the spatial position relationship between the objects; The spatial reasoning module is used to perform spatial reasoning on the dynamic structured scene image according to the task command to obtain the task action sequence and the object spatial state information corresponding to each action in the task action sequence.

2. The method according to claim 1, characterized in that The visually assisted extractor includes: an object detection network and a depth estimation network; the visually assisted extractor is used to perform feature extraction on the node identification information to obtain scene visual features corresponding to the static scene image, including: Using the object detection network, performing object detection based on the node identification information to generate attribute information of each object in the static scene image; The depth estimation network is used to perform depth estimation according to the node identification information to obtain depth estimation information of the task scene, and the scene visual features include: attribute information of each object and the depth estimation information.

3. The method according to claim 1, characterized in that The step of extracting nodes from the static scene image using the graphic node extractor to obtain node identification information in the static scene image includes: If the task command is a reorganization task command, the graphic node extractor is used to extract nodes from the static scene image to obtain identification information of each node in the static scene image, and the identification information of each node includes: attribute information of each node and space occupancy information of each node.

4. The method according to claim 1, wherein The method further comprises: Obtaining an input first question for the target robot and a first scene image captured by the target robot based on the first question; Based on the first question and the first scene image, using the preset visual-spatial reasoning model, generating first visual-spatial reasoning information of the first scene image under the first question; the first visual-spatial reasoning information includes: spatial state information of each object in the first scene image; Generate first answer information corresponding to the first question using the preset task generator according to the first visual-spatial reasoning information; Control the target robot to play the first answer information.

5. The method according to claim 1, wherein The method further comprises: Obtaining sample video data and preset question-answer pairs corresponding to the sample video data in an input preset spatial data set; generating, based on the second question in the preset question-answer pair and the second scene image corresponding to the second question in the sample video data, second visual-spatial reasoning information of the second scene image under the second question using the preset visual-spatial reasoning model; the second visual-spatial reasoning information including spatial state information of each object in the second scene image; Generate second answer information corresponding to the second question using the preset task generator according to the second visual-spatial reasoning information; The preset visual-spatial reasoning model is tested according to the second answer information and the preset answer corresponding to the second question in the preset question-answer pair.

6. The method according to claim 5, characterized in that Before obtaining the sample video data and the preset question-answer pairs corresponding to the sample video data in the input preset spatial data set, the method further includes: Acquire the sample video data and the preset task category corresponding to the sample video data; Based on the preset task category and the sample video data, a preset question-answer pair corresponding to the sample video data is generated.

7. A robot control device, characterized in that: The device comprises: An acquisition module is used to acquire an input task command for a target robot and a static scene image of the task scene in which the target robot is located, which is collected based on the task command; a generation module, configured to generate, based on the static scene image and the task command, a task action sequence corresponding to the task command and object spatial state information corresponding to each action in the task action sequence using a preset visual-spatial reasoning model; wherein the object spatial state information corresponding to each action is used to represent spatial state change information of each object in the static scene image under each action; The generating module is configured to generate an action sequence instruction corresponding to the task command using a preset task generator according to the task action sequence and the object space state information corresponding to each action; A control module, configured to sequentially control the target robot to perform corresponding actions according to each action instruction in the action sequence instruction; The preset visual-spatial reasoning model includes: a graphic node extractor, a visual aid extractor, a scene image generator, and a spatial reasoning module; The generation module is specifically used to: use a graphic node extractor to extract nodes from a static scene image to obtain node identification information in the static scene image; use the visual auxiliary extractor to perform feature extraction on the node identification information to obtain scene visual features corresponding to the static scene image; use the scene image generator to generate a dynamic structured scene image corresponding to the static scene image based on the scene visual features, and the dynamic structured scene image includes: attribute information of each object and the spatial position relationship between the objects; use the spatial reasoning module to perform spatial reasoning on the dynamic structured scene image according to the task command to obtain the task action sequence and the object spatial state information corresponding to each action in the task action sequence.

8. A robot control device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the control device of the robot is running, the processor executes the machine-readable instructions to perform the steps of the robot control method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the robot control method according to any one of claims 1 to 6.

Images