Robot operation method and device based on semantic and spatial reasoning and electronic equipment

Abstract

The invention relates to the technical field of robots, in particular to a robot operation method and device based on semantic and spatial reasoning and electronic equipment, and the method comprises the steps: obtaining a language instruction of an operation task and image data around a robot; identifying a visual object in the image data, and performing semantic and spatial reasoning on the language instruction and the image data to obtain an association level between the visual object and the operation task; and taking the visual object of which the association level is greater than a preset level as a target object related to the operation task, predicting an operation position of the robot according to the operation task and the target object, and controlling the robot to grab or place the target object at the operation position. Therefore, the problems that in the prior art, a robot operation algorithm can only process a simple visual language operation task, and applicability and practicability are low are solved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot operation method, device and electronic equipment based on semantics and spatial reasoning. Background Art

[0002] In the robot operation scenario, reinforcement learning and imitation learning enable the robot to have the ability to understand images and language instructions, and can output reasonable actions. Both methods are end-to-end learning methods of visual language conditional control. Reinforcement learning learns the optimal strategy through online exploration to maximize the reward of robot operation, while imitation learning imitates large-scale behavior data sets in an offline manner. These data can be obtained by expert teaching in the simulator.

[0003] With the development of large basic vision and language models, and the progress of multimodal representation learning, many works use pre-trained vision or language encoders to complete basic operation tasks. By leveraging the powerful capabilities of pre-trained vision and language models, some state-of-the-art end-to-end models can effectively base the semantic concepts of natural language on physical scenes, and can even show a certain generalization ability for unseen scenes and objects. On the other hand, some works use the planning capabilities of large language models to complete complex robotic operation tasks.

[0004] In related technologies, end-to-end learning only needs to understand simple semantic concepts to complete simple visual language operation tasks. However, since processing complex language instructions (including spatial relationships between multiple objects) requires reasoning about more fine-grained spatial relationships between different objects, current end-to-end learning still lacks the ability to process complex language instructions (including spatial relationships between multiple objects) and has low applicability and practicality. Summary of the invention

[0005] The present application provides a robot operation method, device and electronic device based on semantics and spatial reasoning to solve the problems that the robot operation algorithm in the related art can only handle simple visual language operation tasks and has low applicability and practicality.

[0006] The first aspect of the present application provides a robot operation method based on semantic and spatial reasoning, comprising the following steps: obtaining language instructions for an operation task and image data around the robot; identifying visual objects in the image data, performing semantic and spatial reasoning on the language instructions and the image data, and obtaining a correlation level between the visual objects and the operation task; taking the visual objects whose correlation level is greater than a preset level as target objects related to the operation task, predicting the operation position of the robot based on the operation task and the target object, and controlling the robot to grasp or place the target object at the operation position.

[0007] Optionally, performing semantic and spatial reasoning on the language instructions and the image data to obtain the association level between the visualization object and the operation task includes: inputting the language instructions and the image data into a semantic spatial reasoning network, and outputting the association level between the visualization object and the operation task.

[0008] Optionally, the semantic space reasoning network includes an encoding module, an association network, a self-attention network and a multi-layer perception network, wherein the encoding module is used to encode the language instructions and the image blocks of the visualized objects in the image data, and splice the linear layer encoding of each image block to the image representation layer, and output word representation and image representation; the association network is used to associate the word representation and image representation with their respective position representation and category representation, and input all representations into the self-attention network; the multi-layer perception network is used to perceive the correlation score of each representation output by the self-attention network, and determine the correlation level based on the correlation score.

[0009] Optionally, predicting the operating position of the robot according to the operating task and the target object includes: generating an attention map according to an image block of the target object; inputting the attention map and the operating task into an operating position prediction network, outputting an operating position prediction map, and determining the pixel coordinates of the operating position of the robot based on the operating position prediction map; converting the pixel coordinates of the operating position into three-dimensional coordinates in the robot coordinate system, and determining the operating position of the robot according to the three-dimensional coordinates.

[0010] The second aspect of the present application provides a robot operation device based on semantic and spatial reasoning, including: an acquisition module, used to acquire language instructions for an operation task and image data around the robot; an identification module, used to identify visual objects in the image data, perform semantic and spatial reasoning on the language instructions and the image data, and obtain the association level between the visual objects and the operation task; a control module, used to take the visual objects with an association level greater than a preset level as target objects related to the operation task, predict the operation position of the robot according to the operation task and the target object, and control the robot to grasp or place the target object at the operation position.

[0011] Optionally, the recognition module is further used to: input the language instruction and the image data into a semantic space reasoning network, and output the association level between the visual object and the operation task.

[0012] Optionally, the recognition module is further used for: the semantic space reasoning network includes an encoding module, an association network, a self-attention network and a multi-layer perception network, wherein the encoding module is used to encode the language instructions and the image blocks of the visualized objects in the image data, and splice the linear layer encoding of each image block to the image representation layer, and output word representation and image representation; the association network is used to associate the word representation and image representation with their respective position representation and category representation, and input all representations into the self-attention network; the multi-layer perception network is used to perceive the correlation score of each representation output by the self-attention network, and determine the correlation level according to the correlation score.

[0013] Optionally, the control module is further used to: generate an attention map based on the image block of the target object; input the attention map and the operation task into an operation position prediction network, output an operation position prediction map, and determine the pixel coordinates of the operation position of the robot based on the operation position prediction map; convert the pixel coordinates of the operation position into three-dimensional coordinates in the robot coordinate system, and determine the operation position of the robot based on the three-dimensional coordinates.

[0014] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot operation method based on semantics and spatial reasoning as described in the above embodiment.

[0015] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the robot operation method based on semantics and spatial reasoning as described in the above embodiments.

[0016] Therefore, this application has at least the following beneficial effects:

[0017] The embodiments of the present application can enable the robot to analyze images at the object level through semantic-spatial reasoning, and screen out related objects at the top level to fully understand the purpose of the task; through operation position prediction, the robot can learn the pixels of the operation with the help of the attention map, and can more effectively use the information obtained by reasoning to output precise instructions, so as to realize the reasoning of the semantics and spatial relationships of multiple objects and achieve precise control, with high applicability and practicality.

[0018] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A flowchart of a robot operation method based on semantics and spatial reasoning provided according to an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a robot operation framework based on semantic-spatial reasoning provided according to an embodiment of the present application;

[0022] Figure 3 An example diagram of a robot operating device based on semantics and spatial reasoning is provided according to an embodiment of the present application;

[0023] Figure 4 It is a schematic diagram of the structure of a vehicle electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0025] In recent years, using natural language to guide robot operations has become a research hotspot in the field of robot learning; if we want to build a robot operating system that can help humans perform tasks in real scenarios, understanding complex language instructions is the key.

[0026] In related technologies, people use end-to-end learning methods based on visual language conditional control in robot operation scenarios, mainly through reinforcement learning and imitation learning to enable robots to have the ability to understand images and language commands. With the development of basic vision and language large models, and the advancement of multimodal representation learning, end-to-end models can effectively establish the semantic concepts of natural language on the basis of physical scenes, and can even show a certain generalization ability for invisible scenes and objects.

[0027] However, although the end-to-end learning method has been successful in relatively simple visual language manipulation tasks, it still lacks the ability to handle complex language instructions, where complex language contains spatial relationships between multiple objects. For example: there are multiple blocks and bowls of the same or different colors on a table, and now a block needs to be placed in a bowl; existing methods can already complete instructions such as "put the cyan block into the green bowl", but cannot handle more complex instructions, such as "place the cyan block between the yellow block in front and the gray bowl in the back"; therefore, solving the former task only requires understanding simple semantic concepts, while solving the latter task requires reasoning about finer-grained spatial relationships between different objects; it is very challenging to learn abstract spatial reasoning and precise manipulation skills at the same time.

[0028] This application solves complex visual language manipulation problems based on object-level semantic-spatial reasoning and pixel-level operation position prediction:

[0029] (1) Object-level semantic-spatial reasoning: An object-centric semantic-spatial reasoning module is proposed, which aims to predict which objects are relevant to completing the manipulation task from language instructions and image observations; the reasoned results pass key area information to the robotic system to facilitate the underlying manipulation position learning process.

[0030] (2) Pixel-level operation position prediction: A pixel-specific operation position prediction module is proposed. The previous module can be used to infer task-related objects and their corresponding top-down views. This module integrates basic language instructions, global visual input, and related object information and uses the underlying operation position prediction framework (CLIPort) to output robot operation instructions.

[0031] Therefore, this application designs a custom pick and place task based on the popular robotics simulation platform Ravens.

[0032] The following describes the robot operation method, device, electronic device and storage medium based on semantic and spatial reasoning of the embodiment of the present application with reference to the accompanying drawings. Specifically, Figure 1 A flowchart of a robot operation method based on semantics and spatial reasoning provided in an embodiment of the present application.

[0033] like Figure 1 As shown, the robot operation method based on semantics and spatial reasoning includes the following steps:

[0034] In step S101, language instructions for the operation task and image data around the robot are acquired.

[0035] It can be understood that the embodiments of the present application can obtain language instructions for the operation task and image data around the robot, so as to facilitate subsequent semantic and spatial reasoning of the language instructions and image data.

[0036] It should be noted that the language instructions for the operation tasks can be obtained by the robot's auditory system or language perception system through a microphone or other means, and the language instructions of the present application involve reasoning about the relative spatial position of the object. The image data around the robot can be obtained by the robot's vision system through sensors such as cameras and lidars to obtain images or point cloud data of the surrounding environment, without specific limitation.

[0037] In step S102, visual objects in the image data are identified, semantic and spatial reasoning is performed on the language instructions and the image data, and the association level between the visual objects and the operation tasks is obtained.

[0038] It can be understood that the embodiments of the present application can identify visual objects in image data, perform semantic and spatial reasoning on language instructions and image data, and obtain the level of association between the visual objects and the operation tasks, so as to subsequently use the visual objects with an association level greater than a preset level as target objects related to the operation tasks.

[0039] It should be noted that the association level is determined based on the softmax (normalized exponential function) layer processing the relevant scores. Without specific limitation, the score of objects with high association will be close to 1, and vice versa.

[0040] In an embodiment of the present application, semantic and spatial reasoning is performed on language instructions and image data to obtain the association level between the visualized object and the operation task, including: inputting the language instructions and image data into a semantic spatial reasoning network, and outputting the association level between the visualized object and the operation task.

[0041] Among them, the semantic space reasoning network includes an encoding module, an association network, a self-attention network and a multi-layer perception network. Among them, the encoding module is used to encode the image blocks of visual objects in the language instructions and image data, and splice the linear layer encoding of each image block to the image representation layer, and output word representation and image representation; the association network is used to associate word representation and image representation with their respective position representation and category representation, and input all representations into the self-attention network; the multi-layer perception network is used to perceive the relevant score of each representation output by the self-attention network, and determine the relevant level according to the relevant score.

[0042] It can be understood that the embodiment of the present application inputs language instructions and image data into the semantic space reasoning network, and outputs the association level between the visual object and the operation task, so as to subsequently use the visual object with an association level greater than the preset level as the target object related to the operation task.

[0043] Specifically, the module inputs language instructions of length L (number of words) and m (total number of objects in the RGB image) image blocks; the language instructions and image blocks are encoded by the pre-trained CLIP (Contrastive Language-Image Pre-training, a pre-training model based on contrastive text-image pairs) model, and the coordinates of each image block are encoded by a linear layer and spliced ​​into the image representation (obtained by CLIP); next, the L word representations and m image representations are connected with their respective position representations and category representations (0 represents language and 1 represents image) and pass through an 8-layer self-attention network; then the L+m representations pass through an MLP (Multi-Layer Perceptron) and output m relevance scores, the larger the score, the greater the relevance of the corresponding object to the task; finally, a softmax layer is used to process the relevance scores, and the scores of objects with large relevance will be close to 1, otherwise they will be close to 0.

[0044] It should be noted that the semantic-spatial reasoning module requires image blocks of all visible objects as input, and adopts an open vocabulary-based object detection model (ViLD) to extract the image blocks of each object and its coordinates in the RGB image; the size of each image block is 50*50*3.

[0045] In step S103, the visualized object with a correlation level greater than a preset level is taken as a target object related to the operation task, the operation position of the robot is predicted according to the operation task and the target object, and the robot is controlled to grasp or place the target object at the operation position.

[0046] Among them, the preset level can be set according to actual conditions without specific limitation.

[0047] It can be understood that the embodiments of the present application can use visual objects with an association level greater than a preset level as target objects related to the operation task, predict the robot's operation position based on the operation task and the target object, and control the robot to grab or place the target object at the operation position. Therefore, the present application uses a semantic-spatial reasoning module to enable the robot to analyze the image at the object level and screen out related objects at the top level to fully understand that the purpose of the task is to pick up and place the task; through the operation position prediction module, the robot learns the pixels of the operation with the help of the attention map, and can more effectively use the information obtained by reasoning to output precise instructions, so as to realize the reasoning of the semantics and spatial relationships of multiple objects and achieve precise control, with high applicability and practicality.

[0048] In an embodiment of the present application, the operation position of the robot is predicted according to the operation task and the target object, including: generating an attention map according to the image block of the target object; inputting the attention map and the operation task into the operation position prediction network, outputting the operation position prediction map, and determining the pixel coordinates of the operation position of the robot based on the operation position prediction map; converting the pixel coordinates of the operation position into three-dimensional coordinates in the robot coordinate system, and determining the operation position of the robot according to the three-dimensional coordinates.

[0049] It can be understood that the embodiment of the present application generates an attention map based on the image block of the target object, inputs the attention map and the operation task into the operation position prediction network, outputs the operation position prediction map, determines the pixel coordinates of the robot's operation position based on the operation position prediction map, converts the pixel coordinates of the operation position into three-dimensional coordinates in the robot coordinate system, and determines the robot's operation position based on the three-dimensional coordinates to facilitate the subsequent implementation of the robot's grasping and placement operations.

[0050] Specifically, the operation position prediction can predict the robot's grasping / placing point based on the underlying operation algorithm, and use the labels of whether the object is relevant or not to build an attention map, as follows:

[0051] 1) Low-level operation algorithm (CLIPort): CLIPort algorithm is used as the low-level operation framework. CLIPort uses U-Net to convert RGB images into Affordance maps (operation position prediction maps); the length and width of the Affordance map are consistent with the RGB input, but there is only one channel, which indicates the possibility of the position being the robot's operation point. The larger the value, the more likely it is to be the position where the robot grabs or places. The CLIPort algorithm will train the affordance maps for grabbing and placing separately, and select the pixel with the largest value as the robot's grabbing / placing point.

[0052] 2) Attention map: Based on CLIPort, an additional attention map input is added to the original RGB image; the size of the attention map is the same as the original RGB image. Figure 1 The same, but only includes objects related to the task; for the m relevant scores obtained in the previous module, a fixed value is set to distinguish whether the object is relevant to the task, and then only the image blocks corresponding to the relevant objects are added to a completely black RGB image according to the original coordinates (the size is the same as the original RGB Figure 1 In this way, we get the attention map; the attention map is also the result of semantic-spatial reasoning.

[0053] According to the robot operation method based on semantic and spatial reasoning proposed in the embodiment of the present application, by obtaining the language instructions of the operation task and the image data around the robot, identifying the visual objects in the image data, and performing semantic and spatial reasoning on the language instructions and the image data, the association level between the visual objects and the operation task is obtained; the visual objects with an association level greater than the preset level are used as target objects related to the operation task, the operation position of the robot is predicted according to the operation task and the target object, and the robot is controlled to grasp or place the target object at the operation position. Therefore, the present application uses a semantic-spatial reasoning module to enable the robot to analyze the image at the object level and screen out related objects at the top level to fully understand that the purpose of the task is to pick up and place the task; the operation position prediction module enables the robot to learn the pixels of the operation with the help of the attention map, and can more effectively use the information obtained by reasoning to output precise instructions, so as to realize the reasoning of the semantics and spatial relationships of multiple objects and achieve precise control, with high applicability and practicality.

[0054] The following will be combined Figure 2 The robot operation method based on semantics and spatial reasoning of the present application is described in detail as follows:

[0055] (1) Object-based semantic-spatial reasoning

[0056] 1) Network Design: We designed a semantic spatial reasoning network based on the Transformer architecture to understand the spatial relationships of objects from natural language instructions and RGB images.

[0057] Specifically, the module inputs language instructions of length L (number of words) and m (total number of objects in the RGB image) image blocks; the language instructions and image blocks are encoded by the pre-trained CLIP model, and the coordinates of each image block are encoded by a linear layer and spliced ​​into the image representation (obtained by CLIP); next, the L word representations and m image representations are connected with their respective position representations and category representations (0 represents language and 1 represents image) and pass through an 8-layer self-attention network; then the L+m representations pass through an MLP (multi-layer perceptron network) and output m relation scores. The larger the score, the greater the correlation between the corresponding object and the task; finally, a softmax layer is used to process the correlation scores. The scores of objects with large correlations will be close to 1, and vice versa.

[0058] 2) Implementation details: The semantic-spatial reasoning module requires image patches of all visible objects as input. An open vocabulary based object detection model (ViLD) is used to extract the image patches of each object and their coordinates in the RGB image, where the size of each image patch is 50*50*3.

[0059] (2) Operation position prediction

[0060] 1) Low-level operation algorithm (CLIPort): CLIPort algorithm is used as the low-level operation framework. CLIPort uses U-Net to convert RGB images into Affordance maps (operation position prediction maps); the length and width of the Affordance map are consistent with the RGB input, but there is only one channel, which indicates the possibility of the position being the robot's operation point. The larger the value, the more likely it is to be the position where the robot grabs or places. The CLIPort algorithm will train the affordance maps for grabbing and placing separately, and select the pixel with the largest value as the robot's grabbing / placing point.

[0061] 2) Attention map: Based on CLIPort, an additional attention map input is added to the original RGB image; the size of the attention map is the same as the original RGB image. Figure 1 The same, but only includes objects related to the task; for the m relevant scores obtained in the previous module, a fixed value is set to distinguish whether the object is relevant to the task, and then only the image blocks corresponding to the relevant objects are added to a completely black RGB image according to the original coordinates (the size is the same as the original RGB Figure 1 In this way, we get the attention map; the attention map is also the result of semantic-spatial reasoning.

[0062] (3) Training and deployment

[0063] 1) Training: The semantic-spatial reasoning and operation position prediction modules are trained separately. The former obtains standard image patches and labels indicating whether the object is relevant in the simulator to train the semantic-spatial reasoning transformer. The latter uses the labels of whether the object is relevant to build an attention map and combines the RGB input to train the CLIPort network.

[0064] 2) Deployment: First, the ViLD model is used to extract the image blocks of each object and the pixel coordinates in the image from the original RGB image. Then, the image blocks are input into the semantic-spatial reasoning module together with the language instructions to obtain the relevant scores of each object. Then, the attention map is obtained based on the score and the pixel coordinates of the grasping and placing are obtained through the operation position prediction module. Finally, the 2D coordinates can be converted into 3D coordinates in the robot coordinate system through hand-eye calibration to realize the robot's grasping and placing operations.

[0065] In summary, this application effectively solves the problem that existing robot operation algorithms lack the ability to process complex language instructions and achieves precise control through two modules: a semantic-spatial reasoning module and an operation position prediction module; in the semantic-spatial reasoning module, the robot analyzes the image at the object level and screens out related objects at the top level to fully understand the purpose of the task; in the operation position prediction module, the robot learns the pixels of the operation with the help of the attention map, so that the information obtained by reasoning can be more effectively used to output precise instructions; the two modules are trained separately and finally closely combined, which can become a good robot operation training & deployment framework in the future.

[0066] Next, a robot operating device based on semantics and spatial reasoning proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0067] Figure 3 It is a block diagram of a robot operating device based on semantics and spatial reasoning according to an embodiment of the present application.

[0068] like Figure 3 As shown, the robot operating device 10 based on semantics and spatial reasoning includes: an acquisition module 100, a recognition module 200 and a control module 300.

[0069] Among them, the acquisition module 100 is used to obtain the language instructions of the operation task and the image data around the robot; the recognition module 200 is used to identify the visual objects in the image data, perform semantic and spatial reasoning on the language instructions and the image data, and obtain the association level between the visual objects and the operation task; the control module 300 is used to take the visual objects with an association level greater than the preset level as the target objects related to the operation task, predict the operation position of the robot according to the operation task and the target object, and control the robot to grasp or place the target object at the operation position.

[0070] In the embodiment of the present application, the recognition module 200 is further used to: input language instructions and image data into a semantic space reasoning network, and output the association level between the visualized object and the operation task.

[0071] In an embodiment of the present application, a semantic space reasoning network includes an encoding module, an association network, a self-attention network, and a multi-layer perception network, wherein the encoding module is used to encode image blocks of visual objects in language instructions and image data, and splice the linear layer encoding of each image block to the image representation layer, and output word representation and image representation; the association network is used to associate word representation and image representation with their respective position representation and category representation, and input all representations into the self-attention network; the multi-layer perception network is used to perceive the correlation score of each representation output by the self-attention network, and determine the correlation level according to the correlation score.

[0072] In an embodiment of the present application, the control module 300 is further used to: generate an attention map based on the image block of the target object; input the attention map and the operation task into the operation position prediction network, output the operation position prediction map, and determine the pixel coordinates of the robot's operation position based on the operation position prediction map; convert the pixel coordinates of the operation position into three-dimensional coordinates in the robot coordinate system, and determine the robot's operation position based on the three-dimensional coordinates.

[0073] It should be noted that the aforementioned explanation of the embodiment of the robot operation method based on semantics and spatial reasoning is also applicable to the robot operation device based on semantics and spatial reasoning in this embodiment, and will not be repeated here.

[0074] According to the robot operation device based on semantic and spatial reasoning proposed in the embodiment of the present application, the language instructions of the operation task and the image data around the robot are obtained, the visual objects in the image data are identified, and the language instructions and the image data are subjected to semantic and spatial reasoning to obtain the association level between the visual object and the operation task; the visual object with an association level greater than the preset level is used as the target object related to the operation task, the operation position of the robot is predicted according to the operation task and the target object, and the robot is controlled to grasp or place the target object at the operation position. Therefore, the present application uses a semantic-spatial reasoning module to enable the robot to analyze the image at the object level and screen out related objects at the top level to fully understand the purpose of the task; the operation position prediction module enables the robot to learn the pixels of the operation with the help of the attention map, and can more effectively use the information obtained by reasoning to output precise instructions, so as to realize the reasoning of the semantic and spatial relationships of multiple objects and realize precise control, and has high applicability and practicality.

[0075] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0076] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0077] When the processor 402 executes the program, the robot operation method based on semantics and spatial reasoning provided in the above embodiment is implemented.

[0078] Furthermore, the electronic device further comprises:

[0079] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0080] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0081] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0082] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0083] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0084] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0085] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned robot operation method based on semantics and spatial reasoning.

[0086] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0087] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0088] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0089] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0090] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

Claims

1. A robot operation method based on semantics and spatial reasoning, characterized in that: The following steps are involved: Obtain language instructions for operation tasks and image data around the robot; Identify a visual object in the image data, perform semantic and spatial reasoning on the language instruction and the image data, and obtain a correlation level between the visual object and the operation task; The visualized object with a correlation level greater than a preset level is taken as a target object related to the operation task, the operation position of the robot is predicted according to the operation task and the target object, and the robot is controlled to grasp or place the target object at the operation position.

2. The robot operation method based on semantics and spatial reasoning according to claim 1, characterized in that: The performing semantic and spatial reasoning on the language instruction and the image data to obtain the association level between the visualized object and the operation task includes: The language instruction and the image data are input into a semantic space reasoning network, and the association level between the visual object and the operation task is output.

3. The robot operation method based on semantics and spatial reasoning according to claim 2, characterized in that: The semantic space reasoning network includes an encoding module, an association network, a self-attention network and a multi-layer perception network, wherein: The encoding module is used to encode the language instructions and the image blocks of the visual objects in the image data, and splice the linear layer encoding of each image block to the image representation layer, and output the word representation and the image representation; The association network is used to associate the word representation and the image representation with their respective position representation and category representation, and input all the representations into the self-attention network; The multi-layer perception network is used to perceive the relevance score of each representation output by the self-attention network, and determine the relevance level according to the relevance score.

4. The robot operation method based on semantics and spatial reasoning according to claim 1, characterized in that: The predicting the operation position of the robot according to the operation task and the target object comprises: Generate an attention map based on the image block of the target object; Inputting the attention map and the operation task into an operation position prediction network, outputting an operation position prediction map, and determining the pixel coordinates of the operation position of the robot based on the operation position prediction map; The pixel coordinates of the operation position are converted into three-dimensional coordinates in the robot coordinate system, and the operation position of the robot is determined according to the three-dimensional coordinates.

5. A robot operating device based on semantics and spatial reasoning, characterized in that: include: An acquisition module, used to acquire language instructions for operation tasks and image data around the robot; A recognition module, used for recognizing a visual object in the image data, performing semantic and spatial reasoning on the language instruction and the image data, and obtaining a correlation level between the visual object and the operation task; A control module is used to take the visualized object with a correlation level greater than a preset level as a target object related to the operation task, predict the operation position of the robot according to the operation task and the target object, and control the robot to grasp or place the target object at the operation position.

6. The device according to claim 5, characterized in that The identification module is further used for: The language instruction and the image data are input into a semantic space reasoning network, and the association level between the visual object and the operation task is output.

7. The device according to claim 6, characterized in that The identification module is further used for: The semantic space reasoning network includes an encoding module, an association network, a self-attention network and a multi-layer perception network, wherein: The encoding module is used to encode the language instructions and the image blocks of the visual objects in the image data, and splice the linear layer encoding of each image block to the image representation layer, and output the word representation and the image representation; The association network is used to associate the word representation and the image representation with their respective position representation and category representation, and input all the representations into the self-attention network; The multi-layer perception network is used to perceive the relevance score of each representation output by the self-attention network, and determine the relevance level according to the relevance score.

8. The device according to claim 5, characterized in that The control module is further configured to: Generate an attention map based on the image block of the target object; Inputting the attention map and the operation task into an operation position prediction network, outputting an operation position prediction map, and determining the pixel coordinates of the operation position of the robot based on the operation position prediction map; The pixel coordinates of the operation position are converted into three-dimensional coordinates in the robot coordinate system, and the operation position of the robot is determined according to the three-dimensional coordinates.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot operation method based on semantics and spatial reasoning as described in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the robot operation method based on semantics and spatial reasoning as described in any one of claims 1 to 4.

Images