Robot moving operation method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a robot moving operation method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: obtaining a semantic segmentation result based on an RGB image; generating a three-dimensional point cloud based on the depth map; associating the semantic segmentation result corresponding to each pixel point with the three-dimensional space coordinates, and projecting the obtained association result to the view angle of the bird's-eye view; based on the semantic map and the name of the task-related object, obtaining a target position where the task-related object is located; evaluating operability of a nearby position of the target position through an operable map prediction model; and controlling the robot to execute operation on the task-related object based on the local operable map. Thus, the operability of the position near the target position where the task-related object is located is evaluated through the operable map prediction model, it can be guaranteed that the robot is navigated to the operable position for avoiding the obstacle, then it can be guaranteed that the robot accurately operates the task-related object, and the operation success rate can be increased.

Description

Technical Field

[0001] The present disclosure relates to the field of robotics, and more particularly, to a method, apparatus, electronic device, storage medium, and computer program product for moving and operating a robot. Background Art

[0002] With the development of science and technology, robots for various purposes have emerged. For example, robots used to complete various household chores are a common type of robot. These robots need to find and manipulate target objects in unfamiliar environments. Although some progress has been made in object navigation and object manipulation in recent years, the lack of targeted exploration of object guidance has led to poor compatibility between robots' navigation and manipulation skills. This makes it still a significant challenge for robots to complete mobile manipulation tasks.

[0003] Currently, learning an end-to-end policy to complete the entire mobile manipulation task of a robot has attracted the attention of many researchers. These end-to-end methods directly map observations to underlying actions, while also inferring the skills to be performed and how to perform them. However, applying end-to-end learning methods to long-term, multi-stage tasks such as mobile manipulation involving thousands of underlying actions is extremely challenging. This is because this process requires tens of thousands of expert demonstrations or hundreds of millions of reinforcement learning steps for training, and collecting this data in the real world is unrealistic. More importantly, the shortcomings of end-to-end methods, such as limited generalization ability and lack of interpretability, make them difficult to apply in the real world.

[0004] Unlike end-to-end approaches, modular approaches decompose the robot's mobile manipulation task into distinct components, such as navigation, grasping, and placing. By decomposing the mobile manipulation task, modular approaches reduce the complexity of the problem and also offer better interpretability and generalization capabilities. However, using modular approaches to accomplish mobile manipulation tasks still faces several challenges. First, without prior knowledge, efficiently finding the target object in a large workspace is difficult due to the sparsity of valid observation points. Furthermore, humans typically leverage their experience in previous environments to explore new environments and quickly locate the target object. For example, when searching for a cup in a new house, users are more likely to search in the kitchen than in the bedroom. Therefore, learning and leveraging these semantic priors is crucial for the deployment of mobile robots. Second, handoff errors between different skills are a key factor hindering the success of modular approaches. For example, when a robot manipulates an object, it needs to reach a suitable position to ensure that the object is within the effective workspace of the robotic arm. However, the current navigation method simply uses a distance threshold for position filtering, which may cause the robot to navigate to a location full of obstacles and unfavorable for operation, resulting in a high failure rate of the robot's mobile operation and making it difficult to meet the needs of actual usage scenarios. Summary of the Invention

[0005] The present disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for robot movement operation, in order to at least solve the problem of high failure rate of robot movement operation in the above-mentioned related technologies.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for moving and operating a robot is provided, comprising: obtaining an RGB image and a depth map at a current viewing angle, wherein the RGB image and the depth map are shooting results at the current viewing angle by a camera provided on the robot; obtaining a semantic segmentation result based on the RGB image, wherein the semantic segmentation result includes the object category to which each pixel point in a plurality of pixel points included in the RGB image belongs; generating a three-dimensional point cloud based on the depth map, wherein the three-dimensional point cloud includes the three-dimensional spatial coordinates of each pixel point in a plurality of pixel points included in the depth map; and dividing each pixel point into a plurality of pixel points. The corresponding semantic segmentation results are associated with the three-dimensional spatial coordinates, and the obtained association results are projected to a bird's-eye view to obtain a two-dimensional semantic map; the target position of the task-related object is obtained based on the semantic map and the name of the task-related object through a large language model; the operability of the position near the target position is evaluated through an operable map prediction model to obtain a local operable map, wherein each of the multiple grids contained in the local operable map corresponds to a value for indicating the difficulty of operating the task-related object in the grid; based on the local operable map, the robot is controlled to perform operations on the task-related object.

[0007] Optionally, obtaining a semantic segmentation result based on the RGB image includes: inputting the RGB image, the name of the task-related object, and the preset space name into an open vocabulary perception model to obtain an open semantic segmentation result, wherein the open vocabulary perception model is used to detect image content corresponding to the task-related object and image content corresponding to the space in the RGB image; inputting the RGB image into a closed vocabulary perception model to obtain a closed semantic segmentation result, wherein the closed vocabulary perception model is used to detect image content related to the container in the RGB image; and merging the open semantic segmentation result and the closed semantic segmentation result to obtain the semantic segmentation result.

[0008] Optionally, the mobile operation method also includes: obtaining a preset container list, a preset space list and a preset prompt, wherein the preset container list includes the names of multiple containers, the preset space list includes the names of multiple spaces, and the preset prompt is used to instruct the large language model to answer the first question of which containers among the multiple containers the task-related object may / may not appear near and the second question of which spaces among the multiple spaces the task-related object may / may not appear in; learning the object placement pattern based on the preset container list, the preset space list and the preset prompt through the large language model; wherein, obtaining the target position of the task-related object based on the semantic map and the name of the task-related object through the large language model includes: obtaining the target position of the task-related object based on the semantic map, the name of the task-related object and the object placement pattern through the large language model.

[0009] Optionally, obtaining the target position of the task-related object through the large language model based on the semantic map, the name of the task-related object and the object placement pattern includes: evaluating the exploration value of each edge point among multiple edge points based on the semantic map, the name of the task-related object and the object placement pattern through the large language model, wherein the exploration value is used to characterize the possibility that the corresponding edge point is the target position, and the multiple edge points are points at the intersection of the current explored area and the current unexplored area contained in the semantic map; selecting the edge point with the highest corresponding exploration value from the multiple edge points as the target position.

[0010] Optionally, the evaluating the operability of the positions near the target position by the operable map prediction model to obtain a local operable map includes: evaluating the operability of the positions near the target position by the operable map prediction model based on the local semantic map near the target position and the association result corresponding to the target position to obtain the local operable map, wherein the local semantic map is a partial semantic map centered on the target position that is intercepted from the semantic map.

[0011] Optionally, the operational map prediction model comprises a deep convolutional neural network, a deep learning network and a decoder; the operational map prediction model evaluates the operability of the position near the target position based on the local semantic map near the target position and the association result corresponding to the target position to obtain the local operational map, comprising: encoding the local semantic map near the target position by the deep convolutional neural network to obtain a first embedding vector ; Encode the association result corresponding to the target position through the deep learning network to obtain a second embedding vector ; The first embedding vector is decoded by the decoder Upsample to obtain decoding features ; The decoding feature and the second embedding vector The local operational map is obtained by merging the local operational map and inputting the merging result into the decoder.

[0012] According to a second aspect of an embodiment of the present disclosure, a mobile operating device of a robot is provided, comprising: an image acquisition module, configured to acquire an RGB image and a depth map at a current viewing angle, wherein the RGB image and the depth map are shooting results at the current viewing angle by a camera provided on the robot; a semantic segmentation result acquisition module, configured to acquire a semantic segmentation result based on the RGB image, wherein the semantic segmentation result includes the object category to which each pixel point in a plurality of pixel points included in the RGB image belongs; a point cloud generation module, configured to generate a three-dimensional point cloud based on the depth map, wherein the three-dimensional point cloud includes the three-dimensional spatial coordinates of each pixel point in a plurality of pixel points included in the depth map; a semantic map acquisition module, configured to obtain the semantic segmentation result based on the RGB image, wherein the semantic segmentation result includes the object category to which each pixel point in a plurality of pixel points included in the depth map belongs; The semantic segmentation result corresponding to each pixel point is associated with the three-dimensional spatial coordinates, and the obtained association result is projected to a bird's-eye view to obtain a two-dimensional semantic map; a target position acquisition module is configured to obtain the target position of the task-related object based on the semantic map and the name of the task-related object through a large language model; an operable map acquisition module is configured to evaluate the operability of the nearby positions of the target position through an operable map prediction model to obtain a local operable map, wherein each of the multiple grids contained in the local operable map corresponds to a value for indicating the difficulty of operating the task-related object in the grid; an operation execution module is configured to control the robot to perform an operation on the task-related object based on the local operable map.

[0013] Optionally, the semantic segmentation result acquisition module is configured to: input the RGB image, the name of the task-related object and the preset space name into an open vocabulary perception model to obtain an open semantic segmentation result, wherein the open vocabulary perception model is used to detect the image content corresponding to the task-related object and the image content corresponding to the space in the RGB image; input the RGB image into a closed vocabulary perception model to obtain a closed semantic segmentation result, wherein the closed vocabulary perception model is used to detect the image content related to the container in the RGB image; merge the open semantic segmentation result and the closed semantic segmentation result to obtain the semantic segmentation result.

[0014] Optionally, the mobile operating device of the robot also includes: a list acquisition module, configured to acquire a preset container list, a preset space list and a preset prompt, wherein the preset container list includes the names of multiple containers, the preset space list includes the names of multiple spaces, and the preset prompt is used to instruct the large language model to answer the first question of which containers among the multiple containers the task-related object may / may not appear near and the second question of which spaces among the multiple spaces the task-related object may / may not appear in; a rule learning module, configured to learn the object placement rule based on the preset container list, the preset space list and the preset prompt through the large language model; the target position acquisition module is configured to: acquire the target position of the task-related object based on the semantic map, the name of the task-related object and the object placement rule through the large language model.

[0015] Optionally, the target position acquisition module is configured to: evaluate the exploration value of each edge point among multiple edge points based on the semantic map, the names of the task-related objects and the object placement rules through the large language model, wherein the exploration value is used to characterize the possibility that the corresponding edge point is the target position, and the multiple edge points are points at the intersection of the current explored area and the current unexplored area contained in the semantic map; select the edge point with the highest corresponding exploration value from the multiple edge points as the target position.

[0016] Optionally, the operable map acquisition module is configured to: evaluate the operability of the location near the target location based on the local semantic map near the target location and the association result corresponding to the target location through the operable map prediction model, and obtain the local operable map, wherein the local semantic map is a partial semantic map centered on the target location that is intercepted from the semantic map.

[0017] Optionally, the operational map prediction model includes a deep convolutional neural network, a deep learning network and a decoder; the operational map acquisition module is configured to: encode the local semantic map near the target location through the deep convolutional neural network to obtain a first embedding vector ; Encode the association result corresponding to the target position through the deep learning network to obtain a second embedding vector ; The first embedding vector is decoded by the decoder Upsample to obtain decoding features ; The decoding feature and the second embedding vector The local operational map is obtained by merging the local operational map and inputting the merging result into the decoder.

[0018] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the robot movement operation method according to the present disclosure.

[0019] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the robot movement operation method according to the present disclosure.

[0020] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, which implements the robot movement operation method according to the present disclosure when the computer program is executed by a processor.

[0021] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects: In the present disclosure, the operability of the positions near the target position of the task-related objects is evaluated by the operable map prediction model, so that the robot can be navigated to an operable position that avoids occlusions and obstacles, that is, the robot can be navigated to a position that is conducive to operation, and then the robot can accurately operate the task-related objects, which can improve the operation efficiency and success rate.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0024] Figure 1 is a flowchart illustrating a moving operation method of a robot according to an exemplary embodiment of the present disclosure; Figure 2 is a schematic diagram illustrating a process of obtaining a local operational map through an operational map prediction model according to an exemplary embodiment of the present disclosure; Figure 3 is a schematic diagram illustrating an implementation process of a moving operation method of a robot according to an exemplary embodiment of the present disclosure; Figure 4 is an exemplary diagram illustrating an implementation process of moving an object from a starting container to a target container according to an exemplary embodiment of the present disclosure; Figure 5 is a flowchart illustrating a specific implementation of moving a target object from a starting container to a target container according to an exemplary embodiment of the present disclosure; Figure 6 is a block diagram illustrating a mobile operating device of a robot according to an exemplary embodiment of the present disclosure; Figure 7 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0026] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0027] It should be noted that the phrase "at least one of the several items" in this disclosure includes three types of parallel situations: "any one of the several items", "a combination of any multiple of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" means the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing both step 1 and step 2.

[0028] Figure 1 is a flowchart illustrating a moving operation method of a robot according to an exemplary embodiment of the present disclosure.

[0029] Reference Figure 1 In step 101, an RGB image and a depth map at the current viewing angle may be obtained, wherein the RGB image and the depth map may be shooting results at the current viewing angle by a camera set on the robot.

[0030] In step 102, a semantic segmentation result can be obtained based on the RGB image. For example, a hybrid perception module can be used to segment different object instances from the RGB image. The semantic segmentation result can include the object category to which each pixel in the RGB image belongs. For example, the object category can include, but is not limited to, a cup, a table, a floor, a refrigerator, and so on.

[0031] According to an exemplary embodiment of the present disclosure, the hybrid perception module may include an open-vocabulary perception model and a closed-vocabulary perception model. Furthermore, the open-vocabulary perception model may be used to detect image content corresponding to task-related objects and image content corresponding to spaces within an RGB image; while the closed-vocabulary perception model may be used to detect image content related to containers within an RGB image.

[0032] It should be noted that "task-related objects" may refer to objects involved in the task. For example, for the task: "Move the sponge from the table to the sofa", task-related objects may include "sponge", "table", and "sofa". The above-mentioned "space" may be, but is not limited to, a room. For example, common rooms may include: "kitchen", "bathroom", "bedroom", "living room", etc. The above-mentioned "container" may be a common piece of furniture in the home that can be used to place small objects, for example, it may include but is not limited to: "cabinet", "refrigerator", "table", etc.

[0033] The RGB image, the names of the task-related objects, and the preset space names can be input into the open vocabulary perception model to obtain the open semantic segmentation results; and the RGB image can be input into the closed vocabulary perception model to obtain the closed semantic segmentation results. Next, the open semantic segmentation results and the closed semantic segmentation results can be merged to obtain the final semantic segmentation results. Specifically:

[0034]

[0035]

[0036] .

[0037] in, Indicates an object, Indicates room, Represents a container, is the semantic mask, is the RGB image under the current viewing angle mentioned above, is a prompt word containing the name of the object, is a prompt word containing the room name. In this disclosure, it can be obtained by connecting the object name with "." , and you can also get it by connecting the room name with "." For example, assuming the objects to be detected are "sofa", "table", "cabinet", the corresponding prompt words are It can be "sofa.table.cabinet".

[0038] It should be noted that the above-mentioned open vocabulary perception model can be, but is not limited to, a target detection method based on weakly supervised learning (Detector with Image Classes, Detic), and there is no need to train the open vocabulary perception model; the above-mentioned closed vocabulary perception model can be, but is not limited to, You Only Look Once (YOLO), which can collect data from the environment for training.

[0039] In step 103 , a three-dimensional point cloud may be generated based on the depth map, wherein the three-dimensional point cloud may include the three-dimensional spatial coordinates of each pixel point among the multiple pixel points included in the depth map.

[0040] In step 104, the semantic segmentation results corresponding to each pixel point can be associated with the three-dimensional spatial coordinates, and the associated results can be projected to a bird's-eye view to obtain a two-dimensional semantic map. In other words, in the present disclosure, the semantic segmentation results can be aligned with the points at corresponding locations in the three-dimensional point cloud, and the semantically labeled point cloud can be projected to a bird's-eye view based on a geometric transformation to obtain a two-dimensional semantic map.

[0041] The above semantic map Can be a binary grid map that can contain ( ) channels. And, Each of the channels can represent the occupancy map of a class of objects, while the other two channels can represent the explored area and obstacles respectively. Can represent the number of grids, each grid can represent ( ) size of floor space.

[0042] In step 105 , the target location of the task-related object may be obtained through the large language model based on the semantic map and the name of the task-related object.

[0043] According to an exemplary embodiment of the present disclosure, a preset container list, a preset space list, and a preset prompt can also be obtained. The preset container list can contain the names of multiple containers, and the preset space list can contain the names of multiple spaces. For example, you can summarize common household furniture to obtain a preset container list. , and you can also summarize the room names to get a preset room list The preset prompts can be used to instruct the large language model to answer the first question of which containers among the multiple containers the task-related object may / may not appear near and the second question of which spaces among the multiple spaces the task-related object may / may not appear in.

[0044] Then, the large language model can be used to learn object placement patterns based on the preset container list, preset space list, and preset prompts. That is, in the present disclosure, the preset container list, preset space list, and preset prompts can be input into the large language model for common sense reasoning to learn object placement patterns.

[0045] Next, the large language model can be used to determine the target location of task-related objects based on the semantic map, the names of task-related objects, and their placement patterns. Specifically, the large language model uses Probability Soft Logic (PSL) to calculate the exploration value of multiple edge points on the semantic map according to soft constraints. Based on the calculated exploration value of each edge point, the target location of the task-related objects can be determined.

[0046] According to exemplary embodiments of the present disclosure, a large language model can be used to evaluate the exploration value of each of multiple edge points based on a semantic map, the names of task-related objects, and the object placement patterns. This exploration value can be used to represent the likelihood that the corresponding edge point is the target location. Furthermore, the aforementioned edge points can be points at the intersection of currently explored and currently unexplored areas included in the semantic map.

[0047] Then, the edge point (x, y) with the highest exploration value can be selected from the multiple edge points as the target location, i.e., the navigation target point. Next, a path planning algorithm can be used to plan a path to the target point based on the determined navigation target point, and the corresponding navigation action can be analyzed. For example, the path planning algorithm in this disclosure can be, but is not limited to, the Fast Matching Method (FMM).

[0048] It should be noted that the specific process of using PSL to calculate the exploration value of each edge point in the semantic map can be as follows: PSL can be used in The probability soft value is used for reasoning, and the following soft constraints can be proposed:

[0049]

[0050]

[0051] in, For task-related objects, is the edge point; express and The co-occurrence relationship of , which can be obtained from a large language model; express and Whether they are close or not can be determined by calculating the distance between edge points and the semantic map. The normalized closest distance of the occupancy map is obtained; express Is it the closest edge points; Represents the weight of the rule, which is used to measure the importance of the corresponding rule; express Exploration value.

[0052] Then, PSL can convert atoms ( ) is mapped to The probabilistic soft truth value of Next, PSL can model the relationship between atoms as a probabilistic graphical model and can be defined in the implementation The probability density function on , which can be achieved by maximizing To infer Then, we can get Select The edge point with the highest value is taken as the exploration target.

[0053] It should be noted that in this disclosure, the above edge points It can be obtained by calculating the boundary between the explored open area and the unexplored area contained in the semantic map; It can be obtained by reasoning with a large language model, and for pairs of objects with different relationships, different values ​​can be assigned to them to represent the strength of the co-occurrence relationship between them. Specifically: for pairs of objects with co-occurrence relationships, the value can be assigned to them ; For pairs of objects with uncertain relationships, values ​​can be assigned to them ; For pairs of objects with an exclusion relationship, you can assign a value In addition, the above assignments can satisfy: 1> > > >0. In step 106, the operability of the locations near the target location can be evaluated using the operability map prediction model to obtain a local operability map. Each of the multiple grids included in the local operability map can correspond to a value indicating the difficulty of operating the task-related object in the grid. For example, the local operability map can be of size ( ) .

[0054] It should be noted that in the present disclosure, the robot can be initialized at a visible point of a task-related object (the angle at which the object's position can be observed). The robot can then execute an operation strategy at that location and record whether the operation on the task-related object was successful. Subsequently, the points of successful operation can be unified in a single coordinate system to obtain the ground truth of the operational map. This ground truth of the operational map can then be used to train the operational map prediction model. Furthermore, the operational map prediction model can be trained using, but not limited to, the Dice loss function.

[0055] According to an exemplary embodiment of the present disclosure, an operable map prediction model may be used to evaluate the operability of locations near a target location based on a local semantic map near the target location and an association result corresponding to the target location, thereby obtaining a local operable map.

[0056] The local semantic map can be a portion of the semantic map centered on the target position extracted from the aforementioned semantic map. For example, the local semantic map can be a local tensor extracted from the aforementioned semantic map by extracting the obstacle map and the task-related object occupancy map and taking the target position of the task-related object as the center. .

[0057] The above “correlation result corresponding to the target position” can also be called “point cloud of the position of the object related to the marking task”, which can be a tensor containing four channels ,in, Indicates the number of points, each point can correspond to a 4-dimensional vector. And the first three dimensions of the 4-dimensional vector can represent the spatial coordinates of the point ,The last dimension can indicate whether the point belongs to a task-related object.

[0058] According to an exemplary embodiment of the present disclosure, the operational map prediction model may include a deep convolutional neural network, a deep learning network, and a decoder. The deep convolutional neural network may be, but is not limited to, ResNet50, and the deep learning network may be, but is not limited to, PointNet++.

[0059] Figure 2FIG. 1 is a schematic diagram illustrating a process of obtaining a local operational map through an operational map prediction model according to an exemplary embodiment of the present disclosure. Figure 2 , the local semantic map near the target location can be encoded through a deep convolutional neural network to obtain the first embedding vector For example, ResNet50 can be used to encode the local semantic map near the target location to obtain the first embedding vector .

[0060] Then, the association result corresponding to the target position can be encoded through the deep learning network to obtain the second embedding vector For example, PointNet++ can also be used to encode the associated result corresponding to the target position, that is, the point cloud of the object position related to the marking task, to obtain the second embedding vector Next, the first embedding vector can be decoded by Upsample to obtain decoding features .

[0061] This decoded feature can then be and the aforementioned second embedding vector The merged results can be fed into the decoder to obtain a local operational map. Specifically, the point cloud features can be fused with the semantic map features in the first three layers of the decoder: First, the point cloud features can be Projected to Then, you can Tiled composition Next, the tiled feature map can be combined with the aforementioned decoded feature map Multiply each element by element to obtain the fused features. It should be noted that in the present disclosure, a skip connection can be used to connect the encoder features of the corresponding layer to the decoder features. Then, the final local operational map can be obtained based on the above fused features.

[0062] In step 107, the robot can be controlled to perform an operation on the task-related object based on the local operational map. For example, a nearby, easily accessible location can be selected based on the local operational map as a navigation target. Based on this navigation target, the robot can then be navigated to the corresponding location to perform the operation on the task-related object. For example, this operation can include, but is not limited to, picking up or placing.

[0063] It should be noted that in the present disclosure, both the picking strategy and the placing strategy can be deterministic strategies, which may specifically include the following steps: (1) Select the pick point (grab point) or placement point For example, the center of the mask of the task-related object can be calculated as the grasping point, or the center of the largest inscribed circle of the blank plane of the target container can be calculated as the placement point.

[0064] (2) Calculate the position difference between the robot's current gripper position and the pick-up point (grasp point) or placement point ; (3) Control the robot's arms to move separately , , to reach the operating point and open or close the gripper to grab or place task-related objects.

[0065] Figure 3 1 is a schematic diagram illustrating an implementation process of a mobile operation method of a robot according to an exemplary embodiment of the present disclosure. Figure 3 The robot movement operation method provided by the present disclosure can mainly include two parts: the process of finding task-related objects based on visual observation results and the process of operating the task-related objects after finding them. These two processes have been explained in detail above and will not be repeated here.

[0066] Figure 4 FIG. 1 is an exemplary diagram illustrating an implementation process of moving an object from a starting container to a target container according to an exemplary embodiment of the present disclosure. Figure 4 , the robot's initial position is (x, y), and its orientation is . Moreover, the implementation process of moving the object from the starting container to the target container can be mainly divided into two parts, namely the process of finding the task-related object based on the RGB image and depth image obtained by shooting, and the process of operating the task-related object after finding it. For example, in the process of finding the task-related object, the robot can be controlled to perform but not limited to the following actions: turn left, turn right, move forward, move backward, and stop; in the process of operating the task-related object, the robot can be controlled to perform but not limited to the following operations: extend the robot arm, lift the robot arm, yaw the gripper, open the gripper, and close the gripper.

[0067] Figure 5 FIG. 1 is a flowchart illustrating a specific implementation of moving a target object from a starting container to a target container according to an exemplary embodiment of the present disclosure. Figure 5 The robot can sequentially perform the following steps: search for the target object, pick up the target object, find the target container, navigate to a convenient location for the target object, and finally place the target object. Furthermore, if a step fails during this process, the robot can return to that step or the previous step until the operation succeeds. For example, if picking up the target object fails, the robot can return to the previous step, "Searching for the Target Object," which precedes "Pick Up the Target Object."

[0068] The present disclosure provides a robot mobile operation method based on large language model guided exploration and operability perception navigation, which allows the robot to efficiently explore the environment in an open vocabulary environment and quickly find task-related objects. Specifically, the large language model can be used to learn the object placement rules to guide the robot to explore the environment to find task-related objects, which has strong generalization capabilities. In addition, the operable map prediction model can also be used to predict the local operable map to navigate the robot to a location where it is easy to operate task-related objects, which can improve the operation success rate and efficiency, thereby improving the robot's mobile operation capabilities in an open vocabulary environment. In addition, the present disclosure can effectively complete mobile operation tasks without consuming a large amount of data and time to train the model, which can save time and computing costs.

[0069] Figure 6 is a block diagram illustrating a mobile operation device 600 of a robot according to an exemplary embodiment of the present disclosure.

[0070] Reference Figure 6 The robot's mobile operation device 600 may include an image acquisition module 601, a semantic segmentation result acquisition module 602, a point cloud generation module 603, a semantic map acquisition module 604, a target position acquisition module 605, an operable map acquisition module 606 and an operation execution module 607.

[0071] The image acquisition module 601 can acquire an RGB image and a depth map at a current viewing angle, wherein the RGB image and the depth map can be shooting results at the current viewing angle by a camera set on the robot.

[0072] The semantic segmentation result acquisition module 602 can acquire semantic segmentation results based on the RGB image. For example, a hybrid perception module can be used to segment different object instances from the RGB image. The semantic segmentation results can include the object category to which each pixel in the RGB image belongs. For example, the object category can include, but is not limited to, a cup, a table, a floor, a refrigerator, and the like.

[0073] According to an exemplary embodiment of the present disclosure, the hybrid perception module may include an open-vocabulary perception model and a closed-vocabulary perception model. Furthermore, the open-vocabulary perception model may be used to detect image content corresponding to task-related objects and image content corresponding to spaces within an RGB image; while the closed-vocabulary perception model may be used to detect image content related to containers within an RGB image.

[0074] The semantic segmentation result acquisition module 602 may input the RGB image, the names of task-related objects, and the preset space names into an open vocabulary perception model to obtain an open semantic segmentation result. Furthermore, the semantic segmentation result acquisition module 602 may input the RGB image into a closed vocabulary perception model to obtain a closed semantic segmentation result. Next, the semantic segmentation result acquisition module 602 may merge the open semantic segmentation result and the closed semantic segmentation result to obtain a final semantic segmentation result.

[0075] The point cloud generation module 603 may generate a three-dimensional point cloud based on the depth map, wherein the three-dimensional point cloud may include the three-dimensional spatial coordinates of each pixel point among the multiple pixel points included in the depth map.

[0076] The semantic map acquisition module 604 can associate the semantic segmentation results corresponding to each pixel with the three-dimensional spatial coordinates and project the obtained association results to a bird's-eye view to obtain a two-dimensional semantic map. Specifically, in the present disclosure, the semantic segmentation results can be aligned with points at corresponding locations in the three-dimensional point cloud, and the semantically labeled point cloud can be projected to a bird's-eye view based on a geometric transformation to obtain a two-dimensional semantic map.

[0077] The target location acquisition module 605 may acquire the target location of the task-related object based on the semantic map and the name of the task-related object through a large language model.

[0078] According to an exemplary embodiment of the present disclosure, the mobile operating device 600 of the robot may further include a list acquisition module and a rule learning module.

[0079] The list acquisition module can also obtain the preset container list, preset space list and preset prompt. The preset container list can contain the names of multiple containers, and the preset space list can contain the names of multiple spaces. For example, you can summarize the common furniture in the home and obtain the preset container list. , and you can also summarize the room names to get a preset room list The preset prompts can be used to instruct the large language model to answer the first question of which containers among the multiple containers the task-related object may / may not appear near and the second question of which spaces among the multiple spaces the task-related object may / may not appear in.

[0080] The pattern learning module can then use the large language model to learn object placement patterns based on the preset container list, preset space list, and preset prompts. Specifically, in the present disclosure, the preset container list, preset space list, and preset prompts can be input into the large language model for common sense reasoning to learn object placement patterns.

[0081] Next, the target location acquisition module 605 can use the large language model to obtain the target location of the task-related object based on the semantic map, the names of the task-related objects, and the object placement patterns. Specifically, the large language model can use Probability Soft Logic (PSL) to calculate the exploration value of multiple edge points included in the semantic map according to soft constraint rules. Based on the calculated exploration value of each edge point, the target location of the task-related object can be determined.

[0082] According to an exemplary embodiment of the present disclosure, the target location acquisition module 605 can use a large language model to evaluate the exploration value of each of the multiple edge points based on the semantic map, the names of task-related objects, and the object placement patterns. This exploration value can be used to represent the likelihood that the corresponding edge point is the target location. Furthermore, the aforementioned multiple edge points can be points at the intersection of the currently explored area and the currently unexplored area included in the semantic map.

[0083] The target location acquisition module 605 can then select the edge point (x, y) with the highest exploration value from the multiple edge points as the target location, i.e., the navigation target point. Next, a path planning algorithm can be used to plan a path to the determined navigation target point, and the corresponding navigation action can be parsed. For example, the path planning algorithm in the present disclosure can be, but is not limited to, a fast marching method.

[0084] The operational map acquisition module 606 can evaluate the operability of locations near the target location using the operational map prediction model to obtain a local operational map. Each of the multiple grids included in the local operational map can correspond to a value indicating the difficulty of operating the task-related object in the grid. For example, the local operational map can be of size ( ) .

[0085] According to an exemplary embodiment of the present disclosure, the operable map acquisition module 606 can evaluate the operability of the location near the target location based on the local semantic map near the target location and the association result corresponding to the target location through the operable map prediction model to obtain a local operable map.

[0086] According to an exemplary embodiment of the present disclosure, the operational map prediction model may include a deep convolutional neural network, a deep learning network, and a decoder. The deep convolutional neural network may be, but is not limited to, ResNet50, and the deep learning network may be, but is not limited to, PointNet++.

[0087] The operational map acquisition module 606 can encode the local semantic map near the target location through a deep convolutional neural network to obtain a first embedding vector Then, the operational map acquisition module 606 can also encode the association result corresponding to the target position through a deep learning network to obtain a second embedding vector Next, the operational map acquisition module 606 can decode the first embedding vector Upsample to obtain decoding features Then, the map acquisition module 606 can be operated to decode the feature and the aforementioned second embedding vector The merging is performed, and the obtained merging result can be input into the decoder to obtain a local operational map.

[0088] The operation execution module 607 can control the robot to perform an operation on a task-related object based on the local operational map. For example, the robot can select a nearby, easily accessible location based on the local operational map as a navigation target point. Based on this navigation target point, the robot can then navigate to the corresponding location to perform the operation on the task-related object. Examples of such operations include, but are not limited to, picking up and placing.

[0089] Figure 7 is a block diagram illustrating an electronic device 700 according to an exemplary embodiment of the present disclosure.

[0090] Reference Figure 7 The electronic device 700 includes at least one memory 701 and at least one processor 702, wherein the at least one memory 701 stores instructions. When the instructions are executed by the at least one processor 702, the robot movement operation method according to the exemplary embodiment of the present disclosure is executed.

[0091] As an example, electronic device 700 may be a PC, tablet device, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 700 is not necessarily a single electronic device, but may also be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces with local or remote devices (e.g., via wireless transmission).

[0092] In electronic device 700, processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, and the like.

[0093] The processor 702 can execute instructions or codes stored in the memory 701, wherein the memory 701 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0094] The memory 701 may be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, the memory 701 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The memory 701 and the processor 702 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that the processor 702 can access files stored in the memory.

[0095] In addition, the electronic device 700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 700 may be connected to each other via a bus and / or a network.

[0096] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-mentioned robot movement operation method. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as a multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device configured to store a computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0097] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, wherein when the computer program is executed by a processor, the robot movement operation method according to the present disclosure is implemented.

[0098] According to the robot mobile operation method, device, electronic device, storage medium and computer program product disclosed in the present invention, the operability of the position near the target position of the task-related object is evaluated through the operable map prediction model, which can ensure that the robot is navigated to an operable position that avoids occlusion and obstacles, that is, it can ensure that the robot is navigated to a position that is conducive to operation, and then the robot can accurately operate the task-related objects, which can improve the operation efficiency and success rate.

[0099] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0100] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for moving a robot, characterized in that: include: Obtaining an RGB image and a depth map at a current viewing angle, wherein the RGB image and the depth map are captured by a camera provided on the robot at the current viewing angle; Obtaining a semantic segmentation result based on the RGB image, wherein the semantic segmentation result includes an object category to which each pixel in a plurality of pixels included in the RGB image belongs; Generating a three-dimensional point cloud based on the depth map, wherein the three-dimensional point cloud includes the three-dimensional spatial coordinates of each pixel point among the multiple pixel points included in the depth map; Associating the semantic segmentation result corresponding to each pixel point with the three-dimensional spatial coordinates, and projecting the obtained association result to a bird's-eye view to obtain a two-dimensional semantic map; Obtaining a target location of the task-related object based on the semantic map and the name of the task-related object through a large language model; Evaluating the operability of locations near the target location using an operable map prediction model to obtain a local operable map, wherein each of a plurality of grids included in the local operable map corresponds to a value indicating the difficulty of operating the task-related object at the grid; The robot is controlled to perform operations on the task-related object based on the local operational map.

2. The mobile operation method according to claim 1, wherein: The obtaining of a semantic segmentation result based on the RGB image includes: Inputting the RGB image, the name of the task-related object, and the preset space name into an open vocabulary perception model to obtain an open semantic segmentation result, wherein the open vocabulary perception model is used to detect image content corresponding to the task-related object and image content corresponding to the space in the RGB image; Inputting the RGB image into a closed vocabulary perception model to obtain a closed semantic segmentation result, wherein the closed vocabulary perception model is used to detect image content related to the container in the RGB image; The open semantic segmentation result and the closed semantic segmentation result are merged to obtain the semantic segmentation result.

3. The mobile operation method according to claim 1, wherein: The mobile operation method further includes: Obtaining a preset container list, a preset space list, and a preset prompt, wherein the preset container list includes the names of multiple containers, the preset space list includes the names of multiple spaces, and the preset prompt is used to instruct the large language model to answer a first question regarding which of the multiple containers the task-related object may or may not appear near, and a second question regarding which of the multiple spaces the task-related object may or may not appear within; learning object placement patterns based on the preset container list, the preset space list, and the preset prompts through the large language model; The acquiring of the target location of the task-related object based on the semantic map and the name of the task-related object by using the large language model includes: The target position of the task-related object is obtained by the large language model based on the semantic map, the name of the task-related object and the object placement pattern.

4. The mobile operation method according to claim 3, wherein: The acquiring, by the large language model based on the semantic map, the names of the task-related objects, and the object placement pattern, of the target locations of the task-related objects includes: Evaluate, by the large language model, the exploration value of each of a plurality of edge points based on the semantic map, the names of the task-related objects, and the object placement pattern, wherein the exploration value is used to represent the likelihood that the corresponding edge point is the target location, and the plurality of edge points are points at the intersection of a currently explored area and a currently unexplored area included in the semantic map; An edge point with the highest corresponding exploration value is selected from the multiple edge points as the target position.

5. The mobile operation method according to claim 1, wherein: The step of evaluating the operability of locations near the target location using an operable map prediction model to obtain a local operable map includes: The operable map prediction model is used to evaluate the operability of positions near the target position based on a local semantic map near the target position and an association result corresponding to the target position, thereby obtaining the local operable map, wherein the local semantic map is a partial semantic map centered on the target position that is intercepted from the semantic map.

6. The mobile operation method according to claim 5, wherein: The operational map prediction model includes a deep convolutional neural network, a deep learning network and a decoder; The step of evaluating the operability of a location near the target location based on a local semantic map near the target location and an association result corresponding to the target location by the operable map prediction model to obtain the local operable map includes: The local semantic map near the target position is encoded by the deep convolutional neural network to obtain a first embedding vector ; The association result corresponding to the target position is encoded by the deep learning network to obtain a second embedding vector ; The first embedding vector is decoded by the decoder Upsample to obtain decoding features ; The decoding feature and the second embedding vector The local operational map is obtained by merging the local operational map and inputting the merging result into the decoder.

7. A mobile operating device for a robot, characterized in that: include: An image acquisition module is configured to acquire an RGB image and a depth map at a current viewing angle, wherein the RGB image and the depth map are captured by a camera provided on the robot at the current viewing angle; a semantic segmentation result acquisition module configured to acquire a semantic segmentation result based on the RGB image, wherein the semantic segmentation result includes an object category to which each pixel in a plurality of pixels included in the RGB image belongs; a point cloud generation module configured to generate a three-dimensional point cloud based on the depth map, wherein the three-dimensional point cloud includes the three-dimensional spatial coordinates of each pixel point among the multiple pixel points included in the depth map; a semantic map acquisition module configured to associate the semantic segmentation result corresponding to each pixel with the three-dimensional spatial coordinates, and project the obtained association result to a bird's-eye view to obtain a two-dimensional semantic map; a target location acquisition module configured to acquire the target location of the task-related object based on the semantic map and the name of the task-related object using a large language model; an operable map acquisition module configured to evaluate the operability of locations near the target location using an operable map prediction model to obtain a local operable map, wherein each of a plurality of grids included in the local operable map corresponds to a value indicating the difficulty of operating the task-related object at the grid; An operation execution module is configured to control the robot to perform an operation on the task-related object based on the local operational map.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the robot movement operation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the movement operation method of the robot according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the robot movement operation method according to any one of claims 1 to 6 is implemented.

Images