Robot movement operation method, device, electronic device, storage medium, and computer program product

Abstract

The present disclosure relates to a robot movement operation method and device, electronic equipment, storage medium and computer program product, the method comprising: obtaining a semantic segmentation result based on an RGB image; generating a three-dimensional point cloud based on a depth map; associating the semantic segmentation result corresponding to each pixel point and the three-dimensional space coordinates, and projecting the obtained association result to a bird's eye view perspective; obtaining a target position of a task-related object based on a semantic map and the name of the task-related object; evaluating the operability of the nearby position of the target position through an operable map prediction model; and controlling the robot to perform an operation on the task-related object based on the local operable map. In this way, by evaluating the operability of the nearby position of the target position of the task-related object through the operable map prediction model, the robot can be navigated to an operable position that avoids obstacles, thereby ensuring that the robot can accurately operate on the task-related object, and the operation success rate can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, and more specifically, to methods, apparatuses, electronic devices, storage media, and computer program products for the movement and operation of robots. Background Technology

[0002] With the development of technology, robots for various purposes have emerged. For example, robots used to perform various household chores are a common type of robot. This type of robot needs to find and manipulate target objects in unfamiliar environments. In recent years, although some progress has been made in object navigation and manipulation, the lack of targeted object-oriented exploration has resulted in poor compatibility between the robot's navigation and manipulation skills. This makes it still very challenging to complete mobile manipulation tasks through robots.

[0003] Currently, learning an end-to-end policy to complete a robot's entire locomotion task has attracted considerable attention from researchers. These end-to-end methods directly map observations to underlying actions and infer the skills to be performed and how to execute them. However, applying end-to-end learning methods to locomotion, a long-term, multi-stage task involving thousands of underlying actions, is extremely challenging. This is because the process requires tens of thousands of expert demonstrations or hundreds of millions of reinforcement learning steps for training, which is impractical to collect in the real world. More importantly, end-to-end methods are limited by their generalization ability and lack of interpretability, making them difficult to apply in the real world.

[0004] Unlike end-to-end methods, modular approaches break down a robot's mobility tasks into distinct parts, such as navigation, grasping, and placement. Modular approaches reduce problem complexity by decomposing mobility tasks and also offer better interpretability and generalization capabilities. However, using modular approaches to accomplish mobility tasks still faces several challenges. First, efficiently locating target objects in a large workspace is difficult due to the sparsity of effective observation points, especially without prior knowledge. Furthermore, humans generally utilize experience from previous environments to explore new environments quickly and locate target objects. For example, when searching for a cup in a new house, a user is more likely to search in the kitchen than the bedroom. Therefore, learning and utilizing these semantic priors is crucial for the deployment of mobile robots. Second, handover errors between different skills are a key factor hindering the success of modular approaches. For instance, when a robot manipulates an object, it needs to reach a suitable position to ensure the object is within the effective workspace of the robotic arm. However, current navigation methods simply use distance thresholds for location filtering, which may lead the robot to a location full of obstacles that is not conducive to operation, resulting in a high failure rate of robot movement operations and making it difficult to meet the needs of real-world use scenarios. Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for the movement and operation of a robot, in order to at least solve the problem of a high failure rate in the movement and operation of robots in the aforementioned related technologies.

[0006] According to a first aspect of the present disclosure, a method for manipulating a robot is provided, comprising: acquiring an RGB image and a depth map from a current viewpoint, wherein the RGB image and the depth map are images captured by a camera mounted on the robot from the current viewpoint; acquiring a semantic segmentation result based on the RGB image, wherein the semantic segmentation result includes the object category to which each pixel in a plurality of pixels contained in the RGB image belongs; generating a three-dimensional point cloud based on the depth map, wherein the three-dimensional point cloud includes the three-dimensional spatial coordinates of each pixel in a plurality of pixels contained in the depth map; and assigning each pixel... The corresponding semantic segmentation results are associated with three-dimensional spatial coordinates, and the obtained association results are projected onto a bird's-eye view to obtain a two-dimensional semantic map. Based on the semantic map and the names of task-related objects, a large language model is used to obtain the target location of the task-related objects. The operability of the vicinity of the target location is evaluated by an operability map prediction model to obtain a local operability map. Each grid in the local operability map corresponds to a value indicating the difficulty of operating the task-related objects in that grid. Based on the local operability map, the robot is controlled to perform operations on the task-related objects.

[0007] Optionally, obtaining the semantic segmentation result based on the RGB image includes: inputting the RGB image, the name of the task-related object, and a preset spatial name into an open vocabulary awareness model to obtain an open semantic segmentation result, wherein the open vocabulary awareness model is used to detect image content corresponding to the task-related object and image content corresponding to the space within the RGB image; inputting the RGB image into a closed vocabulary awareness model to obtain a closed semantic segmentation result, wherein the closed vocabulary awareness model is used to detect image content related to the container within the RGB image; and merging the open semantic segmentation result and the closed semantic segmentation result to obtain the semantic segmentation result.

[0008] Optionally, the movement operation method further includes: obtaining a preset container list, a preset space list, and preset prompts, wherein the preset container list contains the names of multiple containers, the preset space list contains the names of multiple spaces, and the preset prompts are used to instruct the large language model to answer a first question about which containers the task-related object may / may not appear near and a second question about which spaces the task-related object may / may not appear in; learning object placement patterns through the large language model based on the preset container list, the preset space list, and the preset prompts; wherein obtaining the target location of the task-related object through the large language model based on the semantic map and the names of the task-related objects includes: obtaining the target location of the task-related object through the large language model based on the semantic map, the names of the task-related objects, and the object placement patterns.

[0009] Optionally, obtaining the target location of the task-related object based on the semantic map, the names of the task-related objects, and the object placement pattern using the large language model includes: evaluating the exploration value of each edge point among multiple edge points using the large language model based on the semantic map, the names of the task-related objects, and the object placement pattern, wherein the exploration value is used to characterize the probability that the corresponding edge point is the target location, and the multiple edge points are points on the boundary between the currently explored area and the currently unexplored area contained in the semantic map; and selecting the edge point with the highest exploration value from the multiple edge points as the target location.

[0010] Optionally, the step of evaluating the operability of the vicinity of the target location through the operability map prediction model to obtain a local operability map includes: evaluating the operability of the vicinity of the target location based on the local semantic map near the target location and the association result corresponding to the target location through the operability map prediction model to obtain the local operability map, wherein the local semantic map is a partial semantic map centered on the target location extracted from the semantic map.

[0011] Optionally, the operable map prediction model includes a deep convolutional neural network, a deep learning network, and a decoder; the step of evaluating the operability of nearby locations based on the local semantic map near the target location and the association results corresponding to the target location, and obtaining the local operable map, includes: encoding the local semantic map near the target location using the deep convolutional neural network to obtain a first embedding vector. The deep learning network encodes the association results corresponding to the target location to obtain a second embedding vector. ; the first embedding vector is processed by the decoder Perform upsampling to obtain decoding features ; the decoding features and the second embedding vector The merged data is then input into the decoder to obtain the locally operable map.

[0012] According to a second aspect of the present disclosure, a mobile operation device for a robot is provided, comprising: an image acquisition module configured to acquire an RGB image and a depth map from a current viewpoint, wherein the RGB image and the depth map are images captured by a camera mounted on the robot from the current viewpoint; a semantic segmentation result acquisition module configured to acquire a semantic segmentation result based on the RGB image, wherein the semantic segmentation result includes the object category to which each pixel in a plurality of pixels contained in the RGB image belongs; a point cloud generation module configured to generate a three-dimensional point cloud based on the depth map, wherein the three-dimensional point cloud includes the three-dimensional spatial coordinates of each pixel in a plurality of pixels contained in the depth map; and a semantic map acquisition module configured to... The semantic segmentation result corresponding to each pixel is associated with its three-dimensional spatial coordinates, and the obtained association result is projected onto a bird's-eye view to obtain a two-dimensional semantic map. The target location acquisition module is configured to obtain the target location of the task-related object based on the semantic map and the name of the task-related object using a large language model. The operable map acquisition module is configured to evaluate the operability of the vicinity of the target location using an operable map prediction model to obtain a local operable map, wherein each grid in the local operable map corresponds to a value indicating the difficulty of operating the task-related object in that grid. The operation execution module is configured to control the robot to perform operations on the task-related object based on the local operable map.

[0013] Optionally, the semantic segmentation result acquisition module is configured to: input the RGB image, the name of the task-related object, and a preset spatial name into an open vocabulary perception model to obtain an open semantic segmentation result, wherein the open vocabulary perception model is used to detect image content in the RGB image corresponding to the task-related object and image content corresponding to the space; input the RGB image into a closed vocabulary perception model to obtain a closed semantic segmentation result, wherein the closed vocabulary perception model is used to detect image content in the RGB image related to the container; and merge the open semantic segmentation result and the closed semantic segmentation result to obtain the semantic segmentation result.

[0014] Optionally, the robot's mobile operation device further includes: a list acquisition module configured to acquire a preset container list, a preset space list, and preset prompts, wherein the preset container list contains the names of multiple containers, the preset space list contains the names of multiple spaces, and the preset prompts are used to instruct the large language model to answer a first question about which containers the task-related object may / may not appear near and a second question about which spaces the task-related object may / may not appear in; a pattern learning module configured to learn object placement patterns based on the preset container list, the preset space list, and the preset prompts using the large language model; and a target location acquisition module configured to acquire the target location of the task-related object based on the semantic map, the names of the task-related objects, and the object placement patterns using the large language model.

[0015] Optionally, the target location acquisition module is configured to: evaluate the exploration value of each edge point among multiple edge points based on the semantic map, the names of the task-related objects, and the object placement patterns using the large language model, wherein the exploration value is used to characterize the probability that the corresponding edge point is the target location, and the multiple edge points are points on the boundary between the currently explored area and the currently unexplored area contained in the semantic map; and select the edge point with the highest exploration value from the multiple edge points as the target location.

[0016] Optionally, the operable map acquisition module is configured to: evaluate the operability of nearby locations of the target location based on the local semantic map near the target location and the association results corresponding to the target location through the operable map prediction model, and obtain the local operable map, wherein the local semantic map is a partial semantic map centered on the target location extracted from the semantic map.

[0017] Optionally, the operable map prediction model includes a deep convolutional neural network, a deep learning network, and a decoder; the operable map acquisition module is configured to: encode a local semantic map near the target location using the deep convolutional neural network to obtain a first embedding vector. The deep learning network encodes the association results corresponding to the target location to obtain a second embedding vector. ; the first embedding vector is processed by the decoder Perform upsampling to obtain decoding features ; the decoding features and the second embedding vector The merged data is then input into the decoder to obtain the locally operable map.

[0018] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a robot movement operation method according to the present disclosure.

[0019] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a robot movement operation method according to the present disclosure.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a robot movement operation method according to the present disclosure.

[0021] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0022] In this disclosure, by using an operable map prediction model to assess the operability of the vicinity of the target location of the task-related object, it is possible to ensure that the robot is navigated to an operable location that avoids obstructions and obstacles, that is, to ensure that the robot is navigated to a location that is conducive to operation. This ensures that the robot can accurately operate on the task-related object, thereby improving operational efficiency and success rate.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0025] Figure 1 This is a flowchart illustrating a method for moving a robot according to an exemplary embodiment of the present disclosure;

[0026] Figure 2 This is a schematic diagram illustrating the process of obtaining a local operable map through an operable map prediction model according to an exemplary embodiment of the present disclosure;

[0027] Figure 3 This is a schematic diagram illustrating the implementation process of a robot movement operation method according to an exemplary embodiment of the present disclosure;

[0028] Figure 4This is an example diagram illustrating an implementation process of moving an object from a starting container to a target container according to an exemplary embodiment of the present disclosure;

[0029] Figure 5 This is a flowchart illustrating a specific implementation of moving a target object from a starting container to a target container according to an exemplary embodiment of the present disclosure;

[0030] Figure 6 This is a block diagram illustrating a mobile manipulation device for a robot according to an exemplary embodiment of the present disclosure;

[0031] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0033] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0034] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0035] Figure 1 This is a flowchart illustrating a method for moving a robot according to an exemplary embodiment of the present disclosure.

[0036] Reference Figure 1 In step 101, an RGB image and a depth map can be obtained from the current viewpoint. The RGB image and depth map can be the results of shooting by a camera set on the robot from the current viewpoint.

[0037] In step 102, semantic segmentation results can be obtained based on the RGB image. For example, a hybrid sensing module can be used to segment different object instances from the RGB image. The semantic segmentation results can include the object category to which each pixel in the RGB image belongs. For example, object categories can be, but are not limited to: cup, table, floor, refrigerator, etc.

[0038] According to exemplary embodiments of this disclosure, the hybrid sensing module described above may include an open-vocabulary sensing model and a closed-vocabulary sensing model. Furthermore, the open-vocabulary sensing model can be used to detect image content within an RGB image corresponding to task-related objects and image content corresponding to space; the closed-vocabulary sensing model can be used to detect image content within an RGB image related to containers.

[0039] It should be noted that "task-related objects" can refer to any object involved in the task. For example, for the task "Move the sponge from the table to the sofa," task-related objects could include "sponge," "table," and "sofa." The aforementioned "space" can be, but is not limited to, a room. For example, common rooms could include "kitchen," "bathroom," "bedroom," "living room," etc. The aforementioned "container" can be common furniture in a home that can be used to place small objects, such as, but not limited to, "cabinet," "refrigerator," and "table," etc.

[0040] An RGB image, the names of task-related objects, and a preset spatial name can be input into an open vocabulary-aware model to obtain open semantic segmentation results; conversely, an RGB image can be input into a closed vocabulary-aware model to obtain closed semantic segmentation results. Next, the open and closed semantic segmentation results can be merged to obtain the final semantic segmentation result. Specifically:

[0041]

[0042]

[0043]

[0044] .

[0045] in, Represents an object, Indicates a room. Indicates a container. For semantic masking, The image is an RGB image from the current perspective. A prompt word that includes the name of the object. This is a prompt word containing the room name. In this disclosure, object names can be obtained by connecting them with a period (."). Furthermore, you can also obtain information by connecting the room name with a period (.). For example, assuming the object to be detected is "sofa", "table", or "cabinet", then the corresponding prompt words will be... It can be "sofa.table.cabinet".

[0046] It should be noted that the above open vocabulary awareness model can be, but is not limited to, a weakly supervised learning-based object detection method (Detector with Image Classes, Detic), and there is no need to train this open vocabulary awareness model; the above closed vocabulary awareness model can be, but is not limited to, You Only Look Once, YOLO, which can be trained by collecting data from the environment.

[0047] In step 103, a three-dimensional point cloud can be generated based on the depth map, wherein the three-dimensional point cloud can contain the three-dimensional spatial coordinates of each pixel in the multiple pixels contained in the depth map.

[0048] In step 104, the semantic segmentation result corresponding to each pixel can be associated with the three-dimensional spatial coordinates, and the obtained association result can be projected onto the bird's-eye view to obtain a two-dimensional semantic map. That is, in this disclosure, the semantic segmentation result can be aligned with the corresponding points in the three-dimensional point cloud, and the point cloud with semantic labels can be projected onto the bird's-eye view based on geometric transformation, thereby obtaining a two-dimensional semantic map.

[0049] The above semantic map It can be a binary grid map, which can contain ( (Number) channels. And, Each of the channels can represent the occupancy map of a class of objects, while the other two channels can represent the explored areas and obstacles, respectively. It can represent the number of grids, and each grid can represent ( A ground space of 1.5 inch in size.

[0050] In step 105, the target location of the task-related object can be obtained by using a large language model based on the semantic map and the name of the task-related object.

[0051] According to exemplary embodiments of this disclosure, a preset container list, a preset space list, and preset prompts can also be obtained. The preset container list can contain the names of multiple containers, and the preset space list can contain the names of multiple spaces. For example, a preset container list can be obtained by categorizing common household furniture. Furthermore, it can also summarize the room names to obtain a preset room list. The aforementioned preset prompts can be used to instruct the large language model to answer the first question about which containers a task-related object might / could not appear in, and the second question about which spaces a task-related object might / could not appear in.

[0052] Then, the large language model can learn the rules of object placement based on the aforementioned preset container list, preset space list, and preset prompts. That is, in this disclosure, the preset container list, preset space list, and preset prompts can be input into the large language model to perform common sense reasoning to learn the rules of object placement.

[0053] Next, the target location of task-related objects can be obtained using a large language model based on the semantic map, the names of task-related objects, and the placement patterns of objects. Specifically, the large language model can use Probability Soft Logic (PSL) to calculate the exploration value of multiple edge points contained in the semantic map according to soft constraint rules, and then determine the target location of task-related objects based on the calculated exploration value of each edge point.

[0054] According to exemplary embodiments of this disclosure, the exploration value of each edge point among multiple edge points can be evaluated using a large language model based on a semantic map, the names of task-related objects, and object placement patterns. This exploration value can be used to characterize the probability that the corresponding edge point is a target location. Furthermore, the aforementioned multiple edge points can be points on the boundary between currently explored and currently unexplored areas contained in the semantic map.

[0055] Then, the edge point (x, y) with the highest exploration value can be selected from multiple edge points as the target location, i.e., the navigation target point. Next, a path planning algorithm can be used to plan a path to the determined navigation target point, and the corresponding navigation actions can be parsed. For example, the path planning algorithm in this disclosure can be, but is not limited to, the Fast Matching Method (FMM).

[0056] It should be noted that the specific process of calculating the exploration value of each edge point among multiple edge points contained in the semantic map using PSL can be described as follows:

[0057] PSL can be used in the location We can use the probability soft value to make inferences and propose the following soft constraints:

[0058]

[0059]

[0060]

[0061] in, For objects related to the task, For edge points; express and The co-occurrence relationships can be obtained from large language models; express and Whether they are close or not can be determined by calculating edge points and their relationship to the semantic map. Obtained by normalized nearest distance from the occupancy graph; express Is it the closest? edge points; This represents the weight of a rule, which is used to measure the importance of the corresponding rule; express Its exploratory value.

[0062] Then, PSL can convert atoms ( ) Mapped to located The probability soft truth value, this mapping process is called realization. Next, PSL can model the relationships between atoms as a probabilistic graphical model, and can define them in the implementation. probability density function on Therefore, it can be maximized To infer Then, it is possible to start from multiple edge points. Select The edge point with the highest value is selected as the exploration target.

[0063] It should be noted that, in this disclosure, the aforementioned edge points This can be obtained by calculating the boundary between explored and unexplored areas contained in the semantic map; as mentioned above. This can be obtained through large language model inference, and for object pairs with different relationships, different values ​​can be assigned to represent the strength of their co-occurrence relationship. Specifically: for object pairs with a co-occurrence relationship, values ​​can be assigned to them. For pairs of objects with uncertain relationships, values ​​can be assigned to them. For object pairs that have a repulsive relationship, values ​​can be assigned to them. Furthermore, the above assignments satisfy the following: 1> > > >0.

[0064] In step 106, the operability of the vicinity of the target location can be evaluated using an operability map prediction model to obtain a local operability map. Each grid in this local operability map can correspond to a value indicating the ease or difficulty of manipulating task-related objects within that grid. For example, this local operability map can be of size (…). The mask matrix of ) .

[0065] It should be noted that in this disclosure, the robot can be initialized at a visible point (an angle from which the position of the task-related object can be observed) of the task-related object, and then the operation strategy can be executed at that location, recording whether the operation on the task-related object was successful. Next, the points of successful operation can be unified under the same coordinate system to obtain the ground truth of the operable map. Then, an operable map prediction model can be trained based on this ground truth. Furthermore, the operable map prediction model can also be trained using, but is not limited to, the Dice loss function.

[0066] According to an exemplary embodiment of this disclosure, an operability map can be obtained by using an operability map prediction model to evaluate the operability of nearby locations based on a local semantic map near the target location and the association results corresponding to the target location.

[0067] A local semantic map can be a portion of the semantic map extracted from the aforementioned semantic map, centered on the target location. For example, a local semantic map can be a local tensor extracted from the aforementioned semantic map, containing an obstacle map and a task-related object occupancy map, and centered on the target location of the task-related object. .

[0068] The aforementioned "association results corresponding to the target location" can also be called "point cloud of the location of the object related to the task," which can be a tensor containing four channels. ,in, This represents the number of points, and each point can correspond to a 4-dimensional vector. Furthermore, the first three dimensions of this 4-dimensional vector can represent the spatial coordinates of the point. The last dimension can indicate whether the point belongs to a task-related object.

[0069] According to exemplary embodiments of this disclosure, the above-described operable map prediction model may include a deep convolutional neural network, a deep learning network, and a decoder. The deep convolutional neural network may be, but is not limited to, ResNet50, and the deep learning network may be, but is not limited to, PointNet++.

[0070] Figure 2This is a schematic diagram illustrating a process for obtaining a local operable map through an operable map prediction model according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 2 A first embedding vector can be obtained by encoding a local semantic map near the target location using a deep convolutional neural network. For example, a ResNet50 can be used to encode a local semantic map near the target location to obtain a first embedding vector. .

[0071] Then, a second embedding vector can be obtained by encoding the association results corresponding to the target location through a deep learning network. For example, PointNet++ can also be used to encode the associated results corresponding to the target location, i.e., the point cloud that marks the locations of task-related objects, to obtain a second embedding vector. Next, the first embedding vector can be decoded. Perform upsampling to obtain decoding features .

[0072] Then, the decoding feature can be... and the aforementioned second embedding vector The points are merged, and the merged result can be input into the decoder to obtain a locally operable map. Specifically, point cloud features and semantic map features can be fused in the first three layers of the decoder: First, the point cloud features can be merged... Projected onto a linear layer Then, the... flat construction The feature map. Next, the tiled feature map can be compared with the aforementioned decoded features. Element-wise multiplication yields the fused features. It should be noted that, in this disclosure, skip connections can be used to connect encoder features of corresponding layers to decoder features. Then, the final locally operable map can be obtained based on the fused features described above.

[0073] In step 107, the robot can be controlled to perform operations on task-related objects based on a locally operable map. For example, the nearest and easily operable location can be selected as a navigation target point based on the locally operable map, and the robot can then be navigated to that location to perform operations on the task-related objects. For example, this operation can be, but is not limited to, picking up or placing objects.

[0074] It should be noted that in this disclosure, both the picking strategy and the placement strategy can be deterministic strategies, which may specifically include the following steps:

[0075] (1) Select the pick-up point (grab point) or the placement point For example, the center of the mask of the task-related object can be calculated as the grab point, or the center of the largest inscribed circle of the blank plane of the target container can be calculated as the placement point.

[0076] (2) Calculate the positional difference between the robot's current gripper position and the pickup point (grasping point) or placement point. ;

[0077] (3) Control the robot's robotic arms to move separately , , It reaches the operation point and opens or closes the grippers to grab or place the task-related object.

[0078] Figure 3 This is a schematic diagram illustrating the implementation process of a robot movement operation method according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 3 The robot movement operation method disclosed herein mainly comprises two parts: the process of finding task-related objects based on visual observation results, and the process of operating on the task-related objects after they have been found. These two processes have been described in detail above and will not be repeated here.

[0079] Figure 4 This is an example diagram illustrating an implementation process of moving an object from a starting container to a target container according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 4 The robot's initial position is (x, y), and its orientation is... Furthermore, the process of moving an object from the starting container to the target container can be mainly divided into two parts: the process of finding task-related objects based on the captured RGB and depth images, and the process of manipulating the task-related objects after they have been found. For example, during the process of finding task-related objects, the robot can be controlled to perform actions including but not limited to: turning left, turning right, moving forward, moving backward, and stopping; during the process of manipulating task-related objects, the robot can be controlled to perform operations including but not limited to: extending the robotic arm, raising the robotic arm, yawing the gripper, opening the gripper, and closing the gripper.

[0080] Figure 5 This is a flowchart illustrating a specific implementation of moving a target object from a starting container to a target container according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 5The robot can execute the following steps sequentially: finding the target object, picking up the target object, finding the target container, navigating to a location suitable for placing the target object, and placing the target object. Furthermore, if a step fails, the robot can revert to that step or the previous step until the operation succeeds. For example, if picking up the target object fails, the robot can return to the step preceding "picking up the target object," which is "finding the target object."

[0081] This disclosure provides a robot mobility manipulation method based on large language model-guided exploration and operability-aware navigation. This method allows the robot to efficiently explore its environment in an open vocabulary context, thereby quickly finding task-related objects. Specifically, a large language model can learn object placement patterns to guide the robot's exploration of the environment to find task-related objects, exhibiting strong generalization ability. Furthermore, an operability map prediction model can be used to predict a local operability map to navigate the robot to locations where task-related objects are easily manipulated, improving the success rate and efficiency of operations, thus enhancing the robot's mobility manipulation capabilities in open vocabulary environments. In addition, this disclosure can effectively complete mobility manipulation tasks without requiring a large amount of data and time to train the model, saving time and computational costs.

[0082] Figure 6 This is a block diagram illustrating a mobile operating device 600 for a robot according to an exemplary embodiment of the present disclosure.

[0083] Reference Figure 6 The robot's mobile operation device 600 may include an image acquisition module 601, a semantic segmentation result acquisition module 602, a point cloud generation module 603, a semantic map acquisition module 604, a target position acquisition module 605, an operable map acquisition module 606, and an operation execution module 607.

[0084] The image acquisition module 601 can acquire RGB images and depth maps from the current viewpoint. The RGB images and depth maps can be the results of shooting by a camera set on the robot from the current viewpoint.

[0085] The semantic segmentation result acquisition module 602 can acquire semantic segmentation results based on RGB images. For example, a hybrid sensing module can be used to segment different object instances from an RGB image. The aforementioned semantic segmentation result can include the object category to which each pixel in the RGB image belongs. For example, object categories can be, but are not limited to: cup, table, floor, refrigerator, etc.

[0086] According to exemplary embodiments of this disclosure, the hybrid sensing module described above may include an open-vocabulary sensing model and a closed-vocabulary sensing model. Furthermore, the open-vocabulary sensing model can be used to detect image content within an RGB image corresponding to task-related objects and image content corresponding to space; the closed-vocabulary sensing model can be used to detect image content within an RGB image related to containers.

[0087] The semantic segmentation result acquisition module 602 can input an RGB image, the names of task-related objects, and preset spatial names into an open vocabulary-aware model to obtain an open semantic segmentation result. Furthermore, the semantic segmentation result acquisition module 602 can input an RGB image into a closed vocabulary-aware model to obtain a closed semantic segmentation result. Next, the semantic segmentation result acquisition module 602 can merge the open and closed semantic segmentation results to obtain the final semantic segmentation result.

[0088] The point cloud generation module 603 can generate a three-dimensional point cloud based on the depth map, wherein the three-dimensional point cloud can contain the three-dimensional spatial coordinates of each pixel in the multiple pixels contained in the depth map.

[0089] The semantic map acquisition module 604 can associate the semantic segmentation result corresponding to each pixel with the three-dimensional spatial coordinates, and can project the obtained association result onto the bird's-eye view to obtain a two-dimensional semantic map. That is, in this disclosure, the semantic segmentation result can be aligned with the corresponding points in the three-dimensional point cloud, and the point cloud with semantic labels can be projected onto the bird's-eye view based on geometric transformation, thereby obtaining a two-dimensional semantic map.

[0090] The target location acquisition module 605 can obtain the target location of task-related objects based on the semantic map and the names of task-related objects using a large language model.

[0091] According to an exemplary embodiment of this disclosure, the robot's mobile operation device 600 may further include a list acquisition module and a pattern learning module.

[0092] The list retrieval module can also retrieve a list of preset containers, a list of preset spaces, and preset prompts. The list of preset containers can contain the names of multiple containers, and the list of preset spaces can contain the names of multiple spaces. For example, it can categorize common home furniture to obtain a list of preset containers. Furthermore, it can also summarize the room names to obtain a preset room list. The aforementioned preset prompts can be used to instruct the large language model to answer the first question about which containers a task-related object might / could not appear in, and the second question about which spaces a task-related object might / could not appear in.

[0093] Then, the pattern learning module can learn the object placement patterns based on the aforementioned preset container list, preset space list, and preset prompts using a large language model. That is, in this disclosure, the preset container list, preset space list, and preset prompts can be input into the large language model to perform common-sense reasoning to learn the object placement patterns.

[0094] Next, the target location acquisition module 605 can obtain the target location of the task-related object based on the semantic map, the names of task-related objects, and the object placement patterns using a large language model. Specifically, the large language model can use Probability Soft Logic (PSL) to calculate the exploration value of multiple edge points contained on the semantic map according to soft constraint rules, and then determine the target location of the task-related object based on the calculated exploration value of each edge point.

[0095] According to an exemplary embodiment of this disclosure, the target location acquisition module 605 can evaluate the exploration value of each of a plurality of edge points based on a semantic map, the names of task-related objects, and object placement patterns using a large language model. This exploration value can be used to characterize the probability that the corresponding edge point is the target location. Furthermore, the aforementioned plurality of edge points can be points on the boundary between currently explored and currently unexplored areas contained in the semantic map.

[0096] Then, the target location acquisition module 605 can select the edge point (x, y) with the highest exploration value from multiple edge points as the target location, i.e., the navigation target point. Next, a path planning algorithm can be used to plan a path to the determined navigation target point, and the corresponding navigation action can be parsed. For example, the path planning algorithm in this disclosure can be, but is not limited to, the fast travel method.

[0097] The operability map acquisition module 606 can evaluate the operability of the vicinity of the target location through an operability map prediction model to obtain a local operability map. Each grid in the local operability map can correspond to a value indicating the ease or difficulty of manipulating task-related objects within that grid. For example, the local operability map can be of size (…). The mask matrix of ) .

[0098] According to an exemplary embodiment of this disclosure, the operable map acquisition module 606 can use an operable map prediction model to evaluate the operability of nearby locations based on the local semantic map near the target location and the association results corresponding to the target location, and obtain a local operable map.

[0099] According to exemplary embodiments of this disclosure, the above-described operable map prediction model may include a deep convolutional neural network, a deep learning network, and a decoder. The deep convolutional neural network may be, but is not limited to, ResNet50, and the deep learning network may be, but is not limited to, PointNet++.

[0100] The operable map acquisition module 606 can encode the local semantic map near the target location using a deep convolutional neural network to obtain the first embedding vector. Then, the operable map acquisition module 606 can also encode the association results corresponding to the target location through a deep learning network to obtain a second embedding vector. Next, the operable map acquisition module 606 can use the decoder to process the first embedding vector. Perform upsampling to obtain decoding features Then, the operable map acquisition module 606 can decode the features. and the aforementioned second embedding vector The merged data can be input into the decoder to obtain a locally operable map.

[0101] The operation execution module 607 can control the robot to perform operations on task-related objects based on a locally operable map. For example, it can select the nearest and easily operable location as a navigation target point based on the locally operable map, and then navigate the robot to the corresponding location to perform operations on the task-related objects based on this navigation target point. For example, this operation can be, but is not limited to, picking up, placing, etc.

[0102] Figure 7 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment of the present disclosure.

[0103] Reference Figure 7 The electronic device 700 includes at least one memory 701 and at least one processor 702. The at least one memory 701 stores instructions that, when executed by the at least one processor 702, perform a robot movement operation method according to an exemplary embodiment of the present disclosure.

[0104] As an example, electronic device 700 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 700 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 700 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0105] In electronic device 700, processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0106] The processor 702 can execute instructions or code stored in the memory 701, which can also store data. Instructions and data can also be sent and received via a network through a network interface device, which can employ any known transmission protocol.

[0107] The memory 701 can be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 701 can include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 701 and the processor 702 can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 702 to read files stored in the memory.

[0108] In addition, the electronic device 700 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 700 can be interconnected via a bus and / or network.

[0109] According to exemplary embodiments of this disclosure, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the aforementioned robot movement operation method. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0110] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including a computer program that, when executed by a processor, implements a method for moving a robot according to the present disclosure.

[0111] According to the robot movement operation method, device, electronic device, storage medium and computer program product disclosed herein, by evaluating the operability of the vicinity of the target location of the task-related object by using an operability map prediction model, it is possible to ensure that the robot is navigated to an operable location that avoids obstructions and obstacles, that is, to ensure that the robot is navigated to a location that is conducive to operation, thereby ensuring that the robot can accurately operate on the task-related object, and improving operation efficiency and success rate.

[0112] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0113] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for manipulating the movement of a robot, characterized in that, include: Acquire an RGB image and a depth map from the current viewpoint, wherein the RGB image and the depth map are captured by a camera mounted on the robot from the current viewpoint; Based on the RGB image, a semantic segmentation result is obtained, wherein the semantic segmentation result includes the object category to which each pixel belongs among the multiple pixels contained in the RGB image; Based on the depth map, a three-dimensional point cloud is generated, wherein the three-dimensional point cloud contains the three-dimensional spatial coordinates of each pixel among the multiple pixels contained in the depth map. The semantic segmentation result corresponding to each pixel is associated with the three-dimensional spatial coordinates, and the obtained association result is projected onto the bird's-eye view to obtain a two-dimensional semantic map. The target location of the task-related object is obtained by using a large language model based on the semantic map and the names of the task-related objects. The operability of the vicinity of the target location is evaluated by an operability map prediction model to obtain a local operability map, wherein each of the multiple grids contained in the local operability map corresponds to a value indicating the ease or difficulty of manipulating the task-related object in that grid. Based on the locally operable map, the robot is controlled to perform operations on the objects related to the task. The moving operation method further includes: Obtain a preset container list, a preset space list, and a preset prompt, wherein the preset container list contains the names of multiple containers, the preset space list contains the names of multiple spaces, and the preset prompt is used to instruct the large language model to answer a first question about which containers the task-related object may / may not appear near and a second question about which spaces the task-related object may / may not appear in. The large language model learns the rules of object placement based on the preset container list, the preset space list, and the preset prompts. The step of obtaining the target location of the task-related object based on the semantic map and the names of the task-related objects using a large language model includes: The target location of the task-related object is obtained by using the large language model based on the semantic map, the names of the task-related objects, and the object placement rules. The step of obtaining the target location of the task-related object based on the semantic map, the names of the task-related objects, and the object placement pattern using the large language model includes: The large language model evaluates the exploration value of each edge point among multiple edge points based on the semantic map, the names of the task-related objects, and the object placement patterns. The exploration value is used to characterize the probability that the corresponding edge point is the target location. The multiple edge points are the points on the boundary between the currently explored area and the currently unexplored area contained in the semantic map. Select the edge point with the highest exploration value from the plurality of edge points as the target location.

2. The mobile operation method as described in claim 1, characterized in that, The step of obtaining semantic segmentation results based on the RGB image includes: The RGB image, the name of the task-related object, and the preset space name are input into the open vocabulary perception model to obtain the open semantic segmentation result. The open vocabulary perception model is used to detect the image content corresponding to the task-related object and the image content corresponding to the space in the RGB image. The RGB image is input into a closed-vocabulary perception model to obtain closed semantic segmentation results, wherein the closed-vocabulary perception model is used to detect image content related to containers within the RGB image; The open semantic segmentation result and the closed semantic segmentation result are merged to obtain the semantic segmentation result.

3. The mobile operation method as described in claim 1, characterized in that, The step of evaluating the operability of the vicinity of the target location using an operability map prediction model to obtain a local operability map includes: The operability map prediction model evaluates the operability of nearby locations based on the local semantic map near the target location and the association results corresponding to the target location, thereby obtaining the local operability map. The local semantic map is a portion of the semantic map centered on the target location, extracted from the semantic map.

4. The mobile operation method as described in claim 3, characterized in that, The operable map prediction model includes a deep convolutional neural network, a deep learning network, and a decoder; The step of using the operability map prediction model to evaluate the operability of locations near the target location based on the local semantic map near the target location and the association results corresponding to the target location, and obtaining the local operability map, includes: The local semantic map near the target location is encoded using the deep convolutional neural network to obtain the first embedding vector. ; The deep learning network encodes the association results corresponding to the target location to obtain the second embedding vector. ; The first embedding vector is processed by the decoder. Perform upsampling to obtain decoding features ; The decoded features and the second embedding vector The merged data is then input into the decoder to obtain the locally operable map.

5. A mobile manipulation device for a robot, characterized in that, include: The image acquisition module is configured to acquire an RGB image and a depth map from the current viewpoint, wherein the RGB image and the depth map are the results of being captured by a camera set on the robot from the current viewpoint; The semantic segmentation result acquisition module is configured to acquire semantic segmentation results based on the RGB image, wherein the semantic segmentation results include the object category to which each pixel belongs among multiple pixels contained in the RGB image; The point cloud generation module is configured to generate a three-dimensional point cloud based on the depth map, wherein the three-dimensional point cloud contains the three-dimensional spatial coordinates of each pixel in the plurality of pixels contained in the depth map. The semantic map acquisition module is configured to associate the semantic segmentation result corresponding to each pixel with the three-dimensional spatial coordinates, and project the obtained association result onto the bird's-eye view to obtain a two-dimensional semantic map. The target location acquisition module is configured to acquire the target location of the task-related object based on the semantic map and the name of the task-related object using a large language model; The operable map acquisition module is configured to evaluate the operability of the vicinity of the target location through an operable map prediction model to obtain a local operable map, wherein each of the multiple grids contained in the local operable map corresponds to a value indicating the ease or difficulty of operating the task-related object in that grid. The operation execution module is configured to control the robot to perform operations on the task-related objects based on the local operable map. The robot's movement control device further includes: The list retrieval module is configured to retrieve a preset container list, a preset space list, and a preset prompt. The preset container list contains the names of multiple containers, the preset space list contains the names of multiple spaces, and the preset prompt is used to instruct the large language model to answer a first question about which containers the task-related object may / may not appear near and a second question about which spaces the task-related object may / may not appear in. The pattern learning module is configured to learn the patterns of object placement based on the preset container list, the preset space list, and the preset prompts using the large language model. The target location acquisition module is configured to: acquire the target location of the task-related object based on the semantic map, the name of the task-related object, and the object placement pattern using the large language model; The target location acquisition module is configured as follows: The large language model evaluates the exploration value of each edge point among multiple edge points based on the semantic map, the names of the task-related objects, and the object placement patterns. The exploration value is used to characterize the probability that the corresponding edge point is the target location. The multiple edge points are the points on the boundary between the currently explored area and the currently unexplored area contained in the semantic map. Select the edge point with the highest exploration value from the plurality of edge points as the target location.

6. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the robot's movement operation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the robot movement operation method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot movement operation method as described in any one of claims 1 to 4.

Images