Control method of robot and robot

Abstract

The application discloses a robot control method and robot, and belongs to the technical field of robots. The method comprises the following steps: acquiring an environment image of an environment in which the robot is located; generating structured information of the environment according to the environment image and task information; determining a target action sequence according to the structured information of the environment; and controlling the robot to perform an action included in the target action sequence. The method can improve the accuracy of the determined target action sequence, thereby improving the accuracy of the control of the robot, ensuring the safety of the robot in performing a task, and improving the success rate of the robot in completing the task.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a robot control method and a robot. Background Technology

[0002] With the continuous advancement of robotics technology, there are more and more types of robots, and robots can do more and more things.

[0003] In existing technologies, robot control methods rely solely on images and feature vectors of selectable action sequences to determine the target action sequence. This results in low accuracy of the determined target action sequence, which in turn leads to low accuracy in robot control. Summary of the Invention

[0004] This application provides a robot control method and a robot, which can be used to solve the problems of low accuracy in determining the target action sequence and low accuracy in robot control in related technologies. The technical solution is as follows: On one hand, embodiments of this application provide a robot control method, the method comprising: Obtain an environmental image of the robot's location; Based on the environmental image and task information, structured information about the environment is generated; Based on the structured information of the environment, determine the target action sequence; Control the robot to perform the actions included in the target action sequence.

[0005] In one possible implementation, generating structured information about the environment based on the environment image and task information includes: Based on the environmental image, a three-dimensional occupancy grid of the environment is obtained. The three-dimensional occupancy grid includes multiple voxels, and each voxel corresponds to a voxel information. Based on the voxel information of each voxel and the task information, the structured information of the environment is generated.

[0006] In one possible implementation, obtaining the three-dimensional occupancy mesh of the environment based on the environment image includes: The environmental image is mapped to the bird's-eye view space to obtain a bird's-eye view feature map; The bird's-eye view feature map is transformed to obtain the three-dimensional occupancy grid of the environment.

[0007] In one possible implementation, generating structured information about the environment based on the voxel information of each voxel and the task information includes: The voxel information of each voxel and the task information are input into a visual language model, which is used to perform semantic reasoning on the voxel information and the task information. The result of semantic reasoning using the visual language model is used as the structured information of the environment.

[0008] In one possible implementation, the robot stores multiple optional action sequences, each of which corresponds to a structured information. Determining the target action sequence based on the structured information of the environment includes: Based on the structured information of the environment and the structured information of each optional action sequence, determine the overall similarity between the environment and each optional action sequence; The target action sequence is determined from among the various optional action sequences based on the overall similarity between the environment and the various optional action sequences.

[0009] In one possible implementation, the structured information includes object semantic attributes, object spatial topological relationships, and scene context; The step of determining the comprehensive similarity between the environment and the various optional action sequences based on the structured information of the environment and the structured information of each optional action sequence includes: Based on the object semantic attributes of the environment and the object semantic attributes of each optional action sequence, determine the semantic similarity between the environment and each optional action sequence; Based on the object space topology of the environment and the object space topology of each optional action sequence, determine the spatial structure similarity between the environment and each optional action sequence; Based on the scene context of the environment and the scene context of each optional action sequence, determine the task similarity between the environment and each optional action sequence; Based on the semantic similarity, spatial structure similarity, and task similarity between the environment and each of the optional action sequences, the overall similarity between the environment and each of the optional action sequences is determined.

[0010] In one possible implementation, determining the semantic similarity between the environment and the various optional action sequences based on the object semantic attributes of the environment and the object semantic attributes of each optional action sequence includes: For any one of the optional action sequences, determine a first feature vector to characterize the semantic attributes of the object in that optional action sequence; Determine a second feature vector to characterize the semantic attributes of objects in the environment; Determine the cosine similarity between the first feature vector and the second feature vector; The cosine similarity is used as the semantic similarity between the environment and any of the optional action sequences.

[0011] In one possible implementation, determining the spatial structural similarity between the environment and the various optional action sequences based on the object spatial topological relationships of the environment and the object spatial topological relationships of the various optional action sequences includes: For any of the optional action sequences, determine the graph edit distance between the object space topology of the environment and the object space topology of any of the optional action sequences; The graph edit distance is used as the spatial structural similarity between the environment and any of the optional action sequences.

[0012] In one possible implementation, determining the comprehensive similarity between the environment and the various optional action sequences based on semantic similarity, spatial structure similarity, and task similarity includes: For any of the optional action sequences, the comprehensive similarity between the environment and any optional action sequence is determined based on the semantic similarity and semantic weight parameter between the environment and the optional action sequence, the spatial structure similarity and spatial structure weight parameter between the environment and the optional action sequence, and the task similarity and task weight parameter between the environment and the optional action sequence.

[0013] In one possible implementation, the object semantic attributes are used to describe the object features of a first object in the environment, the first object being the object corresponding to the task, and the object features include at least one of object category, object size, or graspable parts of the object. The object spatial topology is used to indicate the relative positions of various objects in the environment; The scenario context includes at least one of the environment type of the environment or the association between the environment and the task.

[0014] In one possible implementation, the structured information further includes interactive attributes, which include at least one of the material of a first object in the environment, the direction of force applied to the first object, the operation mode of the first object, or the material of a second object, wherein the first object is the object corresponding to the task, and the second object is an obstacle for performing the task.

[0015] In one possible implementation, determining the target action sequence from among the various optional action sequences based on the comprehensive similarity between the environment and the various optional action sequences includes: A reference action sequence is determined from the various optional action sequences, wherein the overall similarity between the reference action sequence and the environment is greater than a similarity threshold; If the reference action sequence is a single sequence, then the reference action sequence is determined to be the target action sequence. If there are multiple reference action sequences, the reference action sequence whose confidence level meets the confidence level requirement is taken as the target action sequence, and the confidence level of the reference action sequence is used to indicate the completion status of the task corresponding to the reference action sequence.

[0016] In one possible implementation, after controlling the robot to execute the actions included in the target action sequence, the method further includes: In response to receiving a feedback signal, the target action sequence is adjusted based on the feedback signal to obtain an adjusted action sequence; Control the robot to execute the actions included in the adjusted action sequence.

[0017] In one possible implementation, after controlling the robot to execute the actions included in the adjusted action sequence, the method further includes: Based on the execution status of the actions included in the adjusted action sequence, the confidence level of the task is determined, and the confidence level is used to indicate whether the task has been completed.

[0018] On the other hand, embodiments of this application provide a robot control device, the device comprising: The acquisition module is used to acquire environmental images of the environment in which the robot is located; A generation module is used to generate structured information about the environment based on the environmental image and task information; The determination module is used to determine the target action sequence based on the structured information of the environment; A control module is used to control the robot to perform the actions included in the target action sequence.

[0019] In one possible implementation, the generation module is configured to obtain a three-dimensional occupancy grid of the environment based on the environment image, the three-dimensional occupancy grid comprising multiple voxels, each voxel corresponding to voxel information; and generate structured information of the environment based on the voxel information of each voxel and the task information.

[0020] In one possible implementation, the generation module is used to map the environment image to a bird's-eye view space to obtain a bird's-eye view feature map; and to transform the bird's-eye view feature map to obtain a three-dimensional occupancy grid of the environment.

[0021] In one possible implementation, the generation module is used to input the voxel information of each voxel and the task information into a visual language model, the visual language model being used to perform semantic reasoning on the voxel information and the task information; and the result obtained by the semantic reasoning of the visual language model is used as the structured information of the environment.

[0022] In one possible implementation, the robot stores multiple optional action sequences, each of which corresponds to a structured information. The determining module is configured to determine the comprehensive similarity between the environment and each optional action sequence based on the structured information of the environment and the structured information of each optional action sequence; and to determine the target action sequence among the optional action sequences based on the comprehensive similarity between the environment and each optional action sequence.

[0023] In one possible implementation, the structured information includes object semantic attributes, object spatial topological relationships, and scene context; The determining module is configured to: determine the semantic similarity between the environment and each optional action sequence based on the object semantic attributes of the environment and the object semantic attributes of each optional action sequence; determine the spatial structure similarity between the environment and each optional action sequence based on the object spatial topology of the environment and the object spatial topology of each optional action sequence; determine the task similarity between the environment and each optional action sequence based on the scene context of the environment and the scene context of each optional action sequence; and determine the comprehensive similarity between the environment and each optional action sequence based on the semantic similarity, spatial structure similarity, and task similarity.

[0024] In one possible implementation, the determining module is configured to, for any one of the optional action sequences, determine a first feature vector representing the semantic attributes of the object in the optional action sequence; determine a second feature vector representing the semantic attributes of the object in the environment; determine the cosine similarity between the first feature vector and the second feature vector; and use the cosine similarity as the semantic similarity between the environment and the optional action sequence.

[0025] In one possible implementation, the determining module is configured to, for any one of the optional action sequences, determine the graph edit distance between the object space topology of the environment and the object space topology of the optional action sequence; and use the graph edit distance as the spatial structural similarity between the environment and the optional action sequence.

[0026] In one possible implementation, the determining module is configured to, for any optional action sequence among the various optional action sequences, determine the comprehensive similarity between the environment and any optional action sequence based on the semantic similarity and semantic weight parameter between the environment and the optional action sequence, the spatial structure similarity and spatial structure weight parameter between the environment and the optional action sequence, and the task similarity and task weight parameter between the environment and the optional action sequence.

[0027] In one possible implementation, the object semantic attributes are used to describe the object features of a first object in the environment, the first object being the object corresponding to the task, and the object features include at least one of object category, object size, or graspable parts of the object. The object spatial topology is used to indicate the relative positions of various objects in the environment; The scenario context includes at least one of the environment type of the environment or the association between the environment and the task.

[0028] In one possible implementation, the structured information further includes interactive attributes, which include at least one of the material of a first object in the environment, the direction of force applied to the first object, the operation mode of the first object, or the material of a second object, wherein the first object is the object corresponding to the task, and the second object is an obstacle for performing the task.

[0029] In one possible implementation, the determining module is configured to determine a reference action sequence among the various optional action sequences, wherein the overall similarity between the reference action sequence and the environment is greater than a similarity threshold; if there is one reference action sequence, the reference action sequence is determined as the target action sequence; if there are multiple reference action sequences, the reference action sequence whose confidence level meets the confidence requirement among the multiple reference action sequences is taken as the target action sequence, and the confidence level of the reference action sequence is used to indicate the completion status of the task corresponding to the reference action sequence.

[0030] In one possible implementation, the device further includes: An adjustment module is used to adjust the target action sequence based on the received feedback signal in response to the feedback signal, so as to obtain an adjusted action sequence. The control module is also used to control the robot to perform the actions included in the adjusted action sequence.

[0031] In one possible implementation, the determining module is further configured to determine the confidence level of the task based on the execution status of the actions included in the adjusted action sequence, wherein the confidence level is used to indicate whether the task has been completed.

[0032] On the other hand, embodiments of this application provide a robot, the robot including a controller, the controller including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement any of the robot control methods described above.

[0033] On the other hand, a computer-readable storage medium is also provided, wherein at least one piece of program code is stored in the computer-readable storage medium, the at least one piece of program code being loaded and executed by a processor to enable a computer device to implement any of the robot control methods described above.

[0034] On the other hand, a computer program or computer program product is also provided, wherein the computer program or computer program product stores at least one computer instruction, which is loaded and executed by a processor to enable a computer device to implement any of the above-described robot control methods.

[0035] The technical solution provided in this application has at least the following beneficial effects: The technical solution provided in this application combines environmental images and task information to generate structured information, and then determines the target action sequence based on the structured information. Compared with the method of determining the target action sequence solely based on environmental images, this method can improve the accuracy of the determined target action sequence. The target action sequence is used to control the robot to execute the actions included in the target action sequence to complete the task. Therefore, it can improve the accuracy of robot control, ensure the safety of robot task execution, and thus increase the success rate of robot task completion. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the implementation environment of a robot control method provided in an embodiment of this application; Figure 2This is a flowchart of a robot control method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the determination of structured information about an environment, as provided in an embodiment of this application. Figure 4 This is a flowchart illustrating the determination of a target action sequence provided in an embodiment of this application; Figure 5 This is an architecture diagram of a robot control system provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a robot control device provided in an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0039] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0040] With the continuous advancement of robotics technology, there are more and more types of robots, and robots can do more and more things.

[0041] In related technologies, robots store multiple optional action sequences. In response to receiving a task, the robot first acquires an image of its environment; determines a first feature vector corresponding to the image; determines a second feature vector corresponding to each optional action sequence; based on the first and second feature vectors, determines a target action sequence from among the optional action sequences; and controls the robot to execute the actions included in the target action sequence so that the robot completes the task.

[0042] However, the above method relies solely on the feature vectors of images and optional action sequences to determine the target action sequence, resulting in low accuracy of the determined target action sequence and consequently, low accuracy of robot control.

[0043] Figure 1 This is a schematic diagram of the implementation environment of a robot control method provided in an embodiment of this application, such as... Figure 1As shown, the implementation environment includes a robot 101, which includes a controller 102. The controller 102 is used to execute the robot control method provided in the embodiments of this application.

[0044] Robot 101 can be a multi-degree-of-freedom legged robot, such as a bipedal robot, quadrupedal robot, hexapod robot, etc. This application embodiment does not limit the robot 101. For example, the robot 101 in this application embodiment is a robot with a height of 173 cm, a weight of 60 kg, 51 degrees of freedom throughout its body, and dexterous hands with 10 active degrees of freedom each, featuring visual and tactile sensing.

[0045] Optionally, the controller 102 acquires an environmental image of the environment in which the robot 101 is located, and generates structured information about the environment based on the environmental image and task information. Based on the structured information of the environment, it determines a target action sequence; and then controls the robot 101 to execute the actions included in the target action sequence so that the robot 101 completes the task.

[0046] This application provides a robot control method, which can be applied to the above-mentioned... Figure 1 The implementation environment shown is as follows: Figure 2 The flowchart shown in this embodiment of the present application illustrates a robot control method. This method can be implemented by... Figure 1 Controller 102 in the middle executes, such as Figure 2 As shown, the method includes the following steps 201 to 204.

[0047] In step 201, an environmental image of the robot's environment is acquired.

[0048] In the exemplary embodiment of this application, the robot is a robot with a height of 173 cm, a weight of 60 kg, 51 degrees of freedom throughout its body, and dexterous hands, each with 10 active degrees of freedom and equipped with a visual sensor and a tactile sensor. Of course, the robot involved in the embodiments of this application can also be other types of robots, and the embodiments of this application do not limit this.

[0049] In one possible implementation, the process of acquiring an environmental image of the robot's environment includes: in response to receiving a task, acquiring an environmental image of the robot's environment.

[0050] Optionally, the robot includes a microphone array for collecting voice data. The microphone array communicates with the controller via a wired or wireless network. After collecting the voice data, the microphone array sends the collected voice data to the controller. The controller converts the voice data into text, and if the text corresponds to a task, the controller receives the task.

[0051] The process of acquiring environmental images of the robot's environment includes: acquiring images captured by image acquisition devices installed on various parts of the robot; and fusing features of each image based on the pose parameters of the image acquisition devices to obtain an environmental image.

[0052] Image acquisition devices are installed in various parts of the robot. The images acquired by these devices can be two-dimensional or three-dimensional, and this application embodiment does not limit this. The various parts of the robot can be the head or the torso, and this application embodiment does not limit this either. When the controller receives a task, it sends an image acquisition request to the image acquisition devices. The image acquisition devices then acquire images based on the request. After acquiring the images, the image acquisition devices send them to the controller, enabling the controller to obtain the images acquired by the image acquisition devices installed in various parts of the robot.

[0053] Optionally, when sending images to the controller, the image acquisition device also sends its pose parameters to the controller. The pose parameters of the image acquisition device include internal pose parameters (intrinsic parameters) and external pose parameters (extrinsic parameters). The internal pose parameters include the lens focal length, image principal point coordinates, distortion coefficients, and pixel scale. The external pose parameters include the translation and rotation parameters of the image acquisition device relative to the robot.

[0054] By acquiring images from multiple image acquisition devices installed on the robot and combining them with corresponding pose parameters to perform feature fusion, an environmental image is obtained. The robot can achieve multi-view image spatial alignment based on the precise pose of each image acquisition device, eliminating the limitations of the field of view of a single image acquisition device and the information loss caused by occlusion. It fully integrates visual features from different directions to generate an environmental image with a wider coverage, more complete object details, and stronger spatial consistency. This provides high-quality visual input for subsequent construction of high-precision 3D occupancy meshes and extraction of accurate scene voxel information, effectively improving the robot's perception and recognition accuracy of surrounding environmental objects and obstacles.

[0055] In step 202, structured information about the environment is generated based on the environmental image and task information.

[0056] In one possible implementation, the process of generating structured information about the environment based on the environment image and task information includes: obtaining a three-dimensional occupancy grid of the environment based on the environment image, the three-dimensional occupancy grid including multiple voxels, each voxel corresponding to voxel information; generating structured information about the environment based on the voxel information of each voxel and task information, the structured information including information obtained by semantic reasoning of the environment and task.

[0057] The voxel information includes at least one of occupancy probability or category. The occupancy probability of any voxel indicates whether the voxel is occupied, and the category of any voxel indicates the category of the object occupying the voxel. For example, if the occupancy probability of a voxel is 1 and the voxel's category is a supermarket shopping cart, then it can be determined that the voxel is occupied by a supermarket shopping cart. As another example, if the occupancy probability of a voxel is 0 and the voxel's category is empty, then it can be determined that the voxel is not occupied.

[0058] This method first generates a 3D occupancy mesh containing voxel information of each voxel based on the environmental image. Then, it combines the task information to generate structured information of the environment, which can completely restore the occupancy, position, and semantic details of objects in the 3D space of the environment. At the same time, it filters and integrates effective environmental data based on task requirements, so that the generated structured information has both accurate 3D spatial representation capabilities and semantic logic that fits the task execution requirements. It can avoid the defects of 2D images losing depth information and environmental description being disconnected from the task, and improve the accuracy and scene adaptability of subsequent target action sequence selection based on structured information.

[0059] In one possible implementation, the process of obtaining the three-dimensional occupancy grid of the environment based on the environment image includes: mapping the environment image to a bird's-eye view space to obtain a bird's-eye view feature map; and transforming the bird's-eye view feature map to obtain the three-dimensional occupancy grid of the environment.

[0060] Optionally, the environmental image is mapped to the bird's-eye view space using bird's-eye view (BEV) technology to obtain a bird's-eye view feature map. The bird's-eye view feature map is then transformed using occupancy network (OCC) technology to obtain a three-dimensional occupancy grid of the environment.

[0061] This method uses bird's-eye view technology to uniformly map environmental images to a top-down space to generate a bird's-eye view feature map, eliminating camera perspective distortion and unifying the spatial coordinates of each object. Then, it uses occupancy network technology to perform 3D spatial inference on the bird's-eye view feature map and outputs a 3D occupancy grid. This can efficiently restore the 3D spatial distribution of the environment and accurately output the occupancy probability and category of each voxel. It makes up for the shortcomings of 2D images, such as lack of depth and inability to represent 3D space. This provides an accurate and complete 3D perception foundation for subsequent generation of structured information including object semantic attributes, object spatial topological relationships, and scene context by combining voxel information and task information, thereby improving the accuracy of the robot's spatial perception of the environment.

[0062] Optionally, task information refers to information that can describe the task, and this application embodiment does not limit this. For example, if the task is that a customer requests the robot to help take a bottle of a specific brand of beverage from the shelf, then the task information is that the customer requests the robot to help take a bottle of a specific brand of beverage from the shelf.

[0063] In one possible implementation, the process of generating structured information about the environment based on the voxel information and task information of each voxel includes: inputting the voxel information and task information of each voxel into a Vision-Language Model (VLM), which is used to perform semantic reasoning on the voxel information and task information of each voxel; and using the result of semantic reasoning by the Vision-Language Model as the structured information of the environment.

[0064] By inputting voxel information and task information into a visual language model for joint semantic reasoning, it is possible to integrate three-dimensional spatial geometric information with task semantic requirements. This overcomes the limitations of a simple three-dimensional occupancy mesh, which can only represent spatial occupancy and lacks task-oriented semantic understanding. Relying on the multimodal understanding capabilities of the visual language model, it analyzes the attributes of objects in the scene, the spatial topological relationships between objects, and the scene context adapted to the current task, generating task-specific structured information. This enables the robot to distinguish between task targets, obstacles, and irrelevant objects, providing accurate and task-compliant environmental semantic basis for subsequent selection of action sequences that are suitable for the task. This significantly improves the rationality and environmental adaptability of robot action planning.

[0065] Optionally, the structured information includes information obtained through semantic reasoning of the environment and task. The structured information includes at least one of object semantic attributes, object spatial topological relationships, or scene context. Specifically, object semantic attributes describe the object features of a first object within the environment, where the first object is the object corresponding to the task. Object features include at least one of object category, object size, or graspable parts of the object. Object spatial topological relationships indicate the relative positions of various objects in the environment. Scene context includes at least one of the environment type or the association between the environment and the task.

[0066] Optionally, the structured information also includes interactive attributes, which include at least one of the following: the material of a first object in the environment, the direction of force applied to the first object, the operation mode of the first object, or the material of a second object. The first object is the object corresponding to the task, and the second object is an obstacle for performing the task.

[0067] For example, the task is for a customer to request the robot to retrieve a bottle of a specific brand of beverage from the shelf. Based on the voxel information of each voxel and the task information, the structured information of the generated environment is as follows: Object semantic attributes: The third shelf is identified as containing the object corresponding to the task. The object is a beverage of a specified brand. The beverage has a cylindrical bottle body that can be grasped, with the neck of the bottle being the graspable part. The beverage is in a plastic bottle and is approximately 12 centimeters high.

[0068] Object spatial topology: The shelf is located 2.5 meters directly in front of the robot. The designated brand of beverage is located on the left side of the third shelf. A shopping cart is blocking part of the aisle in front of the designated brand of beverage. There is a pedestrian standing about 0.8 meters to the left of the shopping cart, with a baby stroller following behind the pedestrian.

[0069] Interaction Attributes: The beverage of the specified brand is made of plastic, which is relatively soft and has a smooth surface. The object is grasped by applying moderate force, and the force direction is radial clamping force or vertical support force. The shopping cart is made of metal, and the baby stroller is made of lightweight and movable material. Both the shopping cart and the baby stroller are moving obstacles that need to be avoided.

[0070] Context: The environment is a supermarket beverage shelf area, and the relationship between the environment and the task is "related". The task type is product retrieval and placement. The first object identified is a beverage of a specified brand, and the second objects are a shopping cart, a baby stroller, and a pedestrian.

[0071] For example, the task is to move newly arrived beverages, snacks, and daily necessities from the warehouse area to the corresponding shelf areas. Based on the voxel information of each voxel and the task information, the structured information of the generated environment is as follows: Object semantic attributes: The objects corresponding to the task were identified as canned carbonated beverages, bagged potato chips, and bottled shampoo. A case of canned carbonated beverages contains 12 cans, each cylindrical, with a volume of 330 ml. The gripping part is the middle of the can. The bagged potato chips have flexible packaging, are rectangular, and measure 236 mm long, 160 mm wide, and 57 mm thick. The gripping part is the middle of the bag. The bottled shampoo comes in a smooth, cylindrical plastic bottle, measuring 15 cm in height. The gripping part is the neck of the bottle. The cans need to be protected from crushing, the bagged potato chips need to be handled gently to avoid breakage, and the shampoo needs to be gripped with appropriate force to prevent slippage.

[0072] The spatial topology of the objects is as follows: the goods to be put on the shelves are located in the storage area, the beverage shelves are located in the east area, the snack shelves are located in the central area, the daily chemical shelves are located in the west area, and a cleaning robot is performing wet mopping operations along the planned path in the aisle where the robot is located. Two AGV (Automated Guided Vehicle) vehicles are located in the central area shelf aisle and the west area shelf aisle respectively. One AGV vehicle is transporting replenishment materials from the storage area to the central area.

[0073] Interactive attributes: The goods box is made of paper with a certain degree of friction on the surface, making it suitable for handling by grasping the sides of the box. The AGV and cleaning robot are moving obstacles and are both made of metal. The cleaning robot moves back and forth along the channel.

[0074] Scenario Context: The environment is an unmanned supermarket during the early morning restocking period, and the relationship between the environment and the task is "related". The task type is autonomous sorting and shelving of multiple product categories. The first objects are canned carbonated drinks, bagged potato chips, and bottled shampoo, and the second objects are AGV carts and cleaning robots.

[0075] like Figure 3 This is a flowchart illustrating the determination of structured information about an environment, as provided in an embodiment of this application. Figure 3 As shown, the process includes: acquiring images from image acquisition devices installed on various parts of the robot; performing feature fusion on each image based on the pose parameters of the image acquisition devices to obtain an environmental image; mapping the environmental image to a bird's-eye view space using BEV technology to obtain a bird's-eye view feature map; converting the bird's-eye view feature map using OCC technology to obtain a three-dimensional occupancy mesh of the environment, the three-dimensional occupancy mesh including multiple voxels, one voxel corresponding to one voxel information; inputting the voxel information of each voxel and task information into the VLM; and using the result of semantic reasoning from the VLM as the structured information of the environment. The details of each step have been described above, and will not be repeated here in this embodiment.

[0076] In step 203, the target action sequence is determined based on the structured information of the environment.

[0077] In one possible implementation, the robot stores multiple optional action sequences, each of which corresponds to a structured information. The process of determining the target action sequence based on the structured information of the environment includes: determining the target action sequence from each optional action sequence based on the structured information of the environment and the structured information of each optional action sequence.

[0078] This application does not limit the method for determining the target action sequence among the various optional action sequences. Optionally, the process of determining the target action sequence among the various optional action sequences based on the structured information of the environment and the structured information of each optional action sequence includes: determining the comprehensive similarity between the environment and each optional action sequence based on the structured information of the environment and the structured information of each optional action sequence; and determining the target action sequence among the various optional action sequences based on the comprehensive similarity between the environment and each optional action sequence.

[0079] By comparing the structured information of the environment with the structured information of each optional action sequence, the comprehensive similarity between the two can be calculated. This allows for the quantification of the adaptability between the environment and each optional action sequence from multiple dimensions, such as object spatial topology, object semantic attributes, and scene context. This avoids planning bias caused by relying on a single dimension for judgment. The solution with the highest matching degree is selected as the target action sequence based on the comprehensive similarity, ensuring that the selected action sequence fits the current task and environment. This improves the rationality of action planning and scene adaptability, reduces robot obstacle avoidance errors and operational lag, and ensures the smooth and efficient completion of the task.

[0080] In one possible implementation, the structured information includes object semantic attributes, object spatial topological relationships, and scene context. The process of determining the comprehensive similarity between the environment and each optional action sequence based on the structured information of the environment and each optional action sequence includes: determining the semantic similarity between the environment and each optional action sequence based on the object semantic attributes of the environment and the object semantic attributes of each optional action sequence; determining the spatial structural similarity between the environment and each optional action sequence based on the object spatial topological relationships of the environment and the object spatial topological relationships of each optional action sequence; determining the task similarity between the environment and each optional action sequence based on the scene context of the environment and the scene context of each optional action sequence; and determining the comprehensive similarity between the environment and each optional action sequence based on the semantic similarity, spatial structural similarity, and task similarity.

[0081] In this implementation, semantic similarity, spatial structure similarity, and task similarity are calculated from three dimensions: object semantic attributes, object spatial topological relationships, and scene context. The three types of similarity are then fused to obtain a comprehensive similarity, which enables a comprehensive and hierarchical quantitative evaluation of scene matching. It can identify whether the interaction target matches, verify whether the relative positions and spatial arrangements between objects are consistent, and combine the task to judge the degree of fit between the action sequence and the task. This avoids the problem of one-sided and distorted matching caused by single-dimensional comparison, so that the comprehensive similarity can completely and objectively reflect the degree of fit between the environment and each candidate action sequence. This allows the target action sequence to be selected, ensuring that the robot's operation actions conform to the requirements of the environment and the task, and reducing the risk of collision and operation failure.

[0082] In one possible implementation, the process of determining the semantic similarity between the environment and each optional action sequence based on the object semantic attributes of the environment and the object semantic attributes of each optional action sequence includes: for any optional action sequence in each optional action sequence, determining a first feature vector to characterize the object semantic attributes of any optional action sequence; determining a second feature vector to characterize the object semantic attributes of the environment; and determining the semantic similarity between the environment and any optional action sequence based on the first feature vector and the second feature vector.

[0083] Optionally, the process of determining the semantic similarity between the environment and any optional action sequence based on the first feature vector and the second feature vector includes: determining the cosine similarity between the first feature vector and the second feature vector; and using the cosine similarity as the semantic similarity between the environment and any optional action sequence.

[0084] In this implementation, by obtaining the first feature vector representing the semantic attributes of the object representing the optional action sequence and the second feature vector representing the semantic attributes of the object representing the environment, and by using cosine similarity to quantify the two sets of vectors, semantic similarity is obtained. This can transform the chaotic and discrete semantic attributes of objects into numerical vectors of a unified dimension, achieving standardized and quantifiable semantic comparison. Relying on cosine similarity, the matching degree between the environment and the optional action sequence is accurately measured, avoiding the defect that textual semantic information is difficult to directly quantify and compare. This makes the calculation results of semantic similarity more objective and provides an accurate and reliable basis for semantic dimension quantification for subsequent fusion of other similarity to solve the comprehensive similarity.

[0085] The process of determining a first feature vector for representing the semantic attributes of an object in any optional action sequence is similar to the process of determining a second feature vector for representing the semantic attributes of an object in the environment. This application embodiment only illustrates the process of determining the first feature vector for representing the semantic attributes of an object in any optional action sequence.

[0086] Optionally, the controller includes a feature vector determination model. This model can be any type capable of determining feature vectors, and this embodiment does not limit its scope. The dimension of the feature vectors determined by the feature vector determination model can be arbitrary, and this embodiment also does not limit its scope. For example, the feature vectors determined by the feature vector determination model have a dimension of 256. Another example is that the feature vectors determined by the feature vector determination model have a dimension of 512.

[0087] Optionally, the process of determining the first feature vector for characterizing the semantic attributes of an object in any optional action sequence includes: inputting the semantic attributes of the object in any optional action sequence into the feature vector determination model; and using the output of the feature vector determination module as the first feature vector.

[0088] In one possible implementation, the process of determining the spatial structural similarity between the environment and each optional action sequence based on the object spatial topology of the environment and the object spatial topology of each optional action sequence includes: for any optional action sequence in each optional action sequence, determining the graph edit distance between the object spatial topology of the environment and the object spatial topology of any optional action sequence; and using the graph edit distance as the spatial structural similarity between the environment and any optional action sequence.

[0089] Optionally, the process of determining the graph edit distance between the object spatial topological relationship of the environment and the object spatial topological relationship of any optional action sequence includes: generating a first image based on the object spatial topological relationship of the environment; generating a second image based on the object spatial topological relationship of any optional action sequence; determining the graph edit distance between the first image and the second image; and using the graph edit distance between the first image and the second image as the graph edit distance between the object spatial topological relationship of the environment and the object spatial topological relationship of any optional action sequence.

[0090] In this context, the object space topology of the first image is the object space topology of the environment. The object space topology of the second image is the object space topology of any selectable action sequence.

[0091] The graph edit distance between the first image and the second image is the shortest number of operations required to transform the first image into the same image as the second image. Alternatively, the graph edit distance between the first image and the second image is the shortest number of operations required to transform the second image into the same image as the first image.

[0092] By calculating the graph edit distance between the spatial topological relationships of objects in the environment and the spatial topological relationships of objects in the selectable action sequence, the spatial structural similarity between the two can be characterized. This method can uniformly quantify the overall difference in the spatial topological relationships of two objects. Based on the graph edit distance, it can fully cover various topological changes such as the addition or removal of objects, changes in object attributes, and changes in the spatial connection relationships between objects. It accurately reflects the degree of matching between the inherent spatial layout of the environment and the spatial layout of the selectable action sequence. Moreover, this quantification method has a unified and calculable numerical standard, which facilitates the rapid selection of target action sequences directly based on the distance value.

[0093] Optionally, the process of determining the task similarity between the environment and each optional action sequence based on the scene context of the environment and the scene context of each optional action sequence includes: for any optional action sequence in each optional action sequence, determining a third feature vector to represent the scene context of any optional action sequence; determining a fourth feature vector to represent the scene context of the environment; determining the cosine similarity between the third feature vector and the fourth feature vector; and using the cosine similarity between the third feature vector and the fourth feature vector as the task similarity between the environment and any optional action sequence.

[0094] The process of determining the third feature vector of the scene context used to characterize any optional action sequence and the process of determining the fourth feature vector of the scene context used to characterize the environment are similar to the process of determining the first feature vector of the semantic attributes of the object used to characterize any optional action sequence, and will not be described again in this embodiment.

[0095] In one possible implementation, the process of determining the comprehensive similarity between the environment and each optional action sequence based on the semantic similarity, spatial structure similarity, and task similarity between the environment and each optional action sequence includes: for any optional action sequence in each optional action sequence, determining the comprehensive similarity between the environment and any optional action sequence based on the semantic similarity, semantic weight parameter, spatial structure similarity, spatial structure weight parameter, task similarity, and task weight parameter between the environment and any optional action sequence.

[0096] The semantic weight parameter, spatial structure weight parameter, and task weight parameter are all set based on experience, or can be flexibly adjusted according to the implementation environment; this embodiment does not limit this. Optionally, the sum of the semantic weight parameter, spatial structure weight parameter, and task weight parameter is a target value. The target value is set based on experience, or can be flexibly adjusted according to the implementation environment; this embodiment does not limit this either. For example, the target value is 1, the semantic weight parameter is 0.3, the spatial structure weight parameter is 0.4, and the task weight parameter is 0.3.

[0097] By configuring the weight parameters for the three types of similarity—semantic similarity, spatial structure similarity, and task similarity—and then calculating the comprehensive similarity using weighted averages, the proportion of each dimension in the matching process can be flexibly adjusted. This takes into account the three matching features of object semantics, spatial topology, and task objectives, and can adapt to the differences in the importance of each dimension under different scenarios, resulting in an overall matching result that fits the actual task requirements and improving the rationality and adaptability of action sequence selection.

[0098] In one possible implementation, the process of determining the target action sequence from among the various optional action sequences based on the comprehensive similarity between the environment and each optional action sequence includes: determining a reference action sequence from among the various optional action sequences, wherein the comprehensive similarity between the reference action sequence and the environment is greater than a similarity threshold; determining the reference action sequence as the target action sequence based on the existence of one reference action sequence; and determining the reference action sequence whose confidence level meets the confidence requirement as the target action sequence based on the existence of multiple reference action sequences, wherein the confidence level of the reference action sequence is used to indicate the completion status of the task corresponding to the reference action sequence.

[0099] The similarity threshold is set based on experience, or it can be flexibly adjusted according to the implementation environment. This application embodiment does not limit this. For example, the similarity threshold is 70%.

[0100] A confidence level that meets the confidence level requirement means that the confidence level is at its highest.

[0101] If there is only one reference action sequence that meets the confidence requirement, then that reference action sequence is taken as the target action sequence.

[0102] If there are multiple reference action sequences that meet the confidence requirements, then the reference action sequence with the highest overall similarity among these multiple reference action sequences that meet the confidence requirements will be taken as the target action sequence.

[0103] This method first uses a similarity threshold to initially screen reference action sequences that meet the overall similarity standard. When there is a single reference action sequence, it can be directly identified as the target action sequence. When there are multiple reference action sequences, a second screening is conducted based on the confidence level representing the task completion status. The hierarchical screening logic is clear, which can quickly filter out invalid action sequences with insufficient similarity and accurately select the target action sequence with the best task completion status from multiple reference action sequences, thereby improving the accuracy and rationality of the selection of target action sequences.

[0104] like Figure 4 This is a flowchart illustrating the determination of a target action sequence provided in an embodiment of this application. For example... Figure 4As shown, the process includes: determining the comprehensive similarity between the environment and each optional action sequence; determining a reference action sequence from the optional action sequences based on the comprehensive similarity between the environment and each optional action sequence; if there is only one reference action sequence, using that reference action sequence as the target action sequence; if there are multiple reference action sequences, using the reference action sequence whose confidence level meets the confidence requirement as the target action sequence. The comprehensive similarity between the environment and the optional action sequences is determined based on semantic similarity, spatial structure similarity, and task similarity. The details of each step have been described above and will not be repeated here.

[0105] In step 204, the robot is controlled to perform the actions included in the target action sequence.

[0106] Optionally, the target action sequence includes at least one of a bipedal locomotion sequence, a bi-arm coordinated action sequence, or a dexterous hand grasping action sequence. The actions included in the target action sequence are at least one of the actions included in the bipedal locomotion sequence, the actions included in the bi-arm coordinated action sequence, or the actions included in the dexterous hand grasping action sequence. The robot is controlled to execute the actions included in the target action sequence so that the robot completes the task.

[0107] In one possible implementation, the target action sequence includes a bipedal locomotion sequence, a bi-arm cooperative action sequence, and a dexterous hand grasping action sequence. The process of controlling the robot to execute the actions included in the target action sequence includes: sending the bipedal locomotion sequence to the bipedal controller, causing the bipedal controller to control the robot's bipedal locomotion to execute the actions included in the bipedal locomotion sequence; sending the bi-arm cooperative control sequence to the bi-arm controller, causing the bi-arm controller to control the robot's bi-arms to execute the actions included in the bi-arm cooperative action sequence; and sending the dexterous hand grasping action sequence to the dexterous hand controller, causing the dexterous hand controller to control the robot's hands to execute the actions included in the dexterous hand grasping action sequence.

[0108] For example, in a task where a customer requests the robot to help retrieve a bottle of a specific brand of beverage from the shelf, the target motion sequence includes a bipedal movement sequence, a bi-arm coordinated movement sequence, and a dexterous hand grasping sequence. The bipedal movement sequence involves the robot starting from its current position, first moving approximately 0.8 meters to the left to avoid a shopping cart parked directly in front, then moving forward to a position 0.5 meters in front of the shelf, avoiding the shopping cart, stroller, and the area where the customer is standing throughout, maintaining a safe distance of at least 0.6 meters. The bi-arm coordinated movement sequence involves, upon reaching the target position, raising the right arm to the third shelf height, with fingers spread in preparation for grasping. The dexterous hand grasping sequence involves adopting a three-finger grasping posture, with an expected gripping force of 2.5 Newtons (N).

[0109] For example, in the task of moving newly arrived beverages, snacks, and daily necessities from the warehouse area to the corresponding shelf areas, the target action sequence includes a bipedal movement sequence, a bi-arm coordinated movement sequence, and a dexterous hand grasping sequence. The dexterous hand grasping sequence includes the following actions: using a wide-finger contact method with uniform force, maintaining a stable gripping force of 1.8N, to move a canned beverage from its location to the AGV cart; once the AGV cart reaches the shelf corresponding to the canned beverage, the canned beverage is placed on the designated shelf. Using a surface contact grasping method with a gripping force controlled at 1.2N, a bag of potato chips is moved from its location to the AGV cart; once the AGV cart reaches the shelf corresponding to the bag of potato chips, the bag of potato chips is placed on the designated shelf. Using a three-finger grasping method with a gripping force controlled at 3.0N, shampoo is moved from its location to the AGV cart; once the AGV cart reaches the shelf corresponding to the shampoo, the shampoo is placed on the designated shelf. The coordinated arm movement sequence includes the following actions: Upon reaching the shelf corresponding to canned beverages, lower both arms to the target height, spread fingers to prepare to grab the can, and then raise the right arm to the height of the shelf where canned beverages are placed, extending it to place the can into the shelf. Upon reaching the shelf corresponding to bagged potato chips, lower the right arm to the target height, spread fingers to prepare to grab the bagged chips, and then raise the right arm to the height of the shelf where bagged potato chips are placed, extending it to place the bagged potato chips into the shelf. Upon reaching the shelf corresponding to shampoo, lower the right arm to the target height, spread fingers to prepare to grab the shampoo, and then raise the right arm to the height of the shelf where shampoo is placed, extending it to place the shampoo into the shelf. The bipedal movement sequence includes the following actions: From the robot's current position, walk forward for 5 seconds to the central shelf, where you will stock bagged potato chips. Then, pause for 2 seconds at the central intersection to wait for the AGV to pass before resuming movement. Walk 50 meters to the east shelf, where you will stock canned beverages. Next, move from the east shelf to the west shelf, where you will stock shampoo.

[0110] In one possible implementation, after the robot executes the actions included in the target action sequence, in response to receiving a feedback signal, the target action sequence is adjusted based on the feedback signal to obtain an adjusted action sequence; the robot is then controlled to execute the actions included in the adjusted action sequence.

[0111] The feedback signal can be a signal sent by the sensing module, a signal sent by the dexterous hand vision module, or a signal sent by the dexterous hand tactile module; this application embodiment does not limit this.

[0112] This method dynamically adjusts the target action sequence based on real-time feedback signals and controls the robot to execute the actions included in the adjusted action sequence. It can promptly correct action deviations according to the actual situation in the environment, adapt to changes in the environment, avoid mechanically running rigidly according to the original action sequence, and effectively improve the stability of robot action execution and the success rate of task completion.

[0113] Optionally, when the perception module detects a change in an object in the environment, it sends a feedback signal to the controller. The feedback signal indicates that the object in the environment has changed. Based on the feedback signal, the controller adjusts the bipedal movement sequence in the target movement sequence to obtain the adjusted bipedal movement sequence, and controls the robot to execute the movements included in the adjusted bipedal movement sequence.

[0114] For example, if the perception module detects that a customer pushing a stroller, originally located in the left aisle of the robot, has moved 0.5 meters toward the robot, the perception module sends a feedback signal to the controller. The controller then adjusts the bipedal movement sequence in the target action sequence. The adjusted bipedal movement sequence changes the radius of movement from 0.8 meters to 1.2 meters, thus ensuring a safe distance of at least 0.8 meters between the robot and the stroller, preventing the robot from colliding with the stroller.

[0115] For example, if the perception module detects an AGV and there is a 0.5-meter gap on the left side of the AGV for pedestrians to pass through, the perception module sends a feedback signal to the controller. The controller then adjusts the bipedal movement sequence in the target action sequence. In the adjusted bipedal movement sequence, the robot adopts a sideways posture so that it can pass through the AGV in a sideways posture.

[0116] Optionally, when the dexterous hand vision module detects a deviation between the actual placement angle of the first object and the initial estimated angle, it sends a feedback signal to the controller. The feedback signal indicates the deviation between the actual placement angle of the first object and the initial estimated angle. Based on the feedback signal, the controller adjusts the dual-arm cooperative motion sequence in the target motion sequence to obtain the adjusted dual-arm cooperative motion sequence, and controls the robot to execute the actions included in the adjusted dual-arm cooperative motion sequence.

[0117] The initial estimated angle is the placement angle of the objects included in the task corresponding to the target action sequence. Alternatively, the initial estimated angle can be set based on experience, or it can be flexibly adjusted according to the implementation environment; this application embodiment does not limit this.

[0118] For example, if the dexterous hand vision module detects that the actual placement angle of the first object on the shelf deviates by 5 degrees from the initial estimated angle, the dexterous hand vision module sends a feedback signal to the controller. The controller adjusts the dual-arm collaborative action sequence in the target action sequence. When the robot executes the actions included in the adjusted dual-arm collaborative action sequence, the robot's hand can be aligned with the graspable area of ​​the first object so that the robot can grasp the first object.

[0119] Optionally, when the dexterous hand touch module detects a deviation between the surface smoothness of the first object and the initial estimated smoothness, it sends a feedback signal to the controller. The feedback signal indicates the deviation between the surface smoothness of the first object and the initial estimated smoothness. Based on the feedback signal, the controller adjusts the dexterous hand grasping action sequence in the target action sequence to obtain the adjusted dexterous hand grasping action sequence, and controls the robot to execute the actions included in the adjusted dexterous hand grasping action sequence.

[0120] The initial estimated smoothness is the surface smoothness of the objects included in the task corresponding to the target action sequence. Alternatively, the initial estimated smoothness can be set based on experience, or it can be flexibly adjusted according to the implementation environment; this embodiment does not limit this.

[0121] For example, if the dexterous hand tactile module detects that the surface smoothness of the first object is smoother than the initial estimated smoothness, the dexterous hand vision module sends a feedback signal to the controller. The controller then adjusts the dexterous hand grasping action sequence in the target action sequence so that the adjusted dexterous hand grasping action sequence includes actions with greater clamping force, in order to grasp the first object.

[0122] In one possible implementation, after the robot executes the actions included in the adjusted action sequence, the confidence level of the task can be determined based on the execution status of the actions included in the adjusted action sequence. The confidence level of the task is used to indicate whether the task has been completed.

[0123] Optionally, based on the completion of actions included in the adjusted action sequence, the confidence level of the task is determined to be 1, where a confidence level of 1 indicates task completion. Conversely, based on the incomplete execution of actions included in the adjusted action sequence, the confidence level of the task is determined to be 0, where a confidence level of 0 indicates task incompleteness.

[0124] Based on the execution status of the robot's target action sequence, a confidence level representing whether the task has been completed is generated. This objectively quantifies the execution result of the current task and provides quantifiable data for selecting action sequences when there are other tasks in the future.

[0125] In one possible implementation, the structured information of the environment, the adjusted action sequence, the task information, and the confidence level of the task are stored accordingly. This allows the robot to determine the action sequence required to complete other tasks when they are received, based on the image of the robot's environment when the other task is received and the stored action sequence.

[0126] In one possible implementation, when the robot's dexterous hand grasps an item, its vision sensor captures an image of the item and sends it to the controller. The controller determines the item's category based on the image. If the item's category differs from the category on the shelf where the robot is located, the controller places the item in a reference area for manual verification. After successful manual verification, the robot then puts the item on the shelf. During the manual verification period, the controller continues to guide the robot to put other items on the shelf.

[0127] The method described above combines environmental images and task information to generate structured information, and then determines the target action sequence based on this structured information. Compared to methods that rely solely on environmental images to determine the target action sequence, this method improves the accuracy of the determined target action sequence. The target action sequence is used to control the robot to execute the actions included in the target action sequence to complete the task. Therefore, it can improve the accuracy of robot control, ensure the safety of robot task execution, and ultimately increase the success rate of robot task completion.

[0128] Figure 5 This is an architecture diagram of a robot control system provided in an embodiment of this application. The system includes: a scene semantic understanding module based on surround view perception, an event retrieval and semantic matching engine based on VLM, a multimodal fusion decision-making and dexterous operation execution module, and a weakly labeled interactive memory graph construction module. Among them, The scene semantic understanding module based on surround view perception is used to acquire environmental images of the robot's environment; the environmental images are mapped to a bird's-eye view space using BEV technology to obtain a bird's-eye view feature map; the bird's-eye view feature map is transformed using OCC technology to obtain a 3D occupancy grid of the environment, which includes multiple voxels, with each voxel corresponding to one voxel's information; semantic reasoning is performed on the voxel information and task information of each voxel using VLM to obtain the structured information of the environment. This process has been described in steps 201 and 202 above, and will not be repeated here in the embodiments of this application.

[0129] The event retrieval and semantic matching engine based on VLM is used to determine the comprehensive similarity between the environment and each optional action sequence based on the structured information of the environment and the structured information of each optional action sequence; based on the comprehensive similarity between the environment and each optional action sequence, a reference action sequence is determined among the optional action sequences; if there is only one reference action sequence, it is used as the target action sequence; if there are multiple reference action sequences, the reference action sequence whose confidence level meets the confidence requirement is used as the target action sequence. This process has been described in step 203 above, and will not be repeated here in the embodiments of this application.

[0130] The multimodal fusion decision-making and dexterous operation execution module is used to control the robot to perform actions included in the target action sequence. This process has been described in step 204 above, and will not be repeated here in this embodiment.

[0131] The weakly labeled interactive memory graph construction module is used to store the task information, the action sequence during task execution, the structured information of the environment, and the confidence level of the task. This process has been described in step 204 above, and will not be repeated here.

[0132] Figure 6 The diagram shown is a structural schematic of a robot control device provided in an embodiment of this application. Figure 6 As shown, the device includes: The acquisition module 601 is used to acquire environmental images of the robot's environment. The generation module 602 is used to generate structured information about the environment based on the environmental image and task information; The determination module 603 is used to determine the target action sequence based on the structured information of the environment; The control module 604 is used to control the robot to perform the actions included in the target action sequence.

[0133] In one possible implementation, the generation module 602 is used to obtain a three-dimensional occupancy mesh of the environment based on the environment image. The three-dimensional occupancy mesh includes multiple voxels, each voxel corresponding to voxel information. Based on the voxel information of each voxel and the task information, the generation module generates structured information of the environment.

[0134] In one possible implementation, the generation module 602 is used to map the environmental image to the bird's-eye view space to obtain a bird's-eye view feature map; and to transform the bird's-eye view feature map to obtain a three-dimensional occupancy grid of the environment.

[0135] In one possible implementation, the generation module 602 is used to input the voxel information and task information of each voxel into the visual language model, and the visual language model is used to perform semantic reasoning on the voxel information and task information; the result of the semantic reasoning of the visual language model is used as the structured information of the environment.

[0136] In one possible implementation, the robot stores multiple optional action sequences, each of which corresponds to a structured piece of information; The determination module 603 is used to determine the comprehensive similarity between the environment and each optional action sequence based on the structured information of the environment and the structured information of each optional action sequence; and to determine the target action sequence among each optional action sequence based on the comprehensive similarity between the environment and each optional action sequence.

[0137] In one possible implementation, the structured information includes object semantic attributes, object spatial topological relationships, and scene context; The determination module 603 is used to determine the semantic similarity between the environment and each optional action sequence based on the semantic attributes of objects in the environment and the semantic attributes of objects in each optional action sequence; to determine the spatial structure similarity between the environment and each optional action sequence based on the spatial topological relationships of objects in the environment and the spatial topological relationships of objects in each optional action sequence; to determine the task similarity between the environment and each optional action sequence based on the scene context of the environment and the scene context of each optional action sequence; and to determine the comprehensive similarity between the environment and each optional action sequence based on the semantic similarity, spatial structure similarity, and task similarity.

[0138] In one possible implementation, the determining module 603 is used to, for any optional action sequence in each optional action sequence, determine a first feature vector representing the semantic attributes of an object in any optional action sequence; determine a second feature vector representing the semantic attributes of an object in the environment; determine the cosine similarity between the first feature vector and the second feature vector; and use the cosine similarity as the semantic similarity between the environment and any optional action sequence.

[0139] In one possible implementation, the determining module 603 is used to determine, for any optional action sequence in each optional action sequence, the graph edit distance between the object space topology of the environment and the object space topology of any optional action sequence; and to use the graph edit distance as the spatial structural similarity between the environment and any optional action sequence.

[0140] In one possible implementation, the determining module 603 is used to determine the comprehensive similarity between the environment and any optional action sequence for any optional action sequence in each optional action sequence, based on the semantic similarity between the environment and any optional action sequence, the semantic weight parameter, the spatial structure similarity between the environment and any optional action sequence, the spatial structure weight parameter, the task similarity between the environment and any optional action sequence, and the task weight parameter.

[0141] In one possible implementation, the object semantic attributes are used to describe the object features of a first object in the environment, which is the object corresponding to the task. The object features include at least one of the following: object category, object size, or graspable parts of the object. Object spatial topology is used to indicate the relative positions of various objects in an environment; Scene context includes at least one of the environment type or the relationship between the environment and the task.

[0142] In one possible implementation, the structured information also includes interactive attributes, which include at least one of the following: the material of a first object in the environment, the direction of force applied to the first object, the operation mode of the first object, or the material of a second object. The first object is the object corresponding to the task, and the second object is an obstacle for performing the task.

[0143] In one possible implementation, the determining module 603 is used to determine a reference action sequence among various optional action sequences, wherein the overall similarity between the reference action sequence and the environment is greater than a similarity threshold; if there is one reference action sequence, the reference action sequence is determined as the target action sequence; if there are multiple reference action sequences, the reference action sequence whose confidence meets the confidence requirement among the multiple reference action sequences is taken as the target action sequence, and the confidence of the reference action sequence is used to indicate the completion status of the task corresponding to the reference action sequence.

[0144] In one possible implementation, the device further includes: The adjustment module is used to respond to the received feedback signal, adjust the target action sequence based on the feedback signal, and obtain the adjusted action sequence. The control module 604 is also used to control the robot to execute the actions included in the adjusted action sequence.

[0145] In one possible implementation, the determining module 603 is further configured to determine the confidence level of the task based on the execution status of the actions included in the adjusted action sequence, the confidence level being used to indicate whether the task has been completed.

[0146] The aforementioned device combines environmental images and task information to generate structured information, and then determines the target action sequence based on this structured information. Compared to methods that rely solely on environmental images to determine the target action sequence, this approach improves the accuracy of the determined target action sequence. The target action sequence is used to control the robot to execute the actions included in the target action sequence to complete the task. Therefore, it improves the accuracy of robot control, ensures the safety of the robot in performing tasks, and ultimately increases the success rate of the robot in completing tasks.

[0147] It should be understood that the above-described apparatus is only illustrated by the division of the functional modules described above when implementing its functions. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0148] In an exemplary embodiment, a robot is also provided, the robot including a controller, the controller including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the robot control method provided in the above method embodiments.

[0149] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one piece of program code that is loaded and executed by a processor to enable a computer device to implement any of the robot control methods described above.

[0150] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0151] In an exemplary embodiment, a computer program or computer program product is also provided, which stores at least one computer instruction, which is loaded and executed by a processor to enable the computer to implement any of the robot control methods described above.

[0152] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0153] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0154] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for controlling a robot, characterized in that, The method includes: Obtain an environmental image of the robot's location; Based on the environmental image and task information, structured information about the environment is generated; Based on the structured information of the environment, determine the target action sequence; Control the robot to perform the actions included in the target action sequence.

2. The method according to claim 1, characterized in that, The step of generating structured information about the environment based on the environmental image and task information includes: Based on the environmental image, a three-dimensional occupancy grid of the environment is obtained. The three-dimensional occupancy grid includes multiple voxels, and each voxel corresponds to a voxel information. Based on the voxel information of each voxel and the task information, the structured information of the environment is generated.

3. The method according to claim 2, characterized in that, The step of obtaining the three-dimensional occupancy grid of the environment based on the environment image includes: The environmental image is mapped to the bird's-eye view space to obtain a bird's-eye view feature map; The bird's-eye view feature map is transformed to obtain the three-dimensional occupancy grid of the environment.

4. The method according to claim 2, characterized in that, The step of generating structured information about the environment based on the voxel information of each voxel and the task information includes: The voxel information of each voxel and the task information are input into a visual language model, which is used to perform semantic reasoning on the voxel information and the task information. The result of semantic reasoning using the visual language model is used as the structured information of the environment.

5. The method according to any one of claims 1 to 4, characterized in that, The robot stores multiple optional action sequences, each of which corresponds to a structured information. Determining the target action sequence based on the structured information of the environment includes: Based on the structured information of the environment and the structured information of each optional action sequence, determine the overall similarity between the environment and each optional action sequence; The target action sequence is determined from among the various optional action sequences based on the overall similarity between the environment and the various optional action sequences.

6. The method according to claim 5, characterized in that, The structured information includes object semantic attributes, object spatial topological relationships, and scene context; The step of determining the comprehensive similarity between the environment and the various optional action sequences based on the structured information of the environment and the structured information of each optional action sequence includes: Based on the object semantic attributes of the environment and the object semantic attributes of each optional action sequence, determine the semantic similarity between the environment and each optional action sequence; Based on the object space topology of the environment and the object space topology of each optional action sequence, determine the spatial structure similarity between the environment and each optional action sequence; Based on the scene context of the environment and the scene context of each optional action sequence, determine the task similarity between the environment and each optional action sequence; Based on the semantic similarity, spatial structure similarity, and task similarity between the environment and each of the optional action sequences, the overall similarity between the environment and each of the optional action sequences is determined.

7. The method according to claim 6, characterized in that, Determining the semantic similarity between the environment and each optional action sequence based on the object semantic attributes of the environment and the object semantic attributes of each optional action sequence includes: For any one of the optional action sequences, determine a first feature vector to characterize the semantic attributes of the object in that optional action sequence; Determine a second feature vector to characterize the semantic attributes of objects in the environment; Determine the cosine similarity between the first feature vector and the second feature vector; The cosine similarity is used as the semantic similarity between the environment and any of the optional action sequences.

8. The method according to claim 6, characterized in that, The step of determining the spatial structural similarity between the environment and the various optional action sequences based on the object spatial topology of the environment and the object spatial topology of each optional action sequence includes: For any of the optional action sequences, determine the graph edit distance between the object space topology of the environment and the object space topology of any of the optional action sequences; The graph edit distance is used as the spatial structural similarity between the environment and any of the optional action sequences.

9. The method according to claim 6, characterized in that, The step of determining the comprehensive similarity between the environment and each optional action sequence based on the semantic similarity, spatial structure similarity, and task similarity between the environment and each optional action sequence includes: For any of the optional action sequences, the comprehensive similarity between the environment and any optional action sequence is determined based on the semantic similarity and semantic weight parameter between the environment and the optional action sequence, the spatial structure similarity and spatial structure weight parameter between the environment and the optional action sequence, and the task similarity and task weight parameter between the environment and the optional action sequence.

10. The method according to claim 6, characterized in that, The object semantic attributes are used to describe the object features of a first object in the environment, where the first object is the object corresponding to the task, and the object features include at least one of object category, object size, or graspable part of the object. The object spatial topology is used to indicate the relative positions of various objects in the environment; The scenario context includes at least one of the environment type of the environment or the association between the environment and the task.

11. The method according to claim 6, characterized in that, The structured information also includes interactive attributes, which include at least one of the following: the material of a first object in the environment, the direction of force applied to the first object, the operation mode of the first object, or the material of a second object. The first object is the object corresponding to the task, and the second object is an obstacle for performing the task.

12. The method according to claim 5, characterized in that, The step of determining the target action sequence from among the various optional action sequences based on the comprehensive similarity between the environment and the various optional action sequences includes: A reference action sequence is determined from the various optional action sequences, wherein the overall similarity between the reference action sequence and the environment is greater than a similarity threshold; If the reference action sequence is a single sequence, then the reference action sequence is determined to be the target action sequence. If there are multiple reference action sequences, the reference action sequence whose confidence level meets the confidence level requirement is taken as the target action sequence, and the confidence level of the reference action sequence is used to indicate the completion status of the task corresponding to the reference action sequence.

13. The method according to any one of claims 1 to 4, characterized in that, After controlling the robot to execute the actions included in the target action sequence, the method further includes: In response to receiving a feedback signal, the target action sequence is adjusted based on the feedback signal to obtain an adjusted action sequence; Control the robot to execute the actions included in the adjusted action sequence.

14. The method according to claim 13, characterized in that, After controlling the robot to execute the actions included in the adjusted action sequence, the method further includes: Based on the execution status of the actions included in the adjusted action sequence, the confidence level of the task is determined, and the confidence level is used to indicate whether the task has been completed.

15. A robot, characterized in that, The robot includes a controller, which includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the robot control method as described in any one of claims 1 to 14.

Images