A method, apparatus and device for controlling movement of a robot

Abstract

The application provides a robot movement control method, device and equipment, relates to the technical field of robots, reconstructs a scene based on a surround view of a target scene, obtains a feature field including geometric features and semantic features, determines an object to be operated from the feature field in response to a natural language instruction, selects a target operation view angle of the object to be operated, matches features of the target operation view angle with features of an actual reach view angle of the robot, calculates an operation pose of the robot, controls the robot to move to a target position corresponding to the actual reach view angle based on the feature field, adjusts the pose of the robot to the operation pose, and controls the robot to complete target operation on the object to be operated through a pre-trained end-to-end control model. In this way, the robot is moved in the direction of approaching the object to be operated, the distance to the object to be operated is reduced, the control range is reduced, the generalization is improved, the data amount of the training end-to-end control model is reduced, and the target operation is completed.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a control method, apparatus, and device for robot movement. Background Technology

[0002] Robots are gradually entering shopping malls, homes, and public places, assisting or replacing human operation of objects in the open world and improving work efficiency.

[0003] As the robot moves and performs its tasks, it can adjust its actions in real time through visual feedback, which can overcome motion errors. However, its small-sample learning characteristic makes its generalization ability in open environments problematic. Summary of the Invention

[0004] In view of this, this application provides a robot movement control method, apparatus and device, which aims to enable the robot to move effectively and complete operations with a small amount of data.

[0005] In a first aspect, this application provides a control method for robot movement, characterized in that the method includes:

[0006] Scene reconstruction is performed based on the loop view of the target scene to obtain the feature field corresponding to the target scene, and the feature field includes geometric features and semantic features;

[0007] In response to a natural language command, an object to be operated is determined from the feature field; a target operation viewpoint of the object to be operated is selected, the target operation viewpoint being determined based on the occlusion of the object to be operated in the feature field;

[0008] Match the features of the target operating viewpoint with the features of the robot's actual reached viewpoint to calculate the robot's operating pose;

[0009] Based on the feature field, the robot is controlled to move to the target position corresponding to the actual arrival viewpoint; the robot's pose is adjusted to the operation pose, and the robot is controlled to complete the target operation on the object to be operated by a pre-trained end-to-end control model.

[0010] Optionally, the scene reconstruction based on the loop view of the target scene to obtain the feature field corresponding to the target scene includes:

[0011] Input the surround view of the target scene into the visual language big model to obtain the semantic features of the surround view;

[0012] The toroidal view and the semantic features are processed using a three-dimensional Gaussian splashing technique to obtain a feature field that integrates geometric and semantic features.

[0013] Optionally, selecting the target operation view of the object to be operated includes:

[0014] Multiple sampling perspectives are sampled for the object to be operated on from the feature field;

[0015] Calculate the target operation view of the object to be operated based on the occlusion status of the object under each sampling view.

[0016] Optionally, the step of calculating the target operation view of the object to be operated based on the occlusion status of the object under each sampling view includes:

[0017] The occlusion of the object to be operated on under each sampling viewpoint is calculated using perspective division.

[0018] The sampling view with the least occlusion is selected as the target operation view of the object to be operated.

[0019] Optionally, the process of matching the features of the target operating viewpoint with the features of the robot's actual reached viewpoint to calculate the robot's operating pose includes:

[0020] Feature extraction is performed on the scene and depth map in the rendered target operation view to obtain the features of the target operation view;

[0021] Feature extraction is performed on scene and depth data from the actual arrival viewpoint to obtain the features of the actual arrival viewpoint;

[0022] The features of the target operating viewpoint and the features of the actual arrived viewpoint are matched to obtain matching point pairs;

[0023] Based on the matching point pairs, the RANSAC-PNP algorithm is used to calculate the robot's operational pose.

[0024] Optionally, controlling the robot to perform the target operation on the object to be operated includes:

[0025] Control the robot to grasp the object to be operated.

[0026] Optionally, the training data for the pre-training of the end-to-end control model includes:

[0027] Under various operational conditions, the robot grasps data corresponding to different types of objects to be operated, wherein the type is one or more of the following: the size, position, shape, color, and texture of the object to be operated.

[0028] Secondly, this application also provides a control device for robot movement, the device comprising:

[0029] The reconstruction module is used to reconstruct the scene based on the surround view of the target scene to obtain the feature field corresponding to the target scene, the feature field including geometric features and semantic features;

[0030] A perspective module is used to respond to natural language commands, determine the object to be operated from the feature field, and select a target operation perspective of the object to be operated, wherein the target operation perspective is determined based on the occlusion of the object to be operated in the feature field.

[0031] The pose module is used to match the features of the target operating viewpoint with the features of the robot's actual reached viewpoint to calculate the robot's operating pose.

[0032] The operation module is used to control the robot to move to the target position corresponding to the actual arrival viewpoint based on the feature field, adjust the robot's pose to the operation pose, and control the robot to complete the target operation on the object to be operated through a pre-trained end-to-end control model.

[0033] Optionally, the reconstruction module is specifically used to input the surround view of the target scene into a large visual language model to obtain the semantic features of the surround view; and to process the surround view and the semantic features using a three-dimensional Gaussian splashing technique to obtain a feature field that integrates geometric and semantic features.

[0034] Thirdly, this application provides an apparatus comprising a memory and a processor, the memory for storing instructions or code, and the processor for executing the instructions or code to cause the apparatus to perform a robot movement control method as described in any of the first aspects above.

[0035] Fourthly, this application provides a computer storage medium storing code, wherein when the code is executed, a device running the code implements a robot movement control method as described in any of the first aspects above.

[0036] This application provides a robot movement control method, apparatus, and device. The method is based on scene reconstruction using a surround view, incorporating geometric and semantic features to form a feature field. From this feature field, the corresponding natural language command indicating the object to be manipulated is determined. The target operation viewpoint is determined based on the occlusion status of the object in the feature field. The features of the target operation viewpoint are matched with the features of the actual arrival viewpoint to calculate the robot's operation pose. During robot control, the robot is first moved closer to the object to be manipulated, reducing the distance to the object, narrowing the control range, improving generalization, and reducing the amount of data required to train the end-to-end control model. This allows the trained end-to-end control model to control the robot to complete the target operation based on the operation pose. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating a robot movement control method provided in an embodiment of this application;

[0039] Figure 2 This is a schematic diagram of the structure of a robot movement control device provided in an embodiment of this application. Detailed Implementation

[0040] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0041] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0042] Unless otherwise stated, the term "multiple" means two or more.

[0043] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0044] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0046] See Figure 1 , Figure 1 This application provides a flowchart illustrating a robot movement control method, which includes:

[0047] S101. Reconstruct the scene based on the panoramic view of the target scene to obtain the feature field corresponding to the target scene. The feature field includes geometric features and semantic features.

[0048] A panoramic view is an image captured by the robot in relation to the target scene.

[0049] The above scenario focuses not only on reconstructing geometric information such as color and texture in the target scene, but also on reconstructing semantic information, so that the reconstructed feature field is an environmental description with both geometric and semantic features.

[0050] S102. In response to a natural language instruction, determine the object to be operated from the feature field; select a target operating viewpoint for the object to be operated, wherein the target operating viewpoint is a viewpoint determined based on the occlusion of the object to be operated in the feature field.

[0051] The system queries the feature field for the object to be operated on, as indicated by the natural language command, and then segments the object from the feature field. For example, if the object to be operated on is a bottle, then the bottle is segmented and selected from the feature field; if the object to be operated on is a green bottle, then the green bottle is segmented and selected from the feature field.

[0052] Optionally, the target operation viewpoint for selecting the object to be operated can be the viewpoint in the feature field that obstructs the object to be operated the least.

[0053] S103. Match the features of the target operating viewpoint with the features of the robot's actual arrival viewpoint, and calculate the robot's operating pose.

[0054] The aforementioned target operation perspective is an ideal perspective that is pre-set based on factors such as operation requirements, target scenario, and robot operation requirements. Specifically, this perspective can be triggered by the object to be operated by the robot and determined after analyzing the target scenario and breaking down the operation task. It is considered to be the perspective most conducive to completing the target operation. For example, when the robot needs to perform operations such as grasping or assembling the object to be operated, it is the best perspective that allows the robot to clearly capture the details, outline, and relationship with the surrounding environment of the object.

[0055] The aforementioned actual arrival viewpoint refers to the actual observation viewpoint obtained after the robot moves to the target position corresponding to the actual arrival viewpoint. It is the true viewpoint determined by the robot's current pose after the robot actually moves to the target position, and is affected by various factors such as the robot's motion control accuracy and environmental interference.

[0056] The features of the target operation viewpoint can be extracted from the scene and depth map in the rendered target operation viewpoint, while the features of the actual arrival viewpoint can be extracted from the scene and depth map in the actual arrival viewpoint.

[0057] Considering factors such as chassis motion errors, it is difficult for a robot to accurately reach the position and posture set by the target viewpoint. Therefore, there is often a certain deviation between the actual reached viewpoint and the target operating viewpoint. This application matches the characteristics of the target operating viewpoint with the characteristics of the robot's actual reached viewpoint to determine the robot's operating pose after moving to the target position corresponding to the actual reached viewpoint, reducing the impact of robot motion errors and improving control accuracy.

[0058] S104. Based on the feature field, control the robot to move to the target position corresponding to the actual arrival viewpoint.

[0059] It should be noted that the above steps can also be performed first, followed by step S103.

[0060] S105. Adjust the robot's pose to the operating pose.

[0061] The operation pose here is the operation pose calculated and determined in step S103. This operation based on the actual arrival viewpoint improves the reliability of the subsequent execution of step S106.

[0062] S106. The robot is controlled by a pre-trained end-to-end control model to complete the target operation on the object to be operated.

[0063] An end-to-end control model is a control model that directly maps inputs to outputs. Robots can use end-to-end control models to directly generate joint movement commands or operations based on their own visual, force, and other sensor information to complete tasks such as object grasping and path planning.

[0064] The aforementioned target operations can be operations that can be performed by robots, such as grasping and assembly, which can assist or replace manual labor.

[0065] Based on steps S101-S106 above, the scene reconstruction based on the panoramic view incorporates geometric and semantic features to form a feature field. From this feature field, which incorporates semantic features, the corresponding natural language command indicating the object to be operated is determined. The target operation viewpoint is determined based on the occlusion status of the object to be operated in the feature field. The features of the target operation viewpoint are matched with the features of the actual arrival viewpoint to calculate the robot's operation pose in the actual arrival viewpoint, reducing the impact of robot motion errors and improving control accuracy. During robot control, the robot is first moved closer to the object to be operated, reducing the distance to the object, narrowing the control range, improving generalization, and reducing the amount of data required to train the end-to-end control model. This allows the trained end-to-end control model to control the robot to complete the target operation based on the operation pose.

[0066] In the embodiments of this application, the above Figure 1 There are several possible implementations of step S101, which will be described below. It should be noted that the implementations given below are merely illustrative examples and do not represent all implementations of the embodiments of this application.

[0067] In step S101, scene reconstruction is performed based on the panoramic view of the target scene to obtain the feature field corresponding to the target scene. Specific steps may include:

[0068] First, the surround view of the target scene is input into the visual language large model to obtain the semantic features of the surround view.

[0069] The aforementioned large-scale visual language model can employ CLIP (Contrastive Language-Image Pretraining) to understand the semantic correspondence between images and text.

[0070] Then, the toroidal view and the semantic features are processed using three-dimensional Gaussian splashing technology to obtain a feature field that integrates geometric and semantic features.

[0071] 3D Gaussian splatting technology can reconstruct 3D scenes from multi-view images or point cloud data. This application combines 3D Gaussian splatting technology with a large visual language model. Besides reconstructing geometric information such as color and texture in the scene, it can also reconstruct the semantic information of the scene using the large visual language model, generating an environmental description with both geometric and semantic features, forming a feature field of the target scene to facilitate subsequent processing of natural language commands. Further, in step S102, in response to a natural language command, determining the object to be operated on from the feature field can be achieved by performing semantic analysis on the natural language command, extracting key information from the natural language command, and determining Gaussian bodies in the feature field that are similar to or match the key information. Since an object may be represented by multiple Gaussian bodies, these Gaussian bodies have a certain spatial correlation. Thus, the position, size, and shape of the Gaussian bodies can be combined to determine which Gaussian bodies belong to the same object. For example, multiple adjacent Gaussian bodies that spatially form the shape of a bottle will be integrated together to complete the segmentation and determination of the object to be operated on, namely, the "bottle".

[0072] In the embodiments of this application, the above Figure 1 There are several possible implementations for step S102, which selects the target operation perspective of the object to be operated on. These will be described below. It should be noted that the implementations given below are merely illustrative examples and do not represent all implementations of the embodiments of this application.

[0073] In step S102, the target operation viewpoint of the object to be operated is selected, and the specific steps may include:

[0074] First, multiple sampling perspectives are sampled from the feature field for the object to be operated on.

[0075] Optionally, multiple sampling perspectives used for uniform sampling should cover as many surfaces and key parts of the object being manipulated as possible to avoid blind spots and to prevent the sampling perspectives from being too concentrated in certain areas. Of course, for semantically important parts, the density of sampling perspectives can be appropriately increased. For example, for a bottle that a robot needs to grasp, the grasping operation parts on the bottle (such as the bottle body) are semantically important, so the number of sampling perspectives near these parts can be increased.

[0076] Then, for each of the sampling viewpoints, the target operation viewpoint of the object to be operated is selected based on the occlusion status of the object to be operated.

[0077] Optionally, perspective division is used to calculate the occlusion of the object to be operated on under each sampling viewpoint. Then, the sampling viewpoint with the least occlusion is selected as the target operation viewpoint for the object to be operated on. In this way, selecting the viewpoint with the least occlusion as the operation viewpoint improves the success rate of the robot in subsequently completing the target operation.

[0078] In the embodiments of this application, the above Figure 1 There are several possible implementations of step S103, which will be described below. It should be noted that the implementations given below are merely illustrative examples and do not represent all implementations of the embodiments of this application.

[0079] In step S103, the features of the target operating viewpoint are matched with the features of the robot's actual reached viewpoint to calculate the robot's operating pose. Specific steps may include:

[0080] First, feature extraction is performed on the scene and depth map in the rendered target operation view to obtain the features of the target operation view.

[0081] The features of the target operation viewpoint can be extracted from the scene and depth map rendered from the target operation viewpoint. The extracted features can include key points that describe the scene and their surrounding local features.

[0082] Secondly, feature extraction is performed on the scene and depth data from the actual arrival perspective to obtain the features of the actual arrival perspective.

[0083] The features of the actual arrival viewpoint can be extracted based on the scene and depth map within the actual arrival viewpoint. The scene within the actual arrival viewpoint can be determined by the robot moving to the target location corresponding to the actual arrival viewpoint and capturing the scene. The depth map can be obtained at the target location using, for example, a depth camera, or it can be estimated using computer vision algorithms.

[0084] Then, the features of the target operating viewpoint and the features of the actual arrived viewpoint are matched to obtain matching point pairs.

[0085] A matching algorithm is used to match the features of the target operation viewpoint and the features of the actual arrival viewpoint to find the correspondence between the two and obtain matching point pairs. Furthermore, based on geometric constraints such as the distance and angle of the matching points, the matching point pairs are filtered to remove some erroneous matching point pairs and improve the accuracy of matching.

[0086] Finally, based on the matched point pairs, the RANSAC-PNP algorithm is used to calculate the robot's operational pose.

[0087] The operational pose calculated using the RANSAC (Random Sample Consensus)-PNP (Perspective-n-Point) algorithm is used as the robot's actual pose relative to the target viewpoint, so that the robot can be adjusted to this operational pose and then begin the next target operation.

[0088] In one specific implementation, the target operation mentioned in this application can be a grasping operation. Furthermore, the training data for the pre-training of the aforementioned end-to-end control model can be,

[0089] Under various operating backgrounds (such as a marble countertop, a white tabletop, etc.), the robot grasps the corresponding data of different types of objects to be operated (such as bottles). The type is one or more of the following: the size of the object to be operated (such as bottles of different sizes such as 300ML and 500ML), position (such as front, middle, left, etc.), shape (such as column, frustum, polygonal prism, etc.), color (such as green, red, transparent, etc.), and texture (such as frosted, smooth, etc.).

[0090] In this way, several sets of data corresponding to different types of substitute objects under different backgrounds are collected to train the end-to-end control model. The end-to-end control model only needs to be trained to perform the target operation after the robot moves to the target position, rather than starting training from the robot's starting position. This reduces the training range, improves generalization, and reduces the amount of training data while completing the target operation.

[0091] The above are some specific implementations of a robot movement control method provided in this application. Based on this, this application also provides a corresponding robot movement control device. The device provided in this application will be described below from the perspective of functional modularity.

[0092] See Figure 2 The diagram shows a structural schematic of a robot movement control device, which includes:

[0093] The reconstruction module 201 is used to reconstruct the scene based on the surround view of the target scene to obtain the feature field corresponding to the target scene, wherein the feature field includes geometric features and semantic features;

[0094] The view module 202 is used to respond to natural language instructions to determine the object to be operated from the feature field; and to select a target operation view of the object to be operated, wherein the target operation view is determined based on the occlusion of the object to be operated in the feature field.

[0095] The pose module 203 is used to match the features of the target operating viewpoint with the features of the robot's actual reached viewpoint to calculate the robot's operating pose.

[0096] The operation module 204 is used to control the robot to move to the target position corresponding to the actual arrival viewpoint based on the feature field, adjust the robot's pose to the operation pose, and control the robot to complete the target operation on the object to be operated through a pre-trained end-to-end control model.

[0097] Based on the aforementioned device, the reconstruction module 201 integrates geometric and semantic features into a feature field based on scene reconstruction using a panoramic view. The viewpoint module 202 and pose module 203 determine the corresponding natural language command-indicated object from the feature field incorporating semantic features. The target operation viewpoint is determined based on the occlusion of the object in the feature field. The features of the target operation viewpoint are matched with the features of the actual arrival viewpoint. The operation module 204 calculates the robot's operation pose within the actual arrival viewpoint, reducing the impact of robot motion errors and improving control accuracy. During robot control, the robot is first moved closer to the object to be operated, reducing the distance to the object, narrowing the control range, improving generalization, and reducing the amount of data required to train the end-to-end control model. This allows the trained end-to-end control model to control the robot to complete the target operation based on the operation pose.

[0098] In one possible implementation, the reconstruction module 201 is specifically used to input the surround view of the target scene into a large visual language model to obtain the semantic features of the surround view; and to process the surround view and the semantic features using a three-dimensional Gaussian splashing technique to obtain a feature field that integrates geometric and semantic features.

[0099] In one possible implementation, the view module 202 is specifically used to sample multiple viewpoints for the object to be operated from the feature field; and to select the target operation viewpoint of the object to be operated based on the occlusion of the object under each sampled viewpoint.

[0100] Optionally, perspective division is used to calculate the occlusion of the object to be operated on under each sampling viewpoint; the sampling viewpoint with the least occlusion is selected as the target operation viewpoint of the object to be operated on.

[0101] In one possible implementation, the pose module 203 is specifically used to extract features from the scene and depth map in the rendered target operation viewpoint to obtain the features of the target operation viewpoint; extract features from the scene and depth data in the actual arrival viewpoint to obtain the features of the actual arrival viewpoint; match the features of the target operation viewpoint and the features of the actual arrival viewpoint to obtain matching point pairs; and calculate the robot's operation pose using the RANSAC-PNP algorithm based on the matching point pairs.

[0102] In one possible implementation, controlling the robot to complete the target operation on the object to be operated means controlling the robot to grasp the object to be operated.

[0103] Optionally, the training data for the pre-training of the end-to-end control model may include: data corresponding to the robot grasping different types of objects to be operated under various operating backgrounds, wherein the type is one or more of the specifications, position, shape, color, and texture of the object to be operated.

[0104] This application also provides corresponding devices and computer storage media for implementing the solutions provided in this application.

[0105] The device includes a memory and a processor. The memory stores instructions or code, and the processor executes the instructions or code to enable the device to perform a robot movement control method according to any embodiment of this application.

[0106] The computer storage medium stores code, and when the code is executed, the device running the code implements a robot movement control method according to any embodiment of this application.

[0107] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0108] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0109] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0110] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.

Claims

1. A control method of a robot movement, characterized by, The method comprises: scene reconstruction based on the surround view of the target scene, to obtain a feature field corresponding to the target scene, the feature field comprising geometric features and semantic features; determining an object to be operated from the feature field in response to a natural language instruction; selecting a target operation view angle of the object to be operated, the target operation view angle being determined according to the occlusion of the object to be operated in the feature field; matching the features of the target operation view angle with the features of the actual reach view angle of the robot, and calculating the operation pose of the robot; based on the feature field, controlling the robot to move to a target position corresponding to the actual reach view angle; adjusting the pose of the robot to the operation pose, and controlling the robot to complete the target operation on the object to be operated through a pre-trained end-to-end control model.

2. The method of claim 1, wherein, The scene reconstruction based on the surround view of the target scene comprises: inputting the surround view of the target scene into a visual language large model to obtain semantic features of the surround view; processing the surround view and the semantic features through three-dimensional Gaussian splashing technology to obtain a feature field fused with geometric features and semantic features.

3. The method of claim 1, wherein, The selection of the target operation view angle of the object to be operated comprises: sampling a plurality of sampling view angles for the object to be operated from the feature field; calculating the target operation view angle of the object to be operated according to the occlusion of the object to be operated under each sampling view angle.

4. The method of claim 3, wherein, The calculation of the target operation view angle of the object to be operated according to the occlusion of the object to be operated under each sampling view angle comprises: calculating the occlusion of the object to be operated under each sampling view angle using perspective division; selecting the sampling view angle with the least occlusion as the target operation view angle of the object to be operated.

5. The method of claim 1, wherein, The matching of the features of the target operation view angle with the features of the actual reach view angle of the robot, and the calculation of the operation pose of the robot comprise: extracting features from the rendered scene and depth map in the target operation view angle to obtain features of the target operation view angle; extracting features from the scene and depth data in the actual reach view angle to obtain features of the actual reach view angle; matching the features of the target operation view angle with the features of the actual reach view angle to obtain matching point pairs; based on the matching point pairs, calculating the operation pose of the robot using the RANSAC-PNP algorithm.

6. The method of claim 1, wherein, The control of the robot to complete the target operation on the object to be operated comprises: controlling the robot to grasp the object to be operated.

7. The method of claim 6, wherein, The pre-training training data of the end-to-end control model comprises: corresponding data of the robot grasping different types of the object to be operated in various operation backgrounds, the types being one or more of the specifications, positions, shapes, colors and textures of the object to be operated.

8. A control device for robot movement, characterized in that The device comprises: a reconstruction module configured to perform scene reconstruction based on a surround view of a target scene to obtain a feature field corresponding to the target scene, the feature field comprising geometric features and semantic features; The perspective module is configured to determine an object to be operated from the feature field in response to the natural language instruction, and determine a target operation perspective of the object to be operated, the target operation perspective being a perspective determined according to an occlusion condition of the object to be operated in the feature field. The pose module is configured to match features in the target operation perspective with features in an actual reaching perspective of the robot, and calculate an operation pose of the robot. The operation module is configured to control the robot to move to a target position corresponding to the actual reaching perspective based on the feature field, adjust the pose of the robot to the operation pose, and control the robot to complete a target operation on the object to be operated by using a pre-trained end-to-end control model.

9. The apparatus of claim 8, wherein, The reconstruction module is specifically configured to input a surround view of the target scene into a visual language large model to obtain semantic features of the surround view, and process the surround view and the semantic features by using a three-dimensional Gaussian splashing technology to obtain a feature field fused with geometric features and semantic features.

10. An apparatus, comprising: The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes to enable the device to perform the control method of the robot movement according to any one of claims 1-7.

Images