Robot movement control method, device and equipment

Abstract

The invention provides a robot movement control method, device and equipment, and relates to the technical field of robots, scene reconstruction is performed based on an all-round view of a target scene, and a feature field including geometric features and semantic features is obtained; determining a to-be-operated object from the feature field in response to a natural language instruction; selecting a target operation view angle of the to-be-operated object; matching the features of the target operation view angle with the features of the actual arrival view angle of the robot, and calculating the operation pose of the robot; based on the characteristic field, controlling the robot to move to a target position corresponding to the actual arrival view angle; the pose of the robot is adjusted to be the operation pose, and the robot is controlled to complete the target operation of the to-be-operated object through the pre-trained end-to-end control model, so that the robot is moved in the direction close to the to-be-operated object, the distance to the to-be-operated object is reduced, the control range is reduced, and generalization is improved; and reducing the data volume of the training end-to-end control model and completing the target operation.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a method, device and equipment for controlling robot movement. Background Art

[0002] Robots are gradually entering shopping malls, homes and public places, assisting or replacing human operators in operating objects in the open world and improving work efficiency.

[0003] When the robot is moving and completing operations, it can adjust operations in real time through visual feedback and overcome motion errors. However, its small sample learning characteristics make its generalization ability in open environments problematic. Summary of the invention

[0004] In view of this, the present application provides a control method, device and equipment for robot movement, aiming to enable the robot to move effectively and complete operations through a small amount of data.

[0005] In a first aspect, the present application provides a method for controlling the movement of a robot, characterized in that the method comprises:

[0006] Reconstructing the scene based on the surround view of the target scene to obtain a feature field corresponding to the target scene, wherein the feature field includes geometric features and semantic features;

[0007] In response to a natural language instruction, determining an object to be operated from the feature field; selecting a target operation viewing angle of the object to be operated, wherein the target operation viewing angle is a viewing angle determined according to an occlusion condition of the object to be operated in the feature field;

[0008] Match the characteristics of the target operation perspective with the characteristics of the robot's actual reach perspective, and calculate the robot's operation posture;

[0009] Based on the feature field, the robot is controlled to move to the target position corresponding to the actual arrival viewing angle; the posture of the robot is adjusted to the operation posture, and the robot is controlled by a pre-trained end-to-end control model to complete the target operation on the object to be operated.

[0010] Optionally, the scene reconstruction based on the surround view of the target scene to obtain a feature field corresponding to the target scene includes:

[0011] Inputting the surround view of the target scene into the visual language model to obtain semantic features of the surround view;

[0012] The surround view and the semantic features are processed by a three-dimensional Gaussian splattering technique to obtain a feature field that integrates geometric features and semantic features.

[0013] Optionally, the selecting a target operation viewing angle of the object to be operated includes:

[0014] Sampling a plurality of sampling viewing angles for the object to be operated from the feature field;

[0015] Under each of the sampling viewing angles, a target operating viewing angle of the object to be operated is selected according to the occlusion condition of the object to be operated.

[0016] Optionally, the calculating, at each sampling viewing angle, selecting a target operating viewing angle of the object to be operated according to an occlusion condition of the object to be operated includes:

[0017] The perspective division method is used to calculate the occlusion of the object to be operated under each sampling perspective;

[0018] A sampling viewing angle with the least occlusion is selected as the target operating viewing angle of the object to be operated.

[0019] Optionally, the matching of the characteristics of the target operation perspective with the characteristics of the robot's actual reach perspective to calculate the robot's operation posture includes:

[0020] Extracting features of the rendered scene and depth map in the target operation perspective to obtain features of the target operation perspective;

[0021] Extract features of the scene and depth data in the actual arrival perspective to obtain features of the actual arrival perspective;

[0022] Matching the characteristics of the target operation perspective with the characteristics of the actual arrival perspective to obtain a matching point pair;

[0023] Based on the matching point pairs, the RANSAC-PNP algorithm is used to calculate the robot's operating posture.

[0024] Optionally, controlling the robot to complete a target operation on the object to be operated includes:

[0025] The robot is controlled to grasp the object to be operated.

[0026] Optionally, the pre-trained training data of the end-to-end control model includes:

[0027] In various operation contexts, the robot captures data corresponding to different types of objects to be operated, where the types are one or more of the specifications, positions, shapes, colors, and textures of the objects to be operated.

[0028] In a second aspect, the present application further provides a control device for robot movement, the device comprising:

[0029] A reconstruction module, used to reconstruct the scene based on the surround view of the target scene to obtain a feature field corresponding to the target scene, wherein the feature field includes geometric features and semantic features;

[0030] A viewing angle module is used to determine the object to be operated from the feature field in response to a natural language instruction; select a target operating viewing angle of the object to be operated, wherein the target operating viewing angle is a viewing angle determined according to an occlusion condition of the object to be operated in the feature field;

[0031] The pose module is used to match the characteristics of the target operation viewpoint with the characteristics of the robot's actual reach viewpoint and calculate the robot's operation pose;

[0032] An operation module is used to control the robot to move to a target position corresponding to the actual arrival angle of view based on the feature field, adjust the robot's posture to the operation posture, and control the robot to complete the target operation on the object to be operated through a pre-trained end-to-end control model.

[0033] Optionally, the reconstruction module is specifically used to input the surround view of the target scene into a large visual language model to obtain semantic features of the surround view; process the surround view and the semantic features through three-dimensional Gaussian splattering technology to obtain a feature field that integrates geometric features and semantic features.

[0034] In a third aspect, the present application provides a device comprising a memory and a processor, wherein the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes a robot movement control method as described in any one of the first aspects above.

[0035] In a fourth aspect, the present application provides a computer storage medium having codes stored therein. When the codes are executed, a device executing the codes implements a method for controlling the movement of a robot as described in any one of the first aspects above.

[0036] The present application provides a control method, device and equipment for robot movement. The method integrates geometric features and semantic features to form a feature field based on scene reconstruction of the surround view; determines the object to be operated indicated by the corresponding natural language instruction from the feature field integrated with the semantic features; determines the target operation perspective according to the occlusion of the object to be operated in the feature field, matches the features of the target operation perspective with the features of the actual arrival perspective, and calculates the robot's operating posture. In the process of robot control, the robot is first moved in the direction close to the object to be operated, the distance from the object to be operated is reduced, the control range is reduced to improve generalization, and the amount of data for training the end-to-end control model is reduced, so that the robot can be controlled by the trained end-to-end control model to complete the target operation based on the operating posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A schematic diagram of a flow chart of a method for controlling robot movement provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of the structure of a robot movement control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0041] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0042] Unless otherwise stated, the term "plurality" means two or more.

[0043] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0044] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] See also Figure 1 , Figure 1 A schematic flow chart of a method for controlling the movement of a robot provided in an embodiment of the present application, wherein the method for controlling the movement of a robot comprises:

[0047] S101. Reconstruct a scene based on a surround view of a target scene to obtain a feature field corresponding to the target scene, wherein the feature field includes geometric features and semantic features.

[0048] The surround view is the image captured by the robot corresponding to the target scene.

[0049] The above-mentioned scene not only focuses on reconstructing geometric information such as color and texture in the target scene, but also reconstructs semantic information so that the reconstructed feature field is an environment description with geometric and semantic features.

[0050] S102, in response to a natural language instruction, determining an object to be operated from the feature field; selecting a target operation viewing angle of the object to be operated, wherein the target operation viewing angle is a viewing angle determined according to an occlusion condition of the object to be operated in the feature field.

[0051] The object to be operated indicated by the natural language instruction is searched in the feature field, and the object to be operated is segmented from the feature field. For example, if the object to be operated indicated by the natural language instruction is a bottle, the bottle is segmented and selected from the feature field; if the object to be operated indicated by the natural language instruction is a green bottle, the green bottle is segmented and selected from the feature field.

[0052] Optionally, the target operation viewing angle of the selected object to be operated may be a viewing angle that least blocks the operation object in the selected feature field.

[0053] S103, matching the characteristics of the target operation perspective with the characteristics of the robot's actual reach perspective, and calculating the robot's operation posture.

[0054] The above-mentioned target operation perspective is an ideal perspective pre-set according to factors such as operation requirements, target scenes, and robot operation requirements. Specifically, this perspective can be triggered by the robot's object to be operated. It is determined after analyzing the target scene and breaking down the operation task. It is considered to be the perspective that is most conducive to completing the target operation. For example, when the robot is required to grasp, assemble, and perform other operations on the object to be operated, the best perspective that allows the robot to clearly capture the object's details, contours, and relationship with the surrounding environment.

[0055] The above-mentioned actual arrival viewing angle refers to the observation viewing angle actually obtained after the robot moves to the target position corresponding to the actual arrival viewing angle. It is the real viewing angle determined by the current posture of the robot after the robot actually moves to the target position, and is affected by many factors such as the robot's motion control accuracy and environmental interference.

[0056] The feature of the target operation perspective may be feature extraction of the scene and depth map in the rendered target operation perspective, and the feature of the actual arrival perspective may be feature extraction based on the scene and depth map in the actual arrival perspective.

[0057] Considering the influence of factors such as the robot's chassis motion error, it is difficult for the robot to completely and accurately reach the position and posture set by the target view angle, so there is often a certain deviation between the actual arrival view angle and the target operation view angle. Therefore, the present application matches the characteristics of the target operation view angle with the characteristics of the robot's actual arrival view angle, determines the operation posture of the robot after it moves to the target position corresponding to the actual arrival view angle, reduces the influence of the robot's motion error, and improves the accuracy of control.

[0058] S104: Based on the feature field, control the robot to move to a target position corresponding to the actual arrival viewing angle.

[0059] It should be noted that, in the above steps, step S104 may be performed first, and then step S103 may be performed.

[0060] S105: Adjust the posture of the robot to the operating posture.

[0061] The operation posture here is the operation posture calculated and determined in step S103. In this way, the operation is performed based on the operation posture in the actual reaching viewing angle, thereby improving the reliability of the execution of the subsequent step S106.

[0062] S106: Control the robot to complete the target operation on the object to be operated through the pre-trained end-to-end control model.

[0063] The end-to-end control model is a control model that directly maps input to output. Through the end-to-end control model, the robot can directly generate joint motion instructions or operating actions based on its own visual, force and other sensor information to complete operational tasks such as object grasping and path planning.

[0064] The above-mentioned target operation may be an operation that can be performed by a robot, such as grasping, assembling, etc., to assist or replace manual labor.

[0065] Based on the above steps S101-S106, it can be known that the scene reconstruction based on the surround view incorporates geometric features and semantic features to form a feature field; the object to be operated indicated by the corresponding natural language instruction is determined from the feature field incorporating semantic features; the target operation perspective is determined according to the occlusion of the object to be operated in the feature field, the features of the target operation perspective are matched with the features of the actual arrival perspective, and the operation posture of the robot in the actual arrival perspective is calculated to reduce the influence of the robot motion error and improve the accuracy of control. In the process of robot control, the robot is first moved in the direction close to the object to be operated, the distance from the object to be operated is reduced, the control range is reduced to improve generalization, and the amount of data for training the end-to-end control model is reduced, so that the robot can be controlled by the trained end-to-end control model to complete the target operation based on the operation posture.

[0066] In the embodiment of the present application, the above Figure 1 There are multiple possible implementations of step S101, which are described below. It should be noted that the implementations described below are only exemplary and do not represent all implementations of the embodiments of the present application.

[0067] In step S101, scene reconstruction is performed based on the surround view of the target scene to obtain a feature field corresponding to the target scene. The specific steps may include:

[0068] Firstly, the surround view of the target scene is input into the visual language model to obtain the semantic features of the surround view.

[0069] The above-mentioned visual language model can adopt CLIP (Contrastive Language-Image Pretraining), which can understand the semantic correspondence between images and texts.

[0070] Then, the surround view and the semantic features are processed by a three-dimensional Gaussian splattering technique to obtain a feature field that integrates geometric features and semantic features.

[0071] The 3D Gaussian Splatting technology can reconstruct a 3D scene from a multi-view image or point cloud data. The present application uses the 3D Gaussian Splatting technology in combination with a large visual language model. In addition to being able to reconstruct geometric information such as color texture in the scene, the semantic information of the scene can also be reconstructed in combination with the large visual language model to generate an environmental description with both geometric features and semantic features, forming a feature field of the target scene, so as to facilitate the subsequent processing of natural language instructions. Further, in step S102, in response to the natural language instruction, the object to be operated is determined from the feature field, which can be to perform semantic analysis on the natural language instruction, extract key information in the natural language instruction, and determine a Gaussian body that is close to or matches the key information in the feature field. Since an object may be represented by multiple Gaussian bodies, these Gaussian bodies have a certain correlation in space. In this way, it is also possible to combine the geometric parameters such as the position, size and shape of the Gaussian body to determine which Gaussian bodies belong to the same object. For example, multiple Gaussian bodies that are adjacent and spatially form a bottle shape will be integrated together to complete the segmentation of the "bottle" object to be operated to determine the object to be operated.

[0072] In the embodiment of the present application, the above Figure 1 There are multiple possible implementations of step S102 to select the target operation angle of the object to be operated, which are introduced below. It should be noted that the implementations given in the following introduction are only exemplary and do not represent all implementations of the embodiments of the present application.

[0073] In step S102, a target operation viewing angle of the object to be operated is selected, and the specific steps may include:

[0074] First, a plurality of sampling viewing angles are sampled from the feature field for the object to be operated.

[0075] Optionally, multiple uniformly sampled sampling perspectives should cover all surfaces and key parts of the operated object as much as possible to avoid blind spots and excessive concentration of sampling perspectives in certain areas. Of course, for semantically important parts, the density of sampling perspectives can be appropriately increased. For example, for a bottle that a robot wants to grasp, the grasping operation part on the bottle (such as the bottle body) has high semantic importance, and the number of sampling perspectives near this part is increased.

[0076] Then, a target operation viewing angle of the object to be operated is selected according to the occlusion condition of the object to be operated under each sampling viewing angle.

[0077] Optionally, perspective division is used to calculate the occlusion of the object to be operated under each of the sampling perspectives. Then, the sampling perspective with the least occlusion is selected as the target operation perspective of the object to be operated. In this way, the perspective with the least occlusion is selected as the operation perspective, which improves the success rate of the robot in completing the target operation later.

[0078] In the embodiment of the present application, the above Figure 1 There are multiple possible implementations of step S103, which are described below. It should be noted that the implementations described below are only exemplary and do not represent all implementations of the embodiments of the present application.

[0079] In step S103, the characteristics of the target operation perspective are matched with the characteristics of the robot's actual reach perspective to calculate the robot's operation posture. The specific steps may include:

[0080] First, feature extraction is performed on the rendered scene and depth map in the target operation perspective to obtain features of the target operation perspective.

[0081] The feature of the target operation perspective may be a feature extraction of the scene and the depth map in the rendered target operation perspective. The extracted features may include key points that can describe the scene and local features around them.

[0082] Secondly, feature extraction is performed on the scene and depth data in the actual arrival perspective to obtain the features of the actual arrival perspective.

[0083] The features of the actual arrival perspective can be extracted based on the scene and depth map in the actual arrival perspective. The scene in the actual arrival perspective can be the scene in the actual arrival perspective determined by shooting the robot when it moves to the target position corresponding to the actual arrival perspective, and the depth map in the actual arrival perspective can be obtained by shooting at the target position through, for example, a depth camera, or can be estimated through a computer vision algorithm.

[0084] Then, the features of the target operation perspective and the features of the actual arrival perspective are matched to obtain a matching point pair.

[0085] A matching algorithm is used to match the characteristics of the target operation perspective and the characteristics of the actual arrival perspective, find the correspondence between the two, and obtain matching point pairs. Furthermore, according to geometric constraints such as the distance and angle of the matching points, the matching point pairs are screened to remove some erroneous matching point pairs and improve the accuracy of matching.

[0086] Finally, based on the matching point pairs, the RANSAC-PNP algorithm is used to calculate the robot's operating posture.

[0087] The operating posture calculated by the RANSAC (Random Sample Consensus)-PNP (Perspective-n-Point) algorithm is used as the actual posture of the robot relative to the target perspective, so that the robot can be adjusted to the operating posture and then start the next target operation.

[0088] In a specific implementation, the target operation mentioned in the present application may be a grabbing operation. Furthermore, the pre-trained training data of the above end-to-end control model may be,

[0089] Under various operating backgrounds (for example, an operating background of a marble countertop, an operating background of a white tabletop, etc.), the robot grabs corresponding data of different types of objects to be operated (for example, bottles), wherein the types are one or more of the specifications of the objects to be operated (for example, bottles of different specifications such as 300ML\500ML, etc.), positions (for example, front, middle, left, etc.), shapes (for example, columnar, truncated cone, polygonal prism, etc.), colors (for example, green, red, transparent, etc.), and textures (for example, frosted, smooth, etc.).

[0090] In this way, several groups of data corresponding to different types of operating objects in different backgrounds are collected to train the end-to-end control model. The end-to-end control model only needs to train the target operation after the robot moves to the target position, rather than the robot starting training from the starting position of the movement, thereby narrowing the training scope, improving generalization, and completing the target operation while reducing the amount of training data.

[0091] The above are some specific implementations of a robot movement control method provided by the embodiment of the present application. Based on this, the present application also provides a corresponding robot movement control device. The device provided by the embodiment of the present application will be introduced from the perspective of functional modularization.

[0092] See also Figure 2 A schematic diagram of a control device for robot movement is shown, the device comprising:

[0093] A reconstruction module 201 is used to reconstruct a scene based on a surround view of a target scene to obtain a feature field corresponding to the target scene, wherein the feature field includes geometric features and semantic features;

[0094] A viewing angle module 202 is used to determine the object to be operated from the feature field in response to a natural language instruction; select a target operating viewing angle of the object to be operated, wherein the target operating viewing angle is a viewing angle determined according to an occlusion condition of the object to be operated in the feature field;

[0095] The posture module 203 is used to match the characteristics of the target operation perspective with the characteristics of the robot's actual reach perspective, and calculate the robot's operation posture;

[0096] The operation module 204 is used to control the robot to move to the target position corresponding to the actual arrival angle based on the feature field, adjust the robot's posture to the operation posture, and control the robot to complete the target operation on the object to be operated through a pre-trained end-to-end control model.

[0097] Based on the above device, the reconstruction module 201 integrates geometric features and semantic features into the scene reconstruction based on the surround view to form a feature field; through the perspective module 202 and the posture module 203, the object to be operated indicated by the corresponding natural language instruction is determined from the feature field integrated with the semantic features, and the target operation perspective is determined according to the occlusion of the object to be operated in the feature field, and the characteristics of the target operation perspective are matched with the characteristics of the actual arrival perspective. Through the operation module 204, the operation posture of the robot in the actual arrival perspective is calculated to reduce the influence of the robot motion error and improve the control accuracy. In the process of robot control, the robot is first moved in the direction close to the object to be operated, the distance from the object to be operated is reduced, the control range is reduced to improve generalization, and the amount of data for training the end-to-end control model is reduced, so that the robot can be controlled by the trained end-to-end control model to complete the target operation based on the operation posture.

[0098] In a possible implementation, the reconstruction module 201 is specifically used to input the surround view of the target scene into a large visual language model to obtain semantic features of the surround view; and process the surround view and the semantic features through a three-dimensional Gaussian splashing technique to obtain a feature field that integrates geometric features and semantic features.

[0099] In a possible implementation, the viewing angle module 202 is specifically configured to sample a plurality of sampling viewing angles for the object to be operated from the feature field; calculate a target operating viewing angle of the object to be operated under each sampling viewing angle, and select the target operating viewing angle of the object to be operated according to the occlusion condition of the object to be operated.

[0100] Optionally, perspective division is used to calculate the occlusion of the object to be operated under each sampling viewing angle; and the sampling viewing angle with the least occlusion is selected as the target operating viewing angle of the object to be operated.

[0101] In a possible implementation, the posture module 203 is specifically used to extract features from the rendered scene and depth map in the target operation perspective to obtain features of the target operation perspective; extract features from the scene and depth data in the actual arrival perspective to obtain features of the actual arrival perspective; match the features of the target operation perspective with the features of the actual arrival perspective to obtain matching point pairs; and calculate the robot's operating posture based on the matching point pairs using the RANSAC-PNP algorithm.

[0102] In a possible implementation manner, controlling the robot to complete the target operation on the object to be operated refers to controlling the robot to grasp the object to be operated.

[0103] Optionally, the pre-trained training data of the end-to-end control model may include: corresponding data of the robot grasping different types of objects to be operated in various operating contexts, and the type is one or more of the specifications, positions, shapes, colors, and textures of the objects to be operated.

[0104] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0105] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes a robot movement control method described in any embodiment of the present application.

[0106] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements a method for controlling the movement of a robot as described in any embodiment of the present application.

[0107] The "first" and "second" in the names such as "first" and "second" (if any) mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.

[0108] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0109] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0110] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A method for controlling robot movement, characterized in that: The method comprises: Reconstructing the scene based on the surround view of the target scene to obtain a feature field corresponding to the target scene, wherein the feature field includes geometric features and semantic features; In response to a natural language instruction, determining an object to be operated from the feature field; selecting a target operation viewing angle of the object to be operated, wherein the target operation viewing angle is a viewing angle determined according to an occlusion condition of the object to be operated in the feature field; Match the characteristics of the target operation perspective with the characteristics of the robot's actual reach perspective, and calculate the robot's operation posture; Based on the feature field, the robot is controlled to move to the target position corresponding to the actual arrival viewing angle; the posture of the robot is adjusted to the operation posture, and the robot is controlled by a pre-trained end-to-end control model to complete the target operation on the object to be operated.

2. The method according to claim 1, characterized in that The scene reconstruction based on the surround view of the target scene to obtain a feature field corresponding to the target scene includes: Inputting the surround view of the target scene into the visual language model to obtain semantic features of the surround view; The surround view and the semantic features are processed by a three-dimensional Gaussian splattering technique to obtain a feature field that integrates geometric features and semantic features.

3. The method according to claim 1, characterized in that The selecting a target operation angle of view of the object to be operated includes: Sampling a plurality of sampling viewing angles for the object to be operated from the feature field; Under each of the sampling viewing angles, a target operating viewing angle of the object to be operated is selected according to the occlusion condition of the object to be operated.

4. The method according to claim 3, characterized in that The calculating, at each of the sampling viewing angles, selecting a target operating viewing angle of the object to be operated according to an occlusion condition of the object to be operated, comprises: The perspective division method is used to calculate the occlusion of the object to be operated under each sampling viewing angle; A sampling viewing angle with the least occlusion is selected as the target operating viewing angle of the object to be operated.

5. The method according to claim 1, characterized in that The matching of the characteristics of the target operation perspective with the characteristics of the robot's actual reach perspective to calculate the robot's operation posture includes: Extracting features of the rendered scene and depth map in the target operation perspective to obtain features of the target operation perspective; Extract features of the scene and depth data in the actual arrival perspective to obtain features of the actual arrival perspective; Matching the characteristics of the target operation perspective with the characteristics of the actual arrival perspective to obtain a matching point pair; Based on the matching point pairs, the RANSAC-PNP algorithm is used to calculate the robot's operating posture.

6. The method according to claim 1, characterized in that The controlling the robot to complete the target operation on the object to be operated includes: The robot is controlled to grasp the object to be operated.

7. The method according to claim 6, characterized in that The pre-trained training data of the end-to-end control model includes: In various operation contexts, the robot captures data corresponding to different types of objects to be operated, where the types are one or more of the specifications, positions, shapes, colors, and textures of the objects to be operated.

8. A control device for robot movement, characterized in that: The device comprises: A reconstruction module, used to reconstruct the scene based on the surround view of the target scene to obtain a feature field corresponding to the target scene, wherein the feature field includes geometric features and semantic features; A viewing angle module is used to determine the object to be operated from the feature field in response to a natural language instruction; select a target operating viewing angle of the object to be operated, wherein the target operating viewing angle is a viewing angle determined according to an occlusion condition of the object to be operated in the feature field; The pose module is used to match the characteristics of the target operation viewpoint with the characteristics of the robot's actual reach viewpoint and calculate the robot's operation pose; An operation module is used to control the robot to move to a target position corresponding to the actual arrival angle of view based on the feature field, adjust the robot's posture to the operation posture, and control the robot to complete the target operation on the object to be operated through a pre-trained end-to-end control model.

9. The device according to claim 8, characterized in that The reconstruction module is specifically used to input the surround view of the target scene into the visual language large model to obtain the semantic features of the surround view; the surround view and the semantic features are processed by three-dimensional Gaussian splashing technology to obtain a feature field that integrates geometric features and semantic features.

10. A device, characterized in that: The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes, so that the device executes the robot movement control method described in any one of claims 1 to 7.

Images