Robot control method, device and system

Abstract

The invention relates to a robot control method, device and system, and belongs to the technical field of intelligent control. The method comprises the steps of obtaining image information; wherein the image information comprises an operation object and an environment where the operation object is located; determining input information of the dexterous operation large model; the input information comprises position and attitude information of the operation object and environment semantic point cloud information of an environment where the operation object is located; inputting the input information into the dexterous operation large model to obtain a target action sequence output by the dexterous operation large model; and controlling the robot to operate the operation object according to the target action sequence. According to the method, more intelligent and comprehensive perception input is introduced, and environment semantic information and a dexterous operation large model are combined, so that the robot is more flexible and efficient in more complex and finer operation tasks.

Description

Technical Field

[0001] The present disclosure belongs to the field of intelligent control technology, and in particular relates to a control method, device and control system for a robot. Background Art

[0002] In recent years, robotics has experienced rapid development, encompassing diverse fields including industrial automation, healthcare, services, logistics, and agriculture. With the continuous advancement of technologies such as deep learning, computer vision, and natural language processing, robotics is developing towards intelligence, diversity, and flexibility.

[0003] Existing robotic control methods typically rely on relatively simple sensor data and employ fixed rules to control robot operations. These methods typically use basic image processing techniques or sensor data (such as position and posture information) to determine the state of the target object and environmental conditions, and then execute tasks according to a pre-set action sequence. While these technologies can achieve simple operations under specific conditions, their shortcomings include a lack of adaptability to complex environments and a lack of comprehensive processing of multimodal information. Furthermore, existing methods are generally unable to effectively handle highly complex and dynamically changing task environments and lack sufficient intelligent decision-making support when performing dexterous and precise operations.

[0004] Therefore, how to overcome the deficiencies in flexibility, precision, and adaptability of existing robot control technologies and enhance the robot's ability to operate in complex and changing environments is a topic worthy of study for those skilled in the art. Summary of the Invention

[0005] In view of the above analysis, the embodiments of the present disclosure aim to provide a robot control method, device and control system, aiming to overcome the deficiencies of the existing technology in flexibility, accuracy and adaptability, and to enhance the robot's operational capabilities in complex and changing environments.

[0006] A first aspect of the present disclosure provides a robot control method, comprising: Acquiring image information; wherein the image information includes an operation object and an environment in which the operation object is located; Determining input information for the dexterous manipulation large model; wherein determining the input information for the dexterous manipulation large model includes: determining input information based on image information, the input information including: position and posture information of the manipulation object, and environmental semantic point cloud information of the environment in which the manipulation object is located; Inputting input information into the dexterous manipulation large model to obtain the target action sequence output by the dexterous manipulation large model; Control the robot to operate the object according to the target action sequence.

[0007] The step of determining input information for the smart operation large model includes: Based on the image information, three-dimensional point cloud information corresponding to the image information is obtained; Segmenting the three-dimensional point cloud information corresponding to the image information to obtain the three-dimensional point cloud information of the operation object and the environmental point cloud information of the environment in which the operation object is located; Determine the position and posture information of the operation object based on the three-dimensional point cloud information of the operation object; The environmental semantic point cloud information is determined according to the environmental point cloud information of the environment in which the operation object is located.

[0008] The step of segmenting the three-dimensional point cloud information corresponding to the image information to obtain the three-dimensional point cloud information of the operation object and the environmental point cloud information of the environment in which the operation object is located includes: Determine the binary mask of the operation object; Determine target point cloud information of the operation object and environmental point cloud information of the environment in which the operation object is located based on the three-dimensional point cloud information corresponding to the image information and the binary mask of the operation object; The step of determining the position and posture information of the operation object according to the three-dimensional point cloud information of the operation object includes: determining three main direction information and 3D position information of the operation object based on the target point cloud information of the operation object as the position and posture information of the operation object; The step of determining the environmental semantic point cloud information according to the environmental point cloud information of the environment in which the operation object is located includes: performing semantic segmentation on the environmental point cloud information to obtain the environmental semantic point cloud information of the environment in which the operation object is located.

[0009] Before the step of acquiring image information, the method further includes: Acquire robot teaching data; wherein the robot teaching data includes at least one of human operation video data and expert teaching data, the human operation video data includes video data of humans operating sample objects, and the expert teaching data includes image data of the sample objects and joint posture data of the robot; Processing the robot teaching data to obtain training data; wherein the training data includes: position and posture information of the sample object, semantic point cloud information of the environment in which the sample object is located, and joint posture information of the robot; Build large models for flexible operation and use training data for model training; Deploy a large model of dexterous manipulation to a robot.

[0010] The step of processing the robot teaching data to obtain training data includes: Based on human operation video data, generate posture information of sample objects and semantic point cloud information of the environment in which the sample objects are located; Human posture information is extracted from human operation video data, mapped to the robot, and the robot's joint posture information is generated.

[0011] The steps of obtaining robot teaching data include: During the remote operation of the sample object, real machine teaching data is obtained; wherein, the image data of the sample object is obtained by an image sensor arranged on the robot, and the joint posture data of the robot includes the joint posture data of the robot during the remote operation of the sample object.

[0012] Before the step of obtaining robot teaching data, the control method further includes: Creating a simulation model of the robot and a simulation task scenario, wherein the simulation task scenario includes a simulation operation object and a simulation environment in which the simulation operation object is located; The method comprises the following steps: obtaining robot teaching data, wherein the robot teaching data includes at least one of human operation video data and expert teaching data, including: obtaining simulation teaching data during the process of a simulation model operating a simulation operation object; wherein image data of a sample object is generated through simulation, and the robot's joint posture data includes the joint posture data of the simulation model during the process of operating the simulation operation object.

[0013] Before the step of acquiring image information, the control method further includes: Get the task information entered by the user; Calling artificial intelligence agents and multimodal large models to analyze and decompose task information and obtain subtask instructions; The step of obtaining image information includes: obtaining image information according to a subtask instruction; The step of determining input information for the dexterous operation large model further includes: determining a subtask instruction as one of the input information.

[0014] A second aspect of the present disclosure provides a robot control device, comprising: An image information acquisition module is configured to acquire image information; wherein the image information includes an operation object and an environment in which the operation object is located; An input information determination module is configured to determine input information of the dexterous manipulation large model; wherein determining the input information of the dexterous manipulation large model includes: determining the input information based on image information, the input information including: position and posture information of the manipulation object, and environmental semantic point cloud information of the environment in which the manipulation object is located; an output module configured to input input information into the dexterous manipulation large model and obtain a target action sequence output by the dexterous manipulation large model; The control module is configured to control the robot to operate the operation object according to the target action sequence.

[0015] The control device further comprises: A task information acquisition module is configured to acquire task information input by a user; The task analysis and decomposition module is configured to call the artificial intelligence agent and the multimodal large model to analyze and decompose task information and obtain subtask instructions; The image information acquisition module is further configured to acquire image information according to the subtask instruction; The input information determination module is further configured to determine the subtask instruction as one of the input information.

[0016] A third aspect of the present disclosure provides a robot control system, comprising: a robot control device, and a cloud server; Among them, the control device is the above-mentioned control device, and the cloud server is deployed with an artificial intelligence agent and a multimodal large model.

[0017] The control method of the robot provided by the present disclosure determines the input multimodal information based on image information, which includes not only the position of the operation object, but also the posture of the operation object and the environmental semantic point cloud information of the environment in which the operation object is located, providing richer contextual information for operation decisions. Through the input of multimodal information, the robot can accurately identify the relationship between the operation object and the environment, and make more accurate control decisions. The dexterous operation large model adopted by the present disclosure is highly intelligent and has the ability to handle high-level cognition and reasoning. The robot can perform dexterous operations in complex operation tasks, especially in tasks that require high precision and coordination. The robot can also dynamically adjust its control strategy and optimize the operation steps according to environmental changes and different targets, and has strong adaptability. It can be seen that the present disclosure combines environmental semantic information with the dexterous operation large model by introducing more intelligent and comprehensive perception input, making the robot more flexible and efficient in more complex and more sophisticated operation tasks.

[0018] The robot control method provided by the present disclosure uses input information for the dexterous manipulation large model, including the posture of the manipulation object and the environmental semantic point cloud information of the manipulation object's environment. Because the posture of the manipulation object and the environmental semantic point cloud information of the manipulation object's environment are highly consistent in both simulation and real environments, the control method provided by the present disclosure enables the dexterous manipulation large model to be trained using massive amounts of simulation environment data, and can be easily migrated to a real robot for use. This makes large-scale expert teaching and data generation feasible in a simulation environment, facilitates the acquisition of massive amounts of training data, thereby improving model generalization and significantly reducing migration costs.

[0019] In addition, the present disclosure also provides a robot control device and control system having the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0021] Figure 1 A flowchart of a specific implementation of the robot control method provided by the present disclosure; Figure 2 A flow chart of a process for determining input information for a smart operating macro model for the present disclosure; Figure 3 A flowchart of the process of using binary masks for segmentation to obtain three-dimensional point cloud information of the operation object and environmental point cloud information of the environment in which the operation object is located; Figure 4 A flowchart of the process of training a large dexterous manipulation model in the robot control method provided by the present disclosure; Figure 5 This is a flow chart of the process of obtaining training data based on human operation video data in the present disclosure; Figure 6 This is a flowchart of the process of obtaining training data using simulation teaching method in the present disclosure; Figure 7 This is a flowchart of another specific implementation of the robot control method provided by the present disclosure before step S101 of acquiring image information; Figure 8 A schematic diagram of the process of the robot control method provided by the present disclosure; Figure 9 A structural block diagram of the control device of the robot provided by the present disclosure; Figure 10 This is a structural block diagram of the robot control system provided by the present disclosure; Figure 11 This is a hierarchical end-to-end model architecture diagram of the robot control system provided by the present disclosure. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. It should be noted that, in the absence of conflict, the embodiments of the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0023] The terms used herein are for the purpose of describing specific embodiments and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, the features, integral bodies, steps, operations, parts, assemblies and / or their groups stated are explained, but the presence or addition of one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups is not excluded. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0024] Some embodiments of the present disclosure provide a method for controlling a robot. Figure 1 A flowchart of a robot control method provided by some embodiments of the present disclosure is shown, and the method specifically includes: S101: Acquire image information; wherein the image information includes an operation object and an environment in which the operation object is located.

[0025] In the disclosed embodiments, the robot acquires image information using an image acquisition device (e.g., an RGB camera, a LiDAR radar, a depth camera, etc.). This image information includes not only visual data of the object being manipulated (e.g., its position and shape), but also overall information about the environment in which the object is being manipulated (e.g., other objects, obstacles, and spatial layout within the scene). This allows the robot to identify and locate all objects and obstacles in the environment, acquiring spatial distribution and semantic information about the environment. For example, image segmentation algorithms can be used to extract important elements from the environment.

[0026] S102: Determine the input information of the dexterous operation large model; wherein, determining the input information of the dexterous operation large model includes: determining the input information based on image information, and the input information includes: position and posture information of the operation object, and environmental semantic point cloud information of the environment in which the operation object is located.

[0027] The position and posture information of the operation object describes the specific state of the operation object in space, providing spatial information for dexterous operation of large models, making the operation more precise.

[0028] The environmental semantic point cloud information of the environment in which the operating object is located is three-dimensional structured data of the scene where the operating object is located, including semantic information of the surrounding environment (such as obstacles, supporting surfaces, etc.), which supports the robot to make intelligent decisions in complex environments, such as avoiding obstacles or choosing the best grasping angle.

[0029] In the embodiment of the present disclosure, the image information obtained may include RGB information and depth information. Specifically, a variety of input methods can be used to obtain RGB information and depth information. For example, through RGB-D images, the color information and corresponding depth value of each pixel can be directly obtained. Binocular images can also be used to simultaneously capture a scene using two cameras to obtain two parallax images. The parallax principle is used to calculate the depth information of each pixel in the scene from the two images. In this way, based on the image information obtained in S101, key feature information of the operation object and environment can be extracted as input for the dexterous operation large model.

[0030] Specifically, the position and posture information of the operating object in three-dimensional space can be determined through computer vision algorithms (such as object detection and posture estimation). The posture information of the operating object can be 6D posture information.

[0031] LiDAR or depth cameras are used to obtain 3D point cloud data of the environment, and semantic segmentation technology is used to extract semantic information about each object in the environment (for example, walls, tabletops, target objects, etc.). This provides robots with a high-dimensional semantic understanding of the environment, helping them make more intelligent decisions.

[0032] S103: Input the input information into the dexterous operation large model to obtain the target action sequence output by the dexterous operation large model.

[0033] The dexterous operation model is a large neural network model based on artificial intelligence, which is responsible for outputting the target action sequence for operating the operation object based on the input information.

[0034] Understandably, the dexterous manipulation model is trained on a large amount of data, learning how to generate reasonable target action sequences based on input information (the position and posture of the manipulated object, and the semantic point cloud information of the environment in which the manipulated object is located). This training data is typically derived from at least one of real-world scenarios and simulated environments.

[0035] In response to input information, the dexterous manipulation model uses a multi-layer neural network to reason and generate a target action sequence. This target action sequence can include a collection of actions that the robot needs to perform over multiple consecutive frames, such as grasping, moving, rotating, and placing.

[0036] S104: Control the robot to operate the operation object according to the target action sequence.

[0037] The robot performs specific operations based on the target action sequence output by the dexterous manipulation model. These operations may include various actions such as grasping, moving, rotating, and combined operations.

[0038] The control method of the robot provided by the present disclosure determines the input multimodal information based on image information, which includes not only the position of the operation object, but also the posture of the operation object and the environmental semantic point cloud information of the environment in which the operation object is located, providing richer contextual information for operation decisions. Through the input of multimodal information, the robot can accurately identify the relationship between the operation object and the environment, and make more accurate control decisions. The dexterous operation large model adopted by the present disclosure is highly intelligent and has the ability to handle high-level cognition and reasoning. The robot can perform dexterous operations in complex operation tasks, especially in tasks that require high precision and coordination. The robot can also dynamically adjust its control strategy and optimize the operation steps according to environmental changes and different targets, and has strong adaptability. It can be seen that the present disclosure combines environmental semantic information with the dexterous operation large model by introducing more intelligent and comprehensive perception input, making the robot more flexible and efficient in more complex and more sophisticated operation tasks.

[0039] Most of the operation models in the prior art use RGB images as input information. Due to the differences between the simulation environment and the actual environment, the simulation training data is difficult to migrate to the real machine. In the control method provided by the present disclosure, the input information of the dexterous operation large model includes the posture of the operation object and the environmental semantic point cloud information of the environment in which the operation object is located. Thanks to the high consistency of the posture of the operation object and the environmental semantic point cloud information of the environment in which the operation object is located in the simulation environment and the real environment, compared with the control method that mainly uses RBG images as the input of the operation large model, the control method provided by the present disclosure enables the dexterous operation large model to be trained using massive simulation environment data and can be easily migrated to the real robot for use. The embodiment of the present disclosure makes it feasible to carry out large-scale expert teaching and data generation in a simulation environment, facilitates the acquisition of massive training data, thereby improving the generalization of the model and greatly reducing the migration cost.

[0040] In some embodiments of the present disclosure, the step of determining the input information of the smart operation large model can be specifically implemented as follows: Figure 2 , the process specifically includes: S201: Based on the image information, obtain three-dimensional point cloud information corresponding to the image information.

[0041] In this step, the acquired image information is converted into corresponding three-dimensional point cloud information.

[0042] One specific implementation method is to use an RGB-D camera to simultaneously capture both the RGB and depth information of an image. The depth information provides the distance of each pixel from the camera, and by combining the two-dimensional information of each pixel, a three-dimensional point cloud can be generated.

[0043] Each pixel in an RGB-D camera records the three primary colors of red (R), green (G), and blue (B), as well as the depth information of a specific 3D point in space corresponding to that pixel. Assuming Z is the depth value of a 3D point in space, u and v are its pixel coordinates, fx and fy are the focal lengths of the lens in the x and y directions (the distance from the imaging plane to the lens), and cx and cy are the coordinates of the optical center (the projection of the lens center point on the imaging plane) in the image coordinate system (with the origin at the upper left corner of the image, horizontal x and vertical y), then the following relationship exists between the 3D point coordinates X, Y, and Z and the pixel coordinates u and v:

[0044] The above formula can be used to obtain the 3D coordinates corresponding to each pixel on the depth map, that is, the three-dimensional point cloud information.

[0045] An optional implementation method may be: using a binocular camera to obtain binocular image information of the operation object, combining binocular depth estimation technology to generate three-dimensional point cloud information.

[0046] An optional implementation method can also be: using a single camera to obtain image information of the operating object from multiple perspectives, combining simultaneous positioning and mapping technology to generate three-dimensional point cloud information.

[0047] S202: Segment the three-dimensional point cloud information corresponding to the image information to obtain the three-dimensional point cloud information of the operation object and the environmental point cloud information of the environment in which the operation object is located.

[0048] This step aims to separate the operation object and the environmental point cloud information from the point cloud, and obtain the three-dimensional point cloud information of the operation object and the environmental point cloud information of the environment in which the operation object is located.

[0049] Specifically, a direct point cloud segmentation method can be used to perform geometric feature analysis (such as distance, normal vector, etc.) on the three-dimensional point cloud information itself to separate the target point cloud and the environment point cloud.

[0050] Deep learning models (such as PointNet) can also be used to perform semantic segmentation on 3D point clouds. Deep learning models are trained to classify point clouds into multiple categories, such as operating objects, desktops, and obstacles.

[0051] In addition, in some embodiments of the present disclosure, a binary mask can be used for segmentation to obtain three-dimensional point cloud information of the operation object and environmental point cloud information of the environment in which the operation object is located. Figure 3 As shown, the process specifically includes: S2021: Determine the binary mask of the operation object.

[0052] Binary masking is a simplified form of image segmentation. By marking the target area of ​​the object (mask value 1) and the background area (mask value 0), it can quickly distinguish the object from the surrounding environment. Each pixel in the image has a corresponding point cloud point. The mask can directly determine which pixels belong to the target area of ​​the object, thereby quickly extracting the 3D point cloud corresponding to these pixels. Compared with complex geometric analysis or semantic segmentation of the entire point cloud, combining binary masking with image information is more intuitive and efficient.

[0053] S2022: Determine target point cloud information of the operation object and environmental point cloud information of the environment in which the operation object is located based on the three-dimensional point cloud information corresponding to the image information and the binary mask of the operation object.

[0054] After determining the binary mask of the operation object, the binary mask is aligned with the corresponding 3D point cloud information. The point cloud data is traversed to filter out the pixel point clouds corresponding to the mask value of 1 and determine them as the target point cloud information of the operation object. The remaining point clouds are automatically classified as the environmental point cloud information of the environment in which the operation object is located.

[0055] During this process, due to imaging and segmentation accuracy issues, the resulting 3D point cloud may contain some noise points, i.e., 3D point clouds that do not belong to the operation object. As an optional implementation, the present disclosure can remove noise through methods such as clustering to obtain a denoised 3D point cloud of the operation object.

[0056] S203: Determine the position and posture information of the operation object according to the three-dimensional point cloud information of the operation object.

[0057] According to the three-dimensional point cloud information of the operation object, the step of determining the position and posture information of the operation object can be specifically as follows: based on the target point cloud information of the operation object, determine the three main direction information and 3D position information of the operation object as the position and posture information of the operation object.

[0058] The core of this process is to use the three-dimensional point cloud information of the operation object to calculate its specific position (3D position information) and direction (3 main direction information) in space through mathematical analysis and geometric methods, thereby providing the robot with an accurate description of the operation object.

[0059] Optionally, the three main directions of the point cloud can be calculated by the principal component analysis (PCA) method. These three main directions constitute the 3D posture of the operation object, and the average value of the point cloud represents the 3D position of the operation object.

[0060] Given a target point cloud X consisting of N 3D points, where X is an N×3 matrix, the calculation steps for the three main directions are: Calculate the center of the target point cloud: .

[0061] Calculate the covariance matrix of the target point cloud: .

[0062] Use the SVD decomposition method to perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues ​​and eigenvectors of the covariance matrix: .

[0063] Sort the eigenvalues ​​from large to small, and select the first three eigenvectors as the three main directions of the target point cloud.

[0064] Together, position and pose information constitute a spatial description of the manipulated object, providing a complete reference framework for the robot to perform grasping, movement, or other operations. The proposed method, based on 3D point cloud analysis, eliminates the need for pre-obtaining a 3D model of the manipulated object. Therefore, even unseen objects can be manipulated correctly, resulting in excellent generalization of the proposed large-scale dexterous manipulation model.

[0065] S204: Determine environmental semantic point cloud information according to the environmental point cloud information of the environment in which the operation object is located.

[0066] This step identifies key semantic information in the operating environment (such as obstacles, support surfaces, and boundaries). Specifically, semantic segmentation is performed on the environmental point cloud to obtain a semantic point cloud of the operating object's environment. The environmental point cloud is decomposed into semantic regions of different categories, and the 3D coordinates of each point cloud point and its corresponding semantic label together form a semantic point cloud.

[0067] This embodiment extracts 3D point cloud information from 2D image information and segments the 3D point cloud to extract the 3D point cloud information of the manipulated object, distinguishing the manipulated object from the rest of the environment. This allows the position and posture information of the manipulated object, as well as the semantic point cloud information of the environment in which the manipulated object resides, to be used as input for the dexterous manipulation model. This allows the dexterous manipulation model to be trained using massive amounts of simulated environment data and easily transferred to a real robot for use, improving model generalization and significantly reducing migration costs.

[0068] In the robot control method provided by the present disclosure, the robot can complete complex tasks autonomously by learning the operation methods of humans or experts. Figure 4 As shown, the process specifically includes the following steps: S401: Obtain robot teaching data. The robot teaching data includes at least one of human operation video data and expert teaching data. The human operation video data includes video data of a human operating a sample object; the expert teaching data includes image data of the sample object and robot joint posture data.

[0069] The purpose of obtaining robot teaching data is to obtain learning samples from human or expert operations and provide real operation data for robot model training.

[0070] Human manipulation video data consists of videos of people manipulating sample objects, recording how they interact with them. Computer vision techniques (such as action recognition or keypoint detection) can be used to extract information such as manipulation steps, object positions, and posture changes from the videos.

[0071] Expert teaching data includes both real-machine teaching data and simulated teaching data, used for robot operations in real and virtual environments, respectively. Real-machine teaching data involves humans directly controlling a real robot via a teleoperation device to complete a task. Simulated teaching data involves humans or automated systems (such as reinforcement learning agents) controlling a virtual robot via a simulation platform to complete a task. By collecting multimodal data from human operation videos, real-machine teaching, and simulated teaching, the comprehensiveness and diversity of the data are improved, enabling the model to generalize more effectively to complex tasks.

[0072] S402: Processing the robot teaching data to obtain training data; wherein the training data includes: position and posture information of the sample object, semantic point cloud information of the environment where the sample object is located, and joint posture information of the robot.

[0073] For human manipulation video data, we obtain video recordings of humans manipulating sample objects, including RGB image sequences and human motion trajectories. We extract the geometric information of the sample objects and the human manipulation trajectories from the videos and map them into data suitable for the robot. For example, we perform open-domain object detection, segmentation, and pose estimation on the human manipulation video data to obtain the sample object poses, semantic point clouds, and sample objects required for the dexterous manipulation large-scale model. We then migrate and redirect this data into robot teaching data, extracting the robot's current joint pose data, thus providing complete input information for training the dexterous manipulation large-scale model.

[0074] For real-machine teaching data, the image data and joint posture data collected when the robot completes the operation in the real environment are obtained, the geometric information of the sample object and the environment is extracted, and the detailed parameters of the robot's movement are recorded.

[0075] For simulation teaching data, virtual data generated by the simulation environment includes images of simulated objects, point cloud information, and robot operation records in the simulation. Object and scene features in the simulation environment, as well as robot action sequences, are extracted.

[0076] S403: Build a large model for smart operation and use the training data to train the model.

[0077] Based on the processed training data, a large model is built that can learn and reason about complex operations. The model accepts multiple input types, including the position and pose information of sample objects, the semantic point cloud information of the environment, and the pose information of the robot joints.

[0078] The large model for smart operation can adopt a Transformer structure with only a decoder, or the latest Transformer-based diffusion model structure, or other structures that can realize the functions of this solution, which are not limited here.

[0079] S404: Deploy the dexterous manipulation large model to the robot.

[0080] Integrating the trained model into the robotic system enables the robot to receive input data and perform actions in real time.

[0081] In this embodiment, robot teaching data (human operation video data and expert teaching data) provides high-quality training data for the dexterous manipulation large-scale model, accelerating model training and convergence, and improving task generalization and robustness. By processing the robot teaching data and extracting multi-dimensional information: the position and posture of the sample object, the semantic point cloud of the environment, and the robot's joint posture information, this provides more comprehensive training data for the dexterous manipulation large-scale model.

[0082] In some optional embodiments, referring to Figure 5, the steps of processing the robot teaching data to obtain training data include the following processes: S501: Generate posture information of a sample object and semantic point cloud information of the environment in which the sample object is located based on human operation video data.

[0083] Vision models (such as object detection and segmentation models) are used to extract key features of the sample object and its environment from the video. Specifically, this involves using the vision model to estimate the 3D position and pose (such as position coordinates and rotation angle) of the sample object. Using semantic segmentation or point cloud generation techniques, a semantic point cloud of the environment is reconstructed from the video, generating semantic point cloud information about the sample object's environment.

[0084] S502: Extracting human posture information from human operation video data, mapping the human posture information to the robot, and generating joint posture information of the robot.

[0085] Based on human operation video data, a visual model is used to identify humans in the video and extract human posture information during the operation (such as arm and finger movements).

[0086] If the human pose information is in 2D, it must be converted to 3D. After obtaining the human's 3D pose information, real2sim migration technology is used to map this 3D pose information to the robot. This mapping process involves kinematic adjustments and scaling, enabling the robot to mimic human movements. Through these steps, the human pose information is converted into joint pose information that the robot can perform. The robot's joint pose information includes the robot's joint angles, current roll, pitch, and yaw velocities, and more.

[0087] In some embodiments of the present disclosure, the step of acquiring robot teaching data may further include acquiring training data using teleoperation. This process specifically includes acquiring real-machine teaching data during teleoperation of a sample object. The image data of the sample object is acquired via an image sensor mounted on the robot, and the robot's joint posture data includes joint posture data of the robot during teleoperation of the sample object.

[0088] Understandably, this process primarily involves collecting motion and environmental data from the robot during actual task execution through teleoperation. Teleoperation refers to human control of a robot through input devices (such as joysticks, touchscreens, and force feedback devices), while the robot performs specific actions during the operation. Using the robot teleoperation system, image information and joint posture data of the robot are recorded in a real-world operating environment, providing high-quality teaching samples for subsequent model training.

[0089] When teleoperating a robot, it can be completely controlled in real time by a human via a remote control device. All operational commands are issued by the human, and the robot serves only as an execution tool. This is a scenario where the human fully controls the robot. Alternatively, the robot can rely on its own perception and decision-making capabilities (such as dexterous manipulation of large models) to complete the task autonomously. However, at critical steps or in failure scenarios, such as when the model fails to operate autonomously or when encountering scenes or objects not seen in the training set, humans intervene and correct the problem in real time. This is a scenario where the robot's autonomous capabilities are combined with human intervention. Providing additional teaching feedback for these challenging scenarios can effectively improve the robustness of the model.

[0090] Optionally, for the above-mentioned teleoperation scenario, an image sensor on the robot body (such as an RGB camera, a depth camera or a lidar) can be used to collect image information in real time, and the robot control system can record the joint posture data of the robot at each time point.

[0091] In some embodiments of the present disclosure, before the step of obtaining robot teaching data, the robot control method further includes: obtaining training data by means of simulation teaching. The difference between simulation teaching and real machine teaching mainly lies in the different manipulation targets. Simulation teaching manipulates the robot model in the simulation environment rather than the real machine. The relevant data recorded by simulation teaching is similar to the manipulation of the real machine, and is simulated data generated in the simulation environment. By performing scene simulation in a virtual environment (simulation scene), the robot model can complete operations in the virtual environment and generate simulation teaching data, providing an efficient, safe, and controllable data source for robot teaching and model training.

[0092] Reference Figure 6 , the process specifically includes: S601: Create a simulation model and a simulation task scenario of the robot. The simulation task scenario includes a simulation operation object and a simulation environment in which the simulation operation object is located.

[0093] The simulation model is a virtual version of the robot, with similar physical properties and control behaviors to the real robot. The simulation task scenario is the virtual environment of the operation task, including the simulated operation objects and environment.

[0094] S602: Acquire simulation teaching data while the simulation model operates the simulation operation object; wherein, the image data of the sample object is generated through simulation, and the joint posture data of the robot includes the joint posture data of the simulation model during the operation of the simulation operation object.

[0095] The virtual camera in the simulation model captures image data of the sample objects in the simulation teaching data. The dynamics module of the simulation model acquires the robot's joint posture data. This method requires no human intervention, offers a higher degree of automation, and can generate massive amounts of training data.

[0096] It can be seen that the input of the dexterous manipulation large model in this disclosure uses more consistent information about the posture of the manipulation object and the environmental semantic point cloud information of the environment in which the manipulation object is located, which solves the migration barrier caused by the inconsistency between the simulation and the real environment brought about by direct RGB image input, making the transition of the model from simulation to real environment more natural. Massive and diverse data can be efficiently generated in the simulation environment, avoiding the high cost and low efficiency of data collection in the real environment. Training in a variety of simulation scenarios can significantly improve the model's ability to cope with complex tasks and scene changes.

[0097] In some embodiments of the present disclosure, before step S101 of acquiring image information, the control method further includes: understanding and decomposing the task by using an artificial intelligence agent and a multimodal large model through user input of task information, and using the analysis results to guide the robot operation process. Figure 7 The robot control method provided in the present disclosure is a flowchart of a specific implementation before step S101 of acquiring image information. The method specifically includes: S701: Obtain task information input by the user.

[0098] The user provides task information to the robot through an interface (such as voice commands or text input). Task information can be a natural language description, image instructions, or other forms of input. For example, "Help me grab the red cup on the table."

[0099] S702: Call the artificial intelligence agent and the multimodal large model to analyze and decompose the task information to obtain subtask instructions.

[0100] The user's high-level task information is broken down into specific subtasks that the robot can perform. The AI ​​agent is responsible for understanding the user's intent, invoking the multimodal big model, and coordinating the analysis of the task information. The multimodal big model combines multiple input forms (such as text, images, and environmental data) to perform task decomposition and reasoning. The task decomposition process breaks down the high-level task into a series of low-level subtask instructions. For example, the task information "Help me grab the red cup on the table" can be further broken down into: Subtask 1: Locate the red cup.

[0101] Subtask 2: Calculate the grasping path.

[0102] Subtask 3: Perform the grasping action.

[0103] S703: The step of obtaining image information includes: obtaining image information according to the subtask instruction.

[0104] Image information is acquired based on the subtask instruction. The image information includes the object being manipulated and the environment in which the object is located. For example, if the subtask instruction is "locate the red cup on the table," the image sensor acquires the required image information, including the red cup and the corresponding desktop area.

[0105] S704: Determining the input information of the dexterous operation large model further includes: determining the subtask instruction as one of the input information.

[0106] In this step, the subtask instructions are fed into the dexterous manipulation model along with other information as part of its input. Based on this input, the dexterous manipulation model generates operational decisions to help the robot perform its target task.

[0107] The subtask instructions can be language instructions. The input of subtask instructions allows the large model to flexibly operate according to different subtask instructions to complete corresponding tasks. This provides the model with clear goal guidance and reduces uncertainty in reasoning.

[0108] This embodiment analyzes and decomposes complex tasks into multiple subtasks, allowing the robot to execute these tasks one by one, thus avoiding confusion and errors when handling complex tasks. By incorporating subtask instructions into the input of the large dexterous manipulation model, image information or other input data can be adaptively selected and processed according to the specific requirements of the task, improving execution accuracy and adaptability.

[0109] In some embodiments of the present disclosure, the input information may also include 2D image information of the object being manipulated. This 2D image information can complement the lack of point cloud information, enabling flexible manipulation of large models to perform specific color and texture operations. For example, if there are multiple colored plates on a table and you want to pick up a specific color, you can combine the 2D image information for comprehensive processing.

[0110] The dexterous manipulation model outputs target action sequences based on these input information. These target action sequences are sent to the general motion control model, which drives the robot to track and execute the target action sequences while ensuring dynamic balance, collision avoidance, and high-precision closed-loop relative control to ultimately complete the task.

[0111] During model training, the disclosed embodiment adopts a strategy of multivariate data mixed training. Figure 8This disclosure provides a process diagram of the robot control method. The multivariate data specifically includes a collection of language instructions, human operation video data, real-machine teaching data, and simulated teaching data. The language instructions primarily describe the task, such as "get a bottle of yogurt from the refrigerator."

[0112] By jointly training real machine and simulation data taught by human experts, human operation video data and real robot operation feedback data, the generalization ability of the dexterous operation large model can be continuously improved.

[0113] In addition, the present disclosure also provides a robot control device 11, such as Figure 9 As shown in the structural block diagram of the robot control device provided in the present disclosure, the control device 11 includes: The image information acquisition module 901 is configured to acquire image information; wherein the image information includes an operation object and the environment in which the operation object is located.

[0114] The input information determination module 902 is configured to determine the input information of the dexterous operation large model; wherein, determining the input information of the dexterous operation large model includes: determining the input information based on image information, and the input information includes: the position and posture information of the operation object, and the environmental semantic point cloud information of the environment in which the operation object is located.

[0115] The output module 903 is configured to input the input information into the dexterous operation large model to obtain the target action sequence output by the dexterous operation large model.

[0116] The control module 904 is configured to control the robot to operate the operation object according to the target action sequence.

[0117] It can be understood that in the control device of the robot provided by the present disclosure: the image information acquisition module 901, the input information determination module 902, the output module 903 and the control module 904, the functions of each module correspond to the various steps in the above-mentioned robot control method. The specific implementation process can be referred to the relevant content in the previous text and will not be repeated here.

[0118] In some embodiments of the present disclosure, the input information determination module 902 includes: The three-dimensional point cloud information determining unit is configured to obtain three-dimensional point cloud information corresponding to the image information based on the image information.

[0119] The segmentation unit is configured to segment the three-dimensional point cloud information corresponding to the image information to obtain the three-dimensional point cloud information of the operation object and the environmental point cloud information of the environment in which the operation object is located.

[0120] The position and posture information determining unit of the operation object is configured to determine the position and posture information of the operation object according to the three-dimensional point cloud information of the operation object.

[0121] The environmental semantic point cloud information determining unit is configured to determine environmental semantic point cloud information based on the environmental point cloud information of the environment in which the operation object is located.

[0122] In some embodiments of the present disclosure, the segmentation unit includes: The binary mask determination subunit is configured to determine a binary mask of an operation object.

[0123] The target point cloud information and environment point cloud information determination subunit is configured to determine the target point cloud information of the operation object and the environment point cloud information of the environment in which the operation object is located based on the three-dimensional point cloud information corresponding to the image information and the binary mask of the operation object.

[0124] The position and posture information determining unit of the operation object is further configured to determine three main direction information and 3D position information of the operation object as the position and posture information of the operation object based on the target point cloud information of the operation object.

[0125] The environment semantic point cloud information determination unit is further configured to: perform semantic segmentation on the environment point cloud information to obtain the environment semantic point cloud information of the environment in which the operation object is located.

[0126] In some embodiments of the present disclosure, before the image information acquisition module 901 acquires image information, the control device 11 further includes: The robot teaching data acquisition module is configured to acquire robot teaching data. The robot teaching data includes at least one of human operation video data and expert teaching data. The human operation video data includes video data of humans operating sample objects, and the expert teaching data includes image data of the sample objects and robot joint posture data.

[0127] The training data generation module is configured to process the robot teaching data to obtain training data. The training data includes: position and posture information of the sample object, semantic point cloud information of the environment in which the sample object is located, and joint posture information of the robot.

[0128] The model training module is configured to build a large model for smart operation and use training data to train the model.

[0129] The deployment module is configured to deploy the dexterous manipulation large model to the robot.

[0130] In some embodiments of the present disclosure, the training data generation module includes: The posture information and environment semantic point cloud information generating unit is configured to generate posture information of the sample object and environment semantic point cloud information of the sample object based on human operation video data.

[0131] The robot's joint posture information generation unit is configured to extract human posture information from human operation video data, map the human posture information to the robot, and generate the robot's joint posture information.

[0132] In some embodiments of the present disclosure, the robot teaching data acquisition module is further configured to: acquire real machine teaching data during remote operation of a sample object; wherein, the image data of the sample object is obtained through an image sensor arranged on the robot, and the joint posture data of the robot includes the joint posture data of the robot during remote operation of the sample object.

[0133] In some embodiments of the present disclosure, before the robot teaching data acquisition module acquires the robot teaching data, the control device 11 further includes: The simulation model and simulation task scene creation module is configured to create a simulation model and simulation task scene of the robot. The simulation task scene includes a simulation operation object and a simulation environment in which the simulation operation object is located.

[0134] The robot teaching data acquisition module is also configured to: acquire simulation teaching data during the process of the simulation model operating the simulation operation object; wherein, the image data of the sample object is generated through simulation, and the robot's joint posture data includes the joint posture data of the simulation model during the process of operating the simulation operation object.

[0135] In some embodiments of the present disclosure, before the robot teaching data acquisition module acquires the robot teaching data, the control device 11 further includes: The task information acquisition module 905 is configured to acquire task information input by the user.

[0136] The task analysis and decomposition module 906 is configured to call the artificial intelligence agent and the multimodal large model to analyze and decompose the task information to obtain subtask instructions.

[0137] The image information acquisition module 901 is further configured to acquire image information according to the subtask instruction.

[0138] The input information determination module 902 is further configured to determine a subtask instruction as one of the input information.

[0139] In addition, the present disclosure also provides a robot control system 1, such as Figure 10As shown in the block diagram of the robot control system provided in the present disclosure, the robot control system 1 includes: a robot control device 11, and a cloud server 12. The control device 11 is the control device 11 described above, and the cloud server 12 is deployed with an artificial intelligence agent and a multimodal large model.

[0140] Reference Figure 11 The hierarchical end-to-end model architecture diagram of the robot control system provided by the present disclosure, the cloud server 12 is deployed with an artificial intelligence agent and a multimodal large model to realize multimodal human-computer interaction, intention understanding and task decomposition planning. The dexterous operation large model adopted by the robot's control device 11 is based on a large language model and is trained with multivariate mixed data to realize the robot's full-body motion trajectory planning at the skill level. The multimodal large model and the dexterous operation large model are seamlessly and deeply coordinated through an embodied agent (AI Agent) using natural language as an interface. The robot's control device 11 adopts a universal motion control model to drive the physical robot to track and execute the skill action trajectory while ensuring dynamic balance, collision avoidance, and high-precision closed-loop relative control to ultimately complete the task.

[0141] It can be seen that the control system provided by the present disclosure adopts a layered end-to-end model, placing the advanced perception and complex cognitive decision-making parts with low computing power requirements and high inference frequency requirements on the cloud server 12, and placing skill motion planning and high-frequency perception closed-loop control on the robot body, which can achieve reasonable distribution of computing power while ensuring high-precision perception and high-frequency closed-loop response.

[0142] In addition, the present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned robot control method when executed by a processor.

[0143] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0144] Professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0145] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0146] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present disclosure in detail. It should be understood that the above description is only a specific implementation method of the present disclosure and is not intended to limit the scope of protection of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A robot control method, characterized in that: include: Acquire image information; wherein the image information includes an operation object and an environment in which the operation object is located; Determine input information of the dexterous operation large model; wherein, the determining the input information of the dexterous operation large model comprises: determining the input information based on the image information, the input information comprising: the position and posture information of the operation object, and the environmental semantic point cloud information of the environment in which the operation object is located; Inputting the input information into the dexterous operation large model to obtain a target action sequence output by the dexterous operation large model; The robot is controlled to operate the operation object according to the target action sequence.

2. The robot control method according to claim 1, wherein: The step of determining the input information of the smart operation large model includes: Based on the image information, obtaining three-dimensional point cloud information corresponding to the image information; Segmenting the three-dimensional point cloud information corresponding to the image information to obtain the three-dimensional point cloud information of the operation object and the environmental point cloud information of the environment in which the operation object is located; Determining the position and posture information of the operation object according to the three-dimensional point cloud information of the operation object; The environmental semantic point cloud information is determined according to the environmental point cloud information of the environment in which the operation object is located.

3. The robot control method according to claim 2, wherein: The step of segmenting the three-dimensional point cloud information corresponding to the image information to obtain the three-dimensional point cloud information of the operation object and the environmental point cloud information of the environment where the operation object is located includes: Determining a binary mask of the operation object; Determine target point cloud information of the operation object and environmental point cloud information of the environment in which the operation object is located based on the three-dimensional point cloud information corresponding to the image information and the binary mask of the operation object; The step of determining the position and posture information of the operation object according to the three-dimensional point cloud information of the operation object comprises: determining three main direction information and 3D position information of the operation object as the position and posture information of the operation object based on the target point cloud information of the operation object; The step of determining the environmental semantic point cloud information according to the environmental point cloud information of the environment in which the operation object is located includes: performing semantic segmentation on the environmental point cloud information to obtain the environmental semantic point cloud information of the environment in which the operation object is located.

4. The control method according to claim 1, wherein: Before the step of acquiring image information, the method further includes: Acquire robot teaching data; wherein the robot teaching data includes at least one of human operation video data and expert teaching data, the human operation video data includes video data of humans operating sample objects, and the expert teaching data includes image data of sample objects and joint posture data of the robot; Processing the robot teaching data to obtain training data; wherein the training data includes: position and posture information of the sample object, semantic point cloud information of the environment where the sample object is located, and joint posture information of the robot; Constructing the dexterous operation large model and using the training data to perform model training; Deploy the dexterous manipulation macromodel to the robot.

5. The robot control method according to claim 4, wherein: The step of processing the robot teaching data to obtain training data comprises: Based on the human operation video data, generating posture information of the sample object and semantic point cloud information of the environment where the sample object is located; Human posture information is extracted from the human operation video data, the human posture information is mapped to the robot, and joint posture information of the robot is generated.

6. The robot control method according to claim 4, wherein: The step of obtaining robot teaching data comprises: During the remote operation of the sample object, real machine teaching data is obtained; wherein, the image data of the sample object is obtained by an image sensor arranged on the robot, and the joint posture data of the robot includes the joint posture data of the robot during the remote operation of the sample object.

7. The robot control method according to claim 4, wherein: Before the step of obtaining robot teaching data, the control method further includes: Creating a simulation model and a simulation task scenario of the robot, wherein the simulation task scenario includes a simulation operation object and a simulation environment in which the simulation operation object is located; The step of obtaining robot teaching data, wherein the robot teaching data includes at least one of human operation video data and expert teaching data, comprises: obtaining simulation teaching data during the process in which the simulation model operates the simulation operation object; wherein the image data of the sample object is generated through simulation, and the joint posture data of the robot includes the joint posture data of the simulation model during the process in which the simulation model operates the simulation operation object.

8. The control method according to any one of claims 1 to 7, wherein: Before the step of acquiring image information, the control method further includes: Get the task information entered by the user; Calling artificial intelligence agents and multimodal large models to analyze and decompose the task information to obtain subtask instructions; The step of acquiring image information includes: acquiring image information according to the subtask instruction; The step of determining the input information of the dexterous operation large model further includes: determining the subtask instruction as one of the input information.

9. A robot control device, characterized in that: include: An image information acquisition module is configured to acquire image information; wherein the image information includes an operation object and an environment in which the operation object is located; The input information determination module is configured to determine the input information of the dexterous operation large model; wherein the determining the input information of the dexterous operation large model comprises: determining the input information based on the image information, the input information comprising: the position and posture information of the operation object, and the environmental semantic point cloud information of the environment where the operation object is located; An output module is configured to input the input information into the dexterous operation large model to obtain a target action sequence output by the dexterous operation large model; The control module is configured to control the robot to operate the operation object according to a target action sequence.

10. The control device according to claim 9, wherein: The control device also includes: A task information acquisition module is configured to acquire task information input by a user; The task analysis and decomposition module is configured to call the artificial intelligence agent and the multimodal large model to analyze and decompose the task information to obtain subtask instructions; The image information acquisition module is further configured to acquire image information according to the subtask instruction; The input information determination module is further configured to determine the subtask instruction as one of the input information.

11. A robot control system, characterized in that: include: The robot's control device and cloud server; Wherein, the control device is the control device as described in claim 10, and the cloud server is deployed with the artificial intelligence agent and the multimodal large model.

Images