A training method and device of a grabbing strategy model, a grabbing method, device and system

Abstract

The embodiment of the application belongs to the technical field of artificial intelligence, and relates to a training method of a grabbing strategy model, which comprises the following steps: obtaining operation text prompt information and current grabbing state data in a current grabbing state; taking the operation text prompt information and at least current image data as inputs of a preset expected grabbing model, and outputting a current expected grabbing posture; taking the operation text prompt information and the current grabbing state data as inputs of the grabbing strategy model, combining the current expected grabbing posture, and iteratively training the grabbing strategy model based on a deep reinforcement learning method until a termination condition is met. The application also provides a grabbing method and related devices and systems. The technical scheme adopted by the application can improve the universality of the grabbing strategy model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus and system for a grasping strategy model. Background Technology

[0002] In robotic grasping, grasping quality is typically used to evaluate grasping stability. Common evaluation methods include using visual recognition to determine success or failure, or using force and / or tactile sensors to detect the gripping strength, force distribution and magnitude, and the contact area between the fingers and the object to determine grasping stability. However, these evaluation methods suffer from the following problems:

[0003] 1. Currently, many evaluation methods are based on single sensor judgments, but these methods may not provide a comprehensive assessment of grasping quality. For example, vision-based methods cannot make accurate judgments under occlusion conditions, and force and tactile sensors usually only provide local information and cannot provide a good indicator of the overall stability of grasping.

[0004] 2. Stability is not the only consideration during robot grasping. The grasping evaluation criteria should be based on the operability of the object after grasping, because the quality of the grasp will affect the stability of subsequent robot operations.

[0005] 3. Currently, these methods also lack good universality and cannot provide different evaluation criteria based on different crawling scenarios and methods. Therefore, during the training process, these methods are unlikely to help train a good and universal crawling strategy. Summary of the Invention

[0006] The purpose of this application is to provide a training method, apparatus and system for a crawling strategy model, so as to improve the versatility of the crawling strategy model.

[0007] In a first aspect, embodiments of this application provide a method for training a crawling strategy model, including the following technical solutions:

[0008] A training method for a grasping strategy model, applied to a grasping training system, includes: a robot, state data acquisition sensors, and a controller; the robot includes: a robotic arm and a grasping actuator disposed at the grasping execution end of the robotic arm; the state data acquisition sensors include an image sensor, a force / tactile sensor disposed on the grasping actuator, and a joint encoder; the force / tactile sensor is disposed on the grasping actuator, and the method includes the following steps:

[0009] The system acquires operation text prompts and current grasping state data. The current grasping state data includes: current image data output by the image sensor; current force data output by the force / tactile sensor; and the robot's current grasping posture calculated based on the joint motion output by the robot's encoder.

[0010] Using the operation text prompt information and at least the current image data as input to the preset desired grasping model, the current desired grasping posture is output.

[0011] The operation text prompts and the current grasping state data are used as inputs to the grasping strategy model. Combined with the current desired grasping posture, the grasping strategy model is iteratively trained based on a deep reinforcement learning method until the termination condition is met.

[0012] Furthermore, in one embodiment, the current desired grasping posture includes: desired force data in the grasping state and the robot's desired grasping posture in the grasping state.

[0013] Furthermore, in one embodiment, the step of using the operation text prompt information and the current grasping state data as input to the grasping strategy model, and combining the current desired grasping posture, to iteratively train the grasping strategy model based on a deep reinforcement learning method until a termination condition is met includes the following steps:

[0014] Using the operation text prompts and the current grasping state data as inputs to the grasping strategy model, the robot's grasping posture at the next moment is output.

[0015] Calculate the attitude difference between the next grasping attitude and the current desired grasping attitude; calculate the force difference between the current force data and the desired force data; calculate the corresponding reward function based on the attitude difference and the force difference;

[0016] Based on the reward function, optimize the crawling strategy model;

[0017] Based on the grasping posture at the next moment, a grasping command is generated to instruct the robot to perform a grasping motion.

[0018] Repeat the process of acquiring operation text prompts and current grasping state data under the current grasping state; use the operation text prompts and at least the current image data as input to a preset desired grasping model; use the operation text prompts and the current grasping state data as input to the grasping strategy model, and output the robot's next grasping posture; calculate the posture difference between the next grasping posture and the current desired grasping posture; calculate the force difference between the current force data and the desired force data; calculate the corresponding reward function based on the posture difference and the force difference; optimize the grasping strategy model based on the reward function; generate a grasping command based on the next grasping posture, and instruct the robot to execute grasping motion steps through the grasping command until the termination condition is met.

[0019] Furthermore, in one embodiment, the current grasping posture includes: the robot's end-effector pose information at the current moment and the grasping actuator's posture data at the current moment; and / or,

[0020] The next grasping posture includes: the robot's end-effector pose information and the grasping actuator's posture data at the next moment;

[0021] The desired grasping posture includes: the desired end-effector pose information of the robot at the current moment and the desired posture data of the grasping actuator at the current moment.

[0022] Furthermore, in one embodiment, before using the operation text prompt information and at least the current image data as input to a preset desired grasping model and outputting the current desired grasping posture, the following steps are further included:

[0023] Acquire training samples; the training samples include: the operation text prompt information and the corresponding training target image data in the grasping state; use the training grasping posture data corresponding to the operation text prompt information as the annotation of the training samples; the training grasping posture data includes the training force data in the grasping state and the training grasping posture of the robot in the grasping state.

[0024] Based on the training samples and the corresponding annotations, a desired crawling model is trained to obtain the preset desired crawling model.

[0025] Furthermore, in one embodiment, before obtaining the training samples, the following steps are also included:

[0026] Based on the operation text prompts, the corresponding annotations and training target image data in the grasping state are generated.

[0027] Secondly, embodiments of this application provide a crawling method for a crawling strategy model obtained by the training method of the crawling strategy model described above, the method comprising the following steps:

[0028] The operation text prompt information and the current grasping state data are obtained; the current grasping state data includes: the current image data output by the image sensor; the current force data output by the force / tactile sensor; and the robot's current grasping posture obtained based on the joint motion output by the robot's encoder.

[0029] Obtain the crawling strategy model;

[0030] Using the operation text prompts and the current grasping posture as inputs to the grasping strategy model, the robot's next grasping posture is output.

[0031] Based on the grasping posture at the next moment, a grasping command is generated to instruct the robot to grasp the target object.

[0032] Repeat the above steps until the target object is successfully grasped.

[0033] Thirdly, embodiments of this application provide a training apparatus for a crawling strategy model, the apparatus comprising:

[0034] The data acquisition module is used to acquire operation text prompts and current grasping state data under the current grasping state; the current grasping state data includes: current image data output by the image sensor; current force data output by the force / tactile sensor; and the robot's current grasping posture calculated based on the joint motion output by the robot's encoder.

[0035] The expectation generation module is used to take the operation text prompt information and at least the current image data as input to a preset expectation grasping model, and output the current expectation grasping posture.

[0036] The model training module is used to take the operation text prompt information and the current grasping state data as input to the grasping strategy model, and combine them with the current expected grasping posture to iteratively train the grasping strategy model based on a deep reinforcement learning method until the termination condition is met.

[0037] Fourthly, embodiments of this application provide a crawling device for a crawling strategy model trained based on the above-described crawling strategy model training device, the device comprising:

[0038] The information acquisition module is used by the text acquisition module to acquire the operation text prompt information and the current grasping state data; the current grasping state data includes: the current image data output by the image sensor; the current force data output by the force / tactile sensor; and the robot's current grasping posture calculated based on the joint motion output by the robot's encoder.

[0039] The model acquisition module is used to acquire the crawling strategy model;

[0040] The posture generation module is used to take the operation text prompt information and the current grasping state data as input to the grasping strategy model, and output the robot's grasping posture at the next moment.

[0041] The instruction generation module is used to generate a grasping instruction based on the grasping posture at the next moment, so as to instruct the robot to grasp the target object through the grasping instruction;

[0042] The step repeat module is used to repeat the above steps until the target object is successfully grasped.

[0043] Fifthly, embodiments of this application provide a grasping system, the system comprising: a robot, a status data acquisition sensor, and a controller; the robot comprising: a robotic arm and a grasping actuator disposed at the grasping execution end of the robotic arm;

[0044] The controller is communicatively connected to both the robot and the status data acquisition sensor.

[0045] The controller is used to implement the training method of the grasping strategy model described above and / or the grasping method described above.

[0046] Compared with the prior art, the embodiments of this application have the following main advantages:

[0047] This application uses multi-dimensional operation text prompts and some current grasping state data as input to a preset expected grasping model, which can output the current expected grasping posture (the preset expected grasping model is a pre-trained expected grasping model, i.e., the current expected grasping posture is obtained based on learning); then, the operation text prompts and current grasping state data are used as input to the grasping strategy model, combined with the current expected grasping posture, and the grasping strategy model is iteratively trained based on a deep reinforcement learning method. This application uses learning to obtain the corresponding expected grasping posture, and calculates the corresponding reward function based on it, which can optimize and obtain a more general grasping strategy model. Attached Figure Description

[0048] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a system architecture diagram of one embodiment of the application system of this application;

[0050] Figure 2 This is a system architecture diagram of one embodiment of the teleoperation-based training sample acquisition system of this application;

[0051] Figure 3 This is a schematic diagram of the structure of an embodiment of the training device for the grasping strategy model of this application;

[0052] Figure 4 This is a flowchart illustrating an embodiment of the training method for the crawling strategy model of this application;

[0053] Figure 5 This is a flowchart illustrating one embodiment of the crawling method of this application;

[0054] Figure 6 This is a schematic diagram of the structure of an embodiment of the training device for the grasping strategy model of this application;

[0055] Figure 7 This is a schematic diagram of the structure of one embodiment of the gripping device of this application;

[0056] Figure 8 This is a schematic diagram of the structure of one embodiment of the computer device of this application. Detailed Implementation

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0058] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0059] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0060] like Figure 1 As shown, Figure 1 This is a system architecture diagram of one embodiment of the application system of this application.

[0061] In one embodiment, this application provides a grasping system, which includes a robot 110, a status data sensor 120, and a controller 130.

[0062] robot

[0063] Specifically, robot 110 may be, but is not limited to: humanoid robot; or robotic arms connected in series or in parallel (e.g., Delta robotic arms, four-axis robotic arms, or six-axis robotic arms).

[0064] The robot includes: a robotic arm and a gripping actuator 111 disposed on the gripping end of the robotic arm; the gripping actuator (e.g., a dexterous hand or gripper).

[0065] Status data sensor

[0066] In one embodiment, the state data sensor 120 may include, but is not limited to, an image sensor 121, a force / tactile sensor 122 disposed on the gripper actuator, and a joint encoder. Other sensors may also be used as needed.

[0067] By employing the aforementioned state data sensor, the embodiments of this application can acquire multi-dimensional state data from three aspects: vision, touch, and robotics, thereby improving the versatility of subsequent models.

[0068] The image sensor 121 can be fixed to the robot (e.g., to the end joint of the robotic arm) or fixed to a preset position outside the robot as needed.

[0069] Typically, there is a pre-defined eye-hand calibration relationship between the image sensor and the robot, which allows us to obtain the coordinate system transformation relationship between the image sensor and the robot.

[0070] Specifically, an image sensor can be any existing or future image acquisition device, such as an RGB camera, a depth camera, a point cloud camera, a camcorder, or any device that includes such a camera or camcorder.

[0071] Force / tactile sensor 122 is disposed on gripper 111.

[0072] The aforementioned force / tactile sensors refer to force sensors and / or tactile sensors.

[0073] Specifically, the force sensor may be, but is not limited to, a two-dimensional or multi-dimensional force sensor used to measure two-dimensional or multi-dimensional force data.

[0074] Specifically, tactile sensors are a type of sensing device that can be placed at the end of an actuator such as a robotic arm and / or robot to measure tactile information while working with the end effector to perform a grasping function.

[0075] To enable the grasping of objects of different shapes and softness, the contact surface between the tactile sensor and the object is typically flexible and has good resilience. Tactile information includes, but is not limited to: arrayed multidimensional force information, surface deformation information, temperature information, and texture information.

[0076] The implementation of a tactile sensor includes a flexible contact surface, sensing circuitry, computing devices, and tactile signal parsing algorithms.

[0077] It should be noted that, in this application embodiment, the signals measured and output by the force / tactile sensor are collectively referred to as force data. Furthermore, for ease of understanding, the following description of this application embodiment mainly uses a tactile sensor as an example.

[0078] Specifically, the force / tactile sensor is pre-calibrated so that the pose conversion relationship between the force / tactile sensor and the end effector can be obtained.

[0079] For example, taking a humanoid robot as an example, the robot's dual robotic arms have dexterous hands equipped with tactile sensors at their ends, with force / tactile sensors distributed in each finger and palm of the dexterous hand; or as... Figure 1 As shown, a gripper 111 is provided at the end of the robotic arm of robot 110, and force / tactile sensors are distributed on the contact surface between the gripper 111 and the object.

[0080] Joint encoders (not shown in the figure) are installed at the robot's motion joints (such as the robotic arm's power-off and finger joints) to collect motion data of the robot's joints.

[0081] controller

[0082] The controller 130 is connected to the robot 110 and the state data sensor 120 via wired or wireless means. For limitations of the controller, please refer to the description of the training method and / or grasping method for the grasping strategy model in the later embodiments.

[0083] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future wireless connection methods.

[0084] The controller in this application embodiment may be, but is not limited to: a computer terminal (PC); an industrial personal computer terminal (IPC); a mobile terminal; a server; a system including a terminal and a server, implemented through interaction between the terminal and the server; a programmable logic controller (PLC); a field-programmable gate array (FPGA); a digital signal processor (DSP); or a microcontroller unit (MCU). The controller generates program instructions based on a pre-defined program, combined with data output from the robot and force / tactile sensors, etc. Exemplarily, it can be applied to, for example,... Figure 8 The computer equipment shown.

[0085] The controller 130 described in this application embodiment can be a standalone controller, or it can be fully or partially integrated into the robot and state data sensors, etc., and this application does not limit it.

[0086] It should be noted that, for the training of the grasping strategy model, the robot and state data sensor mentioned above can be real devices or virtual devices in a virtual environment. When they are virtual devices, the controller can first complete the training of the grasping strategy model based on the virtual devices, and then port it to the real training environment.

[0087] like Figure 2 As shown, Figure 2 This is a system architecture diagram of one embodiment of the teleoperation-based training sample acquisition system of this application.

[0088] In one embodiment, this application provides a teleoperation-based training sample acquisition system 200, which includes: a master-end interactive device 210, a slave-end robot 220, a slave-end image sensor 230, a force / tactile sensor 240, and a controller 250.

[0089] Main Interactor

[0090] In one embodiment, the host interactor 210 is used to collect action data of the operator during the execution of the target operation and send the action data to the controller 250.

[0091] The master interactor 210 may be, but is not limited to, a master sensor, an actuator body for setting the master sensor, etc.

[0092] Specifically, the master sensor can be any sensor capable of acquiring motion data, such as an IMU, image sensor, position encoder, or biochemical sensor (e.g., an electromyography slave sensor). The IMU, or inertial measurement unit, is used to measure motion data related to the operator, such as acceleration and angular velocity.

[0093] In one embodiment, the aforementioned master-end interactor can be directly fixed to a preset key part of the operator. Alternatively, the master-end interactor can be preset at the execution end of the actuator body, allowing the operator to control the movement of the actuator body based on subjective operation, thereby collecting the operator's motion data through a master-end sensor located on the actuator body.

[0094] In another embodiment, the master-end interactor can be pre-set on the actuator body (e.g., master robot, wearable device (e.g., exoskeleton or data glove)), and the actuator body can be moved based on the operator's subjective operation, thereby collecting the operator's motion data through the master-end sensor set on the actuator body.

[0095] In one embodiment, the master interactive device and the operator have a preset calibration relationship, which allows the operator's movement to be reflected directly or indirectly based on the motion data collected by the master interactive device.

[0096] For example, taking an arm exoskeleton as an example, multiple links constitute the actuator body. An IMU can be set at the position of the arm joint corresponding to the actuator body. The arm exoskeleton is worn on the operator's arm, so that the motion data of the corresponding joints during the operator's arm movement can be collected through the IMU.

[0097] It should be noted that the operator described in the embodiments of this application is not limited to a human being, and may also be other living beings as needed. For ease of understanding, the embodiments of this application mainly use a human being as an example for explanation.

[0098] Slave robot

[0099] Specifically, the end-user robot may be, but is not limited to, a humanoid robot; or a robotic arm connected in series or in parallel (e.g., a Delta manipulator, a four-axis manipulator, or a six-axis manipulator).

[0100] Force / tactile sensor 240 is installed at the actuator end of the slave robot 220.

[0101] In one embodiment, the force / tactile sensor can be placed in the gripper or dexterous hand of the robot actuator. For example, Figure 2 As shown, taking a humanoid robot as an example, the robot's two arm ends (i.e., the execution ends) are equipped with dexterous hands 221 with tactile sensors, and force / tactile sensors 240 are distributed in each finger joint and palm of the dexterous hand.

[0102] From end image sensor

[0103] In one embodiment, the end image sensor 230 is used to acquire images of the object. The object's pose and the relative pose of the robot's operating parts (e.g., finger joints) to the object can be determined from the object images.

[0104] Typically, the image sensor 230 and the robot 220 have a preset calibration relationship, so that the object image acquired by the image sensor can be mapped to the robot.

[0105] In one embodiment, taking a robot as an example, the image sensor and the robot arm can be pre-calibrated through eye-hand calibration.

[0106] controller

[0107] The controller 250 communicates with the master interactor 210, the slave robot 220, and the slave image sensor 230 via wired or wireless means.

[0108] For details regarding the limitations of the controller, please refer to the description of the training method and / or grasping method of the grasping strategy model in the later embodiments, which will not be repeated here.

[0109] In another embodiment, the teleoperation-based training sample acquisition system further includes: Demo 240.

[0110] The controller is also configured to acquire object images captured by an end image sensor; generate a spatial virtual image of the object based on the object images; and send the virtual image to a demonstrator to display the spatial virtual image of the object to an operator via the demonstrator.

[0111] This application embodiment employs a teleoperation-based training sample acquisition system, using operation text information, along with corresponding image data and force data information, as training sample samples for the robot operation availability definition model, thereby obtaining more comprehensive and accurate training samples.

[0112] It should be noted that the training method for the crawling strategy model provided in this application embodiment is generally based on the above embodiments. Figure 1 The controller 130 of the grasping system described herein executes the operation, and correspondingly, the device for training the grasping strategy model is generally located in the controller 130 of the grasping training system.

[0113] like Figure 3 and 4 As shown, Figure 3 This is a schematic diagram of the structure of an embodiment of the training device for the grasping strategy model of this application; Figure 4 This is a flowchart illustrating an embodiment of the training method for the crawling strategy model of this application. This application provides a training method for a crawling strategy model, which may include the following steps:

[0114] Step 310: Obtain the operation text prompt information and the current grasping state data under the current grasping state; the current state data includes: the current image data output by the image sensor; the current force data output by the force / tactile sensor; and the robot's current grasping posture obtained based on the joint motion output by the robot's encoder.

[0115] Step 320 takes the operation text prompt information and at least the current image data as input to the preset expected grasping model, and outputs the current expected grasping posture.

[0116] Step 330 uses the operation text prompts and the current grasping state data as input to the grasping strategy model, and combines them with the current expected grasping posture to iteratively train the grasping strategy model based on deep reinforcement learning methods until the termination condition is met.

[0117] This application uses multi-dimensional operation text prompts and some current grasping state data as input to a preset expected grasping model, which can output the current expected grasping posture (the preset expected grasping model is a pre-trained expected grasping model, i.e., the current expected grasping posture is obtained based on learning); then, the operation text prompts and current grasping state data are used as input to the grasping strategy model, combined with the current expected grasping posture, and the grasping strategy model is iteratively trained based on a deep reinforcement learning method. This application uses learning to obtain the corresponding expected grasping posture, and calculates the corresponding reward function based on it, which can optimize and obtain a more general grasping strategy model.

[0118] To facilitate understanding, the above methods and steps will be explained in further detail below.

[0119] Step 310: Obtain the operation text prompt information and the current grasping state data under the current grasping state; the current grasping state data includes: the current image data output by the image sensor; the current force data output by the force / tactile sensor; and the robot's current grasping posture obtained based on the joint motion data output by the robot's encoder.

[0120] The operation text prompts can be instructions for performing different operation tasks on the target object. For example, the operation text prompts for a cup can be, but are not limited to, prompts for pouring water, filling with water, placing and / or cleaning. In this embodiment of the application, the operation text prompts can describe the specific content of prompting the operator to perform the corresponding operation tasks in a more intuitive way.

[0121] Object information can be a description of the object's type, shape, and / or material. For example, object information can be the name of the object's type, such as: cup.

[0122] In one embodiment, the controller responds to a manually or automatically generated grasping start command, retrieves operation text prompt information from a memory or server at a preset address (the operation text prompt information can be obtained based on operator input, selected from preset options, or obtained in any other necessary manner), and retrieves the current grasping status data output by the status data sensor at the current moment from the memory or server at a preset address. This current grasping status data is the status data at the current moment during the grasping attempt.

[0123] In one embodiment, based on the above embodiments, the state data sensor 120 may include, but is not limited to: an image sensor 121, a force / tactile sensor 122 disposed on the gripper actuator, and a joint encoder.

[0124] In one embodiment, the controller retrieves the current image data output by the image sensor from the memory or server according to a preset address;

[0125] In one embodiment, the controller retrieves current force data regarding the force distribution on the gripping surface of the gripping actuator, output by a force / tactile sensor, from a memory or server according to a preset address; and,

[0126] In one embodiment, the controller retrieves the current joint motion data output by the joint encoder from the memory or server according to a preset address, and the current grasping posture of the robot can be determined based on the joint motion data.

[0127] It should be noted that the current grasping posture of the robot mentioned above may include, but is not limited to, the pose information of the end effector joints of the robotic arm and the posture information of the grasping actuator. For example, the end effector posture of the robotic arm can be obtained based on the motion of each joint and combined with kinematic equations; the posture information of the grasping actuator, such as the contact point or angle of the finger relative to the target object, can be obtained based on the motion of each joint of the finger.

[0128] By employing the aforementioned state data sensor, the embodiments of this application can acquire multi-dimensional state data from three aspects: vision, touch, and robotics, thereby improving the versatility of subsequent models.

[0129] It should be noted that the "current capture status data" described in the embodiments of this application is not limited to the data listed in the above embodiments, and may also include other required status data as needed, all of which fall within the scope of protection of this application.

[0130] Step 320 takes the operation text prompt information and at least the current image data as input to the preset expected grasping model, and outputs the current expected grasping posture.

[0131] In one embodiment, the controller retrieves a pre-trained, preset desired grasping model from a memory or server according to a preset address; it takes the operation text prompt information and at least the current image data as input to the preset desired grasping model, and then outputs the current desired grasping posture.

[0132] In one embodiment, the aforementioned current desired grasping posture may include, but is not limited to: desired force data in the grasping state and desired robot posture data in the grasping state.

[0133] It should be noted that the "current desired grasping posture of the robot" mentioned above may include, but is not limited to, the desired pose information of the robot's end effector and the desired posture data of the grasping actuator when grasping an object.

[0134] The training process for the preset expected grasping model will be described in further detail in later embodiments.

[0135] It should be noted that, in addition to the operation text prompts and current image data, other current state data, such as current force data and the robot's current grasping posture, can also be used as input to the desired grasping posture model along with the operation text prompts and current image data, as needed. In this embodiment, the operation text prompts primarily aim to more accurately describe the grasping target. The image data mainly enables the robot to understand the scene, including information such as the target object, its location, and obstacles, thereby better completing the grasping of the target object.

[0136] Step 330 uses the operation text prompts and the current grasping state data as input to the grasping strategy model, and combines them with the current expected grasping posture to iteratively train the grasping strategy model based on deep reinforcement learning methods until the termination condition is met.

[0137] It should be noted that the crawling strategy model described in this application embodiment can adopt various existing or future network models, such as: Feed-Forward Networks, RNN, LSTM, Transformer, GNN, GAN, AE, Convolutional Neural Network (CNN). Common CNN models may include, but are not limited to: LeNet, AlexNet, ZFNet, VGG, GoogLeNet, Residual Net, DenseNet, R-CNN, SPP-NET, Fast-RCNN, Faster-RCNN, FCN, Mask-RCNN, YOLO, SSD, GCN.

[0138] In one embodiment, step 330 may include the following method steps:

[0139] Step 331 uses the operation text prompt information and the current grasping state data as input to the grasping strategy model, and outputs the robot's grasping posture at the next moment.

[0140] In one embodiment, the next-moment grasping posture includes, but is not limited to: the robot's end-effector pose information and the grasping actuator's posture data at the next moment.

[0141] Step 332: Calculate the attitude difference between the next grasping attitude and the current expected grasping attitude; calculate the force difference between the current force data and the expected force data; calculate the corresponding reward function based on the attitude difference and force difference.

[0142] This application embodiment combines posture difference and force difference to calculate the reward function, so that the grasping posture of the grasping strategy model output at the next moment is close to the current expected grasping posture, and the grasping posture at the next moment meets the requirement that the current force data and the expected force data are distributed as consistent as possible. Thus, the grasping posture at the next moment can be limited from multiple dimensions, thereby improving the versatility of the grasping strategy model.

[0143] Specifically, the reward function can be set arbitrarily as needed. For example, the reward function can be defined as follows: the smaller the difference, the greater the reward.

[0144] Step 333 optimizes the crawling strategy model based on the reward function.

[0145] Step 334 generates a grasping command based on the grasping posture at the next moment to instruct the robot to perform the grasping motion.

[0146] In one embodiment, the controller generates a grasping instruction based on the grasping posture at the next moment to instruct the robot to perform grasping motion, thereby applying the grasping posture at the next moment to a real or virtual physical environment, and then acquiring the next frame of state data output by the state sensor, using the next frame of state data as the current state data in step 331 above.

[0147] Step 335: Repeat the steps 310 to 334 above until the termination condition is met.

[0148] Specifically, the preset termination condition can be the highest number of iterations or the difference reaching a preset threshold.

[0149] In this embodiment, when learning an operation task (e.g., grasping an object) using reinforcement learning, a reward function is required. This reward function needs to evaluate the different grasping strategies attempted by the robot. The quality of the reward function affects the final learned grasping effect. This reward function needs to define the optimal grasping posture for the finger and the object. The definition of this optimal posture should be given according to different operation scenarios and objects, and it also needs to consider factors such as the contact point, force, and stability between the finger and the object. However, designing such a reward function is very difficult and complex. Therefore, this solution proposes to calculate the corresponding reward using a learning method, thereby reducing the difficulty of model training and improving the versatility of the grasping strategy model.

[0150] This application uses multi-dimensional operation text prompts and some current grasping state data as input to a preset expected grasping model, which can output the current expected grasping posture (the preset expected grasping model is a pre-trained expected grasping model, i.e., the current expected grasping posture is obtained based on learning); then, the operation text prompts and current grasping state data are used as input to the grasping strategy model, combined with the current expected grasping posture, and the grasping strategy model is iteratively trained based on a deep reinforcement learning method. This application uses learning to obtain the corresponding expected grasping posture, and calculates the corresponding reward function based on it, which can optimize and obtain a more general grasping strategy model.

[0151] In one embodiment, before step 320 uses the operation text prompt information and at least the current image data as input to the preset desired grasping model, the following steps may also be included to train the desired grasping model.

[0152] Step 340: Obtain training samples; training samples include: operation text prompts and target image data in the grasping state; use the training grasping state data corresponding to the operation text prompts as the annotation of the training samples; training grasping state data includes training force data in the grasping state and training robot grasping posture in the grasping state.

[0153] In one embodiment, the controller retrieves pre-generated training samples from the memory access server according to a preset address. The method for generating training samples can be found in the following embodiments.

[0154] Step 350: Based on the training samples and corresponding annotations, train the desired crawling model to obtain the preset desired crawling model.

[0155] In one embodiment, step 350 may include the following method steps:

[0156] Step 351 uses the current training sample as the input to the desired grasping model and outputs the desired grasping posture.

[0157] Step 352: Calculate the difference between the expected grasping posture and the annotation corresponding to the current training sample;

[0158] Step 353 updates the parameters of the desired capture model iteratively based on the difference until the preset training termination condition is met.

[0159] Specifically, the above training termination conditions can be set arbitrarily as needed, such as the difference meeting a preset range or meeting a preset number of iterations, etc.

[0160] The aforementioned expected grasping model can be any existing or future model capable of grasping pose recognition, such as: Feed-Forward Networks, RNN, LSTM, Transformer, GNN, GAN, AE, Convolutional Neural Networks (CNN). Common CNN models may include, but are not limited to: LeNet, AlexNet, ZFNet, VGG, GoogLeNet, Residual Net, DenseNet, R-CNN, SPP-NET, Fast-RCNN, Faster-RCNN, FCN, Mask-RCNN, YOLO, SSD, and GCN.

[0161] This application employs supervised learning to train the desired grasping model, thereby obtaining the desired grasping posture. Therefore, this desired grasping posture is universally applicable. Furthermore, training the desired grasping model using supervised learning is simpler than training using reinforcement learning.

[0162] In one embodiment, prior to step 340, the following method steps may be included to generate the training samples described above.

[0163] Step 360 generates corresponding annotation data and target image data in the capture state based on the operation text prompt information.

[0164] It should be noted that the process of "generating the corresponding desired grasping posture based on the operation text prompts" can be based on various existing or future methods and steps. For example, it can be done through teleoperation, where the operator is instructed to grasp and release an object based on the operation text prompts, thus obtaining the grasping posture. This will be explained in more detail in later embodiments. Alternatively, the temperature distribution of the object's surface can be obtained through a thermal imager, and the grasping posture can be obtained based on the temperature distribution. Continuing with the operation text prompt "pouring water + cup" as an example, the operator, corresponding to the task of pouring water and the target cup, will grasp the handle of the cup. At this time, the temperature of the operator's hand can be transferred to the object. Therefore, by recording the temperature distribution on the object's surface, it is possible to know how the object was grasped by the operator during the operation, thus obtaining the corresponding grasping posture of the operator's hand, which can then be used as the robot's grasping posture.

[0165] In one embodiment, step 360 may include the following method steps. These method steps are generally derived from the above embodiments. Figure 2 The controller 230 of the teleoperation-based training sample request system described herein executes the operation.

[0166] Step 361 sends an operation text prompt to the operator so that the operator can perform a grabbing operation on the target object based on the operation text prompt.

[0167] For example, based on the text prompt "pour water", the operator can observe the specific location of the water cup at the end with their naked eye, or watch a virtual image of the water cup in three-dimensional space shown by a demonstrator, and then subjectively perform the action of grabbing the water cup according to the content of the text prompt. Here, since the purpose is to pour water, the operator will usually subjectively grab the handle of the water cup, rather than the rim. In another example, when the text prompt is "wash the cup", the operator will usually grab the rim of the water cup.

[0168] Step 362: Obtain action data of the operator performing the target operation through the master-end interactive device; generate robot grasping instructions based on the action data to instruct the slave robot to follow and grasp the target object.

[0169] Step 363: After the slave robot completes the operation of following the target, acquire the grasping posture in the completed grasping state and the state data output by the slave state sensor.

[0170] In one embodiment, the grasping posture of the robot when grasping an object can be identified based on the target image data in the completed grasping state output from the end image sensor.

[0171] The aforementioned state data may include, but is not limited to: target image data in the grasping state output by the image sensor (this target image data, together with the operation text prompt information, can be used as training samples for the expected grasping model to train the expected grasping model); training force data in the grasping state output by the force / tactile sensor; and the robot's training grasping posture in the grasping state obtained based on the joint encoder. For the robot's training grasping posture data, please refer to the above embodiments, which will not be repeated here.

[0172] Step 364 uses the operation text prompts, the target image data in the completed grasping state, and the corresponding labeled data (training force data and the robot's training grasping posture in the grasping state) as training samples for the robot.

[0173] This application embodiment can quickly obtain operation text prompts, the corresponding robot grasping posture when grasping an object, and the image data under the grasping state by means of teleoperation. The operation text prompts are used as input, and the grasping posture and image data under the grasping state are used as annotations.

[0174] like Figure 4 As shown, Figure 4 This is a flowchart illustrating one embodiment of the crawling method of this application.

[0175] Based on the crawling strategy model trained by the crawling strategy model training method described in the above embodiments, this application also provides a crawling method, which may include the following method steps:

[0176] It should be noted that the capture method provided in this application embodiment is generally based on the above embodiments. Figure 1 The target object grasping system described herein is executed by the controller 130, and correspondingly, the device for the grasping method is generally provided in the controller 130.

[0177] Step 410: Obtain operation text prompts and current grasping status data. The current grasping status data includes: current image data output by the image sensor; current force data output by the force / tactile sensor; and the robot's current grasping posture calculated based on the joint motion output by the robot's encoder.

[0178] For a description of the current capture status data, please refer to the above embodiment, which will not be repeated here.

[0179] Step 420: Obtain the crawling strategy model.

[0180] Step 430 uses the operation text prompts and grasping status data as input to the grasping strategy model, and outputs the grasping posture and image data under the grasping status.

[0181] The grasping posture includes force data during the grasping state and the robot's grasping posture data.

[0182] Step 440 generates a grasping command based on the grasping posture, so as to instruct the robot to grasp the target object through the grasping command.

[0183] Step 450: Repeat the steps from 410 to 440 until the target object is successfully grasped.

[0184] In one embodiment, the success of object grasping can be determined based on a reward function. Specifically, the "operation text prompts" and "image data in the grasping state" from step 410 can be used as inputs to the aforementioned preset expected grasping model, outputting the expected grasping posture; the difference between the expected grasping posture and the grasping posture output in step 430 is calculated; a reward function is obtained based on the difference, and the success of grasping is determined based on the value of the reward function. For example, when the reward function is less than a preset threshold, the grasping is considered successful. This embodiment of the application, based on the expected grasping model, can not only complete the training of the grasping strategy model but also be used to determine whether the object grasping is successful during actual target grasping operations, thus further improving the versatility of grasping operations. In addition, methods such as collecting images or force data can also be used to determine whether the object grasping is successful, all of which fall within the scope of protection of this application.

[0185] In this embodiment, the grasping strategy model is input based on the operation text prompts and the image data captured and output by the image sensor at the current moment, and the grasping posture at the current moment is output. Steps 410 to 440 are repeated until the object is grasped. The current grasping posture obtained when the object is grasped is taken as the final grasping posture, and this grasping posture is sent to the robot to instruct the robot to complete the grasping of the object.

[0186] This application uses multi-dimensional operation text prompts and some current grasping state data as input to a preset expected grasping model, which can output the current expected grasping posture (the preset expected grasping model is a pre-trained expected grasping model, i.e., the current expected grasping posture is obtained through learning). Then, the operation text prompts and current grasping state data are used as input to the grasping strategy model. Combined with the current expected grasping posture, the grasping strategy model is iteratively trained based on a deep reinforcement learning method. This application uses a learning method to obtain the corresponding expected grasping posture and calculates the corresponding reward function based on it, which can optimize and obtain a more general grasping strategy model, thereby improving the versatility of the target object grasping method.

[0187] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0188] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0189] Further reference Figure 6 As a response to the above Figure 3 The implementation of the method shown in this application provides an embodiment of a training device for a grasping strategy model, which is similar to... Figure 3 Corresponding to the training method embodiment of the grasping strategy model shown, this device can be specifically applied to various controllers.

[0190] like Figure 6 As shown, the training device 300 for the crawling strategy model in this embodiment includes:

[0191] The data acquisition module 310 is used to acquire operation text prompt information and current grasping state data under the current grasping state; the current grasping state data includes: current image data output by the image sensor; current force data output by the force / tactile sensor; and the robot's current grasping posture obtained based on the joint motion output by the robot's encoder.

[0192] The expectation generation module 320 is used to take the operation text prompt information and at least the current image data as input to a preset expectation grasping model, and output the current expectation grasping posture.

[0193] The model training module 330 is used to take the operation text prompt information and the current grasping state data as input to the grasping strategy model, and combine them with the current expected grasping posture to iteratively train the grasping strategy model based on a deep reinforcement learning method until the termination condition is met.

[0194] Please refer to details. Figure 8 To address the aforementioned technical problems, this application also provides a controller (taking computer device 6 as an example).

[0195] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only the computer device 6 with components 61-63 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0196] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0197] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may include both the internal storage unit and its external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as the training of the crawling strategy model and / or the program code of the crawling method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.

[0198] In some embodiments, the processor 62 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run program code stored in the memory 61 or process data, for example, to run program code for training a crawling strategy model and / or a crawling method.

[0199] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 6 and other electronic devices.

[0200] This application also provides another embodiment, namely, providing a computer-readable storage medium storing a crawling strategy model training and / or crawling program, the crawling strategy model training and / or crawling program being executable by at least one processor to cause the at least one processor to perform the steps of the crawling strategy model training and / or crawling method as described above.

[0201] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0202] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A training method for a grasping strategy model, applied to a grasping training system, comprising: A robot, state data acquisition sensors, and a controller; the robot includes: a robotic arm and a gripping actuator disposed at the gripping end of the robotic arm; the state data acquisition sensors include an image sensor, a force and / or tactile sensor disposed on the gripping actuator, and a joint encoder; characterized in that the method includes the following steps: The system acquires operation text prompts and current grasping state data. The current grasping state data includes: current image data output by the image sensor; current force data output by the force and / or tactile sensor; and the robot's current grasping posture calculated based on the joint motion output by the robot's encoder. Using the operation text prompt information and at least the current image data as input to the preset desired grasping model, the current desired grasping posture is output. The operation text prompts and the current grasping state data are used as inputs to the grasping strategy model. Combined with the current desired grasping posture, the grasping strategy model is iteratively trained based on a deep reinforcement learning method until the termination condition is met. The current desired grasping posture includes: the desired force data in the grasping state and the robot's desired grasping posture in the grasping state. The step of using the operation text prompt information and the current grasping state data as input to the grasping strategy model, and combining the current desired grasping posture, to iteratively train the grasping strategy model based on a deep reinforcement learning method until the termination condition is met includes the following steps: Using the operation text prompts and the current grasping state data as inputs to the grasping strategy model, the robot's grasping posture at the next moment is output. Calculate the attitude difference between the next grasping attitude and the current desired grasping attitude; calculate the force difference between the current force data and the desired force data; calculate the corresponding reward function based on the attitude difference and the force difference; Based on the reward function, optimize the crawling strategy model; Based on the grasping posture at the next moment, a grasping command is generated to instruct the robot to perform a grasping motion.

2. The training method for the crawling strategy model according to claim 1, characterized in that, The step of using the operation text prompt information and the current grasping state data as input to the grasping strategy model, and combining the current desired grasping posture, to iteratively train the grasping strategy model based on a deep reinforcement learning method until the termination condition is met includes the following steps: Repeat the process of obtaining the operation text prompt information and the current grasping state data under the current grasping state; use the operation text prompt information and at least the current image data as input to the preset expected grasping model; use the operation text prompt information and the current grasping state data as input to the grasping strategy model, and output the robot's grasping posture at the next moment. The method involves: calculating the attitude difference between the next grasping posture and the current desired grasping posture; calculating the force difference between the current force data and the desired force data; calculating the corresponding reward function based on the attitude difference and the force difference; optimizing the grasping strategy model based on the reward function; and generating a grasping instruction based on the next grasping posture to instruct the robot to perform grasping motion steps until the termination condition is met.

3. The training method for the grasping strategy model according to claim 2, characterized in that, The current grasping posture includes: the robot's end effector pose information at the current moment and the grasping actuator's posture data at the current moment; and / or, The next grasping posture includes: the robot's end-effector pose information and the grasping actuator's posture data at the next moment; The desired grasping posture includes: the desired end-effector pose information of the robot at the current moment and the desired posture data of the grasping actuator at the current moment.

4. The training method for the grasping strategy model according to claim 1, characterized in that, Before the step of using the operation text prompt information and at least the current image data as input to the preset desired grasping model and outputting the current desired grasping posture, the following steps are also included: Acquire training samples; the training samples include: the operation text prompt information and the corresponding training target image data in the grasping state; use the training grasping posture data corresponding to the operation text prompt information as the annotation of the training samples; the training grasping posture data includes the training force data in the grasping state and the training grasping posture of the robot in the grasping state. Based on the training samples and the corresponding annotations, a desired crawling model is trained to obtain the preset desired crawling model.

5. The training method for the grasping strategy model according to claim 4, characterized in that, Before obtaining the training samples, the following steps are also included: Based on the operation text prompts, the corresponding annotations and training target image data in the grasping state are generated.

6. A grasping method based on the grasping strategy model obtained by the training method of any one of claims 1 to 5, characterized in that, The method includes the following steps: The operation text prompt information and the current grasping state data are obtained; the current grasping state data includes: the current image data output by the image sensor; the current force data output by the force and / or tactile sensor; and the robot's current grasping posture obtained based on the joint motion output by the robot's encoder. Obtain the crawling strategy model; Using the operation text prompts and the current grasping state data as inputs to the grasping strategy model, the robot's grasping posture at the next moment is output. Based on the grasping posture at the next moment, a grasping command is generated to instruct the robot to grasp the target object. Repeat the above steps until the target object is successfully grasped.

7. A training device for a grasping strategy model, characterized in that, The device includes: The data acquisition module is used to acquire operation text prompts and current grasping state data under the current grasping state; the current grasping state data includes: current image data output by the image sensor; current force data output by the force and / or tactile sensors set on the grasping actuator; and the robot's current grasping posture calculated based on the joint motion output by the robot's encoder. The expectation generation module is used to take the operation text prompt information and at least the current image data as input to a preset expectation grasping model, and output the current expectation grasping posture. The model training module is used to take the operation text prompt information and the current grasping state data as input to the grasping strategy model, and combine them with the current expected grasping posture to iteratively train the grasping strategy model based on a deep reinforcement learning method until the termination condition is met; the current expected grasping posture includes: expected force data in the grasping state and the robot's expected grasping posture in the grasping state. The step of using the operation text prompt information and the current grasping state data as input to the grasping strategy model, and combining the current desired grasping posture, to iteratively train the grasping strategy model based on a deep reinforcement learning method until the termination condition is met includes the following steps: Using the operation text prompts and the current grasping state data as inputs to the grasping strategy model, the robot's grasping posture at the next moment is output. Calculate the attitude difference between the next grasping attitude and the current desired grasping attitude; calculate the force difference between the current force data and the desired force data; calculate the corresponding reward function based on the attitude difference and the force difference; Based on the reward function, optimize the crawling strategy model; Based on the grasping posture at the next moment, a grasping command is generated to instruct the robot to perform a grasping motion.

8. A grasping device for obtaining the grasping strategy model trained based on the training device of the grasping strategy model according to claim 7, characterized in that, The device includes: The information acquisition module is used by the text acquisition module to acquire the operation text prompt information and the current grasping state data; the current grasping state data includes: current image data output by the image sensor; current force data output by the force and / or tactile sensor; and the robot's current grasping posture obtained based on the joint motion output by the robot's encoder. The model acquisition module is used to acquire the crawling strategy model; The posture generation module is used to take the operation text prompt information and the current grasping state data as input to the grasping strategy model, and output the robot's grasping posture at the next moment. The instruction generation module is used to generate a grasping instruction based on the grasping posture at the next moment, so as to instruct the robot to grasp the target object through the grasping instruction; The step repeat module is used to repeat the above steps until the target object is successfully grasped.

9. A grasping system, characterized in that, The system includes: a robot, status data acquisition sensors, and a controller; the robot includes: a robotic arm and a gripping actuator disposed at the gripping execution end of the robotic arm; The controller is communicatively connected to both the robot and the status data acquisition sensor. The controller is used to implement the training method of the grasping strategy model according to any one of claims 1 to 5 and / or the grasping method according to claim 6.

Images