Object grasping method, object grasping device, electronic device, and storage medium

Abstract

The application provides an object grasping method, an object grasping device, an electronic device and a storage medium, and belongs to the technical field of robot control. The first object state is determined by a preset neural network to determine an initial grasping action corresponding to the first object state. The first object state is updated according to the initial grasping action to obtain a second object state. The first and second object states are subjected to object dispersion degree evaluation to obtain first and second dispersion scores. The first reward data obtained by executing the initial grasping action is determined according to the first and second object states and the first and second dispersion scores. The preset neural network is subjected to parameter optimization according to the first reward data and preset reference reward data to obtain a grasping action generation network. The target object state is subjected to grasping action generation by the grasping action generation network to obtain a target grasping action. The object is grasped according to the target grasping action, and accurate object grasping can be realized in a cluttered scene.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to an object grasping method, an object grasping device, an electronic device, and a storage medium. Background Technology

[0002] Currently, robotic grasping technology is widely used in many scenarios, not only improving the efficiency of production and daily life but also reducing costs and manpower requirements. In manufacturing, robots can automate many tasks, such as grasping, assembling, and packaging parts on the production line. In warehousing and logistics, robots are used for automated grasping and handling tasks in warehouses and logistics. In the home, robots can be used for automated grasping and delivery of items or to perform household chores.

[0003] However, in cluttered scenarios, visual clutter can prevent robots from accurately detecting targets during the grasping process, leading to failures in grasping. Therefore, achieving accurate robot grasping in cluttered environments has become a pressing issue. Summary of the Invention

[0004] The main objective of this application is to provide an object grasping method, object grasping device, electronic device, and storage medium, which aims to enable robots to accurately grasp objects in cluttered scenes.

[0005] To achieve the above objectives, a first aspect of this application provides an object grasping method, the method comprising:

[0006] Get the state of the first object;

[0007] The initial grasping action corresponding to the state of the first object is determined by a preset neural network.

[0008] The state of the first object is updated based on the initial grasping action to obtain the state of the second object;

[0009] The first object state is evaluated for its discreteness to obtain a first discreteness score, and the second object state is evaluated for its discreteness to obtain a second discreteness score.

[0010] Based on the first object state, the second object state, the first discrete score, and the second discrete score, determine the first reward data obtained by performing the initial grasping action;

[0011] The preset neural network is trained by optimizing the parameters of the preset neural network based on the first reward data and the preset benchmark reward data to obtain the grasping action generation network.

[0012] The state of the target object is obtained, and the grasping action generation network is used to generate a grasping action from the state of the target object to obtain the target grasping action. The object is then grasped according to the target grasping action.

[0013] In some embodiments, the initial grasping action corresponding to the state of the first object is determined by a preset neural network, including:

[0014] Determine the set of grasping actions corresponding to the state of the first object; the set of grasping actions includes multiple grasping actions;

[0015] Obtain data on the behavior of the greedy strategy;

[0016] If the greedy strategy behavior data is less than or equal to the behavior threshold, then a grabbing action is randomly selected from the grabbing action set through the preset neural network to obtain the initial grabbing action.

[0017] In some embodiments, after acquiring the greedy strategy behavior data, the object grasping method further includes:

[0018] If the greedy strategy behavior data is greater than the behavior threshold, then the preset neural network is used to evaluate each grabbing action in the grabbing action set to obtain the degree of action execution of each grabbing action.

[0019] Based on the degree of execution of the action, a grabbing action is selected from the grabbing action set to obtain the initial grabbing action.

[0020] In some embodiments, the step of evaluating the object discreteness of the first object state to obtain a first discreteness score includes:

[0021] The first object state is segmented into instances to obtain a predicted detection box;

[0022] Calculate the intersection-union ratio (IUR) between the predicted detection box and the preset detection box;

[0023] The first object state is evaluated for its dispersion based on the intersection-union ratio (IUU) data to obtain the first dispersion score.

[0024] In some embodiments, determining the first reward data obtained by performing the initial grasping action based on the first object state, the second object state, the first dispersion score, and the second dispersion score includes:

[0025] Calculate the degree of change between the second object state and the first object state;

[0026] The difference between the first dispersion score and the second dispersion score is calculated to obtain the score deviation data;

[0027] The first reward data is determined based on the change level data and the scoring deviation data.

[0028] In some embodiments, optimizing the parameters of the preset neural network based on the first reward data and preset benchmark reward data to train the preset neural network and obtain a grasping action generation network includes:

[0029] Obtain the execution status of the initial grasping action, and determine the second reward data obtained by performing the initial grasping action based on the execution status;

[0030] Determine the intermediate grasping action corresponding to the second object state, and calculate the third reward data for executing the intermediate grasping action;

[0031] The first reward data, the second reward data, and the third reward data are summed to obtain the predicted reward data.

[0032] The target loss data is obtained based on the predicted reward data and the preset benchmark reward data;

[0033] The parameters of the preset neural network are optimized based on the target loss data to train the preset neural network and obtain the grasping action generation network.

[0034] In some embodiments, obtaining the target loss data based on the predicted reward data and the preset baseline reward data includes:

[0035] The predicted reward data and the preset benchmark reward data are subtracted to obtain the reward deviation data;

[0036] If the reward deviation data is less than or equal to a preset deviation threshold, then the target loss data is obtained based on the reward deviation data;

[0037] If the reward deviation data is greater than the preset deviation threshold, then the target loss data is obtained based on the reward deviation data and the preset deviation threshold.

[0038] To achieve the above objectives, a second aspect of this application provides an object grasping device, the device comprising:

[0039] The acquisition module is used to acquire the state of the first object;

[0040] The first determining module is used to determine the initial grasping action corresponding to the state of the first object through a preset neural network.

[0041] The state update module is used to update the state of the first object according to the initial grasping action to obtain the state of the second object;

[0042] The evaluation module is used to evaluate the object dispersion of the first object state to obtain a first dispersion score, and to evaluate the object dispersion of the second object state to obtain a second dispersion score.

[0043] The second determining module is used to determine the first reward data obtained by performing the initial grasping action based on the first object state, the second object state, the first discrete score, and the second discrete score.

[0044] The training module is used to optimize the parameters of the preset neural network based on the first reward data and the preset benchmark reward data, so as to train the preset neural network and obtain the grasping action generation network.

[0045] The object grasping module is used to obtain the state of the target object, generate grasping actions from the state of the target object through the grasping action generation network to obtain the target grasping action, and grasp the object according to the target grasping action.

[0046] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the object grasping method described in the first aspect.

[0047] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the object grasping method described in the first aspect.

[0048] The object grasping method, object grasping device, electronic device, and computer-readable storage medium proposed in this application consider the clutter level of the robot's working scene and characterize the clutter level of the scene by the dispersion of objects in the scene. The dispersion level of objects is quantified through a dispersion score, and reward data is obtained using the dispersion score. A preset neural network is trained based on the reward data to obtain a grasping action generation network optimized for dispersion. This grasping action generation network generates a target grasping action, and the object is grasped according to the grasping position and grasping angle represented by the target grasping action, enabling accurate object grasping even in cluttered scenes. Attached Figure Description

[0049] Figure 1 This is a flowchart of the object grasping method provided in the embodiments of this application;

[0050] Figure 2 yes Figure 1 The flowchart of step S120 in the middle;

[0051] Figure 3 This is another flowchart of the object grasping method provided in the embodiments of this application;

[0052] Figure 4 yes Figure 1 The flowchart of step S140 in the middle;

[0053] Figure 5 yes Figure 1 The flowchart of step S150 in the middle;

[0054] Figure 6 yes Figure 1 The flowchart of step S160 in the process;

[0055] Figure 7 yes Figure 6 The flowchart of step S640 in the middle;

[0056] Figure 8 This is a schematic diagram of the object grasping device provided in the embodiments of this application;

[0057] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0061] Currently, robotic grasping technology is widely used in many scenarios, not only improving the efficiency of production and daily life but also reducing costs and manpower requirements. In manufacturing, robots can automate many tasks, such as grasping, assembling, and packaging parts on the production line. In warehousing and logistics scenarios, robots are used for automated grasping and handling tasks in warehouses and logistics. In the home, robots can also be used to automate the grasping and delivery of items or perform household chores.

[0062] However, robots may encounter many challenges and drawbacks in cluttered environments. In some relatively cluttered scenes, robots often struggle to accurately detect targets during the grasping process. Visual clutter prevents the robot from accurately estimating the target's position and orientation, leading to grasping failure. Therefore, achieving accurate robot grasping in cluttered environments has become an urgent problem to be solved.

[0063] Based on this, embodiments of this application provide an object grasping method, an object grasping device, an electronic device, and a computer-readable storage medium, aiming to enable robots to accurately grasp objects in cluttered scenes and improve the accuracy of robot object grasping.

[0064] The object grasping method, object grasping device, electronic device, and computer-readable storage medium provided in this application are specifically described through the following embodiments. First, the object grasping method in the embodiments of this application is described.

[0065] The object grasping method provided in this application relates to the field of robot control technology. This object grasping method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the object grasping method, but is not limited to the above forms.

[0066] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0067] Figure 1 This is an optional flowchart of the object grasping method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S110 to S170.

[0068] Step S110: Obtain the state of the first object;

[0069] Step S120: Determine the initial grasping action corresponding to the state of the first object through a preset neural network;

[0070] Step S130: Update the state of the first object based on the initial grasping action to obtain the state of the second object;

[0071] Step S140: Evaluate the object dispersion of the first object state to obtain a first dispersion score; evaluate the object dispersion of the second object state to obtain a second dispersion score.

[0072] Step S150: Determine the first reward data obtained by performing the initial grasping action based on the first object state, the second object state, the first discrete score, and the second discrete score.

[0073] Step S160: Optimize the parameters of the preset neural network based on the first reward data and the preset benchmark reward data to train the preset neural network and obtain the grasping action generation network.

[0074] Step S170: Obtain the state of the target object, generate a grasping action from the state of the target object through a grasping action generation network, obtain the target grasping action, and grasp the object according to the target grasping action.

[0075] Steps S110 to S170, as illustrated in this embodiment, quantify the clutter level of the robot's work scene by scoring the dispersion of objects. Reward data is obtained based on the dispersion score, and a preset neural network is optimized using this reward data to obtain a dispersion-based grasping action generation network. This grasping action generation network generates a target grasping action, and the robot performs object grasping based on the target grasping action, thus improving the accuracy of object grasping in cluttered scenes.

[0076] In step S110 of some embodiments, a suitable robotic arm is selected, and an experimental platform based on robotic arm grasping is built in a real environment. The robotic arm can be a UR5E robotic arm. A ROS robot simulation platform for collaborative grasping is constructed in a simulation environment, and the simulation platform is highly consistent with the real platform. A communication connection is established between the server used for simulation and the real robotic arm, so that the robotic arm can execute the instructions sent by the server to grasp the target object. The experimental platform is equipped with a visual sensor, which can be a depth camera. The visual sensor is placed on a bracket in front of the robotic arm's workbench to collect relevant visual information.

[0077] This application addresses the problem of object grasping in cluttered scenes using deep reinforcement learning. To train the reinforcement learning model, a visual grasping dataset is first prepared, comprising multiple scene state images. Each scene state image is a set of RGB-D images, including color and depth images of the same size. Each scene state image has a corresponding object grasping label. The object grasping label includes information such as the object's position in the robot's working scene, the object's pose, and the robotic arm's grasping posture. The object's pose can be represented by its tilt angle relative to a reference point, which can be the origin of a Cartesian coordinate system established based on the robot's working scene. The robotic arm's grasping posture can be represented by the grasping position and angle of the tool central point (TCP) of the end effector. The visual grasping dataset is divided into training, validation, and test sets to evaluate model performance during training.

[0078] It should be noted that the process involves acquiring color and depth images of the current work scene using a depth camera, aligning the color and depth images to ensure they are the same size, and then merging them into an RGB-D image. The color and depth images can be 320×320 pixels, and the RGB-D image can be 224×224 pixels.

[0079] Scene state images are acquired from the training set, and features are extracted from these images using a deep convolutional neural network (DNN) to obtain state information such as object color, object edge contours, and object position. This state information is used as the first object state. The feature extraction process is an unsupervised feature learning process. The DNN includes convolutional layers, pooling layers, and fully connected layers. Convolutional layers extract features from the scene state images to obtain a state feature map. Pooling layers reduce the size of the state feature map. Fully connected layers perform feature mapping on the pooled state feature map, outputting the first object state. Each pixel in the object state feature map is mapped to the real space through intrinsic and extrinsic parameter matrices. The DNN can use a pre-trained network or be retrained. During training, the neural network optimizes model parameters by backpropagating errors to improve model accuracy. Since the scene state image is composed of a color image and a depth image, the first object state can be represented as s = (s... RGB ,s D ), where s RGB For the features of a color image, s D Features of the depth image. To improve the stability and convergence speed of the model, batch normalization and residual connections can be introduced into the network layers.

[0080] Please see Figure 2 In some embodiments, step S120 may include, but is not limited to, steps S210 to S230:

[0081] Step S210: Determine the set of grasping actions corresponding to the state of the first object; the set of grasping actions includes multiple grasping actions;

[0082] Step S220: Obtain the greedy strategy behavior data;

[0083] Step S230: If the greedy strategy behavior data is less than or equal to the behavior threshold, a grabbing action is randomly selected from the grabbing action set through a preset neural network to obtain the initial grabbing action.

[0084] In step S210 of some embodiments, when generating a visual grasping strategy, general grasping strategies often suffer from low grasping efficiency due to inaccurate target object positioning or non-standard grasping angles and routes. Furthermore, grasping strategies designed for a specific task cannot be generalized to general tasks. This results in the robotic arm only being able to perform a single task during the grasping process, thus affecting generalization. To address the above problems, embodiments of this application employ a scene diffusion-based deep reinforcement learning method to determine the grasping action.

[0085] The initial grasping action corresponding to the first object state is determined by a pre-defined neural network, which uses a Deep Q-Network (DQN). The set of grasping actions is determined based on the feature size of the first object state and the number of grasping directions. The tool center point of the robot's end effector can rotate from -180° to +180°. Since DQN is a discrete input, discrete output algorithm, the rotatable range of the tool center point needs to be discrete. The rotation range is discretized into 32 directions at 11.25° intervals, resulting in 32 different grasping directions. If the feature size of the first object state is M×N and the number of grasping directions is 32, then the set of grasping actions is an M×N×32 state-action space, where M×N represents the action space and 32 represents the state directions. The set of grasping actions includes M×N×32 grasping actions.

[0086] In step S220 of some embodiments, the deep Q-network employs an ε-greedy exploration strategy to select a grasping action, where ε is a behavior threshold representing the probability that the robot will perform the grasping action next. A larger ε value indicates a higher probability that the grasping action will be selected. A random number between [0.0, 1.0] is randomly generated and used as the behavior data for the greedy strategy.

[0087] In step S230 of some embodiments, if the greedy strategy behavior data is less than or equal to the behavior threshold, it means that the probability of the grabbing action being selected is ε. Then, a grabbing action is randomly selected from the grabbing action set through a deep Q-network to obtain the initial grabbing action. The behavior threshold is a small value, such as 0.1.

[0088] In steps S210 to S230 above, the action exploration is carried out through the ε greedy exploration strategy, which can select the initial grasping action from multiple grasping actions to obtain a new action, so as to avoid missing grasping actions.

[0089] Please see Figure 3 In some embodiments, after step S220, the object grasping method may also include, but is not limited to, steps S310 to S320:

[0090] Step S310: If the greedy strategy behavior data is greater than the behavior threshold, then the action evaluation of each grasping action in the grasping action set is performed through a preset neural network to obtain the action execution degree of each grasping action.

[0091] Step S320: Select a grasping action from the grasping action set according to the degree of action execution to obtain the initial grasping action.

[0092] In step S310 of some embodiments, if the greedy strategy behavior data is greater than the behavior threshold, it means that the probability of the grabbing action being selected is 1-ε. Then, the deep Q network is used to evaluate each grabbing action in the set of grabbing actions corresponding to the first object state to obtain the action execution degree of each grabbing action. The action execution degree is the Q value, which is used to represent the probability that the action is executed.

[0093] In step S320 of some embodiments, the grasping action with the largest Q value is selected from the grasping action set as the initial grasping action, i.e., the initial grasping action a. t =argmax(Q(s) t ,a t ,θ)), s t Let θ represent the state of the object at time t, and θ be the grasping direction.

[0094] Steps S310 to S320 above utilize an ε-greedy exploration strategy to select an initial grasping action from multiple grasping actions, thus obtaining the currently estimated optimal action. Simultaneously, the ε-greedy exploration strategy balances the trade-off between exploration and utilization, performs well in reinforcement learning tasks, and ensures the accuracy of grasping action selection, thereby guaranteeing the accuracy of object grasping.

[0095] In step S130 of some embodiments, in the first object state s t The initial grabbing action a is executed below. t To perform object grasping, a depth camera is used to acquire a scene state image after the initial grasping action. A deep convolutional neural network is then used to extract features from this scene state image to update the first object state s. t The second object state s is obtained. t+1 .

[0096] Please see Figure 4 In some embodiments, step S140 may include, but is not limited to, steps S410 to S430:

[0097] Step S410: Perform instance segmentation on the first object state to obtain the predicted detection box;

[0098] Step S420: Calculate the intersection-union ratio (IU) data between the predicted detection box and the preset detection box;

[0099] Step S430: Evaluate the object dispersion degree of the first object state based on the cross-union ratio data to obtain the first dispersion score.

[0100] In step S410 of some embodiments, an end-to-end Degree of Decentralization Evaluate Network (DDEN) is designed. The DDEN network evaluates the degree of dispersion between objects based on the real-time states of objects in the current robot working scene. The first object state and the second object state are input into the Degree of Decentralization Evaluate Network to evaluate the object dispersion, obtaining a first dispersion score corresponding to the first object state and a second dispersion score corresponding to the second object state. The calculation method for the second dispersion score is the same as that for the first dispersion score. The criterion for the degree of dispersion is the proportion of the object in the robot working scene relative to the size of the workspace, used to indicate whether the objects are densely packed.

[0101] The DDEN network can employ a VIT-based Mask R-CNN network to fully consider the state information of objects. The basic idea of ​​a VIT-based Mask R-CNN network is that if an object cannot be detected under certain confidence conditions, it indicates that the edges are too occluded or not clear enough, thus determining whether objects in the current scene are scattered.

[0102] The object state exists as a feature map. A pre-trained VIT-based Mask R-CNN network is used to segment the first object state into instances, obtaining predicted detection boxes including masks and categories. The VIT-based Mask R-CNN network includes a Region Proposal Network (RPN), a Region of Interest (ROI) alignment layer, fully connected layers, and a Fully Convolutional Network (FCN). Multiple ROIs are defined for each pixel in the first object state. These ROIs are input into the RPN network for binary classification and regression, filtering out some ROIs. The ROI alignment layer aligns the remaining ROIs with the pixels of the first object state, mapping the pixels of the ROIs to the pixels of the first object state. The fully connected layer performs N-class classification and bounding box regression on the aligned ROIs to obtain the object category and detection boxes. The fully convolutional layer performs full convolution on the aligned ROIs to generate a mask. N is an integer greater than 2.

[0103] It should be noted that the mask is a binary mask, used to indicate whether a given pixel is part of an object. A pixel is marked as 1 when it belongs to an object, and as 0 when it does not belong to an object.

[0104] Non-maximum suppression (NMS) is applied to the predicted detection boxes. First, a confidence threshold is set, and predicted detection boxes with confidence scores below the threshold are removed, while those with confidence scores greater than or equal to the threshold are retained. Then, predicted detection boxes of the same object category are sorted from highest to lowest confidence score, and the predicted detection box with the highest confidence score is selected as the first predicted detection box for that object category. The intersection-union ratio (IUR) is calculated between the first predicted detection box and the other predicted detection boxes in the same object category. Finally, predicted detection boxes with IUR scores greater than a preset IUR threshold are removed. The IUR can be defined as the ratio of the overlapping area to the merged area between two predicted detection boxes. The IUR threshold is also the detection threshold. To evaluate the dispersion of objects, the confidence threshold and detection threshold can be appropriately increased to obtain the most complete object as a marker of dispersion evaluation.

[0105] In step S420 of some embodiments, the intersection-over-union (IoU) data is used to represent the degree of overlap between the predicted detection box and the ground truth detection box, and can be calculated using a first mask of the predicted detection box and a second mask of the ground truth detection box. The IoU data is calculated as shown in formula (1).

[0106]

[0107] Where A is the set of pixels of the first mask; B is the set of pixels of the second mask; A∩B is the sum of the number of pixels where the first mask and the second mask intersect; A∪B is the sum of the number of pixels in the union of the first mask and the second mask.

[0108] In step S430 of some embodiments, the average value of the mask IOU obtained by DDEN detection is subtracted from the number of undetected objects to evaluate the object dispersion of the first object state and obtain a first dispersion score. The calculation method of the first dispersion score is shown in formula (2).

[0109]

[0110] Where V is the evaluation value of the current state, that is, the evaluation of the degree of dispersion of objects in the current scene. N curr This represents the number of objects detected by DDEN in the current robot's working scene, i.e., the number of effective first masks. curr It represents the number of undetected objects, IOU. i It is the cross-union ratio of the i-th mask. α is an adjustable parameter.

[0111] It is understandable that the discreteness scores of an object state that changes continuously within a preset time range are also continuous.

[0112] Steps S410 to S430 above can evaluate the dispersion of objects in the robot's working scene. By considering the dispersion of objects, accurate grasping of objects in cluttered scenes can be achieved.

[0113] Please see Figure 5 In some embodiments, step S150 may include, but is not limited to, steps S510 to S530:

[0114] Step S510: Calculate the change in the state of the second object compared to the state of the first object;

[0115] Step S520: Subtract the first dispersion score and the second dispersion score to obtain the score deviation data;

[0116] Step S530: Determine the first reward data based on the change level data and the scoring deviation data.

[0117] In step S510 of some embodiments, both the first object state and the second object state exist as feature maps. The grayscale difference between corresponding pixels in the two feature maps is calculated, and the sum of these grayscale differences is used as the degree of change data. If the first object state is represented as s... t The state of the second object is represented as s t+1 Both the first and second object states have K pixels, and the grayscale difference of the i-th pixel is represented as... The degree of change data is represented as

[0118] In step S520 of some embodiments, the first dispersion score is represented as V(s) t The second dispersion score is represented as V(s) t+1 The scoring deviation data is represented as: M t =V(s) t+1 )-V(s t ).

[0119] In step S530 of some embodiments, if the change level data is greater than a preset change threshold, it indicates that the robotic arm has performed an effective operation on the object, and the object's state has changed after the action is executed. That is, after performing the initial grasping action in the first object state, the object in the second object state is more dispersed, and a reward is required. The first reward data is determined based on the scoring deviation data, and at this time, the first reward data is greater than 0. If the change level data is less than or equal to the preset change threshold, it indicates that after performing the initial grasping action in the first object state, the object in the second object state is not more dispersed, and a penalty is required. The penalty is taken as a negative number as the first reward data, and at this time, the first reward data is less than 0. The preset change threshold can be set according to the actual situation.

[0120] The calculation method for the first reward data is shown in formula (3).

[0121]

[0122] Among them, R s This refers to the penalty or reward given after the grabbing action is performed, depending on whether the objects in the scene become more scattered or more clustered.

[0123] Steps S510 to S530 above construct a reward function based on the evaluation value of the dispersion degree, and update the model parameters through reward data to generate a grasping action generation network with better grasping performance in cluttered scenes.

[0124] Please see Figure 6 In some embodiments, step S160 may include, but is not limited to, steps S610 to S650:

[0125] Step S610: Obtain the execution status of the initial grabbing action, and determine the second reward data obtained by executing the initial grabbing action based on the execution status;

[0126] Step S620: Determine the intermediate grasping action corresponding to the second object state, and calculate the third reward data for executing the intermediate grasping action;

[0127] Step S630: Perform data summation on the first reward data, the second reward data, and the third reward data to obtain the predicted reward data;

[0128] Step S640: Obtain target loss data based on predicted reward data and preset benchmark reward data;

[0129] Step S650: Optimize the parameters of the preset neural network based on the target loss data to train the preset neural network and obtain the grasping action generation network.

[0130] In step S610 of some embodiments, the execution state is the open / closed state of the end effector gripper. An execution state indicating that the gripper is fully closed means that the object was not successfully grasped after the initial grasping action, the grasping success rate after one grasp is 0, and a penalty needs to be imposed, which is then taken as a negative number as the second reward data. An execution state indicating that the gripper is not fully closed means that the object was successfully grasped after the initial grasping action, the grasping success rate after the first grasp is 1, and a reward needs to be imposed, which is then taken as the second reward data. The calculation method for the second reward data is shown in formula (4).

[0131]

[0132] Among them, R gThe reward or penalty obtained after a capture.

[0133] In step S620 of some embodiments, the set of grasping actions corresponding to the second object state is determined by referring to the method of step S210, and the grasping action with the largest Q value is selected as the intermediate grasping action from the grasping action set. The Q value of the intermediate grasping action is multiplied by the reward parameter to obtain the third reward data. That is, the third reward data is expressed as:

[0134] γ×Q(s t+1 argmax(Q(s) t+1 , a))) Formula (5)

[0135] Where γ is the reward parameter; a is the set of all possible actions in the second object state, including 24×24×32 actions.

[0136] In step S630 of some embodiments, the first reward data, the second reward data, and the third reward data are added together to obtain the predicted reward data. The method for calculating the predicted reward data is shown in formula (6).

[0137] R total =R s +R g +γ×Q(s t+1 argmax(Q(s) t+1 , a))) Formula (6)

[0138] Among them, R total For predicting reward data; R s This is the first reward data; R g This is the second reward data; Q(s) t+1 argmax(Q(s) t+1 ,a))) is the third reward data.

[0139] In step S640 of some embodiments, a preset neural network can be continuously trained guided by the predicted reward data. When the predicted reward data is maximized, the optimal grasping action is obtained. To improve training efficiency, the preset neural network can also be trained using stochastic gradient descent, and the target loss data is obtained through the predicted reward data and the preset baseline reward data. The preset baseline reward data is the Q-value obtained by inputting the object state and grasping action label into a deep Q-network. This deep Q-network is not the same as the first deep Q-network inputting the first object state. The target loss data can be the square of the difference between the predicted reward data and the preset baseline reward data, expressed as:

[0140] Loss=(R s +R g +γ×Q(s t+1 argmax(Q(s)t+1 a t+1 )))-Q(s t a t )) 2 Formula (7)

[0141] In step S650 of some embodiments, the parameters of a preset neural network are optimized using the target loss data to train the preset neural network. Training of the preset neural network stops when the target loss data reaches its minimum or the number of iterations reaches its maximum, resulting in a Grasping Strategy Generation Network (GSGN). Every k steps (i.e., every k iterations), the network parameters of the preset neural network are copied to a second deep Q-network. The Grasping Strategy Generation Network can generate a strategy for grasping position and angle, enabling the robot to successfully grasp and remove target objects, thus causing objects in the robot's workspace to tend to disperse after each grasping action. The Grasping Strategy Generation Network proposed in this application embodiment can adapt to new grasping tasks without requiring additional supervised training.

[0142] Through the above steps S610 to S650, a grasping action generation network with good generalization ability, good stability and fast learning speed can be generated.

[0143] Please see Figure 7 In some embodiments, step S640 may include, but is not limited to, steps S710 to S730:

[0144] Step S710: Perform data difference processing on the predicted reward data and the preset benchmark reward data to obtain reward deviation data;

[0145] Step S720: If the reward deviation data is less than or equal to the preset deviation threshold, then the target loss data is obtained based on the reward deviation data.

[0146] Step S730: If the reward deviation data is greater than the preset deviation threshold, the target loss data is obtained based on the reward deviation data and the preset deviation threshold.

[0147] In step S710 of some embodiments, the target loss data is calculated by the Huber function, and the calculation method of the target loss data is shown in formula (8).

[0148]

[0149] Where y^ represents the predicted reward data; y represents the preset baseline reward data; and δ represents the hyperparameter of the Huber function, i.e., the preset deviation threshold, which is usually set to 1.

[0150] The absolute value of the difference between the predicted reward data and the preset benchmark reward data is used as the reward deviation data, which is |y^-y|.

[0151] In step S720 of some embodiments, if the reward deviation data is less than or equal to a preset deviation threshold, then half of the square of the reward deviation data is used as the target loss data.

[0152] In step S730 of some embodiments, if the reward deviation data is greater than a preset deviation threshold, the preset deviation threshold and the reward deviation data are multiplied to obtain the first data, half of the square of the preset deviation threshold is used as the second data, and the first data and the second data are subtracted to obtain the target loss data.

[0153] Steps S710 to S730 above obtain different target loss data based on the deviation between the preset reward data and the preset benchmark reward data, which can improve the network training efficiency.

[0154] In step S170 of some embodiments, after the model training is complete, the model can be used to perform an object grasping task. For each grasping attempt, the robot performs a series of actions, such as moving the robotic arm and gripper, until it successfully grasps the object or reaches the maximum number of attempts. At each time step, the robot selects actions based on the current object state and the action probabilities generated by the grasping action generation network. When selecting actions, the grasping actions are sorted from highest to lowest probability, and actions are selected and executed sequentially according to the sorting order until the object is successfully grasped or the maximum number of attempts is reached.

[0155] To enable robots to learn effective grasping strategies in complex scenarios, a technique called experience-based replay can be used. This allows the robot to learn from previous experiences and improve its grasping success rate. Specifically, after each grasping attempt, the robot will replay the experience of that attempt. t ,a t ,r t ,s t+1 Stored in an experience pool, r t The total reward data obtained for performing the grasping action. During each training session, the model randomly draws some experience from the experience pool for training, so that the robot can learn more effective grasping strategies from previous experiences.

[0156] The object grasping method in this application considers the dispersion of objects in the real-time workspace image during the robot's grasping action. Based on dispersion optimization, a grasping strategy generation network is constructed to generate grasping positions and angles. Compared to direct grasping methods, this grasping strategy enables the robot to successfully grasp objects and has better grasping performance. Moreover, this object grasping method has a fast learning speed, good generalization ability, and good stability, exhibiting excellent versatility and providing a novel solution to the robot grasping problem in cluttered scenes.

[0157] Please see Figure 8 This application also provides an object grasping device that can implement the above-described object grasping method. The device includes:

[0158] The acquisition module 810 is used to acquire the state of the first object;

[0159] The first determining module 820 is used to determine the initial grasping action corresponding to the state of the first object through a preset neural network.

[0160] The state update module 830 is used to update the state of the first object based on the initial grasping action to obtain the state of the second object.

[0161] The evaluation module 840 is used to evaluate the object dispersion of the first object state to obtain a first dispersion score, and to evaluate the object dispersion of the second object state to obtain a second dispersion score.

[0162] The second determining module 850 is used to determine the first reward data obtained by performing the initial grasping action based on the first object state, the second object state, the first discrete score, and the second discrete score.

[0163] The training module 860 is used to optimize the parameters of the preset neural network based on the first reward data and the preset benchmark reward data, so as to train the preset neural network and obtain the grasping action generation network.

[0164] The object grasping module 870 is used to obtain the state of the target object, generate grasping actions from the target object state through a grasping action generation network, obtain the target grasping action, and grasp the object according to the target grasping action.

[0165] The specific implementation of this object grasping device is basically the same as the specific implementation of the object grasping method described above, and will not be repeated here.

[0166] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described object grasping method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0167] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0168] The processor 910 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0169] The memory 920 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910 using the object grasping method of the embodiments of this application.

[0170] The input / output interface 930 is used to implement information input and output;

[0171] The communication interface 940 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0172] Bus 950 transmits information between various components of the device (e.g., processor 910, memory 920, input / output interface 930, and communication interface 940);

[0173] The processor 910, memory 920, input / output interface 930 and communication interface 940 are connected to each other within the device via bus 950.

[0174] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described object grasping method.

[0175] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0176] The object grasping method, object grasping device, electronic device, and computer-readable storage medium provided in this application quantify the clutter level of the scene by scoring the dispersion of objects in the robot's working environment. Reward data is obtained based on the dispersion score, and a preset neural network is optimized based on the reward data to obtain a dispersion-based grasping action generation network. A target grasping action is generated through this grasping action generation network, and the object is grasped according to the target grasping action, thus improving the accuracy of object grasping by the robot in cluttered scenes.

[0177] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0178] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0179] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0181] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0182] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0184] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0186] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for grasping objects, characterized in that, The method includes: Get the state of the first object; The initial grasping action corresponding to the state of the first object is determined by a preset neural network. The state of the first object is updated based on the initial grasping action to obtain the state of the second object; The first object state is evaluated for its discreteness to obtain a first discreteness score, and the second object state is evaluated for its discreteness to obtain a second discreteness score. Based on the first object state, the second object state, the first discrete score, and the second discrete score, determine the first reward data obtained by performing the initial grasping action; The preset neural network is trained by optimizing the parameters of the preset neural network based on the first reward data and the preset benchmark reward data to obtain the grasping action generation network. The target object state is obtained, and the grasping action generation network is used to generate grasping actions based on the target object state to obtain the target grasping action. The object is then grasped based on the target grasping action. The step of evaluating the object's discreteness in the first object state to obtain a first discreteness score includes: The first object state is segmented into instances to obtain a predicted detection box; the intersection-union ratio (IUR) between the predicted detection box and a preset detection box is calculated; the object dispersion of the first object state is evaluated based on the IUR to obtain the first dispersion score. The step of determining the first reward data obtained by performing the initial grasping action based on the first object state, the second object state, the first discrete score, and the second discrete score includes: Calculate the degree of change between the second object state and the first object state; calculate the difference between the first discrete score and the second discrete score to obtain the score deviation data; determine the first reward data based on the degree of change data and the score deviation data; The step of optimizing the parameters of the preset neural network based on the first reward data and preset benchmark reward data to train the preset neural network and obtain a grasping action generation network includes: The execution state of the initial grasping action is obtained, and the second reward data obtained by executing the initial grasping action is determined based on the execution state; the intermediate grasping action corresponding to the second object state is determined, and the third reward data of executing the intermediate grasping action is calculated; the first reward data, the second reward data, and the third reward data are summed to obtain predicted reward data; the target loss data is obtained based on the predicted reward data and the preset benchmark reward data; the parameters of the preset neural network are optimized based on the target loss data to train the preset neural network and obtain the grasping action generation network.

2. The object grasping method according to claim 1, characterized in that, The initial grasping action corresponding to the state of the first object is determined by a preset neural network, including: Determine the set of grasping actions corresponding to the state of the first object; the set of grasping actions includes multiple grasping actions; Obtain data on the behavior of the greedy strategy; If the greedy strategy behavior data is less than or equal to the behavior threshold, then a grabbing action is randomly selected from the grabbing action set through the preset neural network to obtain the initial grabbing action.

3. The object grasping method according to claim 2, characterized in that, After acquiring the greedy strategy behavior data, the object grasping method further includes: If the greedy strategy behavior data is greater than the behavior threshold, then the preset neural network is used to evaluate each grabbing action in the grabbing action set to obtain the degree of action execution of each grabbing action. Based on the degree of execution of the action, a grabbing action is selected from the grabbing action set to obtain the initial grabbing action.

4. The object grasping method according to claim 1, characterized in that, The step of obtaining the target loss data based on the predicted reward data and the preset baseline reward data includes: The predicted reward data and the preset benchmark reward data are subtracted to obtain the reward deviation data; If the reward deviation data is less than or equal to a preset deviation threshold, then the target loss data is obtained based on the reward deviation data; If the reward deviation data is greater than the preset deviation threshold, then the target loss data is obtained based on the reward deviation data and the preset deviation threshold.

5. An object grasping device, characterized in that, The device includes: The acquisition module is used to acquire the state of the first object; The first determining module is used to determine the initial grasping action corresponding to the state of the first object through a preset neural network. The state update module is used to update the state of the first object according to the initial grasping action to obtain the state of the second object; The evaluation module is used to evaluate the object dispersion of the first object state to obtain a first dispersion score, and to evaluate the object dispersion of the second object state to obtain a second dispersion score. The second determining module is used to determine the first reward data obtained by performing the initial grasping action based on the first object state, the second object state, the first discrete score, and the second discrete score. The training module is used to optimize the parameters of the preset neural network based on the first reward data and the preset benchmark reward data, so as to train the preset neural network and obtain the grasping action generation network. The object grasping module is used to obtain the state of the target object, generate grasping actions from the state of the target object through the grasping action generation network to obtain the target grasping action, and grasp the object according to the target grasping action. The device is also used for: The first object state is segmented into instances to obtain a predicted detection box; the intersection-union ratio (IUR) between the predicted detection box and a preset detection box is calculated; the object dispersion of the first object state is evaluated based on the IUR to obtain the first dispersion score. Calculate the degree of change between the second object state and the first object state; calculate the difference between the first discrete score and the second discrete score to obtain the score deviation data; determine the first reward data based on the degree of change data and the score deviation data; The execution state of the initial grasping action is obtained, and the second reward data obtained by executing the initial grasping action is determined based on the execution state; the intermediate grasping action corresponding to the second object state is determined, and the third reward data of executing the intermediate grasping action is calculated; the first reward data, the second reward data, and the third reward data are summed to obtain predicted reward data; the target loss data is obtained based on the predicted reward data and the preset benchmark reward data; the parameters of the preset neural network are optimized based on the target loss data to train the preset neural network and obtain the grasping action generation network.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the object grasping method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the object grasping method according to any one of claims 1 to 4.

Images