A flexible object grasping method and device, electronic equipment and storage medium

Abstract

The application provides a flexible object grasping method and device, electronic equipment and storage medium, wherein the flexible object grasping method comprises the following steps: adopting a position evaluation strategy model to determine the Q value of each grasping position in a flexible object image; adopting a posture grasping strategy model to determine a grasping target point and a grasping posture according to the Q value of each grasping position; and controlling a mechanical arm to perform a grasping operation on the grasping target point by adopting the grasping posture. The value of each grasping position of the flexible object is obtained by adopting the position evaluation strategy model, so that the mechanical arm can select the best grasping position when grasping the object; and the posture grasping strategy model is adopted to generate appropriate mechanical arm actions, so as to realize stable grasping and operation of the flexible object, thereby effectively improving the grasping effect of the flexible object.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a flexible object grasping method, device, electronic device, and storage medium. Background Technology

[0002] With the continuous advancement of artificial intelligence technology, an increasing number of industrial and household service robots are gradually penetrating people's production and daily lives, playing an increasingly important role and providing strong support for economic development and improving people's quality of life. However, more and more application scenarios require robots to be able to independently complete tasks involving the manipulation of flexible objects such as soft fabrics, cables, and bags.

[0003] When using traditional robotic arm control algorithms to control robotic arms to grasp flexible objects, the grasping effect is not good. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for grasping flexible objects, so as to improve the grasping effect of flexible objects.

[0005] In a first aspect, embodiments of this application provide a flexible object grasping method, comprising: employing a position evaluation strategy model to determine the Q value of each grasping position in a flexible object image; the Q value being used to characterize the probability that the grasping position is grasped; employing an attitude grasping strategy model to determine a grasping target point and a grasping attitude based on the Q value of each grasping position; and controlling a robotic arm to perform a grasping operation on the grasping target point using the grasping attitude.

[0006] In the implementation of the above scheme, a position evaluation strategy model is used to obtain the value of each gripping position of the flexible object so that the robotic arm can select the best gripping position when gripping the object; an attitude gripping strategy model is used to generate appropriate robotic arm movements to achieve stable gripping and manipulation of the flexible object, thereby effectively improving the gripping effect of the flexible object.

[0007] In one implementation of the first aspect, the step of employing a position evaluation strategy model to obtain the grasping position of the flexible object and the Q value of each grasping position based on the flexible object image includes: employing a first convolutional layer of the position evaluation strategy model to extract a first low-level feature of the flexible object in the flexible object image; and employing a first fully connected layer of the position evaluation strategy model to determine the Q value of each grasping position in the flexible object image based on the first low-level feature.

[0008] In the implementation of the above scheme, the position evaluation strategy model adopts a structure that combines convolutional layers and fully connected layers, so that the position evaluation strategy model can obtain the value of each grasping position of the flexible object, so that the robotic arm can select the best grasping position when grasping the object, thereby effectively improving the grasping effect of the flexible object.

[0009] In one implementation of the first aspect, the first convolutional layer employing the position evaluation strategy model extracts the first low-level features of the flexible object in the flexible object image, comprising: extracting the first low-level features of the flexible object in the flexible object image after meshing processing by employing the first convolutional layer employing the position evaluation strategy model.

[0010] In the implementation of the above scheme, by processing the flexible object image into a grid, the computational and storage overhead of the flexible object grasping method is effectively reduced, thereby improving the processing efficiency of the flexible object grasping method.

[0011] In one implementation of the first aspect, after determining the Q-values ​​of each grasping position in the flexible object image based on the first low-level features in the first fully connected layer of the position evaluation strategy model, the method further includes: processing the flexible object image into a heatmap based on the Q-values ​​to obtain a flexible object heatmap; the flexible object heatmap is used to characterize the magnitude of the Q-values ​​of each grasping position; the step of using a posture grasping strategy model to determine the grasping target point and grasping posture based on the Q-values ​​of each grasping position includes: using a posture grasping strategy model to determine the grasping target point and grasping posture based on the flexible object heatmap used to characterize the magnitude of the Q-values ​​of each grasping position.

[0012] In the implementation of the above scheme, the Q-value is used to process the flexible object image into a heat map, thereby reducing the computation and storage overhead of the posture grasping strategy model and effectively improving the processing efficiency of the above flexible object grasping method.

[0013] In one implementation of the first aspect, the step of employing a posture-based grasping strategy model to determine the grasping target point and grasping posture based on the Q-values ​​of each grasping position includes: employing a second convolutional layer of the posture-based grasping strategy model to extract second low-level features based on the Q-values ​​of each grasping position; and employing a second fully connected layer of the posture-based grasping strategy model to determine the grasping target point and grasping posture based on the second low-level features.

[0014] In the implementation of the above scheme, the posture grasping strategy model adopts a structure that combines convolutional layers and fully connected layers, which enables the posture grasping strategy model to generate appropriate robotic arm movements to achieve stable grasping and manipulation of flexible objects, thereby effectively improving the grasping effect of flexible objects.

[0015] In one implementation of the first aspect, before the second convolutional layer of the posture grasping strategy model extracts the second low-level features based on the Q-values ​​of each grasping position, the method further includes: obtaining the current position and current posture of the robotic arm; the second fully connected layer of the posture grasping strategy model determines the grasping target point and grasping posture based on the second low-level features, including: obtaining the grasping target point and grasping posture based on the second fully connected layer of the posture grasping strategy model, the second low-level features, and the current position and current posture of the robotic arm.

[0016] In the implementation of the above scheme, by inputting additional auxiliary information into the posture grasping strategy model, the posture grasping strategy model can quickly obtain the grasping target point and the grasping posture, which effectively improves the processing efficiency of the posture grasping strategy model.

[0017] In one implementation of the first aspect, before determining the Q-value of each grasping position in the flexible object image using the position evaluation strategy model, the method further includes: training the position evaluation strategy model and the posture grasping strategy model using the Proximal Policy Optimization (PPO) method.

[0018] In the implementation of the above scheme, the near-end policy optimization method (PPO) is used to train the posture grasping strategy model, so that the posture grasping strategy model can output the optimal grasping strategy, thereby improving the grasping effect on flexible objects.

[0019] Secondly, embodiments of this application provide a flexible object grasping device, including: a position evaluation module, a grasping strategy acquisition module, and a grasping control module, wherein...

[0020] The position evaluation module is used to determine the Q value of each grasping position in the flexible object image by adopting a position evaluation strategy model; the Q value is used to characterize the probability that the grasping position is grasped.

[0021] The grasping strategy acquisition module is used to determine the grasping target point and grasping posture based on the Q value of each grasping position using an attitude grasping strategy model.

[0022] The grasping control module is used to control the robotic arm to grasp the target point using the grasping posture.

[0023] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0024] Fourthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other through the communication bus; the memory stores computer program instructions that can be executed by the processor, and the computer program instructions are read and executed by the processor to perform the method provided in the first aspect or any possible implementation of the first aspect.

[0025] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating the flexible object grasping method provided in this application embodiment;

[0028] Figure 2 This is a schematic diagram of the structure of the location evaluation strategy model provided in the embodiments of this application;

[0029] Figure 3 This is a schematic diagram of the structure of the posture grasping strategy model provided in the embodiments of this application;

[0030] Figure 4 This is a schematic diagram of the structure of the PPO algorithm provided in the embodiments of this application;

[0031] Figure 5 A flowchart illustrating the application of the flexible object grasping method provided in this application to a task of flattening a soft cloth;

[0032] Figure 6 A schematic diagram illustrating the average reward calculated based on performance indicators in a comparative experiment provided for an embodiment of this application;

[0033] Figure 7 This is a schematic diagram of the flexible object grasping device provided in the embodiments of this application;

[0034] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of this application, and are therefore merely examples and should not be used to limit the scope of protection of this application.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0037] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0038] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0039] Robotic flexible object manipulation tasks refer to the tasks involved in manipulating flexible objects. Flexible objects are typically deformable objects made of elastic bodies, soft materials, etc., and are characterized by variable shape, rapid deformation response, and poor recovery ability after deformation.

[0040] Currently, methods for manipulating flexible objects in related technologies include:

[0041] (1) An image Jacobi proportional integral derivative PID control strategy based on a simplified physical model is adopted. The visual measurement values ​​of the feature points on the surface of the flexible object are continuously fed back to the controller, and the PID control law is used to make the image position of the feature points continuously approach the desired value in order to control the deformation position of the flexible object.

[0042] (2) An adaptive visual servo controller is used to operate the flexible object. This method can estimate the unknown deformation model and camera parameters online, and then control the deformation position of the unknown flexible object in an unstructured environment.

[0043] The aforementioned methods for manipulating flexible objects have the drawback of being simple to perform but having poor operational results.

[0044] Based on this, embodiments of this application provide a flexible object grasping method. This method uses a position evaluation strategy model to obtain the value of each grasping position of the flexible object, so that the robotic arm can select the best grasping position when grasping the object; and uses a posture grasping strategy model to generate appropriate robotic arm movements to achieve stable grasping and operation of the flexible object, thereby effectively improving the grasping effect of the flexible object.

[0045] The aforementioned flexible object grasping method can be applied to robot operation scenarios involving flexible object manipulation tasks, such as flexible component manipulation, food processing, textile manufacturing, object packaging, medical surgical robots, and companion robots.

[0046] Please see Figure 1 This application provides a method for grasping flexible objects, including:

[0047] Step S110: Using a position evaluation strategy model, determine the Q value of each grasping position in the flexible object image; the Q value is used to characterize the probability that the grasping position is grasped.

[0048] Step S120: Using the posture grasping strategy model, determine the grasping target point and grasping posture based on the Q value of each grasping position;

[0049] Step S130: Control the robotic arm to perform a grasping operation on the target point using a grasping posture.

[0050] The following is a detailed description of steps S110 to S130:

[0051] First, step S110 will be described in detail:

[0052] It is understandable that the position evaluation strategy model in step S110 is a deep learning model, whose main task is to obtain the Q value of each grasping position on the flexible object. The input of the model is the image of the flexible object, and the output is the Q value of each grasping position in the image of the flexible object.

[0053] It is understood that the aforementioned flexible object image can be obtained by directly photographing the flexible object by the electronic device performing the aforementioned flexible object grasping method, or it can be obtained by the flexible object image sent by other electronic devices by the electronic device performing the aforementioned flexible object grasping method.

[0054] As an optional implementation of the above-mentioned flexible object grasping method, step S110 employs a position evaluation strategy model to obtain the grasping position of the flexible object and the Q value of each grasping position based on the flexible object image. This includes: using the first convolutional layer of the position evaluation strategy model to extract the first low-level features of the flexible object in the flexible object image; and using the first fully connected layer of the position evaluation strategy model to determine the Q value of each grasping position in the flexible object image based on the first low-level features.

[0055] Please see Figure 2 The position evaluation strategy model in this application embodiment includes a first convolutional layer and a first fully connected layer. It should be noted that the "first" and "second" in the first convolutional layer and the first fully connected layer in this application embodiment, as well as the second convolutional layer and the second fully connected layer that appear in the following description of the pose grasping strategy model, are not limitations on the number of convolutional layers and the number of fully connected layers, but are used to distinguish the structure of the position evaluation strategy model and the structure of the pose grasping strategy model.

[0056] It is understandable that, such as Figure 2 As shown, the first convolutional layer in the position evaluation strategy model includes convolutional layers C1, C2, and C3. The first fully connected layer includes a fully connected layer FC1, an activation function layer LR1, a fully connected layer FC2, an activation function layer LR2, and a fully connected layer FC3. After the flexible object image is input into the position evaluation strategy model, it first undergoes convolution processing through convolutional layers C1, C2, and C3 in sequence. Convolutional layers C1, C2, and C3 are used to extract the first low-level features related to shape and texture in the flexible object image. After processing by the convolutional layers, the first low-level features are fed into the first fully connected layer so that the position evaluation strategy model can learn more discriminative abstract information from the first low-level features. Finally, the position evaluation strategy model outputs the Q-values ​​of each grasping position in the flexible object image.

[0057] Optionally, before the location evaluation strategy model passes the first bottom-level features into the first fully connected layer, it can also flatten the first bottom-level features through the flattening layer C4.

[0058] Optionally, after the first low-level features are flattened by the flattening layer C4, new first low-level features can be obtained by superimposing the features input from the additional input layer C5, and then the new first low-level features are passed into the first fully connected layer. The features input from the additional input layer C5 may include: pixel position information.

[0059] It should be noted that when performing flexible object manipulation tasks, if each operation of the robotic arm is regarded as a subtask, and if the robotic arm does not reset after completing the current subtask but directly executes the next subtask, the position evaluation strategy model can consider the distance between the current position of the robotic arm and the grasping position of the flexible object, as well as the change in posture of the robotic arm when reaching the grasping position, when calculating the Q value of each grasping position in the flexible object image. In this case, the features input to the additional input layer C5 can include the current position and current posture information of the robotic arm, in addition to the position of the pixel.

[0060] The network structure of the above location evaluation strategy model can be adopted as shown in Table 1.

[0061] Table 1. Network Structure of the Location Evaluation Strategy Model

[0062] number of floors Layer description Layer type Input dimensions Output size nuclear size Step length filling 1 conv.Conv2d Conv2d (3,64,64) (64,32,32) (3,3) (2,2) (1,1) 2 conv.LayerNorm LayerNorm (64,32,32) (64,32,32) - - - 3 conv.LeakyReLU LeakyReLU (64,32,32) (64,32,32) - - - 4 conv.Conv2d Conv2d (64,32,32) (64,16,16) (3,3) (2,2) (1,1) 5 conv.LayerNorm LayerNorm (64,16,16) (64,16,16) - - - 6 conv.LeakyReLU LeakyReLU (64,16,16) (64,16,16) - - - 7 conv.Conv2d Conv2d (64,16,16) (4,8,8) (3,3) (2,2) (1,1) 8 conv.LayerNorm LayerNorm (4,8,8) (4,8,8) - - - 9 conv.LeakyReLU LeakyReLU (4,8,8) (4,8,8) - - - 10 head.Linear Linear 356 256 - - - 11 head.LeakyReLU LeakyReLU 256 256 - - - 12 head.Linear Linear 256 256 - - - 13 head.LeakyReLU LeakyReLU 256 256 - - - 14 head.Linear Linear 256 1 - - -

[0063] As an optional implementation of the above-mentioned flexible object grasping method, step S110 uses the first convolutional layer of the position evaluation strategy model to extract the first low-level features of the flexible object in the flexible object image. This includes: using the first convolutional layer of the position evaluation strategy model to extract the first low-level features of the flexible object in the meshed flexible object image. For example, before extracting the first low-level features of the flexible object, the flexible object image is meshed. The meshing method can be equal-interval meshing, adaptive meshing, or non-uniform meshing. Equal-interval meshing divides the image into square or rectangular grids of equal size, each grid containing the same number of pixels; adaptive meshing dynamically selects the grid size according to the image size and complexity, thereby preserving image details to the greatest extent; non-uniform meshing divides the image into grids of different sizes, the size of each grid depending on its position and texture features.

[0064] As an optional implementation of the above-mentioned flexible object grasping method, after determining the Q value of each grasping position in the flexible object image based on the first bottom layer features in the first fully connected layer of the position evaluation strategy model, the method further includes: processing the flexible object image into a heat map based on the Q value to obtain a flexible object heat map; the flexible object heat map is used to characterize the magnitude of the Q value of each grasping position.

[0065] At this time, step S120 adopts an attitude grasping strategy model to determine the grasping target point and grasping attitude based on the Q value of each grasping position, including: adopting an attitude grasping strategy model to determine the grasping target point and grasping attitude based on the flexible object heat map used to characterize the Q value of each grasping position.

[0066] Regarding the Q-value mentioned in step S110, three concepts in reinforcement learning are introduced:

[0067] (1) Q-value, or Q value, refers to the expected value of the long-term reward of performing a certain action in a certain state. It is a scalar.

[0068] (2) Q-table refers to a state-action table that records the Q value of each action in each state. It is a two-dimensional array.

[0069] (3) Q-map, combined with Q-value and Q-table above, refers to an image-action correspondence function. The input of this function is an image, and the output is a matrix of the same size as the input image. Each element of the matrix represents the Q value of the corresponding pixel in the input image.

[0070] It is understandable that the Q-value of each grasping position in the flexible object image determined by the position evaluation strategy model in step S110 is the aforementioned Q-value, which is scalar data.

[0071] To facilitate processing by the posture grasping strategy model, the flexible object image can be processed into a heatmap based on the Q-value. For example, areas with larger Q-values ​​can be designated as red areas, while areas with smaller Q-values ​​can be designated as blue areas, using color to distinguish the Q-values ​​at different grasping positions. It can be understood that the position evaluation strategy model can learn a Q-map function based on the Q-values ​​of each grasping position in the flexible object image, and then directly output the flexible object image as a flexible object heatmap using the Q-map function.

[0072] The following details step S120:

[0073] It is understandable that the posture grasping strategy model in step S120 is a deep learning model. The input of this model can be a heat map of a flexible object, and the output is the grasping target point and grasping posture of the robotic arm.

[0074] As an optional implementation of the above-mentioned flexible object grasping method, step S120 adopts a posture grasping strategy model to determine the grasping target point and grasping posture based on the Q value of each grasping position, including: using the second convolutional layer of the posture grasping strategy model to extract the second low-level features based on the Q value of each grasping position; and using the second fully connected layer of the posture grasping strategy model to determine the grasping target point and grasping posture based on the second low-level features.

[0075] Please see Figure 3The pose capture strategy model in this embodiment includes a second convolutional layer and a second fully connected layer. The second convolutional layer comprises convolutional layers C1', C2', and C3', and the first fully connected layer comprises a fully connected layer FC1', an activation function layer ReLU1, a fully connected layer FC2', an activation function layer ReLU2, and a fully connected layer FC3'. After inputting an image or matrix, it is sequentially processed through convolutional layers C1', C2', and C3' to obtain second low-level features. After processing by the convolutional layers, the second low-level features are fed into the first fully connected layer so that the pose capture strategy model can learn higher-level abstract features. Finally, the pose capture strategy model outputs the target point and the capture pose.

[0076] It is understandable that if the input to the posture capture strategy model is a heatmap of a flexible object, then the second low-level features can include the color features of the heatmap of the flexible object.

[0077] Optionally, before the pose capture strategy model passes the second bottom-level features into the second fully connected layer, it can also flatten the second bottom-level features through the flattening layer C4'.

[0078] Optionally, after the second bottom layer features are flattened by the flattening layer C4', new second bottom layer features can be obtained by superimposing the features input by the additional input layer C5', and then the new second bottom layer features are passed into the first fully connected layer.

[0079] As an optional implementation of the above-mentioned flexible object grasping method, before extracting the second low-level features based on the Q value of each grasping position in the second convolutional layer of the posture grasping strategy model, the method further includes: obtaining the current position and current posture of the robotic arm.

[0080] The second fully connected layer of the posture grasping strategy model determines the grasping target point and grasping posture based on the second bottom layer features, including: the second fully connected layer of the posture grasping strategy model obtains the grasping target point and grasping posture based on the second bottom layer features, the current position of the robotic arm, and the current posture.

[0081] It is understandable that the current position and current posture of the robotic arm can be superimposed on the flattened second bottom layer features through the additional input layer C5'.

[0082] It is understandable that the additional information input into the additional input layer C5', in addition to the current position and current posture information of the robotic arm, may also include the position information of the pixels.

[0083] The network structure of the above posture grasping strategy model can be adopted as shown in Table 2.

[0084] Table 2 Network Structure of the Attitude Grasping Strategy Model

[0085] number of floors Layer description Layer type Input dimensions Output size nuclear size Step length filling 1 conv.Conv2d Conv2d (3,64,64) (64,32,32) (3,3) (2,2) (1,1) 2 conv.LayerNorm LayerNorm (64,32,32) (64,32,32) - - - 3 conv.LeakyReLU LeakyReLU (64,32,32) (64,32,32) - - - 4 conv.Conv2d Conv2d (64,32,32) (64,16,16) (3,3) (2,2) (1,1) 5 conv.LayerNorm LayerNorm (64,16,16) (64,16,16) - - - 6 conv.LeakyReLU LeakyReLU (64,16,16) (64,16,16) - - - 7 conv.Conv2d Conv2d (64,16,16) (4,8,8) (3,3) (2,2) (1,1) 8 conv.LayerNorm LayerNorm (4,8,8) (4,8,8) - - - 9 conv.LeakyReLU LeakyReLU (4,8,8) (4,8,8) - - - 10 head.Linear Linear 356 256 - - - 11 head.LeakyReLU LeakyReLU 256 256 - - - 12 head.Linear Linear 256 256 - - - 13 head.LeakyReLU LeakyReLU 256 256 - - - 14 head.Linear Linear 256 6 - - -

[0086] As shown in Table 2, the output of the posture grasping strategy model is a vector of length 6, which includes the coordinates of the grasping target point on the x-axis, y-axis and z-axis, as well as the three Euler angles of the robotic arm around the x-axis, y-axis and z-axis when grasping the flexible object.

[0087] As an optional implementation of the above-described flexible object grasping method, step 110, which uses a position evaluation strategy model to determine the Q-value of each grasping position in the flexible object image, further includes: training the position evaluation strategy model and the posture grasping strategy model using the Proximal Policy Optimization (PPO) method. For example, in this implementation, the position evaluation strategy model and the posture grasping strategy model are cascaded together; therefore, the PPO method can be used to jointly train the position evaluation strategy model and the posture grasping strategy model.

[0088] It is understandable that, in addition to the Proximal Policy Optimization (PPO) method mentioned above, the position evaluation strategy model and pose grasping strategy model can also be trained using methods such as DQN (Deep Q-Network) and A3C (Asynchronous Advantage Actor-Critic).

[0089] The following is a detailed introduction to the Proximity Optimization (PPO) method described above:

[0090] PPO (Proximal Policy Optimization) is a policy gradient algorithm in reinforcement learning. It can learn from historical data multiple times, transforming online learning into offline learning, and can be used in both continuous and discrete action spaces.

[0091] PPO (Programmatic Policy Object) limits the magnitude of policy updates to avoid data bias and policy collapse. It uses two networks: an Actor network for outputting actions and a Critic network for calculating the value function. The algorithm aims to achieve data efficiency and performance stability while using only first-order optimization methods. The core idea of ​​the PPO algorithm is to ensure that the new policy does not deviate too far from the old policy when updating it. Please see [link to relevant documentation]. Figure 4 A PPO consists of four main parts:

[0092] (1) Policy network Actor: It is a neural network that can output an action probability distribution based on the current state input. The parameters of the policy network are optimized by the PPO objective function.

[0093] (2) Value function network Critic: This is another neural network that can output an estimated value of a state value function based on the current state input. The parameters of the value function network are optimized by the mean squared error loss function.

[0094] (3) Environment: It is a simulated or real reinforcement learning task scenario that can return the next state and reward signal based on the action output by the policy network;

[0095] (4) Experience Buffer Replay Buffer: It is a data structure that stores sampled trajectory data (state, action, reward, etc.) and can be used to calculate the advantage function and update the policy network and value function network.

[0096] The general process involves the policy network outputting an action based on the current state returned by the environment. Then, a certain amount of trajectory data is sampled from the environment and stored in an experience buffer. Finally, the data in the buffer updates the Critic and Actor networks. Simultaneously, the Critic network provides the Actor network with an estimate of the advantage function to optimize the objective function. This process is repeated until convergence. The definitions of the objective function and the advantage function are as follows:

[0097]

[0098] A(s t a t )=Q(s t a t )-V(s t )

[0099] in, Indicates ...; r t (θ) represents...; It means...; π θold (·|s t ) represents the old strategy; π θ (·|s t ) represents the new strategy; β is an adaptive adjustment coefficient; To maximize the KL divergence; objective function L KL The meaning of (θ) is: when updating the policy network, in addition to maximizing the expected reward, it is also necessary to minimize the KL divergence with the old policy; Q(s t a t ) is the action-value function, representing the state s. tNext, execute action a t Expected return; V(s) t ) is the state-value function, representing the state value in state s. t The expected return following the current strategy; the advantage function A(s) t a t The advantage function can be approximated by the generalized advantage estimation (GAE) which measures how well an action is performed relative to the average level.

[0100] Please see Figure 5 This application provides a flowchart illustrating the application of the aforementioned flexible object grasping method to a flexible object manipulation task involving a flat, soft fabric. The main steps include: First, dividing the original image of the soft fabric into a grid, with each grid corresponding to a grasping position strategy. Each frame of the image is fixed at 64×64×3 and input into the position evaluation strategy model to calculate the current evaluation value of each grid point on the flexible object; the redder the color, the higher the evaluation value (i.e., the higher the grasping probability). Then, the image after position evaluation is input into the posture grasping strategy model to calculate the optimal grasping posture at each grid point (including the coordinates of the grasping target point and the Euler angles of the robotic arm). Finally, the robotic arm only needs to select the position based on the evaluation value and then use the posture to solve the inverse problem to execute the grasping task.

[0101] Understandably, in addition to the reward value, a performance metric can be set for the flexible object manipulation task at each event step to detect the degree of task completion. Here, the time step and reward are the same, without any scaling. At the beginning of each round, the upper bound u and lower bound l of the performance metric need to be calculated.

[0102] Taking the task of manipulating a flexible object, such as flattening a soft cloth, as an example, the initial state is a randomly placed soft cloth. The performance metric is the calculated coverage area when the cloth is flattened on a table. When performing a flexible object manipulation task, the maximum coverage area can only be achieved when the cloth is completely flattened; conversely, a greater degree of unevenness will result in a smaller coverage area on the table. Therefore, the upper limit of the performance metric for this task is when the cloth is completely flattened, and the lower limit is close to but greater than zero. For standardization, considering the upper bound u and lower bound l of the task, this application embodiment sets the following formula to standardize the performance at each time step:

[0103]

[0104] Where s is the original performance value before normalization; These are the normalized performance values;

[0105] This application also includes a control experiment to verify the effectiveness of the above-described flexible object grasping method. The control experiment includes:

[0106] Three benchmark models were set up: CURL, PlaNet, and DrQ. CURL is a convolutional neural network based on a U-Net architecture, employing bidirectional convolution operations and path aggregation techniques. PlaNet is an image segmentation model based on a fully convolutional neural network, employing global pooling and multi-scale feature fusion techniques. DrQ is an image segmentation model based on a graph convolutional neural network, employing adaptive graph convolution and attention mechanisms. During training, the maximum number of training steps was set to 4e5, with data recorded every 1e4 steps. The average reward calculated based on performance metrics is as follows: Figure 6 As shown in the figure, during the task of manipulating a flexible object like a flattened cloth, the aforementioned flexible object grasping method remained stable, with its average reward consistently around 85, significantly outperforming the three traditional benchmark algorithms. Furthermore, the average unfolded area (maximum 1.0) and training time of the four algorithms are shown in Table 3 below. The PlaNet algorithm had the longest training time, reaching 167 hours. The flexible object grasping method provided in this embodiment has a longer training time, but it achieves the best performance per round, with a total average unfolded area of ​​0.94, nearly reaching a fully unfolded state.

[0107] Table 3 Comparison of average unfolded area and training time

[0108] CURL PlaNet DrQ This application Average unfolded area 0.51 0.86 0.52 0.94 Training time (h) 53.1 167.5 84.7 143.3

[0109] Based on the same inventive concept, please refer to Figure 7 This application also provides a flexible object grasping device 200, which includes:

[0110] The position evaluation module 210 is used to determine the Q value of each grasping position in the flexible object image by adopting a position evaluation strategy model; the Q value is used to characterize the probability that the grasping position is grasped.

[0111] The grasping strategy acquisition module 220 is used to determine the grasping target point and grasping posture based on the Q value of each grasping position using an attitude grasping strategy model.

[0112] The grasping control module 230 is used to control the robotic arm to perform a grasping operation on the grasping target point using the grasping posture.

[0113] As an optional implementation of the aforementioned flexible object grasping device, the position evaluation module 210 includes:

[0114] The first low-level feature acquisition unit is used to extract the first low-level features of the flexible object from the image of the flexible object by employing the first convolutional layer of the position evaluation strategy model.

[0115] The Q-value acquisition unit is used to determine the Q-value of each grasping position in the flexible object image based on the first fully connected layer of the position evaluation strategy model and the first bottom layer features.

[0116] As an optional implementation of the above-mentioned flexible object grasping device, the first low-level feature acquisition unit is specifically used to: extract the first low-level features of the flexible object from the flexible object image after it has been meshed, using the first convolutional layer of the position evaluation strategy model.

[0117] As an optional implementation of the aforementioned flexible object grasping device, the position evaluation module 210 further includes:

[0118] A heatmap acquisition unit is used to process the image of the flexible object into a heatmap based on the Q value, thereby acquiring a heatmap of the flexible object; the heatmap of the flexible object is used to characterize the Q value of each of the grasping positions;

[0119] At this time, the grasping strategy acquisition module 220 is specifically used to: adopt an attitude grasping strategy model and determine the grasping target point and grasping attitude based on the flexible object heat map that represents the Q value of each grasping position.

[0120] As an optional implementation of the above-mentioned flexible object grasping device, the grasping strategy acquisition module 220 includes:

[0121] The second low-level feature acquisition unit is used to extract the second low-level features based on the Q value of each grasping position using the second convolutional layer of the pose grasping strategy model.

[0122] The strategy acquisition unit is used to determine the target point and the grasping posture based on the second fully connected layer of the posture grasping strategy model and the second bottom layer features.

[0123] As an optional implementation of the above-mentioned flexible object grasping device, the grasping strategy acquisition module 220 further includes:

[0124] An additional information acquisition unit is used to acquire the current position and current posture of the robotic arm;

[0125] At this time, the strategy acquisition unit is specifically used to: adopt the second fully connected layer of the posture grasping strategy model, and obtain the grasping target point and grasping posture based on the second bottom layer features, the current position of the robotic arm and the current posture.

[0126] As an optional embodiment of the above-mentioned flexible object grasping device, the flexible object grasping device 200 further includes:

[0127] The posture grasping strategy model training module is used to train the position evaluation strategy model and the posture grasping strategy model using the Proximal Policy Optimization (PPO) method.

[0128] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. (Refer to...) Figure 8 The electronic device 300 includes a processor 310, a memory 320, and a communication interface 330. These components are interconnected and communicate with each other via a communication bus 340 and / or other forms of connection mechanism (not shown).

[0129] The memory 320 includes one or more (only one is shown in the figure), which may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The processor 310 and other possible components may access the memory 320 to read and / or write data therein.

[0130] Processor 310 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 310 described above can be a general-purpose processor, including a central processing unit (CPU), a microcontroller unit (MCU), a network processor (NP), or other conventional processors; it can also be a special-purpose processor, including a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0131] Communication interface 330 includes one or more (only one is shown in the figure) and can be used to communicate directly or indirectly with other devices to exchange data. For example, communication interface 330 can be an Ethernet interface; it can be a mobile communication network interface, such as an interface for 3G, 4G, or 5G networks; or it can be other types of interfaces with data transmission and reception functions.

[0132] One or more computer program instructions may be stored in the memory 320, and the processor 310 may read and run these computer program instructions to implement the flexible object grasping method provided in the embodiments of this application and other desired functions.

[0133] Understandable. Figure 8 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown. Figure 8 The components shown can be implemented using hardware, software, or a combination thereof. For example, electronic device 300 can be a single server (or other device with computing power), a combination of multiple servers, a cluster of a large number of servers, etc., and can be either a physical device or a virtual device.

[0134] This application also provides a computer-readable storage medium storing computer program instructions. These computer program instructions are read and executed by a computer's processor to perform the flexible object grasping method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 8 The memory 320 in the electronic device 300.

[0135] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0136] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0138] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for grasping flexible objects, characterized in that, The method includes: A position evaluation strategy model is used to determine the Q value of each grasping position in the image of the flexible object; the Q value is used to characterize the probability that the grasping position is grasped. An attitude-based grasping strategy model is adopted to determine the grasping target point and grasping attitude based on the Q value of each grasping position; wherein, the position evaluation strategy model and the attitude-based grasping strategy model are cascaded. The robotic arm is controlled to perform a grasping operation on the target point using the grasping posture described above. The method of using a position evaluation strategy model to obtain the grasping position of the flexible object and the Q value of each grasping position based on the flexible object image includes: using a first convolutional layer of the position evaluation strategy model to extract the first low-level features of the flexible object in the flexible object image; and using a first fully connected layer of the position evaluation strategy model to determine the Q value of each grasping position in the flexible object image based on the first low-level features. After determining the Q-value of each grasping position in the flexible object image based on the first low-level features in the first fully connected layer of the position evaluation strategy model, the method further includes: processing the flexible object image into a heatmap based on the Q-value to obtain a flexible object heatmap; the flexible object heatmap is used to characterize the magnitude of the Q-value of each grasping position. The adoption of the posture grasping strategy model, which determines the grasping target point and grasping posture based on the Q value of each grasping position, includes: adopting the posture grasping strategy model to determine the grasping target point and grasping posture based on the flexible object heat map used to characterize the Q value of each grasping position.

2. The flexible object grasping method of claim 1, wherein, The first convolutional layer of the position evaluation strategy model extracts the first low-level features of the flexible object from the flexible object image, including: The first convolutional layer of the position evaluation strategy model is used to extract the first low-level features of the flexible object from the image of the flexible object after it has been processed into a grid.

3. The flexible object grasping method of claim 1, wherein, The posture-based grasping strategy model determines the grasping target point and grasping posture based on the Q-value of each grasping position, including: The second convolutional layer of the pose-grabbing strategy model extracts the second low-level features based on the Q-values ​​of each grabbing position. The second fully connected layer of the posture grasping strategy model is used to determine the grasping target point and grasping posture based on the second bottom layer features.

4. The flexible object grasping method of claim 3, wherein, Before extracting the second low-level features based on the Q-values ​​of each grasping position in the second convolutional layer of the pose-grabbing strategy model, the method further includes: Obtain the current position and current orientation of the robotic arm; The second fully connected layer of the posture grasping strategy model determines the grasping target point and grasping posture based on the second bottom-layer features, including: The second fully connected layer of the posture grasping strategy model is used to obtain the grasping target point and grasping posture based on the second bottom layer features, the current position and the current posture of the robotic arm.

5. The flexible object grasping method according to any one of claims 1 to 4, characterized in that, Before employing the position evaluation strategy model to determine the Q-values ​​of each grasping position in the flexible object image, the method further includes: The position evaluation strategy model and the posture grasping strategy model are trained using the Proximal Policy Optimization (PPO) method.

6. A flexible object grasping device, characterized by The device includes: a location assessment module, a grasping strategy acquisition module, and a grasping control module, wherein... The position evaluation module is used to determine the Q value of each grasping position in the flexible object image by adopting a position evaluation strategy model; the Q value is used to characterize the probability that the grasping position is grasped. The grasping strategy acquisition module is used to determine the grasping target point and grasping posture based on the Q value of each grasping position using an attitude grasping strategy model; wherein, the position evaluation strategy model and the attitude grasping strategy model are cascaded. The grasping control module is used to control the robotic arm to grasp the target point using the grasping posture; The location assessment module includes: The first low-level feature acquisition unit is used to extract the first low-level features of the flexible object from the image of the flexible object by employing the first convolutional layer of the position evaluation strategy model. The Q-value acquisition unit is used to determine the Q-value of each grasping position in the flexible object image based on the first fully connected layer of the position evaluation strategy model and the first bottom layer features. The location assessment module also includes: A heatmap acquisition unit is used to process the image of the flexible object into a heatmap based on the Q value, thereby acquiring a heatmap of the flexible object; the heatmap of the flexible object is used to characterize the Q value of each of the grasping positions; The grasping strategy acquisition module is specifically used to: adopt an attitude grasping strategy model and determine the grasping target point and grasping attitude based on the flexible object heat map that represents the Q value of each grasping position.

7. An electronic device, comprising: include: A processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1 to 5 by calling the program instructions.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 5.

Images