An accelerated relational grasping method based on deep reinforcement learning

Abstract

The application discloses an acceleration relational type grabbing method based on deep reinforcement learning, first, the RGB-D image information of a scene is collected and converted to a given overhead perspective, then the learnable grabbing embedding is extracted and the received RGB-D image and the historical grabbing embedding are sent into the grabbing network, the grabbing network introduces the encoder-decoder architecture and the improved attention mechanism to finally output the Q graph, the optimal grabbing point is extracted from the Q graph and the final grabbing coordinates are obtained through corresponding transformation for the grabbing of the mechanical arm. The application can improve the generalization of the grabbing method, enhance the spatial and temporal perception ability and improve the reasoning speed of the model, and ensure that the intelligent agent completes the grabbing task in a complex scene.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and machine learning, and in particular to an accelerated relation-based grasping method based on deep reinforcement learning. Background Technology

[0002] Deep learning (DL) is widely used in robotic grazing due to its powerful feature learning and representation capabilities and its end-to-end learning characteristics. However, deep learning-based grazing methods typically require large amounts of labeled data for training and often underutilize this data, both spatially and temporally. Spatially, objects are typically placed in specific ways; for example, cups and cutlery are usually placed on tables, while shoes and trash cans are usually placed on the ground. A lack of understanding of spatial relationships weakens the model's generalization ability. Temporally, data presented as images lacks temporal correlation, limiting the model's ability to learn dynamic grazing tasks. Therefore, the spatial independence of data requires grazing models to fit different grazing scenarios with a large amount of data, resulting in insufficient generalization with limited samples. The temporal independence of data limits the robot's decision-making ability in complex grazing tasks. Furthermore, traditional deep learning methods are insufficient in handling unstructured environments and changing scenarios. Improving the generalization of crawling methods, enhancing spatial and temporal perception capabilities, and increasing the inference speed of models have become hot research topics in crawling methods. Summary of the Invention

[0003] The purpose of this invention is to provide an accelerated relational grasping method based on deep reinforcement learning. This invention can improve the generalization of the grasping method, enhance spatial and temporal perception capabilities, and improve the inference speed of the model, ensuring that the agent can complete the grasping task in complex scenarios.

[0004] The technical solution provided by this invention is as follows: A method for accelerating relation-based data crawling based on deep reinforcement learning, comprising the following steps:

[0005] Step 1: Construct a crawling network and model the crawling problem as a Markov decision process;

[0006] Step 2: Determine the pose of the image acquisition device and acquire RGB-D image information. Then, transform the acquired RGB-D image to the viewpoint of the given virtual camera through perspective transformation.

[0007] Step 3: Extract learnable grasping embeddings from the converted RGB-D image, and input the historical grasping embeddings and the grasping embeddings extracted from the RGB-D image into the grasping network. The encoder of the grasping network generates the corresponding feature representation, which is a set of several abstract entities.

[0008] Step 4: Input the abstract entity into the relation module to extract relations and generate the corresponding relation vector;

[0009] Step 5: The generated relation vector constitutes the feature representation of the integrated relation. The decoder of the grasping network receives this feature representation and outputs a Q map with the same width and height as the original RGB-D image. The position with the largest Q value in the Q map is the pixel coordinate of the grasping point in the virtual camera. Then, the camera coordinate of the grasping point in the image acquisition device is obtained based on the pixel coordinate. Finally, the base coordinate of the grasping point in the robotic arm is obtained through the hand-eye calibration method for the robotic arm to grasp.

[0010] The aforementioned accelerated relation-based crawling method based on deep reinforcement learning uses a crawling network based on the DQN network in deep reinforcement learning. It leverages the interaction and adaptive adjustment capabilities of reinforcement learning with the environment to address the problem of scarce crawling data, improves the processing capability for unstructured environments, introduces the encoder-decoder architecture from deep learning, improves the output layer of the DQN network, and introduces an improved attention mechanism as a relation module to enhance the model's ability to capture spatial relations.

[0011] The aforementioned method for accelerating relational grasping based on deep reinforcement learning models the grasping problem as a Markov decision process, where the state is a scene representation containing historical grasping information, the action is a tuple consisting of grasping operation and grasping position, and the reward indicates whether the grasping operation is successful and the effect of the grasping operation.

[0012] The aforementioned method for accelerating relation-based data crawling based on deep reinforcement learning, specifically the process of extracting learnable crawling embeddings from the converted RGB-D image, is as follows:

[0013] First, the converted RGB-D image is tiled, and then passed through the corresponding multilayer perceptron φ. i Encoding the tiled RGB-D image:

[0014] g i =φ i (flatten(h)), i∈{o, p, t},

[0015] Where h is the converted RGB-D image, and flatten(·) is the tiling operation.

[0016] The resulting capture embedding g is a triple:

[0017] g=(g o g p g t );

[0018] Among them, g o To capture object embedding, g p For location embedding, gt is for task embedding.

[0019] In the aforementioned method for accelerating relation-based data extraction based on deep reinforcement learning, step 4 involves inputting abstract entities into the relation module to extract relations and generate corresponding relation vectors.

[0020] First, a two-layer MLP is used to process each abstract entity e. i Extract the corresponding query vector q i Key vector k i Sum vector v i Then, use MLP For the key vector set k 1：N Value vector set v 1：N Perform compression:

[0021]

[0022] Where, x 1：N Let N be the set of vectors before compression. Let be the compressed vector set with capacity O(n). when Explanation of MLP Features of the core region in the scene were extracted.

[0023] Subsequently, each abstract entity e i The corresponding query vector q i Both will be associated with the compressed key vector set Calculate the attention weight vector

[0024]

[0025] Where <·> represents the inner product of two vectors;

[0026] Received Normalization needs to be performed using the softmax function, resulting in:

[0027]

[0028] in, The weights are the normalized weights.

[0029] In the aforementioned method for accelerating relation-based data crawling based on deep reinforcement learning, step 5 involves the crawling network's decoder receiving a set of entities incorporating relational features. Output the Q-map Q(s), from which the pixel coordinates of the optimal capture point in the virtual camera are obtained:

[0030] p = argmax Q(s); then, the camera coordinates of the grasping point in the image acquisition device are obtained through the inverse perspective transformation in step 2; finally, the base coordinates p of the grasping point in the robotic arm are obtained through the hand-eye calibration method. base Ultimately, the grasping action is composed of a grasping operation and a grasping position tuple.

[0031] Compared to existing technologies, this invention focuses on unstructured and dynamic complex environments to improve the generalization of grasping methods, enhance spatial and temporal perception capabilities, and increase the inference speed of the model, ensuring that the agent can complete grasping tasks in complex scenarios. This invention uses Deep Q Networks (DQNs) from deep reinforcement learning as a baseline method, introducing encoder-decoder architectures from deep learning and attention mechanisms from natural language processing. The deep reinforcement learning method in this invention integrates the representation learning capabilities of deep learning into reinforcement learning, improving the perception capabilities in complex grasping scenarios. The attention mechanism can effectively handle spatial object relationships. By introducing this technology into the grasping network, the method proposed in this invention can focus on the relationships between core objects in the environment, which are preserved even when non-core parts of the environment change, thus possessing stronger generalization capabilities than traditional deep learning methods. Furthermore, recognizing the sparsity of grasping points in grasping tasks, this invention improves the attention mechanism to accelerate the inference of object relationships. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the scene in the example;

[0033] Figure 2 This is a schematic diagram of the ablation experiment results for the compression ratio of the relation module in the example. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0035] Example: A method for accelerating relation-based data crawling based on deep reinforcement learning, comprising the following steps:

[0036] Step 1: Construct a crawling network to model the crawling problem as a Markov decision process. The crawling network proposed in this invention draws on the DQN network in deep reinforcement learning. It utilizes the interaction capabilities of reinforcement learning with the environment and its adaptive adjustment capabilities to solve the problem of scarce crawling data, improve the processing capability of unstructured environments, introduces the encoder-decoder architecture from deep learning, improves the output layer of the DQN network, and introduces an improved attention mechanism as a relation module to enhance the model's ability to capture spatial relationships.

[0037] In this step, the crawling problem is modeled as an MDP tuple. Where S represents the state (containing a scene representation with historical capture information), and A represents the action set (containing capture operations and capture positions, etc.). Let S be the state transition matrix (S represents the current state, s′ represents the next state, and a represents the action performed at the current moment), r be the reward (indicating whether the grabbing operation was successful and its effect), and γ be the discount factor. The scenario in this example is as follows: Figure 1 As shown, the scene contains two objects: a hollow cylinder (representing a container) and a solid cube (representing goods). This example requires the agent to learn the correct grasping order (first pick up the goods, then place them into the container). In this example, r ∈ {-1, 0, 1}. When the robotic arm successfully grasps the object and the grasping order is correct (e.g., picking up the goods first), r = 1; when the robotic arm successfully grasps the object but the grasping order is incorrect (e.g., picking up the container first) or the robotic arm collides with the object, r = -1; when the robotic arm fails to grasp the object, r = 0. Furthermore, in this example, γ is set to 0.9.

[0038] Step 2: Determine the pose of the image acquisition device and acquire RGB-D image information. Transform the acquired RGB-D image to the viewpoint of a given virtual camera (viewing the robotic arm's operating space from above) using perspective transformation. Extract a learnable grasping embedding from the transformed RGB-D image, and input the historical grasping embedding g′ (which can be acquired) along with the RGB-D image into the grasping network. By introducing historical grasping embeddings, the grasping network can learn from the contextual information of grasping, enhancing the model's temporal perception and improving the robot's decision-making ability in complex grasping tasks.

[0039] The crawling and embedding process in this step is as follows:

[0040] First, the converted RGB-D image is tiled, and then passed through the corresponding multi-layer perceptron (MLP) φ iEncoding the tiled RGB-D image:

[0041] g i =φ i (flatten(h)), i∈{o, p, t},

[0042] Where h is the converted RGB-D image, and flatten(·) is the tiling operation.

[0043] The resulting capture embedding g is a triple:

[0044] g=(g o g p g t );

[0045] Among them, g o To capture object embedding, g p To capture the location embedding, g t Embedded for task crawling.

[0046] Step 3: Extract learnable crawling embeddings from the converted RGB-D image, and input the historical crawling embeddings and the crawling embeddings extracted from the RGB-D image into the crawling network. The encoder of the crawling network generates the corresponding feature representation. This feature representation has smaller width and height dimensions and deeper feature channels than the input RGB-D image, and can be understood as a set of several "abstract entities". In this step, the encoder of the crawling network (with the same architecture as the encoder in FCN) receives the RGB-D image and the historical crawling embeddings g, and generates the corresponding feature representation, which can be understood as a set e of N "abstract entities". 1：N In this example, N is set to 16.

[0047] Step 4: Input the generated "abstract entities" into the relation module to extract relations. The relation module generates a corresponding relation vector for each "abstract entity." Specifically, the "abstract entities" are first compressed to obtain the "abstract entities" of the core regions in the scene. Then, each "abstract entity" obtains its corresponding relation vector under the influence of the "abstract entities" of the core regions. When the scene changes, the core regions usually remain unchanged, and the core relations in the scene are preserved. The model possesses the ability to reason about spatial relations and handle changing scenes, thereby improving the generalization of the crawling. In addition, the extraction of the "abstract entities" of the core regions simplifies the calculation of relations and improves the inference speed of the model.

[0048] Specifically, a two-layer MLP is first used for each abstract entity e. i Extract the corresponding query vector q i Key vector k i Sum vector v iThen, use MLP For the key vector set k 1：N Value vector set v 1：N Perform compression:

[0049]

[0050] Where, x 1：N Let N be the set of vectors before compression. Let be the compressed vector set with capacity O(n). when Explanation of MLP Features of the core region in the scene were extracted.

[0051] Subsequently, each abstract entity e i The corresponding query vector q i Both will be associated with the compressed key vector set Calculate the attention weight vector

[0052]

[0053] Where <·> represents the inner product of two vectors;

[0054] Received Normalization needs to be performed using the softmax function, resulting in:

[0055]

[0056] in, The weights are the normalized weights.

[0057] Relationship module compression rate The results of the ablation experiment are as follows Figure 2 As shown, since there are two objects in this example, there are also two core regions. Therefore, when the compression ratio... At that time, the relation module performed best, with the model requiring the fewest inference steps of 56.

[0058] Step 5: The generated relation vectors constitute the feature representation of the incorporation relation. The decoder of the grasping network receives this feature representation and outputs a Q-map with the same width and height as the original RGB-D image. The Q-map stores the cumulative reward value at the corresponding position. The position with the largest Q-value in the Q-map is the pixel coordinate of the grasping point in the virtual camera. The pixel coordinates of the grasping point are transformed into camera coordinates in the virtual camera through the intrinsics of the virtual camera. Then, the camera coordinates of the grasping point in the image acquisition device are obtained through the inverse perspective transformation in Step 2. Finally, the base coordinates of the grasping point in the robotic arm are obtained through the hand-eye calibration method for the robotic arm's grasping. The reward value of the grasping point is obtained according to the reward mechanism, and the Q-value at the corresponding position in the Q-map is updated through reinforcement learning. In this step, the decoder of the grasping network (with the same architecture as the decoder in FCN) receives the entity set of incorporation relation features. Output the Q-map Q(s). From Q(s), obtain the pixel coordinates of the optimal grasping point in the virtual camera: p = argmaxQ(s), and through appropriate transformations, obtain the grasping coordinates p in the robotic arm base coordinate system. base Finally, the action is generated. In this example, the grab operation and grab position are a pair.

[0059] In summary, this invention can improve the generalization of grasping methods, enhance spatial and temporal perception capabilities, and improve the inference speed of the model, ensuring that the agent can complete grasping tasks in complex scenarios.

Claims

1. A method for accelerating relation-based data crawling based on deep reinforcement learning, characterized in that: Includes the following steps: Step 1: Construct a crawling network and model the crawling problem as a Markov decision process; Step 2: Determine the pose of the image acquisition device and acquire RGB-D image information. Then, transform the acquired RGB-D image to the viewpoint of the given virtual camera through perspective transformation. Step 3: Extract learnable grasping embeddings from the converted RGB-D image, and input the historical grasping embeddings and the grasping embeddings extracted from the RGB-D image into the grasping network. The encoder of the grasping network generates the corresponding feature representation, which is a set of several abstract entities. Step 4: Input the abstract entity into the relation module to extract relations and generate the corresponding relation vector; Step 5: The generated relation vector constitutes the feature representation of the integrated relation. The decoder of the grasping network receives this feature representation and outputs a Q map with the same width and height as the original RGB-D image. The position with the largest Q value in the Q map is the pixel coordinate of the grasping point in the virtual camera. Then, the camera coordinate of the grasping point in the image acquisition device is obtained based on the pixel coordinate. Finally, the base coordinate of the grasping point in the robotic arm is obtained through the hand-eye calibration method for the robotic arm to grasp. In step 4, the abstract entity is input into the relation module to extract relations and generate the corresponding relation vector: First, a two-layer MLP is used to process each abstract entity. Extract the corresponding query vector Key vector Sum value vector Then, use MLP For key vector set Value vector set Perform compression: ； in, Let be the set of vectors before compression, with a capacity of . , Let be the compressed vector set with capacity O(n). ;when Explanation of MLP Features of the core region in the scene were extracted. Subsequently, each abstract entity Corresponding query vector Both will be associated with the compressed key vector set Calculate the attention weight vector : ； in, This represents the dot product of two vectors; Received Need to use After normalizing the function, the final result is: ， in, The weights are the normalized weights.

2. The accelerated relation-based data crawling method based on deep reinforcement learning according to claim 1, characterized in that: The described crawling network is based on the DQN network in deep reinforcement learning. It utilizes the interaction and adaptive adjustment capabilities of reinforcement learning with the environment to solve the problem of scarce crawling data, improves the processing capability of unstructured environments, introduces the encoder-decoder architecture from deep learning, improves the output layer of the DQN network, and introduces an improved attention mechanism as a relation module to enhance the model's ability to capture spatial relationships.

3. The accelerated relation-based data crawling method based on deep reinforcement learning according to claim 2, characterized in that: The grasping problem is modeled as a Markov decision process, where the state is a scene representation containing historical grasping information, the action is a tuple consisting of grasping operation and grasping position, and the reward indicates whether the grasping operation is successful and the effect of the grasping operation.

4. The accelerated relation-based data crawling method based on deep reinforcement learning according to claim 1, characterized in that: The process of extracting learnable crawling embeddings from the converted RGB-D image is as follows: First, the converted RGB-D image is tiled, and then passed through the corresponding multilayer perceptron. Encoding the tiled RGB-D image: ， in For the converted RGB-D image, For tiling operation; The obtained crawling embedding For a triple: ； in, To embed objects, To capture the location embedding, Embedded for task crawling.

5. The accelerated relation-based data crawling method based on deep reinforcement learning according to claim 1, characterized in that: In step 5, the decoder of the crawling network receives the set of entities incorporating relational features. Output Q graph ,Depend on Obtain the pixel coordinates of the optimal capture point in the virtual camera: Then, the camera coordinates of the grasping point in the image acquisition device are obtained through the inverse perspective transformation in step 2. Finally, the base coordinates of the grasping point in the robotic arm are obtained through the hand-eye calibration method. Ultimately, the grasping action is composed of a grasping operation and a grasping position tuple.

Images