Deep q-learning based local observation grasping view selection system and method

Abstract

A local observation grasping viewpoint selection system and method based on deep Q-learning includes: a point cloud resolver, a viewpoint encoder, a long short-term memory unit, an attention unit, and a Q-network. This invention models the viewpoint selection problem in multi-view robotic arm grasping as a sequential decision problem and uses reinforcement learning to construct a strategy to solve the local observation problem. In each viewpoint selection, all information from the beginning to the previous time step is fully considered, thereby obtaining a higher grasping success rate than existing methods.

Description

Technical Field

[0001] This invention relates to a technology in the field of robot control, specifically a local observation and grasping perspective selection system and method based on deep Q-learning for robotic arm grasping and robot active perception. Background Technology

[0002] In recent years, robot control, especially robotic arm grasping tasks, has become increasingly widely used in industry. However, in most grasping task settings, the camera is fixed at an external position to collect visual information about the work area, and the computer vision model only improves its ability to understand visual information. An improvement is to introduce multiple observation points to acquire information from multiple fields of view, thus filling in the blind spots of a single perspective and significantly improving the agent's grasping ability. Existing multi-view active perception methods all make a strong assumption—that the agent has a global perspective that can obtain information from all perspectives, or that the information from subsequent perspectives is a subset of that from preceding perspectives. These methods all rely on information entropy and only consider the selection of a single-step field of view. Summary of the Invention

[0003] To address the shortcomings of existing technologies and their inapplicability to multi-view grasping scenarios, this invention proposes a local observation grasping viewpoint selection system and method based on deep Q-learning. The viewpoint selection problem in multi-view robotic arm grasping is modeled as a sequential decision problem, and a reinforcement learning method is used to construct a strategy to solve the local observation problem. In each viewpoint selection, all information from the beginning to the previous moment is fully considered, thereby achieving a higher grasping success rate than existing methods.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a local observation and grasping perspective selection system based on deep Q-learning, comprising: a point cloud parser, a perspective encoder, a long short-term memory unit, an attention unit, and a Q-network. Specifically: the point cloud parser reconstructs n point cloud data into m feature points with features using PointNet++, transposes the reconstructed data, and then passes it through a multilayer perceptron to obtain a visual encoding. The perspective encoder obtains feature vectors based on all available perspectives in the perspective candidate pool using word embedding. The long short-term memory unit concatenates the current visual information features with the perspective encodings of all perspectives into a vector, and then passes it through a feedforward network to obtain state space encoding and action space encoding. The attention unit, based on the state space encoding and action space encoding, employs a low-order bilinear pooling attention mechanism and uses fully connected operations to convert the state encoding and action encoding to a length consistent with the hidden layers before calculating the state-action encoding. The Q-network, based on the state-action encoding information, obtains the expected reward for each action in the current state, i.e., the predicted grasping accuracy for selecting different perspectives.

[0006] The point cloud parser includes a point cloud aggregation unit, an extraction and transformation unit, and a multilayer perception unit. Specifically, the point cloud convolution unit performs aggregation operations on the point cloud data acquired by the depth camera through the backbone network of PointNet++ to obtain the reconstructed point cloud and the features of each point; the extraction and transformation unit transposes the obtained points and features, increasing the scale of the features from a single point to the entire point cloud space; and the multilayer perception unit compresses the dimension of multiple points in the features to 1 to obtain the feature vector of the entire point cloud.

[0007] The view encoder includes a one-hot encoding unit, a linear mapping unit, and a nonlinear activation unit, wherein: the one-hot encoding unit numbers the viewpoints in circumferential order according to the viewpoints to obtain the one-hot encoding; the linear mapping unit performs a linear transformation on the one-hot encoding to obtain a low-dimensional mapping of the one-hot encoding; and the nonlinear activation unit passes the low-dimensional linear representation through an activation function to obtain the final view encoding of each viewpoint in that dimension.

[0008] The Long Short-Term Memory (LSTM) unit comprises an input unit, a forgetting unit, and an output unit. The input unit performs linear transformation and nonlinear activation based on the visual point cloud input features acquired at the current moment to obtain the encoding within the memory unit for the current viewpoint. The forgetting unit performs linear transformation and nonlinear activation based on the hidden layer state encoding from the previous moment and the visual point cloud input features at the current moment to obtain a weakened state encoding for hidden layer information that is too far removed from the past. The output unit performs linear transformation and nonlinear activation based on the encoding information of the input layer and hidden layer generated by the input unit and the forgetting unit to obtain the output number of the viewpoint to be acquired at the next moment.

[0009] Preferably, in order to process the different action space at each moment, the view encoding information transmitted between the view encoder and the attention unit follows the structure of the pointer network, that is, all actions are pre-encoded, and the state-action code is calculated one by one according to the remaining view in the candidate pool at each moment.

[0010] This invention relates to a local observation and grasping viewpoint selection method based on the above-mentioned system. A point cloud resolver acquires point cloud data from a depth camera at each selected viewpoint and processes it into feature vectors, which serve as the visual input data for the currently selected viewpoint. The viewpoint encoder takes unselected viewpoints as candidate viewpoints, i.e., the action space at that moment. The viewpoint code is concatenated with its corresponding visual point cloud code and input into a long short-term memory unit to obtain the state code at the moment of viewpoint selection. The state code is then input into an attention unit along with the actions in the action space at that moment to obtain the state-action code for that moment. A Q-network is used to calculate the predicted long-term reward (Q-value) for each action under the state-action code at that moment, and the maximum value is taken as the action output for that moment. After selecting a new viewpoint based on argmax, the visual information of that viewpoint is obtained as the observation for the next moment. Simultaneously, the action space is updated and iteratively updated until the required number of viewpoints are selected, thus completing the viewpoint selection task in multi-view grasping.

[0011] The splicing refers to: visual information features with a dimension of 512, viewpoint encoding with a dimension of 128, and a spliced ​​vector length of 640.

[0012] The low-order bilinear pooling attention mechanism refers to the following: First, the state code and view code grids output by the long short-term memory unit are aligned to the same dimension after linear transformation and tanh activation function. Then, an add operation is performed on both in the same dimension, followed by tanh activation, to obtain the code of each selectable view after reassigning attention weights to the new view information.

[0013] Technical effect

[0014] This invention models the viewpoint selection problem in multi-view robotic arm grasping as a sequential decision problem and uses reinforcement learning to construct a strategy to solve the local observation problem; compared with the rule / heuristic-based baseline algorithm, it achieves a 10% improvement in grasping accuracy; compared with the brute-force enumeration method, it only requires 1 / 40 of the time to approach the global optimum. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention;

[0016] Figure 2 This is a schematic diagram of the system of the present invention;

[0017] Figure 3 This is a schematic diagram of the training curve after removing two key modules in the embodiment;

[0018] Figure 4 This is a schematic diagram comparing the actual crawling performance of the present invention with that of various baseline algorithms;

[0019] In the figure: (a) random method, (b) mean method, (c) the present invention, (d) greedy method, (e) select all method. Detailed Implementation

[0020] This embodiment is based on a multi-view crawling network dataset, which is an extension of the GraspNet dataset. In this embodiment, the crawling network dataset is reconstructed into a reinforcement learning-friendly environment, which helps determine the selection of multiple views. In this environment, multiple views can be combined into a scenario based on reinforcement learning methods.

[0021] like Figure 2 As shown, this embodiment illustrates a local observation and grasping viewpoint selection system based on deep Q-learning, comprising: a point cloud resolver, a viewpoint encoder, a long short-term memory unit, an attention unit, and a Q-network. To simplify the learning-oriented motion primitives, this embodiment divides a 360-degree circle into k = 16 viewpoints, i.e., 22.5° / viewpoint. This is also the number of viewpoints in the candidate pool. The initial 16 viewpoints are encoded as the action space. Whenever the local observation and grasping viewpoint selection system selects a viewpoint, a pointer network is used to remove that viewpoint from the selectable viewpoint set.

[0022] like Figure 1 As shown, this embodiment relates to a method for selecting a local observation capture perspective based on the above system, including:

[0023] Step 1: Read the initial viewpoint point cloud data and obtain its visual encoding through the point cloud parser; and convert all the selectable views in the viewpoint candidate pool into feature vectors through the viewpoint encoder.

[0024] Step 2: Combine the above two data points as the observation at that moment, and convert them into a state code for that moment through a long short-term memory unit;

[0025] Step 3: Perform attention operations on the state code and all actions in the action space at the current moment to calculate the state-action code at that moment;

[0026] Step 4: Encode the state and actions using a Q-network to obtain the predicted long-term reward for each action in the current state;

[0027] Step 5: Perform argmax operation on the prediction results of the previous step to obtain the action at the current moment, i.e., the selected viewpoint number;

[0028] Step Six: Based on the selected viewpoint, perform the actual field of view selection operation, adjust the camera position to obtain the visual information of the corresponding position, and delete the viewpoint selected in this step from the viewpoint candidate pool to update the candidate pool.

[0029] Step 7: Repeat the steps until the desired number of fields of view are reached.

[0030] like Figure 3 As shown, when only point cloud data is fed into the Long Short-Term Memory network instead of being concatenated with viewpoint encoding, the convergence speed during training is significantly slower, and the final grasping accuracy is slightly inferior to the version with viewpoint encoding; another version without attention units experiences a huge drop in grasping accuracy. This indicates that attention units are crucial for the system of this invention to find a suitable viewpoint.

[0031] Through specific practical experiments in the GraspNet-aroundview environment, this method was compared with baseline algorithms such as average and random. Simultaneously, a greedy algorithm based on brute-force enumeration and a select-all method exceeding the selectable limit were tested, resulting in the data shown in Table 1. Tests showed that compared to rule-based / heuristic baseline algorithms, this invention achieved a 10% improvement in crawling accuracy; compared to the brute-force enumeration method, it only requires 1 / 40th of the time to approach the global optimum.

[0032] Table 1

[0033]

[0034] Random: The random method randomly selects 5 perspectives from 16 candidate perspectives for each task, which can be considered as the lower bound of the multi-view selection task in multi-view capture.

[0035] Average: The averaging method selects 5 perspectives at equal intervals from 16 perspectives for each task. Different starting points are chosen to represent the average performance. The averaging method is the most common and fundamental solution for multi-view object grasping.

[0036] Greedy: The greedy method selects the top 5 performing viewpoints from 16 prepared viewpoints for each task (by exploring all possible combinations) based on the information gain of all viewpoints. This method is similar to brute-force search, meaning the system has global information at the start.

[0037] Fixed: The full-fill method combines all 16 views for each task.

[0038] This method outperforms all baseline algorithms except the greedy algorithm because it directly greedily selects the highest-scoring solution from all candidate view combinations after exhaustive search, but this is extremely wasteful of computation time. Compared with existing technologies, this invention achieves the highest level while remaining within the limitations of local observations and significantly reducing computational overhead. As shown in Table 2, due to the computational overhead of neural networks, this invention is slightly slower than random and mean methods; however, due to the huge overhead of the greedy method's brute-force search, this invention achieves approximately 35 times the computational efficiency while reaching a similar level of grasping accuracy.

[0039] Table 2

[0040] Random methods 12.10±0.19 Mean method 12.24±0.44 This invention 12.27±0.36 Greedy method 434.91±0.06 Select All 17.69±0.39

[0041] Compared to existing technologies, this method solves the multi-view selection problem through learning rather than heuristic design, achieving a 10% improvement in grasping accuracy compared to rule / heuristic-based baseline algorithms; and approaching the global optimum in just 1 / 40th of the time compared to brute-force enumeration methods.

[0042] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A local observation and capture viewpoint selection system based on deep Q-learning, characterized in that, include: The system consists of a point cloud parser, a view encoder, a long short-term memory (LSTM) unit, an attention unit, and a Q-network. Specifically: The point cloud parser reconstructs n point cloud data into m feature points using PointNet++, transposes the reconstructed data, and then passes it through a multilayer perceptron to obtain visual encoding. The view encoder uses word embedding to obtain feature vectors based on all available viewpoints in the viewpoint candidate pool. The LSM unit concatenates the current visual information features with the viewpoint encodings of all viewpoints into a vector, and then passes it through a feedforward network to obtain state-space and action-space state encodings. The attention unit uses a low-order bilinear pooling attention mechanism based on the state-space and action-space state encodings, and then uses fully connected operations to convert the state and action encodings to the same length as the hidden layers before calculating the state-action encoding. The Q-network uses the state-action encoding information to obtain the expected reward for each action in the current state, i.e., the prediction accuracy for different viewpoints.

2. The local observation and capture viewpoint selection system based on deep Q-learning according to claim 1, characterized in that, The point cloud parser includes a point cloud aggregation unit, an extraction and transformation unit, and a multilayer perception unit. Specifically, the point cloud convolution unit performs aggregation operations on the point cloud data acquired by the depth camera through the backbone network of PointNet++ to obtain the reconstructed point cloud and the features of each point; the extraction and transformation unit transposes the obtained points and features, increasing the scale of the features from a single point to the entire point cloud space; and the multilayer perception unit compresses the dimension of multiple points in the features to 1 to obtain the feature vector of the entire point cloud.

3. The local observation and capture viewpoint selection system based on deep Q-learning according to claim 1, characterized in that, The view encoder includes a one-hot encoding unit, a linear mapping unit, and a nonlinear activation unit, wherein: the one-hot encoding unit numbers the viewpoints in circumferential order according to the viewpoints to obtain the one-hot encoding; the linear mapping unit performs a linear transformation on the one-hot encoding to obtain a low-dimensional mapping of the one-hot encoding; and the nonlinear activation unit passes the low-dimensional linear representation through an activation function to obtain the final view encoding of each viewpoint in that dimension.

4. The local observation and capture viewpoint selection system based on deep Q-learning according to claim 1, characterized in that, The Long Short-Term Memory (LSTM) unit comprises an input unit, a forgetting unit, and an output unit. The input unit performs linear transformation and nonlinear activation based on the visual point cloud input features acquired at the current moment to obtain the encoding within the memory unit for the current viewpoint. The forgetting unit performs linear transformation and nonlinear activation based on the hidden layer state encoding from the previous moment and the visual point cloud input features at the current moment to obtain a weakened state encoding for hidden layer information that is too far removed from the past. The output unit performs linear transformation and nonlinear activation based on the encoding information of the input layer and hidden layer generated by the input unit and the forgetting unit to obtain the output number of the viewpoint to be acquired at the next moment.

5. A method for selecting a local observation capture perspective based on the system described in any one of claims 1-4, characterized in that, The point cloud parser acquires point cloud data from the depth camera at each selected viewpoint and processes it into feature vectors, which serve as the visual input data for the currently selected viewpoint. The view encoder takes unselected viewpoints as candidate viewpoints, i.e., the action space at that moment. The viewpoint code is concatenated with its corresponding visual point cloud code and input into the Long Short-Term Memory unit to obtain the state code at the moment of viewpoint selection. The state code is then input into the attention unit along with the actions in the action space at that moment to obtain the state-action code at that moment. The Q-network calculates the predicted long-term reward (Q-value) for each action under the state-action code at that moment and takes the maximum value as the action output at that moment. After selecting a new viewpoint based on argmax, the visual information of that viewpoint is obtained as the observation for the next moment. At the same time, the action space is updated and iteratively updated until the required number of viewpoints are selected, thus completing the viewpoint selection task in multi-view capture.

6. The method for selecting a local observation capture perspective according to claim 5, characterized in that, The splicing refers to: visual information features with a dimension of 512, viewpoint encoding with a dimension of 128, and a spliced ​​vector length of 640.

7. The method for selecting a local observation capture perspective according to claim 5, characterized in that, The low-order bilinear pooling attention mechanism refers to the following: First, the state code and view code grids output by the long short-term memory unit are aligned to the same dimension after linear transformation and tanh activation function. Then, an add operation is performed on both in the same dimension, followed by tanh activation, to obtain the code of each selectable view after reassigning attention weights to the new view information.

Images