A training method, recognition method, system, and device for a crawling detection model.

Abstract

The application discloses a kind of training method of grabbing detection model, identification method, system, equipment.The method is by obtaining and carries out point cloud conversion processing to RGB-D image training set, obtains point cloud dataset;RGB-D image training set is carried out multi-scale feature extraction processing, obtains global feature map set;Respectively, feature interpolation processing and size balance processing are carried out to point cloud dataset, and interpolation feature dataset and local feature dataset are obtained;Local feature dataset and interpolation feature dataset are carried out feature fusion processing, obtain and carry out multi-angle prediction processing to semantic space feature set, obtain training prediction candidate set;According to training prediction candidate set, the parameter of initialized grabbing detection model is updated, and grabbing detection model is obtained.The method can effectively improve the distinguishing ability of grabbing detection model to the edge of stacked object, effectively improve the identification accuracy of grabbing detection model.The application can be widely applied in artificial intelligence technical field.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method, recognition method, system, and device for a grasping and detection model. Background Technology

[0002] In recent years, with the continuous development of social technology, the application of robots / robotic arms to automatically grasp target objects in stacked scenarios has become increasingly common.

[0003] Currently, there are two main traditional methods for automatically identifying and grasping objects in stacked scenarios:

[0004] The first method is to directly output one or more object grasping boxes from a planar RGB image. This can be achieved using the YOLO algorithm. The grasping parameters mainly include the position of the center point of the object to be grasped and the rotation angle. This method is suitable for the robotic arm to grasp from top to bottom and works well for a single object. However, the more complex the scene, the worse the performance of the algorithm becomes. It is more likely to misjudge the edges of the target objects in the scene and lacks depth information, which makes the robotic arm unable to grasp accurately and may even damage the surface of the object.

[0005] The second method uses scene point clouds obtained directly or indirectly (RGB-D images) as input. It leverages the semantic and geometric information of the point clouds to feed them into a backbone network to obtain the 6D (six degrees of freedom) pose of the object to be grasped and then performs the grasping. However, this method is too idealistic and heavily reliant on the environment (lighting around the camera and the quality of the point cloud data). In stacked scenes, it easily treats different objects in contact as a single entity, and it has low tolerance for small or irregularly shaped objects. In scenes where collision-free conditions are required, the algorithm's assessment of grasping quality is too low, leading to the failure to successfully grasp objects in close contact within the scene.

[0006] Therefore, the problems existing in the current technology still need to be solved and optimized. Summary of the Invention

[0007] The purpose of this invention is to at least partially solve one of the technical problems existing in the related art.

[0008] Therefore, the first objective of this invention is to provide a training method for a grasping detection model, which can effectively improve the grasping detection model's ability to distinguish the edges of stacked objects and effectively improve the recognition accuracy of the grasping detection model.

[0009] The second objective of this application is to provide a method for recognizing a grasping detection model.

[0010] The third objective of this application is to provide a training system for a grasping detection model.

[0011] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of this application include:

[0012] In a first aspect, embodiments of this application provide a training method for a grasping detection model, the training method comprising:

[0013] Obtain a labeled RGB-D image training set, and perform point cloud transformation processing on the labeled RGB-D image training set to obtain a point cloud dataset;

[0014] Multi-scale feature extraction processing is performed on the labeled RGB-D image training set to obtain a global feature map set;

[0015] Based on the global feature map set, feature interpolation processing is performed on the point cloud dataset to obtain an interpolated feature set, and size balancing processing is performed on the point cloud dataset to obtain a local feature dataset;

[0016] The local feature dataset and the interpolated feature dataset are subjected to feature fusion processing to obtain a semantic space feature set, and the semantic space feature set is subjected to multi-angle prediction processing to obtain a training prediction candidate set.

[0017] The parameters of the initialized crawling detection model are updated based on the training prediction candidate set to obtain the trained crawling detection model.

[0018] In addition, the training method according to the above embodiments of this application may also have the following additional technical features:

[0019] Furthermore, in one embodiment of this application, the step of performing multi-scale feature extraction processing on the labeled RGB-D image training set to obtain a global feature map includes:

[0020] The first feature extraction process is performed on the RGB-D image training set to obtain the first feature map set;

[0021] The first feature map set is subjected to a second feature extraction process to obtain a second feature map set;

[0022] The second feature map set is subjected to third feature extraction processing to obtain the third feature map set;

[0023] Semantic aggregation processing is performed on the first feature map set, the second feature map set, and the third feature map set to obtain a global feature map set.

[0024] Furthermore, in one embodiment of this application, the step of performing feature interpolation processing on the point cloud dataset based on the global feature map set to obtain an interpolated feature set includes:

[0025] The point cloud dataset is subjected to upsampling and instance segmentation processes in sequence to obtain the first intermediate dataset;

[0026] The first intermediate dataset is downsampled to obtain the second intermediate dataset;

[0027] The second intermediate dataset is subjected to feature interpolation processing based on the global feature map set to obtain the interpolated feature set.

[0028] Furthermore, in one embodiment of this application, the step of performing size balancing processing on the point cloud dataset to obtain a local feature dataset includes:

[0029] Obtain a preset multi-layered annular cylinder;

[0030] The point cloud dataset is grouped according to the multi-layered annular cylinder to obtain a grouped dataset.

[0031] The local feature dataset is obtained by performing perceptual processing on the grouped dataset based on the first intermediate dataset.

[0032] Furthermore, in one embodiment of this application, the step of performing multi-angle prediction processing on the semantic space feature set to obtain a training prediction candidate set includes:

[0033] The semantic space feature set is upsampled to obtain a third intermediate dataset;

[0034] The third intermediate dataset is subjected to width processing to obtain a width candidate set;

[0035] The third intermediate dataset is processed to obtain a candidate set of angles;

[0036] The third intermediate dataset is subjected to crawling probability processing to obtain a probability candidate set;

[0037] The training prediction candidate set is generated based on the width candidate set, the angle candidate set, and the probability candidate set.

[0038] Furthermore, in one embodiment of this application, the step of updating the parameters of the initialized crawling detection model based on the trained prediction candidate set to obtain a trained crawling detection model includes:

[0039] Obtain the stacked object labels corresponding to the training prediction candidate set;

[0040] The training loss value is determined based on the training prediction candidate set and the stacked object labels;

[0041] Based on the training loss value, the parameters of the initialized crawling detection model are updated to obtain the trained crawling detection model.

[0042] Secondly, embodiments of this application provide a method for identifying a crawling detection model, including:

[0043] Acquire the RGB-D image to be detected;

[0044] The RGB-D image to be detected is input into the grasping and detection model as described in any of the first aspects above to obtain a prediction candidate set;

[0045] A disorder analysis is performed on the predicted candidate set to obtain a first predicted candidate box. The first predicted candidate box is used to characterize the predicted candidate box with the lowest disorder in the predicted candidate set.

[0046] Thirdly, embodiments of this application provide a training system for a grasping detection model, comprising:

[0047] The acquisition module is used to acquire a labeled RGB-D image training set and to perform point cloud transformation processing on the labeled RGB-D image training set to obtain a point cloud dataset;

[0048] The first processing module is used to perform multi-scale feature extraction processing on the labeled RGB-D image training set to obtain a global feature map set.

[0049] The second processing module is used to perform feature interpolation processing on the point cloud dataset based on the global feature map set to obtain an interpolated feature set, and to perform size balancing processing on the point cloud dataset to obtain a local feature dataset.

[0050] The third processing module is used to perform feature fusion processing on the local feature dataset and the interpolated feature dataset to obtain a semantic space feature set, and to perform multi-angle prediction processing on the semantic space feature set to obtain a training prediction candidate set.

[0051] The update module is used to update the parameters of the initialized crawling detection model based on the training prediction candidate set, so as to obtain the trained crawling detection model.

[0052] Fourthly, embodiments of this application also provide a computer device, including:

[0053] At least one processor;

[0054] At least one memory for storing at least one program;

[0055] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0056] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a processor-executable program, which, when executed by the processor, is used to implement the above-described method.

[0057] The advantages and beneficial effects of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application:

[0058] This application discloses a training method for a grasping detection model. The method involves acquiring a labeled RGB-D image training set and performing point cloud transformation on the labeled RGB-D image training set to obtain a point cloud dataset. Multi-scale feature extraction is then performed on the labeled RGB-D image training set to obtain a global feature map. Feature interpolation is then performed on the point cloud dataset based on the global feature map to obtain an interpolated feature set, and size balancing is performed on the point cloud dataset to obtain a local feature dataset. Feature fusion is then performed on the local feature dataset and the interpolated feature dataset to obtain a semantic space feature set, and multi-angle prediction is performed on the semantic space feature set to obtain a training prediction candidate set. Finally, the parameters of an initialized grasping detection model are updated based on the training prediction candidate set to obtain a trained grasping detection model. This training method can effectively improve the grasping detection model's ability to distinguish the edges of stacked objects and effectively improve the recognition accuracy of the grasping detection model. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0060] Figure 1 A flowchart illustrating a training method for a crawling detection model provided in an embodiment of this application;

[0061] Figure 2 A schematic diagram illustrating the principle of a feature extraction network provided in an embodiment of this application;

[0062] Figure 3 This application provides a schematic diagram of the model principle corresponding to step 130 in an embodiment of the present application.

[0063] Figure 4 A schematic diagram illustrating the principle of a size balancing network provided in an embodiment of this application;

[0064] Figure 5 This is a schematic diagram of the model principle corresponding to step 140 provided in an embodiment of this application;

[0065] Figure 6 This is a schematic diagram of the structure of a training system for a grasping detection model provided in an embodiment of this application;

[0066] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0067] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0069] Currently, there are two main traditional methods for automatically identifying and grasping objects in stacked scenarios:

[0070] The first method is to directly output one or more object grasping boxes from a planar RGB image. This can be achieved using the YOLO algorithm. The grasping parameters mainly include the position of the center point of the object to be grasped and the rotation angle. This method is suitable for the robotic arm to grasp from top to bottom and works well for a single object. However, the more complex the scene, the worse the performance of the algorithm becomes. It is more likely to misjudge the edges of the target objects in the scene and lacks depth information, which makes the robotic arm unable to grasp accurately and may even damage the surface of the object.

[0071] The second method uses scene point clouds obtained directly or indirectly (RGB-D images) as input. It leverages the semantic and geometric information of the point clouds to feed them into a backbone network to obtain the 6D (six degrees of freedom) pose of the object to be grasped and then performs the grasping. However, this method is too idealistic and heavily reliant on the environment (lighting around the camera and the quality of the point cloud data). In stacked scenes, it easily treats different objects in contact as a single entity, and it has low tolerance for small or irregularly shaped objects. In scenes where collision-free conditions are required, the algorithm's assessment of grasping quality is too low, leading to the failure to successfully grasp objects in close contact within the scene.

[0072] In view of this, embodiments of the present invention provide a training method for a grasping detection model, which can effectively improve the grasping detection model's ability to distinguish the edges of stacked objects and effectively improve the recognition accuracy of the grasping detection model.

[0073] Reference Figure 1 In this embodiment of the application, a method for training a grasping detection model is provided, the training method comprising:

[0074] Step 110: Obtain the labeled RGB-D image training set, and perform point cloud transformation processing on the labeled RGB-D image training set to obtain a point cloud dataset;

[0075] It is understandable that the RGB-D image training set can be divided into batches of RGB images (color images) and batches of depth images. The RGB images are used to supplement the contour information of objects in the scene and the global semantic information of the scene, while the depth images are used for object pose recognition. Furthermore, the labeled RGB-D image training set contains labels corresponding to the objects in the RGB-D images. It is also understandable that the point cloud dataset is used to enrich scene features. There are already various specific implementations for generating point cloud data from RGB-D images, which will not be elaborated upon here.

[0076] Reference Figure 2 Step 120: Perform multi-scale feature extraction processing on the labeled RGB-D image training set to obtain a global feature map set;

[0077] Step 120 involves performing multi-scale feature extraction processing on the labeled RGB-D image training set to obtain a global feature map, including:

[0078] Step 121: Perform a first feature extraction process on the RGB-D image training set to obtain a first feature map set;

[0079] Step 122: Perform second feature extraction processing on the first feature map set to obtain the second feature map set;

[0080] Step 123: Perform third feature extraction processing on the second feature map set to obtain the third feature map set;

[0081] Step 124: Perform semantic aggregation processing on the first feature map set, the second feature map set, and the third feature map set to obtain a global feature map set.

[0082] It is understood that, in the embodiments of this application, multi-scale feature extraction processing of the RGB-D image training set can be achieved by using the feature extraction network in the capture detection model, thereby obtaining a global feature map. The feature extraction network in this application includes a 7x7x128 convolutional layer, a 3x3 max pooling layer, 4 basic blocks, a global max pooling layer (GMP), a global average pooling layer (GAP), an attention channel module, and an upsampling layer. Specifically, taking an RGB-D image from the RGB-D image training set as an example, the feature extraction network outputs a 16x feature map of the RGB-D image at the third base layer and uses this 16x feature map as the first feature map. Then, at the fourth base layer, it outputs a 32x feature map of the RGB-D image and uses this 32x feature map as the second feature map. Next, at the global max pooling layer, it outputs a max pooling feature map of the RGB-D image and uses this max pooling feature map as the third feature map. Then, the first, second, and third feature maps undergo channel shifting through a 1×1 convolutional layer with 128 channels to obtain the same input format. Next, the first feature map passes through a global average pooling layer, a first attention channel module, a 1×1 convolutional layer with 128 channels, and a second attention channel convolutional layer. The process for the second feature map after channel shifting is similar to that of the first feature map, and will not be elaborated further here. Finally, the first, second, and third feature maps are used as inputs for semantic aggregation. Local information of different sizes extracted from the network is integrated to obtain scene-rich semantic context information, and a global feature map set is output in the second upsampling layer of the feature extraction network.

[0083] Reference Figure 3 Step 130: Perform feature interpolation processing on the point cloud dataset based on the global feature map set to obtain an interpolated feature set, and perform size balancing processing on the point cloud dataset to obtain a local feature dataset.

[0084] Step 130, performing feature interpolation processing on the point cloud dataset based on the global feature map set to obtain an interpolated feature set, includes:

[0085] Step 131: Perform upsampling and instance segmentation processing on the point cloud dataset in sequence to obtain the first intermediate dataset;

[0086] Step 132: Downsample the first intermediate dataset to obtain the second intermediate dataset;

[0087] Step 133: Perform feature interpolation processing on the second intermediate dataset based on the global feature map set to obtain the interpolated feature set.

[0088] It is understood that in this embodiment, the point cloud dataset is first upsampled to increase its feature dimension. Then, the K-nearest neighbor algorithm is used for feature extraction and point cloud analysis, obtaining the object type and feature information for each point, resulting in a segmented point cloud. The segmented point cloud is then downsampled to obtain a second intermediate dataset belonging to several instances of the same type. This second intermediate dataset includes the object type and corresponding feature information of the point cloud data within the scene. Notably, the second intermediate dataset, obtained after upsampling, instance segmentation, and downsampling, is used to represent the local features of objects. It is also understood that this embodiment obtains the scene's point cloud features (i.e., the interpolated feature set) by performing feature interpolation on the first intermediate dataset based on a global feature map.

[0089] Reference Figure 3 and Figure 4 Step 130, performing size balancing on the point cloud dataset to obtain a local feature dataset, includes:

[0090] Step 135: Obtain the preset multi-layered annular cylinder;

[0091] Step 136: Group the point cloud dataset according to the multi-layer annular cylinder to obtain a grouped dataset;

[0092] Step 137: Perform perceptual processing on the grouped dataset based on the first intermediate dataset to obtain the local feature dataset.

[0093] It is understood that, in this embodiment of the application, the point cloud dataset can be size-balanced using a size-balanced network. The size-balanced network mainly consists of multi-layer circular cylinders and multi-layer perceptrons (MLPs), wherein the multi-layer circular cylinders can be represented as:

[0094] M c =(r,h,k)

[0095] Where r is the grasping radius of the robotic arm, r∈(d min d width ), d min Let d be the minimum grasping radius of the robotic arm. widthis the maximum gripping width of the robotic arm, h is the height of the annular cylinder, and k is the number of layers in the annular cylinder.

[0096] It is understandable that this is achieved through a multi-layered annular cylinder M. c The obtained point cloud dataset is grouped. Based on the minimum wrapping cylinder corresponding to the size of the object in the scene, each object is classified and its corresponding local features are extracted to obtain the grouped dataset. The grouped dataset is used to represent the multi-size local features of the scene.

[0097] Understandably, after obtaining the grouped dataset, a multilayer perceptron is used to encode the point cloud data of different annular cylinders. In order to further improve the semantic information of the grouped dataset, the grouped dataset is interpolated based on the first intermediate dataset for continuous grasping and detection, thereby obtaining the local feature dataset of the point cloud data in the scene.

[0098] Reference Figure 5 Step 140: Perform feature fusion processing on the local feature dataset and the interpolation feature dataset to obtain a semantic space feature set, and perform multi-angle prediction processing on the semantic space feature set to obtain a training prediction candidate set.

[0099] It is understood that, in the embodiments of this application, the multi-angle prediction processing of the semantic space feature dataset can be achieved by a crawling prediction network. First, the local feature dataset and the interpolated feature dataset can be fused to obtain the semantic space feature set. Then, the semantic space feature set is input into a 1×1 convolutional layer with 128 channels to reduce the channel dimension of the semantic space feature set. The semantic space feature set with reduced channel dimension is then input into the crawling prediction network for multi-angle prediction processing.

[0100] Continue to refer to Figure 5 Step 140, performing multi-angle prediction processing on the semantic space feature set to obtain a training prediction candidate set, includes:

[0101] Step 141: Upsample the semantic space feature set to obtain a third intermediate dataset;

[0102] Step 142: Perform width processing on the third intermediate dataset to obtain a width candidate set;

[0103] Step 143: Perform angle processing on the third intermediate dataset to obtain an angle candidate set;

[0104] Step 144: Perform crawling probability processing on the third intermediate dataset to obtain a probability candidate set;

[0105] Step 145: Generate the training prediction candidate set based on the width candidate set, the angle candidate set, and the probability candidate set.

[0106] It is understood that in the embodiments of this application, the grasping prediction network performs preliminary processing on the semantic space feature set by using two transposed convolutional layers with 128 channels each, so that the image size in the semantic space feature set is the same as the size of the initial input RGB-D image. Then, after upsampling processing to obtain the third intermediate dataset, three branches are used to process the grasping width, grasping angle, and grasping probability, respectively. Each branch has a 1×1 convolutional layer and a 3×3 convolutional layer.

[0107] Specifically, for a given RGB-D image input to the capture and detection model, its corresponding training prediction candidate set can be represented by an approximation function, which can be expressed as follows:

[0108] G=(Γ,W,Q)∈R 3×H×w

[0109] Where Γ is the robot's angle grasping configuration corresponding to the angle candidate set, W is the robot's width grasping configuration corresponding to the width candidate set, Q is the robot's grasping probability configuration corresponding to the probability candidate set, H is the height of the output predicted candidate box, w is the width of the output predicted candidate box, and R... 3×H×w G is used to characterize all predicted candidate results output by the grab detection model. G is used to characterize the predicted candidate set including the angle grab configuration Γ, the width grab configuration W, and the grab probability configuration Q.

[0110] Step 150: Update the parameters of the initialized crawling detection model according to the training prediction candidate set to obtain the trained crawling detection model.

[0111] Step 150, updating the parameters of the initialized crawling detection model based on the training prediction candidate set to obtain the trained crawling detection model, includes:

[0112] Step 151: Obtain the stacked object labels corresponding to the training prediction candidate set;

[0113] Step 152: Determine the training loss value based on the training prediction candidate set and the stacked object labels;

[0114] Step 153: Update the parameters of the initialized grabbing and detection model according to the training loss value to obtain the trained grabbing and detection model.

[0115] It is understood that in this embodiment, the detection model needs to be trained before it is put into use to adjust its internal parameters and achieve better prediction results. Specifically, when training the model, a batch of RGB-D images can be acquired. Each RGB-D image includes image data of stacked objects, and the corresponding stacked object label is also acquired. This label is used to characterize the true type of the stacked objects in the RGB-D image. Then, each RGB-D image and its corresponding stacked object label can be used as a set of training data. The input data of the model is the RGB-D image. The model predicts the RGB-D images, and the output data of the model is the training prediction candidate box. After obtaining the training prediction candidate box output by the model, the accuracy of the model's prediction can be evaluated based on the training prediction candidate box and the stacked object label, thereby updating the model's parameters. It is worth mentioning that the training prediction candidate box in this embodiment is one of the recognition results in the training prediction candidate set.

[0116] Specifically, for machine learning models, the accuracy of model predictions can be measured by a loss function. The loss function is defined on a single training data point and measures the prediction error of that data point. Specifically, the loss value is determined by the label of the individual training data point and the model's prediction result for that data point. However, in actual training, a training dataset contains many data points. Therefore, a cost function is generally used to measure the overall error of the training dataset. The cost function is defined on the entire training dataset and calculates the average prediction error of all training data points, providing a better measure of the model's prediction performance. For general machine learning models, the aforementioned cost function, plus a regularization term to measure model complexity, serves as the training objective function. Based on this objective function, the loss value of the entire training dataset can be calculated. Many types of loss functions are commonly used, such as 0-1 loss, squared loss, absolute loss, logarithmic loss, and cross-entropy loss, which will not be elaborated upon here. In this embodiment, any one of these loss functions can be selected to determine the training loss value, such as the cross-entropy loss function. Based on the training loss value, the backpropagation algorithm is used to update the model parameters. After several iterations, a well-trained grasping and detection model can be obtained. The specific number of iterations can be preset, or training can be considered complete when the accuracy requirement is met on the test set.

[0117] In summary, the embodiments of this application obtain more accurate semantic information in the scene through a feature extraction network structure, and perform feature interpolation on the point cloud dataset through a global feature map set to obtain local features of the scene. Furthermore, the first intermediate dataset obtained by point cloud data instance segmentation and the multilayer perceptron are used to participate in size balancing processing to obtain richer local features of the scene. The above feature data are then input into the grasping detection network for predictive analysis, which can generate more accurate grasping postures.

[0118] In addition, embodiments of this application propose a method for identifying a grasping detection model, including:

[0119] Step 210: Obtain the RGB-D image to be detected;

[0120] Step 220: Input the RGB-D image to be detected into any of the aforementioned grasping and detection models to obtain a prediction candidate set;

[0121] Step 230: Perform a disorder analysis on the prediction candidate set to obtain a first prediction candidate box. The first prediction candidate box is used to characterize the prediction candidate box with the lowest disorder in the prediction candidate set.

[0122] It is understandable that, for the RGB-D image to be detected input to the grasping detection model, after obtaining the prediction candidate set, compared with the traditional method of selecting the maximum value of the prediction grasping candidate box for execution, the embodiments of this application can integrate the prediction candidate boxes that are greater than the threshold by setting a fixed threshold in advance, which ranges from 0 to 1, to form a prediction candidate set. The disorder of the prediction candidate set is analyzed by combining the contextual semantics of the scene, and the robot arm is guided to perform object grasping based on the first prediction candidate box with the lowest disorder, instead of relying solely on the dataset. This can effectively improve the success rate of object grasping in stacked scenarios.

[0123] Specifically, in this embodiment of the application, in order to intuitively measure the clutter level of the surrounding environment of the target item, the clutter level can be used to represent the clutter level of the corresponding semantic category in the prediction candidate box, which can be expressed as:

[0124]

[0125] Where S is the number of pixels of the object in the capture box, c is the semantic category, H(c) is the disorder of semantic category c, and t(c) is the degree of disorder. i ) represents the i-th pixel with semantic category c within the region, h i h represents the height of the i-th object. t This represents the average height of all objects within the scene area. This represents the height coefficient, used to measure the impact of objects of different heights around the target object on the difficulty of grasping it.

[0126] Reference Figure 6 This application also provides a training system for a grasping detection model, characterized in that it includes:

[0127] The acquisition module 101 is used to acquire a labeled RGB-D image training set and to perform point cloud conversion processing on the labeled RGB-D image training set to obtain a point cloud dataset;

[0128] The first processing module 102 is used to perform multi-scale feature extraction processing on the labeled RGB-D image training set to obtain a global feature map set.

[0129] The second processing module 103 is used to perform feature interpolation processing on the point cloud dataset based on the global feature map set to obtain an interpolated feature set, and to perform size balancing processing on the point cloud dataset to obtain a local feature dataset.

[0130] The third processing module 104 is used to perform feature fusion processing on the local feature dataset and the interpolation feature dataset to obtain a semantic space feature set, and to perform multi-angle prediction processing on the semantic space feature set to obtain a training prediction candidate set.

[0131] The update module 105 is used to update the parameters of the initialized crawling detection model according to the training prediction candidate set, so as to obtain the trained crawling detection model.

[0132] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0133] Reference Figure 7 This application also provides a computer device, including:

[0134] At least one processor 201;

[0135] At least one memory 202 is used to store at least one program;

[0136] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the method embodiment described above.

[0137] Similarly, it can be understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] This application also provides a computer-readable storage medium storing a program executable by a processor 201, which, when executed by the processor 201, is used to implement the above-described method embodiments.

[0139] Similarly, the content of the above method embodiments is applicable to the present computer-readable storage medium embodiments. The specific functions implemented by the present computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0141] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0142] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0144] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0145] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0146] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0147] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0148] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for training a grasp detection model, the method comprising: The training method comprises: obtaining a labeled RGB-D image training set, and performing point cloud conversion processing on the labeled RGB-D image training set to obtain a point cloud data set; performing multi-scale feature extraction processing on the labeled RGB-D image training set to obtain a global feature map set; performing feature interpolation processing on the point cloud data set according to the global feature map set to obtain an interpolated feature data set, and performing size balancing processing on the point cloud data set to obtain a local feature data set; performing feature fusion processing on the local feature data set and the interpolated feature data set to obtain a semantic spatial feature set, and performing multi-angle prediction processing on the semantic spatial feature set to obtain a training prediction candidate set; updating parameters of an initialized grasp detection model according to the training prediction candidate set to obtain a trained grasp detection model. 2.The method of claim 1, wherein, The multi-scale feature extraction processing on the labeled RGB-D image training set to obtain a global feature map set comprises: performing first feature extraction processing on the RGB-D image training set to obtain a first feature map set; performing second feature extraction processing on the first feature map set to obtain a second feature map set; performing third feature extraction processing on the second feature map set to obtain a third feature map set; performing semantic aggregation processing on the first feature map set, the second feature map set and the third feature map set to obtain a global feature map set. 3.The method of claim 1, wherein, The feature interpolation processing on the point cloud data set according to the global feature map set to obtain an interpolated feature data set comprises: performing upsampling processing and instance segmentation processing on the point cloud data set in sequence to obtain a first intermediate data set; performing downsampling processing on the first intermediate data set to obtain a second intermediate data set; performing feature interpolation processing on the second intermediate data set according to the global feature map set to obtain the interpolated feature data set.

4. The method of claim 3, wherein, The size balancing processing on the point cloud data set to obtain a local feature data set comprises: obtaining a preset multi-layer annular cylinder; performing grouping processing on the point cloud data set according to the multi-layer annular cylinder to obtain a grouped data set; performing perception processing on the grouped data set according to the first intermediate data set to obtain the local feature data set.

5. The method of claim 1, wherein, The multi-angle prediction processing on the semantic spatial feature set to obtain a training prediction candidate set comprises: performing upsampling processing on the semantic spatial feature set to obtain a third intermediate data set; performing grasp width processing on the third intermediate data set to obtain a width candidate set; performing grasp angle processing on the third intermediate data set to obtain an angle candidate set; performing grasp probability processing on the third intermediate data set to obtain a probability candidate set; generating the training prediction candidate set according to the width candidate set, the angle candidate set and the probability candidate set.

6. The method of claim 1, wherein, The parameter updating of the initialized grasp detection model according to the training prediction candidate set to obtain a trained grasp detection model comprises: obtaining a stacked object label corresponding to the training prediction candidate set; determining a training loss value according to the training prediction candidate set and the stacked object label; According to the training loss value, the initialized grasp detection model is updated in parameters to obtain a trained grasp detection model.

7. A recognition method of a grasp detection model, characterized by, The method comprises: obtaining an RGB-D image to be detected; inputting the RGB-D image to be detected into the grasp detection model according to any one of claims 1-6 to obtain a prediction candidate set; performing chaos degree analysis on the prediction candidate set to obtain a first prediction candidate box, wherein the first prediction candidate box is used to represent a prediction candidate box with the lowest chaos degree in the prediction candidate set. 8.A training system of a grasp detection model, characterized by, The method comprises: an acquisition module configured to acquire an annotated RGB-D image training set, and perform point cloud conversion processing on the annotated RGB-D image training set to obtain a point cloud data set; a first processing module configured to perform multi-scale feature extraction processing on the annotated RGB-D image training set to obtain a global feature map set; a second processing module configured to perform feature interpolation processing on the point cloud data set according to the global feature map set to obtain an interpolated feature data set, and perform size balancing processing on the point cloud data set to obtain a local feature data set; a third processing module configured to perform feature fusion processing on the local feature data set and the interpolated feature data set to obtain a semantic spatial feature set, and perform multi-angle prediction processing on the semantic spatial feature set to obtain a training prediction candidate set; an update module configured to update an initialized grasp detection model in parameters according to the training prediction candidate set to obtain a trained grasp detection model.

9. A computer device, comprising: The method comprises: at least one processor; at least one memory configured to store at least one program; when the at least one program is executed by the at least one processor, the at least one processor is caused to implement the method according to any one of claims 1-7.

10. A computer readable storage medium having stored therein a program which is executable by a processor, characterized in that, The program executable by the processor, when executed by the processor, is used to implement the method according to any one of claims 1-7.

Images