Grabbing attitude generation method and system based on multi-scale feature fusion and point-by-point prediction

Abstract

The invention provides a grabbing attitude generation method and system based on multi-scale feature fusion and point-by-point prediction, and the method comprises the steps: collecting a grabbing data set, and obtaining a trained grabbing region detection model; constructing a target recognition network combined with multi-scale point cloud feature fusion; according to the trained capture region detection model, using the preprocessed point cloud data as network input, outputting a high-capture confidence region, and mapping the high-capture confidence region to a reference coordinate system of the input point cloud; and a point cloud bounding box is constructed for the output high-grabbing confidence region, a diversified grabbing posture candidate set is generated, and the grabbing posture with the highest quality is selected from the grabbing posture candidate set to serve as the posture of the robot for grabbing the target by calculating grabbing candidate scores. Through construction of a lightweight target recognition network, multi-scale point cloud feature fusion, an efficient training and verification strategy, point cloud data preprocessing and enhancement, accurate grabbing area recognition and attitude estimation and an optimized grabbing selection strategy, the interaction ability and grabbing precision of the mechanical arm to the environment are improved.

Description

Technical Field

[0001] The present invention belongs to the field of robot grasping control, and in particular relates to a grasping posture generation method and system based on multi-scale feature fusion and point-by-point prediction. Background Art

[0002] It is crucial to improve the robot's autonomous grasping ability in complex environments. This can not only improve production efficiency and reduce labor costs, but also has important strategic significance for the realization of smart logistics. Compared with the existing point cloud target detection network, there are generally problems such as complex network structure and time-consuming training. Therefore, how to build a lightweight and efficient point cloud target area recognition network and how to generate 6-DOF grasping postures in complex stacking environments have become research hotspots in the field of robot disordered grasping.

[0003] The network architecture based on MLP (multi-layer perceptron) is an important technical architecture for point cloud object recognition and pose estimation. PointNet++ is an important milestone in point cloud processing technology and has been widely used in the field of robot point cloud detection. For 6-DOF grasping pose detection in complex environments, the traditional point cloud grasping estimation method captures point cloud features based on the stacking of multiple layers of MLP, which will lead to gradient explosion or disappearance caused by the deepening of the point cloud processing network, as well as slow model convergence. For example, the grasping pose generation network based on the MLP point cloud processing architecture ("Liang H, Ma X, Li S, et al.PointNetGPD: Detecting Grasp Configurations from Point Sets [J]. IEEETransactions on Automation Science and Engineering, 2020, 17(4)"), this method uses the PointNet++ network to enhance the local feature extraction of point cloud data, thereby improving the grasping accuracy. However, due to the high degree of reliance on the multi-layer stacked MLP, the model complexity increases, the training converges slowly, and the efficiency is low when processing a large number of point cloud data, and it is impossible to directly generate stable grasping pose predictions in complex scenes. Summary of the invention

[0004] The purpose of the present invention is to provide a grasping posture generation method and system based on multi-scale feature fusion and point-by-point prediction to solve the above problems.

[0005] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a grasping posture generation method based on multi-scale feature fusion and point-by-point prediction, comprising: Collect the grasping data set, annotate it and use it as the training set and verification set, use the training set and verification set to train and verify the preset grasping area detection model, and obtain the trained grasping area detection model; Construct a target recognition network that combines multi-scale point cloud feature fusion, obtain point cloud data containing the target object, pre-process the point cloud data, and input the pre-processed point cloud data into the target recognition network model for grasping area recognition; According to the trained grasping region detection model, the preprocessed point cloud data is used as the network input, and the high grasping confidence region is output and mapped to the reference coordinate system of the input point cloud; The point cloud bounding box is constructed for the output high grasping confidence area to generate a diverse grasping posture candidate set. By calculating the grasping candidate score, the grasping candidate posture with the highest quality score is selected from the grasping posture candidate set as the posture of the robot grasping the target.

[0006] Furthermore, we collected crawled data sets and annotated them as training and validation sets, including: Complex stacked point cloud data in a simulation environment, and data on stacking of common objects in life collected in a real environment; the data set is labeled according to the highest point grabbing strategy, and the data set labeled in the simulation environment is randomly divided into training set and verification set in proportion; the data set is collected in real scenes and simulation scenes respectively, and the training set consists entirely of single objects in the simulation scene.

[0007] Furthermore, the training set and the validation set are used to train and validate the preset grasping area detection model to obtain the trained grasping area detection model, including: The proposed grasping area detection model is trained using the training set data, and the back propagation algorithm and the standard gradient-based optimization algorithm are used to optimize the objective function so that the difference between the detected grasping area and the true value is minimized. At the same time, the grasping area detection model is tested using the validation set to adjust the learning rate during the grasping area detection model training process. The loss function during training is defined as:

[0008] in It is an improved traditional cross entropy loss function; is a predefined weighting factor; heights represents the point Height information along the center of gravity; MAX(heights) indicates the maximum height and is used to normalize the height.

[0009] Furthermore, a target recognition network combined with multi-scale point cloud feature fusion is constructed, including: a data preprocessing module, a multi-scale feature extraction module, a target local feature extraction module, and a decoding module connected in sequence.

[0010] Furthermore, point cloud data containing the target object is obtained, and the point cloud data is preprocessed, and the preprocessed point cloud data is input into the target recognition network model for grasping area recognition, including: The point cloud data containing the target object is obtained through a single-view point cloud camera, and the obtained point cloud is downsampled, filtered, noise removed, and density clustered to obtain the ideal point cloud for the input model; after the input point cloud data is enhanced, it is input into the target recognition network model for grasping area recognition; The camera is a point cloud camera or a depth camera; the data enhancement operation uses normalization and centering operations to reduce the model's sensitivity to changes in data size and the complexity of adapting to data distribution in different locations.

[0011] Furthermore, a point cloud bounding box is constructed for the output high grasping degree grasping area, including: Construct a point cloud bounding box for the output high-grasp grasping area and calculate the edge points within the bounding box And the symmetric points relative to the bounding box , and perform posture estimation based on direction constraints for each edge point; and and The direction of the connecting line is used as the parallel direction of the grasping clamp, and the midpoint of the connection is connected As a grabbing point.

[0012] Furthermore, a diverse set of grasping posture candidates is generated, and by calculating the grasping candidate scores, the grasping candidate postures with the highest quality scores are selected from the grasping posture candidate set as the postures of the robot grasping the target, including: Remove possible collision points from the grasping posture candidate set, predict the grasping posture point by point for non-collision points in the high grasping confidence area, and generate a diverse grasping posture candidate set; calculate the Euclidean distance between the grasping center point and the centroid of the point cloud in the high grasping confidence area, and select the grasp with the highest quality as the position of the robot grasping target by calculating the grasping candidate score; the grasping candidate posture meets two conditions: The grasping posture is located in a position that does not contain the target point cloud, that is, the robot arm gripper does not collide with the target point cloud; The grasping posture is closest to the centroid of the high grasp confidence region.

[0013] In a second aspect, the present invention provides a grasping posture generation system based on multi-scale feature fusion and point-by-point prediction, comprising: The model building module is used to collect the grasping data set, annotate it as the training set and verification set, and use the training set and verification set annotated by the top grasping-based annotation strategy to train and verify the preset grasping area detection model, so as to obtain a grasping detection model that can autonomously identify high grasping confidence areas after training; The grasping area recognition module is used to build a target recognition network that combines multi-scale point cloud feature fusion, obtain point cloud data containing the target object, pre-process the point cloud data, and input the pre-processed point cloud data into the target recognition network model for grasping area recognition; The mapping module is used to output the high grasp confidence area according to the trained grasp area detection model, using the preprocessed point cloud data as the network input, and mapping it to the reference coordinate system of the input point cloud; The posture output module is used to construct a point cloud bounding box for the output high-grasp degree grasping area, generate a diverse grasping posture candidate set, and select the grasp with the highest quality in the grasping posture candidate set as the posture of the robot grasping the target by calculating the grasping candidate score.

[0014] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for generating a grasping posture based on multi-scale feature fusion and point-by-point prediction when executing the computer program.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the grasping posture generation method based on multi-scale feature fusion and point-by-point prediction.

[0016] Compared with the prior art, the present invention has the following technical effects: The present invention uses the local feature extraction module and the local aggregation (LA) layer to capture the local information of the original point cloud at different scales. In order to efficiently extract the local features of the target point cloud, a five-layer KAN linear layer with an inverted bottleneck structure is used to map the original local information to a high-dimensional space and perform feature extraction. This multi-scale feature fusion strategy improves the network's recognition ability and anti-interference ability of high-grasp confidence areas in unstructured environments.

[0017] By collecting data from both simulation and real-world environments and annotating them based on the highest point grabbing strategy, this solution enhances the model's cross-domain adaptability. Data from simulation environments are used for training, and data from real-world environments are used for verification and testing, ensuring the reliability of the model in practical applications.

[0018] The back propagation algorithm and the standard gradient-based optimization algorithm are used to optimize the objective function, so that the difference between the detected grasping area and the true value is minimized. At the same time, the model is tested using the validation set, the learning rate is adjusted and overfitting is avoided, ensuring the training efficiency and generalization ability of the model.

[0019] The input point cloud is preprocessed by downsampling, filtering, noise removal and clustering to obtain ideal point cloud data. Then, through data enhancement processing (such as normalization and centering), the model's sensitivity to data size changes and position distribution is reduced, and the model's adaptability and stability are improved.

[0020] The target recognition network model is used to identify the grasping area of ​​the point cloud data and output the high grasping confidence area. Subsequently, the point cloud bounding box is constructed, the edge points are calculated, the symmetric points are calculated, and the posture estimation based on the direction constraint is performed on the high grasping confidence area to generate a diverse set of grasping posture candidates. This process ensures the accurate identification of the grasping area and the accurate estimation of the grasping posture.

[0021] By calculating the Euclidean distance between the grasp center point and the centroid of the point cloud in the high grasp confidence area and combining it with the grasp candidate score, the grasp with the highest quality is selected as the pose of the robot grasping the target. This optimization strategy ensures the feasibility and optimality of the grasping pose.

[0022] This technical solution provides a cross-domain adaptive point cloud disordered grasping prediction method based only on single-view point cloud input, achieving accurate grasping in complex stacking environments. By constructing a lightweight target recognition network, multi-scale point cloud feature fusion, efficient training and verification strategies, point cloud data preprocessing and enhancement, accurate grasping area recognition and posture estimation, and optimized grasping selection strategies, the robot arm's interactive ability with the environment and grasping accuracy are improved. This method has broad application prospects and can be applied to automated warehousing, intelligent manufacturing, service robots and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The figure is an overall logic block diagram of the method disclosed in the present invention.

[0024] Figure 2 A dataset for multi-object point cloud grasping.

[0025] Figure 3 This is the target area detection result map.

[0026] Figure 4 Schematic diagram of grasping posture under direction constraints.

[0027] Figure 5 This is the 6-DOF grasping posture prediction diagram.

[0028] Figure 6It is a flow chart of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings: Example 1, please refer to Figure 6 The present invention provides a grasping posture generation method based on multi-scale feature fusion and point-by-point prediction, comprising: Collect the grasping data set, annotate it and use it as the training set and verification set, use the training set and verification set to train and verify the preset grasping area detection model, and obtain the trained grasping area detection model; Construct a target recognition network that combines multi-scale point cloud feature fusion, obtain point cloud data containing the target object, pre-process the point cloud data, and input the pre-processed point cloud data into the target recognition network model for grasping area recognition; According to the trained grasping area detection model, the preprocessed point cloud data is used as the network input, and the high grasping confidence area is output and mapped to the coordinate system; The point cloud bounding box is constructed for the output high-grasp degree grasping area to generate a diverse grasping posture candidate set. By calculating the grasping candidate score, the grasp with the highest quality is selected from the grasping posture candidate set as the position of the robot grasping the target.

[0030] By collecting and annotating the grasping data set, the model is provided with abundant training samples and verification data, which helps the model learn the grasping features of objects in different scenarios and improves the generalization ability and accuracy of the model.

[0031] Using the training set to train the preset grab area detection model and verifying the model with the validation set can ensure the stability and reliability of the model during the training process. This data-driven model training method enables the model to better adapt to actual application scenarios.

[0032] Constructing a target recognition network combined with multi-scale point cloud feature fusion can make full use of point cloud feature information at different scales in point cloud data and improve the model's ability to recognize features such as shape, size, and position of the grasped object. The multi-scale feature fusion strategy enables the network to capture the connection between global and local features, thereby improving the accuracy and robustness of grasping area recognition.

[0033] By using the target recognition network to identify the grasping area of ​​the preprocessed point cloud data, the high grasping confidence area in complex stacking scenes can be accurately located. The identified grasping area is mapped to the coordinate system, providing accurate spatial position information for subsequent grasping posture generation.

[0034] By constructing a point cloud bounding box for the output high grasping degree grasping area, multiple possible grasping posture candidates can be generated. This diversity provides more options for subsequent grasping posture selection. By calculating the grasping candidate score, the grasp with the highest quality can be selected from the grasping posture candidate set as the posture of the robot grasping the target. This selection strategy ensures the feasibility and optimality of the grasping posture.

[0035] The grasping posture generation method provided by the present invention can enable the robot to accurately grasp objects in complex environments, thereby improving the robot's interactive ability and autonomy, and enabling the robot to better adapt to different application scenarios and task requirements.

[0036] In summary, the grasping posture generation method based on multi-scale feature fusion and point-by-point prediction provided by the present invention provides strong technical support for the robot's precise grasping through the technical effects of data-driven model training and verification, target recognition network with multi-scale point cloud feature fusion, accurate grasping area recognition and mapping, diverse grasping posture candidate set generation and improving the robot's grasping ability.

[0037] Embodiment 2, the present invention provides a grasping posture generation method based on multi-scale feature fusion and point-by-point prediction, which specifically includes: like Figure 1 As shown in FIG. 1 , it is an overall flow chart of the method of the present invention. The present invention discloses a method for generating disordered grasping postures of point clouds based on multi-scale feature fusion and point-by-point prediction, which mainly includes ten steps: The implementation of the present invention requires a monocular point cloud camera and a GPU. In the specific implementation process, Revopoint's Tracer series 3D industrial camera and a laptop computer equipped with an NVIDIA 3070 graphics card are used.

[0038] The specific steps include the following: Step 1: Construct a lightweight target recognition network PointKan that combines multi-scale point cloud feature fusion to identify the grasping area of ​​the input point cloud; Specifically, the grab area recognition model includes a data processing module, a multi-scale feature extraction module, a target local feature extraction module, and a decoding module connected in sequence. Specifically, the grab detection model may include: The data processing module uses normalization and centralization methods on the input point cloud to reduce the model's sensitivity to changes in data size and the complexity of adapting to data distribution in different locations.

[0039] The multi-scale feature extraction module uses the FPS farthest point sampling strategy to ensure that the sampling points are evenly distributed in space. Each sampling point fuses the feature information of its neighborhood points at different scales through MLP, thereby capturing the global structural information of the input point cloud and using the residual layer to avoid the gradient vanishing problem.

[0040] The target local feature extraction module consists of a local feature aggregation (LA) layer and a KAN linear layer. The LA layer is used to capture the local geometric features around each point and provide the original local information for subsequent deep feature extraction. The target point cloud is transformed and aggregated point by point through the KAN linear layer. Finally, a convolution module is used to organize the output feature dimensions.

[0041] The decoder module uses the FPS strategy to sample a portion of points from the high-level point cloud data as the basis for upsampling. According to the coordinate information of these sampling points, the high-level feature representation is interpolated to obtain the feature values ​​of the corresponding positions. The interpolated feature values ​​are concatenated with the low-level feature representation to generate low-dimensional information that integrates high-dimensional information.

[0042] Step 2: Collect the grasping data set, which includes complex stacking point cloud data in the simulation environment and data of common objects in life stacking collected in the real environment. According to the highest point grasping strategy, the data set is annotated and randomly divided into training set and validation set according to the proportion; Specifically, in order to reduce the difficulty of dataset production, the dataset does not annotate the grasping posture, and only uses the highest point grasping strategy to annotate the highest area of ​​the point cloud scene. The training data is generated in a simulation environment, consisting of a single rectangular block of the YCB dataset (such as Figure 2 The objects in each set of point cloud data in the training set are randomly placed, and the types, shapes, quantities and postures of the objects vary in different scenes. In order to fully verify the feasibility of the algorithm and the cross-domain recognition capability, point cloud test datasets with different stacking complexities are constructed in the CoppelimSim simulation environment and real-world scenes.

[0043] Step 3: Use the training set data to train the proposed grasping area detection model, and use the back propagation algorithm and the standard gradient-based optimization algorithm to optimize the objective function so that the difference between the detected grasping area and the true value is minimized; at the same time, use the validation set to test the grasping area detection model to adjust the learning rate during the grasping area detection model training process and avoid overfitting of the grasping detection model to a certain extent; Specifically, the loss function during training is defined as:

[0044] in It is an improved traditional cross entropy loss function; is a predefined weighting factor; heights represents the point Height information along the center of gravity; MAX(heights) represents the maximum height, which is used to normalize the height to ensure the stability and efficiency of the loss function.

[0045] Step 4: Obtain point cloud data containing the target object through a single-view point cloud camera; Step 5: Downsample the point cloud, filter the point cloud, remove noise, and cluster the point cloud to obtain the ideal point cloud of the target object; Step 6: After the input point cloud data is enhanced, it is input into the target recognition network model to identify the grasping area; Specifically, the data enhancement operation uses normalization and centering operations to reduce the model's sensitivity to changes in data size and the complexity of adapting to data distribution in different locations. Step 7: Based on the trained grasp detection model, the real point cloud data after preprocessing is used as the network input. The model outputs high grasp confidence areas and maps them to real-world coordinates.

[0046] Step 8: Perform point cloud plane fitting on the output high grasping degree grasping area and calculate the edge points within the fitting plane And the symmetric points relative to the bounding box , and perform orientation constraint-based pose estimation for each edge point.

[0047] Specifically, the RANSAC point cloud plane fitting method is used to obtain two candidate plane sub-point sets, namely the inner point set and external point set ,in and are the center points of the two sub-regions and are the directional eigenvectors , are the edge points of the fitting plane, for Symmetrical points with respect to the short side of the fitting plane; extract The three-dimensional oriented boundary of the image is obtained and the corresponding geometric parameters are calculated, including the long edge vertices. and and the center point , based on this information, calculate the long side vector This process provides the necessary geometric basis for subsequent posture estimation. The geometric information of the parameters can be found in Figure 2 ; The directional eigenvectors are calculated and With preset reference vector The cosine similarity of . The specific formula is as follows:

[0048] According to the similarity calculation results, the most suitable sub-region is selected The selection strategy is as follows:

[0049] After determining the grasping area, further estimate its six-degree-of-freedom pose , which is the three-dimensional coordinate of the grasping point. The calculation formula for this step is as follows:

[0050]

[0051]

[0052]

[0053] A reference vector is introduced in the calculation of the directional eigenvector To constrain the grasping approach direction. In addition, to avoid the “parallel grasping forceps” problem, a threshold mechanism is set: and When the result of the calculation is too low (for example, below 0.01), the direction vector is replaced by a fixed perpendicular vector To ensure that the robotic arm can grasp in a vertical position to the ground.

[0054] Using edge points Symmetrical to the edge The connection vector As the gripping direction of the gripper, the gripper opening and closing distance For vector The calculation formula is:

[0055]

[0056] Step 9: Perform point-by-point grasping posture anti-collision detection in the grasping posture candidate set to remove possible collision points where collision may occur; Specifically, the edge point , symmetric point as well as With as the center of the circle and radius R as the radius, calculate whether the object point cloud is inside the sphere. This is used to perform collision detection. The calculation formula is:

[0057] Step 10: Calculate the Euclidean distance between the grasp center point and the centroid of the point cloud in the high grasp confidence area. By calculating the grasp candidate score, select the grasp with the highest quality as the pose of the robot grasping the target; The present invention provides a point cloud disordered grasping prediction method with cross-domain adaptation based only on single-view point cloud input. The method can realize cross-domain recognition in a complex stacking environment, and can stably generate six-degree-of-freedom grasping postures using only single-view point cloud data input, thereby improving the robot arm's ability to interact with the environment.

[0058] In yet another embodiment of the present invention, a grasping posture generation system based on multi-scale feature fusion and point-by-point prediction is provided, which can be used to implement the above-mentioned grasping posture generation method based on multi-scale feature fusion and point-by-point prediction. Specifically, the system includes: The model building module is used to collect the grasping data set, annotate it as a training set and a verification set, and use the training set and the verification set to train and verify the preset grasping area detection model to obtain the trained grasping area detection model; The grasping area recognition module is used to build a target recognition network that combines multi-scale point cloud feature fusion, obtain point cloud data containing the target object, pre-process the point cloud data, and input the pre-processed point cloud data into the target recognition network model for grasping area recognition; The mapping module is used to output the high grasp confidence area according to the trained grasp area detection model, using the preprocessed point cloud data as the network input, and mapping it to the reference coordinate system of the input point cloud; The posture output module is used to construct a point cloud bounding box for the output high-grasp degree grasping area, generate a diverse grasping posture candidate set, and select the grasp with the highest quality in the grasping posture candidate set as the posture of the robot grasping the target by calculating the grasping candidate score.

[0059] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0060] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the grasping posture generation method based on multi-scale feature fusion and point-by-point prediction.

[0061] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both a built-in storage medium in a computer device and an extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the grasping posture generation method based on multi-scale feature fusion and point-by-point prediction in the above embodiment.

[0062] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A grasping posture generation method based on multi-scale feature fusion and point-by-point prediction, characterized in that: include: Collect the grasping data set, annotate it and use it as the training set and verification set, use the training set and verification set to train and verify the preset grasping area detection model, and obtain the trained grasping area detection model; Construct a target recognition network that combines multi-scale point cloud feature fusion, obtain point cloud data containing the target object, pre-process the point cloud data, and input the pre-processed point cloud data into the target recognition network model for grasping area recognition; According to the trained grasping region detection model, the preprocessed point cloud data is used as the network input, and the high grasping confidence region is output and mapped to the reference coordinate system of the input point cloud; A point cloud bounding box is constructed for the output high grasping confidence area to generate a diverse grasping posture candidate set. By calculating the grasping candidate score, the grasping candidate posture with the highest quality score is selected from the grasping posture candidate set as the posture of the robot grasping the target.

2. The method for generating grasping posture based on multi-scale feature fusion and point-by-point prediction according to claim 1, characterized in that: Collect crawled data sets and annotate them as training and validation sets, including: Complex stacked point cloud data in a simulation environment, and data on stacking common objects in life collected in a real environment; the data set is labeled according to the highest point grabbing strategy, and the data set labeled in the simulation environment is randomly divided into training set and verification set in proportion; the data set is collected in real scenes and simulation scenes respectively, and the training set consists entirely of single objects in the simulation scene.

3. The method for generating grasping posture based on multi-scale feature fusion and point-by-point prediction according to claim 2, characterized in that: The training set and the validation set are used to train and validate the preset grasping area detection model, and the trained grasping area detection model is obtained, including: The proposed grasping area detection model is trained using the training set data, and the back propagation algorithm and the standard gradient-based optimization algorithm are used to optimize the objective function so that the difference between the detected grasping area and the true value is minimized. At the same time, the grasping area detection model is tested using the validation set to adjust the learning rate during the grasping area detection model training process. The loss function during training is defined as: in It is an improved traditional cross entropy loss function; is a predefined weighting factor; heights represents the point Height information along the center of gravity; MAX(heights) indicates the maximum height and is used to normalize the height.

4. The method for generating grasping posture based on multi-scale feature fusion and point-by-point prediction according to claim 1, characterized in that: A target recognition network combining multi-scale point cloud feature fusion is constructed, including: a data preprocessing module, a multi-scale feature extraction module, a target local feature extraction module, and a decoding module connected in sequence.

5. The method for generating grasping posture based on multi-scale feature fusion and point-by-point prediction according to claim 4, characterized in that: Obtain point cloud data containing the target object, preprocess the point cloud data, and input the preprocessed point cloud data into the target recognition network model for grasping area recognition, including: The point cloud data containing the target object is obtained through a single-view point cloud camera, and the obtained point cloud is downsampled, filtered, noise removed, and clustered to obtain the ideal point cloud of the target object; after the input point cloud data is enhanced, it is input into the target recognition network model for grasping area recognition; The camera is a point cloud camera or a depth camera; the data enhancement operation uses normalization and centering operations to reduce the model's sensitivity to changes in data size and the complexity of adapting to data distribution at different locations.

6. The method for generating grasping posture based on multi-scale feature fusion and point-by-point prediction according to claim 1, characterized in that: Construct a point cloud bounding box for the output high grasp confidence area, including: Construct a point cloud bounding box for the output high grasp confidence area and calculate the edge points within the bounding box And the symmetric points relative to the bounding box , and perform posture estimation based on direction constraints for each edge point; and and The direction of the connecting line is used as the parallel direction of the grasping clamp, and the midpoint of the connection is connected As the center point of grabbing.

7. The method for generating grasping posture based on multi-scale feature fusion and point-by-point prediction according to claim 1, characterized in that: Generate a diverse set of grasping posture candidate sets, calculate the grasping candidate scores, and select the grasping candidate poses with the highest quality scores from the grasping posture candidate set as the posture of the robot grasping the target, including: Remove possible collision points from the grasping posture candidate set, predict the grasping posture point by point for edge points and symmetric points in the high grasping confidence area, and generate a rich and diverse grasping posture candidate set; calculate the Euclidean distance between the grasping center point and the centroid of the point cloud in the high grasping confidence area, use the distance between the grasping center point and the centroid as the grasping quality evaluation standard, and select the grasp with the highest quality as the position of the robot grasping target; the grasping candidate posture meets two conditions: The grasping posture is located in a position that does not contain the target point cloud, that is, the robot arm gripper does not collide with the target point cloud; The grasping posture is closest to the centroid of the high grasp confidence region.

8. A grasping posture generation system based on multi-scale feature fusion and point-by-point prediction, characterized in that: include: The model building module is used to collect the grasping data set, annotate it as the training set and verification set, and use the training set and verification set annotated by the top grasping-based annotation strategy to train and verify the preset grasping area detection model, so as to obtain a grasping detection model that can autonomously identify high grasping confidence areas after training; The grasping area recognition module is used to build a target recognition network that combines multi-scale point cloud feature fusion, obtain point cloud data containing the target object, pre-process the point cloud data, and input the pre-processed point cloud data into the target recognition network model for grasping area recognition; The mapping module is used to output the high grasp confidence area according to the trained grasp area detection model, using the preprocessed point cloud data as the network input, and mapping it to the reference coordinate system of the input point cloud; The posture output module is used to construct a point cloud bounding box for the output high-grasp degree grasping area, generate a diverse grasping posture candidate set, and select the grasp with the highest quality in the grasping posture candidate set as the posture of the robot grasping the target by calculating the grasping candidate score.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for generating a grasping posture based on multi-scale feature fusion and point-by-point prediction as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a grasping posture based on multi-scale feature fusion and point-by-point prediction as claimed in any one of claims 1 to 7 are implemented.

Images