Mechanical arm farmland operation space perception method based on 3D target detection

Abstract

The application discloses a mechanical arm farmland operation space perception method based on 3D target detection, and the method comprises the following steps: acquiring images collected by a camera and radar point cloud information; encoding the radar point cloud information by using VoxelNet to obtain a Swin Transformer deep learning model; the VoxelNet is an end-to-end 3D target detection framework based on point cloud; a hierarchical scene modeling module is designed to integrate multi-scale features of the point cloud and the image; an instance guided fusion module (IGF) is designed to guide feature fusion and enhance the perception ability of the target instance; the target instance comprises a construction vehicle; finally, a decoder and a detection head are used to perform boundary box regression and classification on the enhanced instance feature representation, so that 3D target information of the operation space is obtained. Dynamic perception of a three-dimensional operation space of a mechanical arm is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a spatial perception method for robotic arms operating in farmland based on 3D target detection. Background Technology

[0002] With the deepening of a new round of technological revolution and industrial transformation, agriculture is accelerating its transition to a stage of intelligent and precision development. Against this backdrop, the intelligent upgrading of agricultural machinery has become a crucial support for promoting agricultural modernization. However, the complex and ever-changing farmland operating environment poses challenges to the precise control of robotic arms. Therefore, how to accurately perceive the working space of robotic arms using sensor information is fundamental to achieving efficient operation.

[0003] Based on the above background, this patent design is based on a 3D target detection model with multimodal information, which performs joint analysis on the information captured by the camera and laser sensor to achieve dynamic perception of the three-dimensional working space of the robotic arm. Summary of the Invention

[0004] This invention provides a method for perceiving the spatial working space of a robotic arm in farmland based on 3D target detection. It performs joint analysis on the information captured by cameras and laser sensors to achieve dynamic perception of the three-dimensional working space of the robotic arm.

[0005] According to one aspect of this disclosure, a spatial perception method for robotic arms operating in farmland based on 3D target detection is provided, characterized in that the method includes: Acquire images captured by the camera and radar point cloud information; VoxelNet is used to encode the radar point cloud information to obtain The SwinTransformer deep learning model encodes the acquired images to obtain... VoxelNet is an end-to-end 3D object detection framework based on point clouds. The hierarchical scene modeling module integrates multi-scale features of point clouds and images; The design instance guides the fusion module IGF to guide feature fusion, enhancing the ability to perceive target instances; target instances include construction vehicles; Finally, the decoder and detector head are used to perform bounding box regression and classification on the enhanced instance feature representation to obtain 3D target information of the work space. The 3D target information is used for the robotic arm to plan its running trajectory.

[0006] In one possible implementation, the hierarchical scene modeling module includes point-to-mesh transformation (P2G) and mesh-to-region transformation (G2R). Point-to-mesh transformation (P2G) transforms scattered point cloud data Mapped to a gridded feature with a defined spatial location The mapping formula is as follows: ,in Represents a grid The k-th point in It is a point cloud feature extraction function. It is the image feature projection function, and MSA is a multi-head self-attention mechanism.

[0007] In one possible implementation, the mesh-to-domain transformation (G2R) utilizes a cross-domain self-attention Shift-MSA mechanism to transform mesh features. Further transformed into regional characteristics: ; ShiftMSA is a cross-region self-attention mechanism that divides the grid feature Rm into several non-overlapping windows Rm={R1,R2,…,Rn}. The cross-region self-attention mechanism calculates MSA(Ri) independently for each window feature and sums them.

[0008] In one possible implementation, the design of the instance-guided fusion module IGF guides feature fusion, including: based on Candidate regions for generating target instance heatmaps are selected, and the top K high-response regions are used to generate a candidate instance set. The feature information of each instance q is fused with scene features through a deformable attention mechanism to obtain an enhanced instance representation. DeformAtt is a deformable attention mechanism. Scene features include 3D coordinate information of the hay truck or the working environment.

[0009] Compared with the prior art, the beneficial effects of the present invention are: This disclosure presents a method for perceiving the spatial needs of a robotic arm operating in farmland based on 3D target detection. It employs a 3D target detection model based on multimodal information, designs a hierarchical scene modeling module to integrate multi-scale features from point clouds and images, and designs an instance-guided fusion module (IGF) to guide feature fusion, enhancing the perception capability of target instances, including construction vehicles. Finally, a decoder and a detection head perform bounding box regression and classification on the enhanced instance feature representation to obtain 3D target information of the operating space. This 3D target information is used for the robotic arm to plan its trajectory. This achieves dynamic perception of the robotic arm's three-dimensional operating space. Attached Figure Description

[0010] Figure 1 This diagram illustrates a spatial perception model architecture block diagram for 3D object detection according to an embodiment of the present disclosure.

[0011] Figure 2A flowchart illustrating a robotic arm spatial perception method for farmland operations based on 3D target detection, according to an embodiment of this disclosure, is shown. Detailed Implementation

[0012] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0013] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0014] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0015] A spatial perception method for robotic arms operating in farmland based on 3D target detection is provided, characterized in that the method includes: Acquire images captured by the camera and radar point cloud information; VoxelNet is used to encode the radar point cloud information to obtain The SwinTransformer deep learning model encodes the acquired images to obtain... VoxelNet is an end-to-end 3D object detection framework based on point clouds. The hierarchical scene modeling module integrates multi-scale features of point clouds and images; The design instance guides the fusion module IGF to guide feature fusion, enhancing the ability to perceive target instances; target instances include construction vehicles; Finally, the decoder and detector head are used to perform bounding box regression and classification on the enhanced instance feature representation to obtain 3D target information of the work space. The 3D target information is used for the robotic arm to plan its running trajectory.

[0016] In one possible implementation, the hierarchical scene modeling module includes point-to-mesh transformation (P2G) and mesh-to-region transformation (G2R). Point-to-mesh transformation (P2G) transforms scattered point cloud data Mapped to a gridded feature with a defined spatial location The mapping formula is as follows: ,in Represents a grid The k-th point in It is a point cloud feature extraction function. It is the image feature projection function, and MSA is a multi-head self-attention mechanism.

[0017] In one possible implementation, the mesh-to-domain transformation (G2R) utilizes a cross-domain self-attention Shift-MSA mechanism to transform mesh features. Further transformed into regional characteristics: ; ShiftMSA is a cross-region self-attention mechanism that divides the grid feature Rm into several non-overlapping windows Rm={R1,R2,…,Rn}. The cross-region self-attention mechanism calculates MSA(Ri) independently for each window feature and sums them.

[0018] In one possible implementation, the design of the instance-guided fusion module IGF guides feature fusion, including: based on Candidate regions for generating target instance heatmaps are selected, and the top K high-response regions are used to generate a candidate instance set. The feature information of each instance q is fused with scene features through a deformable attention mechanism to obtain an enhanced instance representation. DeformAtt is a deformable attention mechanism. Scene features include 3D coordinate information of the hay truck or the working environment.

[0019] This disclosure presents a method for perceiving the spatial needs of a robotic arm operating in farmland based on 3D target detection. It employs a 3D target detection model based on multimodal information, designs a hierarchical scene modeling module to integrate multi-scale features from point clouds and images, and designs an instance-guided fusion module (IGF) to guide feature fusion, enhancing the perception capability of target instances, including construction vehicles. Finally, a decoder and a detection head perform bounding box regression and classification on the enhanced instance feature representation to obtain 3D target information of the operating space. This 3D target information is used for the robotic arm to plan its trajectory. This achieves dynamic perception of the robotic arm's three-dimensional operating space.

[0020] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A spatial perception method for robotic arms in farmland operations based on 3D target detection, characterized in that, The method includes: Acquire images captured by the camera and radar point cloud information; VoxelNet is used to encode the radar point cloud information to obtain The Swin Transformer deep learning model encodes the acquired images to obtain... VoxelNet is an end-to-end 3D object detection framework based on point clouds. The hierarchical scene modeling module integrates multi-scale features of point clouds and images; The design instance guides the fusion module IGF to guide feature fusion, enhancing the ability to perceive target instances; target instances include construction vehicles; Finally, the decoder and detector head are used to perform bounding box regression and classification on the enhanced instance feature representation to obtain 3D target information of the work space. The 3D target information is used for the robotic arm to plan its running trajectory.

2. The method for spatial perception of robotic arm farmland operations based on 3D target detection according to claim 1, characterized in that, The hierarchical scene modeling module includes point-to-mesh transformation (P2G) and mesh-to-region transformation (G2R); Point-to-mesh transformation (P2G) transforms scattered point cloud data Mapped to a gridded feature with a defined spatial location The mapping formula is as follows: ,in Represents a grid The k-th point in It is a point cloud feature extraction function. It is the image feature projection function, and MSA is a multi-head self-attention mechanism.

3. The method for spatial perception of robotic arm farmland operations based on 3D target detection according to claim 2, characterized in that, Mesh-to-Region Transformation (G2R) utilizes a cross-region self-attention Shift-MSA mechanism to transform mesh features. Further transformed into regional characteristics: ; ShiftMSA is a cross-region self-attention mechanism that divides the grid feature Rm into several non-overlapping windows Rm={R1,R2,…,Rn}. The cross-region self-attention mechanism calculates MSA(Ri) independently for each window feature and sums them.

4. The method for spatial perception of robotic arm farmland operations based on 3D target detection according to claim 3, characterized in that, The design example guides the fusion module IGF to guide feature fusion, including: based on Candidate regions for generating target instance heatmaps are selected, and the top K high-response regions are used to generate a candidate instance set. The feature information of each instance q is fused with scene features through a deformable attention mechanism to obtain an enhanced instance representation. DeformAtt is a deformable attention mechanism. Scene features include 3D coordinate information of the hay truck or the working environment.

Images