Unmanned aerial vehicle oriented millimeter wave radar point cloud and image fusion decision clustering method

Abstract

The application discloses a kind of unmanned aerial vehicle-oriented millimeter wave radar point cloud and image fusion decision clustering method, the method includes: camera and millimeter wave radar are jointly calibrated;Target crop point cloud data is obtained using millimeter wave radar, and point cloud data is merged into aggregate frame point cloud and excludes background noise point;The point cloud data after background noise point exclusion is clustered;The point cloud data after clustering is completed is optimized using target crop based target ID feature;Image data of target crop is obtained using visual sensor, and target crop in image is detected using trained YOLOv7 network;The center coordinates of target crop bounding box are calculated, and the center coordinates are projected on radar coordinates to obtain the elevation information of target crop, and the decision fusion of image and millimeter wave radar point cloud is completed.The application can adaptively adjust the mode of unmanned aerial vehicle pesticide spraying according to the growth characteristics of crops.

Description

Technical Field

[0001] This invention relates to the field of smart agriculture, specifically to a decision clustering method for millimeter-wave radar point cloud and image fusion for unmanned aerial vehicles (UAVs). Background Technology

[0002] With the advancement of agricultural modernization, drone technology is emerging in agriculture. Using drones for pesticide spraying has become a popular research direction for improving production efficiency and reducing environmental pollution. Traditional spraying methods suffer from uneven spraying and pesticide waste; through positioning and intelligent control systems, drones achieve precise pesticide delivery, increasing crop yields. Millimeter-wave radar and visual sensors, as advanced sensing technologies, have been widely applied in various fields.

[0003] Among them, 4D millimeter-wave radar has advantages such as high precision, all-weather performance, long-range detection, high-speed processing, strong anti-interference ability, and multi-target tracking. Visual sensors, on the other hand, can accurately detect targets. Fusion of these two sensors for use in drone pesticide spraying can improve the all-weather performance and accuracy of drone-based crop pesticide spraying. A single sensor is insufficient for accurate environmental perception during drone-based crop pesticide spraying, and fusion technology still faces challenges such as data integration and processing, and intelligent flight control systems. Furthermore, during data fusion, clustering methods are sensitive to outliers and noise points; some excessively large outliers can have a significant impact. Summary of the Invention

[0004] The purpose of this invention is to provide a millimeter-wave radar point cloud and image fusion decision clustering method for UAVs, which aims to improve the spraying height and angle of UAVs that are adaptively adjusted relative to crops.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A decision clustering method for fusing millimeter-wave radar point clouds and images for unmanned aerial vehicles (UAVs) includes the following steps:

[0007] Joint calibration of camera and millimeter-wave radar;

[0008] Use millimeter-wave radar to acquire point cloud data of target crops, merge the point cloud data into aggregated frame point cloud and remove background noise points;

[0009] Clustering is performed on point cloud data after background noise points have been excluded;

[0010] The clustered point cloud data is optimized using target ID features based on the target crop.

[0011] Image data of the target crop is acquired using a visual sensor, and the target crop in the image is detected using a trained YOLOv7 network.

[0012] The center coordinates of the target crop bounding box are calculated, and the center coordinates are projected onto the radar coordinates to obtain the elevation information of the target crop, thus completing the decision fusion of the image and the millimeter-wave radar point cloud.

[0013] Furthermore, the point cloud data is merged into aggregate frames and background noise points are removed, specifically including the following steps:

[0014] The original point cloud frame is specifically defined as:

[0015] P ij (i=1,2,3,…,N,j=1,2,3,…,M i (3)

[0016] Where N represents the total number of point cloud frames, M i P represents the total number of points obtained through preprocessing in the i-th frame. ij This represents the j-th point in the i-th frame;

[0017] Furthermore, the points obtained from processing every five original point cloud frames are merged to form an aggregated frame point cloud. The aggregated frame point cloud is specifically defined as follows:

[0018]

[0019] in, Indicates the aggregate frame index;

[0020] Background noise points are eliminated by setting a reflection intensity threshold T. The specific formula is as follows:

[0021] A′ k ={p∈A k |R p >T} (5)

[0022] Among them, A' k This represents the aggregated frame after reflection intensity filtering, where p represents A. k Point R in p This represents the reflection intensity at point p.

[0023] Furthermore, the point cloud data after background noise points have been excluded is clustered, specifically including the following steps:

[0024] Radar point cloud data A' after excluding background noise points k The coordinates (x', y', z') of the midpoint are used to calculate the distance between points using a modified distance formula, where the modified distance formula is:

[0025]

[0026] Where, p i and p j Let d represent two distinct points in space, where α and β represent the weighted values ​​of the distances in the vertical and horizontal directions, respectively. ij Point p i and point p j The distance between them;

[0027] Based on the distance d between points ij Establish the distance matrix M between points k The specific formula is as follows:

[0028]

[0029] Where n represents the number of points;

[0030] Using the Gaussian kernel function, the distances between points are mapped to a larger similarity space, resulting in a similarity matrix M. s The specific formula is as follows:

[0031]

[0032] Where σ represents M k Standard deviation, similarity matrix M s The specific formula is as follows:

[0033]

[0034] Among them, s ij p i and point p j The similarity between them;

[0035] Based on the similarity matrix M s Calculate the sum of the weights Σ of the edges within the cluster. in The specific formula is as follows:

[0036] Σ in =∑ i,j∈C s ij (10)

[0037] Where C represents the cluster to which the current node belongs, and initially the node itself belongs to a cluster;

[0038] Calculate the sum of the weights Σ of the edges between different clusters. tot The specific formula is as follows:

[0039]

[0040] Data points are clustered based on modularity gain. The specific formula for calculating modularity gain is as follows:

[0041]

[0042] Where m represents the sum of the total weights of all edges in the point cloud data center cluster;

[0043] Each point is merged with its neighboring points in turn, their maximum modularity gain is calculated, and it is determined whether it is greater than 0. If it is greater than 0, the point is placed in the cluster of the neighboring node with the largest modularity increment. This process is continued until the cluster to which the point belongs no longer changes.

[0044] Furthermore, the feature is that the clustered point cloud data is optimized using target ID features based on the target crop, specifically including the following steps:

[0045] Assign an ID to each clustered point and add an associated object to each target point, using the following formula:

[0046] P = (x, y, AO, ID) T (13)

[0047] Where x and y represent the coordinates of the target point P, and AO represents the ID of the point associated with point P;

[0048] Points with the same related objects are grouped into clusters. The specific expression of a cluster is:

[0049] C n,m ={P i}, i∈m (14)

[0050] Where n represents the clustered aggregate frame index, m represents the number of clusters in the aggregate frame, and the most frequently associated objects in each cluster are found and the positions of these points are recorded in a list;

[0051] The formula for calculating the difference in the average x-coordinate of all points belonging to the cluster of this associated object is as follows:

[0052]

[0053] Among them, C n1,m and C n2,m It is a cluster with the same associated objects, k1 represents C n1,m The number of midpoints, k2 represents C n2,m The number of midpoints;

[0054] If |Δx|<1, then the positional relationships between the two clusters will be merged.

[0055] Furthermore, the center coordinates of the target crop bounding box are calculated, and these center coordinates are projected onto radar coordinates to obtain the elevation information of the target crop. This process includes the following steps:

[0056] The bounding box coordinates of the target crop are (x1, x2, y1, y2), where (x1, y1) are the coordinates of the top-left corner of the bounding box, and (x2, y2) are the coordinates of the bottom-right corner of the bounding box. The center coordinates (x1, x2, y1, y2) of the target crop are calculated from these bounding box coordinates. c ,y c The specific formula is:

[0057]

[0058] The center coordinates (x) of the target crop c ,y c The center coordinates (x, y) are obtained by projecting them onto the radar point cloud data of the target crop. r ,y r ,z r According to (x) r ,y r ,z r The coordinates determine the cluster C in which the point belongs. n And calculate the average coordinate (x) of all points in the cluster. p ,y p ,z p The specific formula is:

[0059]

[0060] Where m represents cluster C n The number of points in the z-axis, based on z p The value estimates the growth height of the target crop, enabling the drone to adaptively adjust based on the growth height.

[0061] According to one aspect of the present invention, a storage medium is provided, wherein the storage medium stores instructions that, when a computer reads the instructions, cause the computer to execute the millimeter-wave radar point cloud and image fusion decision clustering method for unmanned aerial vehicles as described in any of the preceding claims.

[0062] According to another aspect of the present invention, an electronic device is provided, comprising a processor and the aforementioned storage medium, wherein the processor executes instructions in the storage medium.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0064] 1. This invention utilizes multi-sensor decision fusion technology, which can adaptively adjust the pesticide spraying method of the drone according to the growth characteristics of the crop, thereby improving the utilization rate of pesticides and the efficiency of pesticide spraying by the drone.

[0065] 2. This invention solves the problems of over-clustering and under-clustering by using a detection enhancement method based on the target ID features of point cloud data of the target crop.

[0066] 3. This invention proposes an improved distance formula and clusters data points based on modular gain, which can better capture the actual relationship between data points and reduce the impact of noise. Attached Figure Description

[0067] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0068] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0069] Figure 2 This is a schematic diagram of the background noise elimination points according to an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of point cloud data clustering according to an embodiment of the present invention;

[0071] Figure 4 This is a general flowchart of an embodiment of the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] like Figure 1 As shown, a decision clustering method for millimeter-wave radar point cloud and image fusion for UAVs specifically includes the following steps:

[0074] Joint calibration of camera and millimeter-wave radar;

[0075] Use millimeter-wave radar to acquire point cloud data of target crops, merge the point cloud data into aggregated frame point cloud and remove background noise points;

[0076] Clustering is performed on point cloud data after background noise points have been excluded;

[0077] The clustered point cloud data is optimized using target ID features based on the target crop.

[0078] Image data of the target crop is acquired using a visual sensor, and the target crop in the image is detected using a trained YOLOv7 network.

[0079] The center coordinates of the target crop bounding box are calculated, and the center coordinates are projected onto the radar coordinates to obtain the elevation information of the target crop, thus completing the decision fusion of the image and the millimeter-wave radar point cloud.

[0080] A 4D millimeter-wave radar and camera were mounted on a drone, and the camera was calibrated to obtain the camera intrinsic parameter coefficients required for coordinate transformation. The calibration was performed using the Zhang Zhengyou calibration method. First, a chessboard pattern was printed, then the camera was fixed and photographed on the chessboard pattern, and finally the data was processed using the toolbox provided by Matlab.

[0081] To establish a corresponding coordinate system for synchronizing the positional relationship between the two sensors when acquiring data, the first step is to use the millimeter-wave radar's installation position plane O as an example. R -X R Y R Z R Establish a millimeter-wave radar coordinate system with the camera mounting position plane O. C -X C Y C Z C Establish a camera coordinate system, based on the three-dimensional world coordinate system O. w -X w Y w Z w A world coordinate system is established as the benchmark. First, the radar coordinate system is transformed into the world coordinate system. Then, through rotation and translation, the world coordinate system is transformed into the camera coordinate system. Finally, based on the reverse thinking of pinhole imaging, the three-dimensional target in the image space coordinate system is transformed into the pixel coordinate system to achieve spatial synchronization of the two sensors. The radar sampling period, which has a lower sampling frequency, is used as the fusion period. Based on multi-threading technology, the radar sampling triggers the camera data sampling to complete the time synchronization of radar and camera data.

[0082] The 4D millimeter-wave radar used performs range-velocity FFT on the two-dimensional matrix of the echo signal from each receiving channel after digital sampling, and then performs CA-CFAR detection on the range-Doppler matrix generated by the range-velocity FFT. CA-CFAR calculates the threshold value by averaging the values ​​around the detection unit. First, the data around the data to be detected in the range and velocity dimensions are extracted, referred to as training units, and the noise power is calculated. Next, the coefficients α for each of the two dimensions are obtained according to the threshold coefficient formula, specifically expressed as follows:

[0083]

[0084] The decision threshold G is then calculated using a formula and set as the threshold value, specifically expressed as follows:

[0085]

[0086] Where N is the number of reference units. If the number of units to be detected is higher than the threshold, it is considered a real target; otherwise, it is considered noise. The sliding window is repeated until all units are detected. After completing 2D-FFT and 2D-CFAR, the complex data of all received channels for each target are arranged by channel number, and FFT is performed on the data. At this time, the data position corresponding to the angle of the target forms a peak. The peak point is obtained by CFAR detection, and the angle value of the target list can be obtained by combining it with the angle measurement formula. The position coordinates of the target point in space can be calculated based on the angle value.

[0087] The data acquired by the 4D millimeter-wave radar is preprocessed to obtain multiple consecutive raw point cloud frames. The specific definition of the raw point cloud frame is as follows:

[0088] P ij (i=1,2,3,…,N,j=1,2,3,…,M i (3)

[0089] Where N is the total number of point cloud frames, M i P is the total number of points obtained through preprocessing in the i-th frame. ij Let j represent the j-th point in the i-th frame. The points obtained from processing every five original point cloud frames are merged to form an aggregated frame point cloud. The specific definition of the aggregated frame point cloud is as follows:

[0090]

[0091] in, It is an aggregated frame index, and background noise points are excluded by setting a reflection intensity threshold T, such as... Figure 2 As shown, the specific representation is as follows:

[0092] A′ k ={p∈A k |R p >T} (5)

[0093] Among them, A' k This represents the aggregated frame after reflection intensity filtering, where p represents A. k Point R in p This represents the reflection intensity at point p.

[0094] like Figure 3 As shown, radar point cloud data A' after background noise points are excluded. k The coordinates (x', y', z') of the midpoint are used to construct the distance matrix M between points using an improved distance formula. k The improved distance formula is as follows:

[0095]

[0096] Where p i and p j Let d represent two distinct points in space. α and β represent the weighted values ​​of the distances in the vertical and horizontal directions, respectively. The values ​​of α and β are set according to the specific application scenario. ij Point p i and point p j The distance between them, matrix M k Represented as:

[0097]

[0098] Where d ij Let M be the distance between point i and point j. To amplify the differences between points, a Gaussian kernel function is used to map the distances between points to a larger similarity space, resulting in a similarity matrix M. k Specifically, it can be expressed as:

[0099]

[0100] Where σ is M k Standard deviation, similarity matrix M s Specifically, it can be expressed as follows:

[0101]

[0102] Where s ij Let Σ represent the similarity between point i and node j, and calculate the sum of the weights of edges within the community based on the similarity matrix. in Specifically, it can be expressed as:

[0103] Σ in =∑ i,j∈C s ij (10)

[0104] Where C represents the cluster to which the current node belongs, and initially the node itself belongs to one cluster. The sum of the weights of the edges between different clusters is Σ. tot Specifically, it can be expressed as:

[0105]

[0106] The data points are then clustered based on the modularity gain, which is specifically expressed as:

[0107]

[0108] Where m represents the sum of the total weights of all edges in a point cloud data center cluster. Initially, each point belongs to a cluster, and the number of clusters is the same as the number of points. Each point is sequentially merged with its neighboring points, and their maximum modularity gain is calculated. If the gain is greater than 0, the point is placed in the cluster of the neighboring node with the largest modularity increment. This process is repeated until the cluster to which the point belongs no longer changes.

[0109] Assign an ID to each clustered point and add an associated object to each target point, specifically as follows:

[0110] P = (x, y, AO, ID) T (13)

[0111] Where x and y are the coordinates of the target point P, and AO is the ID of the points associated with that point. Points with the same associated object form a cluster, which is specifically represented as follows:

[0112] C n,m ={P i}, i∈m (14)

[0113] Where n represents the clustered aggregate frame index, and m represents the number of clusters in the aggregate frame, the most frequently associated objects in each cluster are found and the positions of these points are recorded in a list. The average x-coordinate difference of all points belonging to the cluster of the associated object is calculated, specifically expressed as:

[0114]

[0115] Among them, C n1,m and C n2,m It is a cluster with the same associated objects, k1 represents C n1,m The number of midpoints, k2 represents C n2,m The number of midpoints; if |Δx|<1, then the positional relationship between the two clusters should be merged.

[0116] Based on image data acquired from a visual sensor, a trained YOLOv7 network model is used to detect target crops, obtaining the bounding box coordinates (x1, x2, y1, y2) of the target crops. Here, (x1, y1) represents the coordinates of the top-left corner of the bounding box, and (x2, y2) represents the coordinates of the bottom-right corner. The center coordinates (x1, x2, y1, y2) of the target crops are then calculated from these bounding box coordinates. c ,y c Specifically, it is expressed as follows:

[0117]

[0118] The center coordinates (x) of the target crop c ,y cThe center coordinates (x, y) are obtained by projecting them onto the radar point cloud data of the target crop. r ,y r ,z r ), according to (x r ,y r ,z r The coordinates determine the cluster C to which this point belongs. n And calculate the average coordinate (x) of all points in the cluster. p ,y p ,z p Specifically, it can be expressed as:

[0119]

[0120] Where m represents cluster C n The number of points in the z-axis, and based on z p The value is used to estimate the growth height of the target crop and the pesticide spraying height of the drone. The height is then adaptively adjusted based on the growth height to ensure the effectiveness and coverage of the pesticide spraying.

[0121] like Figure 4 As shown, the overall process includes:

[0122] Joint calibration of the camera and 4D millimeter-wave radar;

[0123] 4D millimeter-wave radar is used to acquire point cloud data of target crops. Multiple original point cloud frames are merged into one aggregate frame and a filtering method based on reflection intensity is used to remove background noise points.

[0124] Clustering of point cloud data is performed using a graph-based method;

[0125] Use target ID features based on the target crop to solve the problems of over-clustering and under-clustering of point cloud data;

[0126] The 4D millimeter-wave radar and vision sensor are jointly calibrated, and the vision sensor is used to acquire image data of the target crop. The trained YOLOv7 network is then used to detect the target crop in the image.

[0127] Based on the obtained target crop detection box, the center coordinates of the detection box are calculated and projected onto the radar coordinates to obtain the elevation information of the target crop. This enables decision fusion of visual sensor and 4D millimeter-wave radar data, achieving adaptive pesticide spraying of crops by drones.

[0128] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A decision clustering method for millimeter-wave radar point cloud and image fusion for unmanned aerial vehicles, characterized in that, Specifically, the following steps are included: Joint calibration of camera and millimeter-wave radar; Use millimeter-wave radar to acquire point cloud data of target crops, merge the point cloud data into aggregated frame point cloud and remove background noise points; Clustering is performed on point cloud data after background noise points have been excluded; The clustered point cloud data is optimized using target ID features based on the target crop. Image data of the target crop is acquired using a visual sensor, and the target crop in the image is detected using a trained YOLOv7 network. The center coordinates of the target crop bounding box are calculated, and the center coordinates are projected onto the radar coordinates to obtain the elevation information of the target crop, thus completing the decision fusion of the image and the millimeter-wave radar point cloud. Merging point cloud data into aggregate frames and excluding background noise points involves the following steps: The original point cloud frame is specifically defined as: (3) in, This represents the total number of point cloud frames. Indicates the first The total number of points in the frame is obtained through preprocessing. Indicates the first The first frame One point; Furthermore, the points obtained from processing every five original point cloud frames are merged to form an aggregated frame point cloud. The aggregated frame point cloud is specifically defined as follows: (4) in, Indicates the aggregate frame index; By setting a reflection intensity threshold To eliminate background noise points, the specific formula is as follows: (5) in, This represents the aggregated frame after reflection intensity filtering. express The point in the middle, Point The intensity of reflection; Clustering of point cloud data after excluding background noise points includes the following steps: Radar point cloud data that excludes background noise points Coordinates of the midpoint The distance between points is calculated using an improved distance formula, which is: (6) in, and They represent two distinct points in space. These represent the weight values ​​for the distances in the vertical and horizontal directions, respectively. Point and points The distance between them; Based on the distance between points Establish the distance matrix between points The specific formula is as follows: (7) in, Indicates the number of points; Using the Gaussian kernel function, the distances between points are mapped to a larger similarity space, resulting in a similarity matrix. The specific formula is as follows: (8) Where σ represents Standard deviation, similarity matrix The specific formula is as follows: (9) in, express and points The similarity between them; Based on the similarity matrix Calculate the sum of the weights of the edges within the cluster. The specific formula is as follows: (10) in, This indicates the cluster to which the current node belongs, and initially, the node itself belongs to a cluster. Calculate the sum of the weights of edges between different clusters. The specific formula is as follows: (11) Data points are clustered based on modularity gain. The specific formula for calculating modularity gain is as follows: (12) Where m represents the sum of the total weights of all edges in the point cloud data center cluster; Each point is merged with its neighboring points in turn, their maximum modularity gain is calculated, and it is determined whether it is greater than 0. If it is greater than 0, the point is placed in the cluster of the neighboring node with the largest modularity increment. This process is continued until the cluster to which the point belongs no longer changes.

2. The method according to claim 1, characterized in that, The clustered point cloud data is optimized using target crop ID features, specifically including the following steps: Assign an ID to each clustered point and add an associated object to each target point, using the following formula: (13) in, Indicates the target point coordinates Point The ID of the associated point; Points with the same related objects are grouped into clusters. The specific expression of a cluster is: (14) in, This represents the clustered aggregate frame index. This indicates the number of clusters in the aggregation frame, finds the most frequently associated objects in each cluster, and records the positions of these points in a list; Calculate the average of all points belonging to the cluster of this associated object. The difference in coordinates is calculated using the following formula: (15) in, and It is a cluster of objects with the same association. express The number of midpoints express The number of midpoints; if If the value is less than 1, then the positional relationship between the two clusters will be merged.

3. The method according to claim 2, characterized in that, The center coordinates of the target crop's bounding box are calculated, and these center coordinates are projected onto radar coordinates to obtain the target crop's elevation information. This process includes the following steps: The bounding box coordinates of the target crop are ,in These are the coordinates of the top-left corner of the bounding box. These are the coordinates of the bottom right corner of the bounding box. The center coordinates of the target crop are calculated from the bounding box coordinates. The specific formula is as follows: (16) center coordinates of the target crop The coordinates of the center are obtained by projecting the data onto the radar point cloud data of the target crop. ,according to The coordinates determine the cluster of the point. And calculate the average coordinates of all points in the cluster. The specific formula is as follows: (17) in, Cluster The number of points in the data is based on The value estimates the growth height of the target crop, enabling the drone to adaptively adjust based on the growth height.

4. A storage medium, characterized in that, The storage medium stores instructions that, when read by a computer, cause the computer to execute the millimeter-wave radar point cloud and image fusion decision clustering method for UAVs as described in any one of claims 1-3.

5. An electronic device, characterized in that, It includes a processor and the storage medium of claim 4, wherein the processor executes instructions in the storage medium.

Images