Cooperative surveying and mapping control method and system based on unmanned ship and unmanned aerial vehicle

Abstract

The invention discloses a cooperative surveying and mapping control method and system based on an unmanned ship and an unmanned aerial vehicle, relates to the technical field of automatic control, and realizes acquisition, processing and fusion of real-time high-precision three-dimensional image data during operation of the system. Multi-source data are synchronously acquired from an unmanned ship and an unmanned aerial vehicle by using LiDAR, sonar and camera sensors, the robustness of feature extraction is enhanced by denoising and calibration preprocessing, ensuring of data quality and application of an ORB algorithm in a feature point extraction module in combination with gradient directions and texture feature points, and an edge detection module identifies the edge in an image, so that the robustness of feature extraction is improved. And generating the boundary set B and marking the edge, thereby improving the accuracy of the edge information. And the feature point matching module performs feature point matching and data fusion by using an RANSAC algorithm to generate a fused image Ifuse, so that the phenomena of dislocation, incomplete fusion, overlapping and overlapping gaps at image joints are reduced as much as possible.

Description

Technical Field

[0001] The present invention relates to the field of automation control technology, and in particular to a method and system for collaborative mapping control based on an unmanned boat and an unmanned aerial vehicle. Background Art

[0002] The collaborative mapping system of unmanned boats and drones belongs to the field of automation and intelligent control technology. This field covers the development and application of various automation equipment and systems, aiming to improve production efficiency and reduce human errors. In the field of automation and intelligent control technology, environmental mapping is an important application direction, mainly used for specific tasks such as terrain mapping, resource exploration, and disaster monitoring. The collaborative mapping of unmanned boats and drones is an emerging sub-field with broad prospects. By combining underwater and aerial data collection, it can provide more comprehensive and accurate environmental information.

[0003] In the existing environmental mapping system, the collaborative work of unmanned boats and drones still faces many challenges. The current situation is that most systems can only use unmanned boats or drones for mapping independently, lacking an effective coordination mechanism, resulting in limited data coverage and accuracy. At the same time, traditional mapping methods usually rely on a single sensor type, such as LiDAR or sonar, which makes it difficult to comprehensively process multi-source data. In addition, the existing system's image fusion and feature matching technologies are relatively rudimentary and cannot fully utilize the rich information obtained from different platforms (unmanned boats and drones). These deficiencies lead to large errors and incomplete data in the mapping results, affecting the effectiveness of environmental monitoring and resource management.

[0004] And due to the limitations in data processing and fusion, in the collaborative mapping process of unmanned boats and drones, the data obtained by the two differ in space and time, and traditional processing methods are difficult to achieve efficient feature point extraction and matching, resulting in image misalignment and incomplete fusion, overlap and gaps at the connection points. In addition, the deficiencies in edge detection and alignment technology also result in obvious seams when images are stitched together, affecting the overall mapping effect. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method and system for collaborative mapping control based on an unmanned boat and an unmanned aerial vehicle, which solves the problems mentioned in the background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a collaborative mapping control system based on unmanned boats and unmanned aerial vehicles, including a data acquisition and preprocessing module, a feature point extraction module, an edge detection module and a feature point matching module;

[0007] The data acquisition and preprocessing module collects real-time mapping three-dimensional image data from unmanned boats and drones by using LiDAR, sonar and camera sensors, and performs denoising preprocessing and calibration preprocessing to form a mapping data group P;

[0008] The feature point extraction module uses the ORB algorithm to extract feature points in the surveying and mapping data set P, and also includes extracting gradient direction feature points and extracting texture feature points in the image, integrating them into the extracted feature data set F i ;

[0009] The edge detection module extracts the feature data set F i Use edge detection algorithm to identify edges in the image, obtain boundary set B, and mark edges according to boundary set B simultaneously;

[0010] The feature point matching module extracts feature data set F according to the boundary set B i The RANSAC algorithm is used to match feature points, and based on the feature point matching results, the mapping data set P of the unmanned boat and the drone is fused to obtain the fused image I. fused .

[0011] Preferably, the data acquisition and preprocessing module includes a sensor acquisition unit and a data preprocessing unit;

[0012] The sensor acquisition unit collects real-time mapping 3D image data from unmanned boats and drones by using LiDAR, sonar and camera sensors, including 3D point cloud data P lidar , underwater terrain sonar data Sonar and two-dimensional image data I;

[0013] The data preprocessing unit processes the collected three-dimensional point cloud data P lidar , underwater terrain sonar data Sonar The denoising preprocessing and calibration preprocessing are performed on the two-dimensional image data I. The denoising preprocessing includes the three-dimensional point cloud data P lidar Mean filter denoising preprocessing acquisition , underwater terrain sonar data Sonar Median filter denoising preprocessing acquisition And two-dimensional image data I Gaussian filtering denoising preprocessing acquisition ,The calibration preprocessing includes calibration under the same time reference, forming a mapping data group P;

[0014] Among them, calibration under the same time base is performed through the following data synchronization methods:

[0015] ;

[0016] In the formula, P represents the mapping data group, and Synchronize represents the synchronization function, which is specifically used to calibrate the data of different sensors under the same time reference. represents the 3D point cloud data after denoising preprocessing, represents the underwater terrain sonar data after denoising preprocessing, Represents denoising preprocessed two-dimensional image data.

[0017] Preferably, the three-dimensional point cloud data P lidar The mean filter denoising preprocessing is performed using the following calculation formula:

[0018] ;

[0019] In the formula, represents the ith 3D point cloud data after denoising, N represents the filter window length, Represents the jth original unprocessed 3D point cloud data;

[0020] Underwater terrain sonar data Sonar The median filter denoising preprocessing is performed using the following calculation formula:

[0021] ;

[0022] In the formula, represents the denoised i-th underwater terrain sonar data, , median represents the median filter function, represents a window of length N, the center of the window is i and i represents the index of the current underwater terrain sonar data, and the underwater terrain sonar data in the window is from arrive , Indicates the half-width of the window;

[0023] The two-dimensional image data I Gaussian filter denoising preprocessing is processed by the following calculation formula:

[0024] ;

[0025] In the formula, Represents the edge pixel coordinates (x, y) of the denoised two-dimensional image data at the position, represents the standard deviation of the Gaussian filter, k represents the window length, specifically half the length of the filter window, Represents the original two-dimensional image data at position The pixel value of represents the normalized coefficient of the Gaussian function, Represents the weight factor of the Gaussian function, based on the offset (p, q) and standard deviation Calculate and obtain, specifically indicating the weight factor Process each pixel value within the window length k.

[0026] Preferably, the feature point extraction module includes a feature point extraction unit and a feature point marking unit;

[0027] The feature point extraction unit uses the ORB algorithm to extract the feature points ORB feature points F in the surveying and mapping data group P ORB , and also includes using the Sobel operator to calculate the gradient of the image in the x and y directions to extract gradient direction feature points And use the gray level co-occurrence matrix GLCM calculation formula to extract the texture feature points T in the image;

[0028] The feature point marking unit marks the three-dimensional position of the feature point, including using a multi-view geometry method to reconstruct the three-dimensional position of the feature point from images of different view angles, and integrates them into an extracted feature data set F i , including ORB feature points F ORB , gradient direction feature points And texture feature points T.

[0029] Preferably, the ORB feature point F ORB Obtained through the following calculation formula:

[0030] ;

[0031] In the formula, F ORB Represents a set of ORB feature points, Represents FAST feature points, which are obtained through the FAST feature point detection algorithm. Indicates the gradient direction of the feature point, which is obtained by calculating the gradient direction of each FAST feature point. The BRIEF descriptor representing the feature point is obtained by generating a descriptor for each feature point using the BRIEF algorithm.

[0032] Preferably, the edge detection module comprises an edge detection unit;

[0033] The edge detection unit extracts the feature data set F i The Canny edge detection algorithm is used to identify edge information in the image, including the two-dimensional coordinates of each edge pixel: edge pixel coordinates (x, y), the gradient magnitude of the edge pixel: gradient magnitude (G(x, y)) and the gradient direction of the edge pixel: gradient direction ( (x, y)), form a boundary set B, mark the edge according to the boundary set B, and then use high and low thresholds to perform double threshold detection on the gradient amplitude in the boundary set B, mark the edge and non-edge, and then use non-maximum suppression to retain the local maximum value and remove non-edge points to form the detected boundary set B.

[0034] Preferably, the gradient magnitude (G(x, y)) and the gradient direction ( (x, y)) is obtained by the following calculation formula:

[0035] ;

[0036] ;

[0037] In the formula, Represents the gradient magnitude at the edge pixel coordinates (x, y), Represents the gradient direction at the edge pixel coordinate (x, y), where and The Sobel operator is used to calculate the gradient of the image in the x and y directions.

[0038] Preferably, the feature point matching module includes a matching unit;

[0039] The matching unit extracts feature data set F according to the boundary set B i Use the RANSAC algorithm to match feature points and find the best matching feature point set {(f i , f j )}, and through the feature point set {(f i , f j )}Composition of drone graphics set and unmanned boat image collection , and fuse it with the surveying and mapping data set P of the unmanned boat and the unmanned aerial vehicle to obtain the fused image I fused , and then use the BFGS algorithm to iteratively adjust the fused image I fused The fusion and intersection area;

[0040] The feature point set {(f i , f j )} is obtained through the following calculation formula:

[0041] ;

[0042] In the formula, and denote the vth pair of feature points A and B respectively, H denotes the fitted model, including the homography matrix fitting model, Represents the loss function, specifically, it is used to calculate the error term;

[0043] The fused image I fused Obtained through the following calculation formula:

[0044] ;

[0045] In the formula, Represents a drone graphics set, represents the image set of the unmanned boat, warpPerspective represents the perspective transformation function, Indicates the fusion preset weight coefficient.

[0046] Preferably, the BFGS algorithm is used to iteratively adjust the fused image I fused Iterate in the following way:

[0047] ;

[0048] In the formula, Indicates that the fused image is The edge pixel coordinates of the position specifically represent the pixel brightness of the current fused image. Indicates that the target image is The edge pixel coordinates of the position specifically represent the pixel brightness of the unfused target image. Represents the accumulation operation of all pixels. Represents the fused image I to be iteratively adjusted fused gather.

[0049] A method for collaborative mapping control based on an unmanned boat and an unmanned aerial vehicle comprises the following steps:

[0050] Step 1: The data acquisition and preprocessing module collects real-time mapping 3D image data from unmanned boats and drones by using LiDAR, sonar and camera sensors, and performs denoising and calibration preprocessing to form a mapping data group P;

[0051] Step 2: The feature point extraction module uses the ORB algorithm to extract feature points from the surveying and mapping data set P, including extracting gradient direction feature points and extracting texture feature points in the image, and integrates them into the extracted feature data set F i ;

[0052] Step 3: The edge detection module extracts the feature data set F i Use edge detection algorithm to identify edges in the image, obtain boundary set B, and mark edges according to boundary set B simultaneously;

[0053] Step 4: The feature point matching module extracts feature data set F based on boundary set Bi The RANSAC algorithm is used to match feature points, and based on the feature point matching results, the mapping data set P of the unmanned boat and the drone is fused to obtain the fused image I. fused .

[0054] The present invention provides a method and system for collaborative mapping control based on unmanned boats and unmanned aerial vehicles, which has the following beneficial effects:

[0055] (1) When the system is running, it realizes the real-time high-precision three-dimensional image data acquisition, processing and fusion. Using LiDAR, sonar and camera sensors, the system can simultaneously collect multi-source data from unmanned boats and drones, and ensure data quality through denoising and calibration preprocessing. The application of ORB algorithm in the feature point extraction module combines gradient direction and texture feature points to enhance the robustness of feature extraction. The edge detection module improves the accuracy of edge information by identifying the edges in the image, generating a boundary set B and marking the edges. The feature point matching module uses the RANSAC algorithm to perform feature point matching and data fusion to generate a fused image I fused , thereby integrating the surveying and mapping data of unmanned boats and drones, and minimizing the misalignment, incomplete fusion, overlap and gaps in the image connection.

[0056] (2) ORB feature points and gradient direction feature points are extracted using the ORB algorithm and Sobel operator, and texture feature points are extracted through gray-level co-occurrence matrix calculation, making feature extraction more comprehensive and accurate. The feature point marking unit reconstructs the three-dimensional position of the feature points through a multi-view geometry method and integrates them into an extracted feature data set, ensuring the spatial consistency of the feature points.

[0057] (3) The RANSAC algorithm is used for feature point matching to effectively identify and match the optimal feature point set of the UAV and the unmanned boat {(f i , f j )}, ensuring the robustness and accuracy of image matching. Combined with the homography matrix model, the system uses the perspective transformation function to achieve the fusion of the UAV graphics set and the unmanned boat image set to obtain the fused image I fused , then, the system uses the BFGS algorithm for iterative optimization and fine-tunes the intersection area of ​​the fused image to ensure the high quality and consistency of the fused image. Through the application of these advanced algorithms, the system solves the problems of inaccurate image matching and low fusion quality in traditional surveying and mapping methods, significantly improving the effect and efficiency of collaborative surveying between unmanned boats and drones, especially in complex environments and changing conditions, ensuring the reliability and accuracy of surveying and mapping results. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1This is a flowchart diagram of a control system based on cooperative mapping between unmanned boats and unmanned aerial vehicles of the present invention;

[0059] Figure 2 The present invention is a schematic diagram of the steps of a collaborative mapping control method based on an unmanned boat and an unmanned aerial vehicle. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0061] The collaborative mapping system of unmanned boats and drones belongs to the field of automation and intelligent control technology. This field covers the development and application of various automation equipment and systems, aiming to improve production efficiency and reduce human errors. In the field of automation and intelligent control technology, environmental mapping is an important application direction, mainly used for specific tasks such as terrain mapping, resource exploration, and disaster monitoring. The collaborative mapping of unmanned boats and drones is an emerging sub-field with broad prospects. By combining underwater and aerial data collection, it can provide more comprehensive and accurate environmental information.

[0062] In the existing environmental mapping system, the collaborative work of unmanned boats and drones still faces many challenges. The current situation is that most systems can only use unmanned boats or drones for mapping independently, lacking an effective coordination mechanism, resulting in limited data coverage and accuracy. At the same time, traditional mapping methods usually rely on a single sensor type, such as LiDAR or sonar, which makes it difficult to comprehensively process multi-source data. In addition, the existing system's image fusion and feature matching technologies are relatively rudimentary and cannot fully utilize the rich information obtained from different platforms (unmanned boats and drones). These deficiencies lead to large errors and incomplete data in the mapping results, affecting the effectiveness of environmental monitoring and resource management.

[0063] And due to the limitations in data processing and fusion, in the collaborative mapping process of unmanned boats and drones, the data obtained by the two differ in space and time, and traditional processing methods are difficult to achieve efficient feature point extraction and matching, resulting in image misalignment and incomplete fusion, overlap and gaps at the connection points. In addition, the deficiencies in edge detection and alignment technology also result in obvious seams when images are stitched together, affecting the overall mapping effect.

[0064] Example 1

[0065] The present invention provides a mapping control system based on the cooperation of unmanned boats and unmanned aerial vehicles. Figure 1 , including data acquisition and preprocessing module, feature point extraction module, edge detection module and feature point matching module;

[0066] The data acquisition and preprocessing module collects real-time mapping three-dimensional image data from unmanned boats and drones by using LiDAR, sonar and camera sensors, and performs denoising preprocessing and calibration preprocessing to form a mapping data group P;

[0067] The feature point extraction module uses the ORB algorithm to extract feature points in the surveying and mapping data set P, and also includes extracting gradient direction feature points and extracting texture feature points in the image, integrating them into the extracted feature data set F i ;

[0068] The edge detection module extracts the feature data set F i Use edge detection algorithm to identify edges in the image, obtain boundary set B, and mark edges according to boundary set B simultaneously;

[0069] The feature point matching module extracts feature data set F according to the boundary set B i The RANSAC algorithm is used to match feature points, and based on the feature point matching results, the mapping data set P of the unmanned boat and the drone is fused to obtain the fused image I. fuse

[0070] In this embodiment, real-time high-precision three-dimensional image data collection, processing and fusion are realized. Using LiDAR, sonar and camera sensors, the system can synchronously collect multi-source data from unmanned boats and drones, and ensure data quality through denoising and calibration preprocessing. The application of ORB algorithm in the feature point extraction module, combined with gradient direction and texture feature points, enhances the robustness of feature extraction. The edge detection module improves the accuracy of edge information by identifying edges in the image, generating a boundary set B and marking edges. The feature point matching module uses the RANSAC algorithm to perform feature point matching and data fusion to generate a fused image I fused , thereby integrating the surveying and mapping data of unmanned boats and drones, and minimizing the misalignment, incomplete fusion, overlap and gaps in the image connection.

[0071] Example 2

[0072] This embodiment is explained in Example 1, please refer to Figure 1 ,Specifically: the data acquisition and preprocessing module includes a sensor acquisition unit and a data preprocessing unit;

[0073] The sensor acquisition unit collects real-time mapping 3D image data from unmanned boats and drones by using LiDAR, sonar and camera sensors, including 3D point cloud data P lidar , underwater terrain sonar data Sonar and two-dimensional image data I;

[0074] The data preprocessing unit processes the collected three-dimensional point cloud data P lidar , underwater terrain sonar data Sonar The denoising preprocessing and calibration preprocessing are performed on the two-dimensional image data I. The denoising preprocessing includes the three-dimensional point cloud data P lidar Mean filter denoising preprocessing acquisition , underwater terrain sonar data Sonar Median filter denoising preprocessing acquisition And two-dimensional image data I Gaussian filtering denoising preprocessing acquisition ,The calibration preprocessing includes calibration under the same time reference, forming a mapping data group P;

[0075] Among them, calibration under the same time base is performed through the following data synchronization methods:

[0076] ;

[0077] In the formula, P represents the mapping data group, and Synchronize represents the synchronization function, which is specifically used to calibrate the data of different sensors under the same time reference. represents the 3D point cloud data after denoising preprocessing, represents the underwater terrain sonar data after denoising preprocessing, Represents denoising preprocessed two-dimensional image data.

[0078] The 3D point cloud data P lidar The mean filter denoising preprocessing is performed using the following calculation formula:

[0079] ;

[0080] In the formula, represents the ith 3D point cloud data after denoising, N represents the filter window length, Represents the jth original unprocessed 3D point cloud data;

[0081] Underwater terrain sonar data Sonar The median filter denoising preprocessing is performed using the following calculation formula:

[0082] ;

[0083] In the formula, represents the denoised i-th underwater terrain sonar data, , median represents the median filter function, represents a window of length N, the center of the window is i and i represents the index of the current underwater terrain sonar data, and the underwater terrain sonar data in the window is from arrive , Indicates the half-width of the window;

[0084] The two-dimensional image data I Gaussian filter denoising preprocessing is processed by the following calculation formula:

[0085] ;

[0086] In the formula, Represents the edge pixel coordinates (x, y) of the denoised two-dimensional image data at the position, represents the standard deviation of the Gaussian filter, k represents the window length, specifically half the length of the filter window, Represents the original two-dimensional image data at position The pixel value of represents the normalized coefficient of the Gaussian function, Represents the weight factor of the Gaussian function, based on the offset (p, q) and standard deviation Calculate and obtain, specifically indicating the weight factor Process each pixel value within the window length k.

[0087] In this embodiment, the system uses LiDAR, sonar and camera sensors to synchronously collect multi-source data from unmanned boats and drones, including three-dimensional point cloud data, underwater terrain sonar data and two-dimensional image data, and significantly improves the quality and accuracy of the data through denoising preprocessing methods such as mean filtering, median filtering and Gaussian filtering.

[0088] Example 3

[0089] This embodiment is explained in Example 2. Please refer to Figure 1 ,Specifically: the feature point extraction module includes a feature point extraction unit and a feature point marking unit;

[0090] The feature point extraction unit uses the ORB algorithm to extract the feature points ORB feature points F in the surveying and mapping data group P ORB , and also includes using the Sobel operator to calculate the gradient of the image in the x and y directions to extract gradient direction feature points And use the gray level co-occurrence matrix GLCM calculation formula to extract the texture feature points T in the image;

[0091] The feature point marking unit marks the three-dimensional position of the feature point, including using a multi-view geometry method to reconstruct the three-dimensional position of the feature point from images of different view angles, and integrates them into an extracted feature data set F i , including ORB feature points F ORB , gradient direction feature points And texture feature points T.

[0092] Among them, the ORB feature point F ORB Obtained through the following calculation formula:

[0093] ;

[0094] In the formula, F ORB Represents a set of ORB feature points, Represents FAST feature points, which are obtained through the FAST feature point detection algorithm. Indicates the gradient direction of the feature point, which is obtained by calculating the gradient direction of each FAST feature point. The BRIEF descriptor representing the feature point is obtained by generating a descriptor for each feature point using the BRIEF algorithm.

[0095] The edge detection module includes an edge detection unit;

[0096] The edge detection unit extracts the feature data set F i The Canny edge detection algorithm is used to identify edge information in the image, including the two-dimensional coordinates of each edge pixel: edge pixel coordinates (x, y), the gradient magnitude of the edge pixel: gradient magnitude (G(x, y)) and the gradient direction of the edge pixel: gradient direction ( (x, y)), form a boundary set B, mark the edge according to the boundary set B, and then use high and low thresholds to perform double threshold detection on the gradient amplitude in the boundary set B, mark the edge and non-edge, and then use non-maximum suppression to retain the local maximum value and remove non-edge points to form the detected boundary set B.

[0097] The gradient magnitude (G(x, y)) and gradient direction ( (x, y)) is obtained by the following calculation formula:

[0098] ;

[0099] ;

[0100] In the formula, Represents the gradient magnitude at the edge pixel coordinates (x, y), Represents the gradient direction at the edge pixel coordinate (x, y), where and The Sobel operator is used to calculate the gradient of the image in the x and y directions.

[0101] In this embodiment, the ORB algorithm and the Sobel operator are used to extract ORB feature points and gradient direction feature points, and the texture feature points are extracted by gray-level co-occurrence matrix calculation, so that the feature extraction is more comprehensive and accurate. The feature point marking unit reconstructs the three-dimensional position of the feature points through a multi-view geometry method and integrates them into an extracted feature data set, ensuring the spatial consistency of the feature points.

[0102] Example 4

[0103] This embodiment is explained in Example 3, please refer to Figure 1 ,Specifically: the feature point matching module includes a matching unit;

[0104] The matching unit extracts feature data set F according to the boundary set B i Use the RANSAC algorithm to match feature points and find the best matching feature point set {(f i , f j )}, and through the feature point set {(f i , f j )}Composition of drone graphics set and unmanned boat image collection , and fuse it with the surveying and mapping data set P of the unmanned boat and the unmanned aerial vehicle to obtain the fused image I fused , and then use the BFGS algorithm to iteratively adjust the fused image I fused The fusion and intersection area;

[0105] The feature point set {(f i , f j )} is obtained through the following calculation formula:

[0106] ;

[0107] In the formula, and denote the vth pair of feature points A and B respectively, H denotes the fitted model, including the homography matrix fitting model, Represents the loss function, specifically, it is used to calculate the error term;

[0108] The fused image I fused Obtained through the following calculation formula:

[0109] ;

[0110] In the formula, Represents a drone graphics set, represents the image set of the unmanned boat, warpPerspective represents the perspective transformation function, Indicates the fusion preset weight coefficient.

[0111] Among them, the BFGS algorithm is used to iteratively adjust the fused image I fused Iterate in the following way:

[0112] ;

[0113] In the formula, Indicates that the fused image is The edge pixel coordinates of the position specifically represent the pixel brightness of the current fused image. Indicates that the target image is The edge pixel coordinates of the position specifically represent the pixel brightness of the unfused target image. Represents the accumulation operation of all pixels. Represents the fused image I to be iteratively adjusted fused gather.

[0114] In this embodiment, the RANSAC algorithm is used to match feature points, effectively identifying and matching the best feature point set {(f i , f j )}, ensuring the robustness and accuracy of image matching. Combined with the homography matrix model, the system uses the perspective transformation function to achieve the fusion of the UAV graphics set and the unmanned boat image set to obtain the fused image I fused , then, the system uses the BFGS algorithm for iterative optimization and fine-tunes the intersection area of ​​the fused image to ensure the high quality and consistency of the fused image. Through the application of these advanced algorithms, the system solves the problems of inaccurate image matching and low fusion quality in traditional surveying and mapping methods, significantly improving the effect and efficiency of collaborative surveying between unmanned boats and drones, especially in complex environments and changing conditions, ensuring the reliability and accuracy of surveying and mapping results.

[0115] Example 5

[0116] A method for collaborative mapping control based on unmanned boats and unmanned aerial vehicles, please refer to Figure 2 , specifically: including the following steps:

[0117] Step 1: The data acquisition and preprocessing module collects real-time mapping 3D image data from unmanned boats and drones by using LiDAR, sonar and camera sensors, and performs denoising and calibration preprocessing to form a mapping data group P;

[0118] Step 2: The feature point extraction module uses the ORB algorithm to extract feature points from the surveying and mapping data set P, including extracting gradient direction feature points and extracting texture feature points in the image, and integrates them into the extracted feature data set F i ;

[0119] Step 3: The edge detection module extracts the feature data set F i Use edge detection algorithm to identify edges in the image, obtain boundary set B, and mark edges according to boundary set B simultaneously;

[0120] Step 4: The feature point matching module extracts feature data set F based on boundary set B i The RANSAC algorithm is used to match feature points, and based on the feature point matching results, the mapping data set P of the unmanned boat and the drone is fused to obtain the fused image I. fused .

[0121] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A collaborative mapping control system based on unmanned boats and unmanned aerial vehicles. Features: It includes data acquisition and preprocessing module, feature point extraction module, edge detection module and feature point matching module; The data acquisition and preprocessing module collects real-time mapping three-dimensional image data from unmanned boats and drones by using LiDAR, sonar and camera sensors, and performs denoising preprocessing and calibration preprocessing to form a mapping data group P; The feature point extraction module uses the ORB algorithm to extract feature points in the surveying and mapping data set P, and also includes extracting gradient direction feature points and extracting texture feature points in the image, integrating them into the extracted feature data set F i ; The edge detection module extracts the feature data set F i Use edge detection algorithm to identify edges in the image, obtain boundary set B, and mark edges according to boundary set B simultaneously; The feature point matching module extracts feature data set F according to the boundary set B i The RANSAC algorithm is used to match feature points, and based on the feature point matching results, the mapping data set P of the unmanned boat and the drone is fused to obtain the fused image I. fused .

2. According to claim 1, a collaborative mapping control system based on unmanned boats and unmanned aerial vehicles, Features: The data acquisition and preprocessing module includes a sensor acquisition unit and a data preprocessing unit; The sensor acquisition unit collects real-time mapping 3D image data from unmanned boats and drones by using LiDAR, sonar and camera sensors, including 3D point cloud data P lidar , underwater terrain sonar data Sonar and two-dimensional image data I; The data preprocessing unit processes the collected three-dimensional point cloud data P lidar , underwater terrain sonar data Sonar The denoising preprocessing and calibration preprocessing are performed on the two-dimensional image data I. The denoising preprocessing includes the three-dimensional point cloud data P lidar Mean filter denoising preprocessing acquisition , underwater terrain sonar data Sonar Median filter denoising preprocessing acquisition And two-dimensional image data I Gaussian filtering denoising preprocessing acquisition ,The calibration preprocessing includes calibration under the same time reference, forming a mapping data group P; Among them, calibration under the same time base is performed through the following data synchronization methods: ； In the formula, P represents the mapping data group, and Synchronize represents the synchronization function, which is specifically used to calibrate the data of different sensors under the same time reference. represents the 3D point cloud data after denoising preprocessing, represents the underwater terrain sonar data after denoising preprocessing, Represents denoising preprocessed two-dimensional image data.

3. According to claim 2, a collaborative mapping control system based on unmanned boats and unmanned aerial vehicles, Features: The 3D point cloud data P lidar The mean filter denoising preprocessing is performed using the following calculation formula: ； In the formula, represents the ith 3D point cloud data after denoising, N represents the filter window length, Represents the jth original unprocessed 3D point cloud data; Underwater terrain sonar data Sonar The median filter denoising preprocessing is performed using the following calculation formula: ； In the formula, represents the denoised i-th underwater terrain sonar data, represents a window of length N, the center of the window is i and i represents the index of the current underwater terrain sonar data. The underwater terrain sonar data in the window is from arrive , Indicates the half-width of the window; The two-dimensional image data I Gaussian filter denoising preprocessing is processed by the following calculation formula: ； In the formula, Represents the edge pixel coordinates (x, y) of the denoised two-dimensional image data at the position, represents the standard deviation of the Gaussian filter, k represents the window length, specifically half the length of the filter window, Represents the original two-dimensional image data at position The pixel value of represents the normalized coefficient of the Gaussian function, Represents the weight factor of the Gaussian function, based on the offset (p, q) and standard deviation Calculate and obtain, specifically indicating the weight factor Process each pixel value within the window length k.

4. According to claim 2, a collaborative mapping control system based on unmanned boats and unmanned aerial vehicles, Features: The feature point extraction module includes a feature point extraction unit and a feature point marking unit; The feature point extraction unit uses the ORB algorithm to extract the feature points ORB feature points F in the surveying and mapping data group P ORB , and also includes using the Sobel operator to calculate the gradient of the image in the x and y directions to extract gradient direction feature points And use the gray level co-occurrence matrix GLCM calculation formula to extract the texture feature points T in the image; The feature point marking unit marks the three-dimensional position of the feature point, including using a multi-view geometry method to reconstruct the three-dimensional position of the feature point from images of different view angles, and integrates them into an extracted feature data set F i , including ORB feature points F ORB , gradient direction feature points And texture feature points T.

5. According to claim 4, a collaborative mapping control system based on unmanned boats and unmanned aerial vehicles, Features: Among them, the ORB feature point F ORB Obtained through the following calculation formula: ； In the formula, F ORB Represents a set of ORB feature points, Represents FAST feature points, which are obtained through the FAST feature point detection algorithm. Indicates the gradient direction of the feature point, which is obtained by calculating the gradient direction of each FAST feature point. The BRIEF descriptor representing the feature point is obtained by generating a descriptor for each feature point using the BRIEF algorithm.

6. According to claim 1, a collaborative mapping control system based on unmanned boats and unmanned aerial vehicles, Features: The edge detection module includes an edge detection unit; The edge detection unit extracts the feature data set F i The Canny edge detection algorithm is used to identify edge information in the image, including the two-dimensional coordinates of each edge pixel: edge pixel coordinates (x, y), the gradient magnitude of the edge pixel: gradient magnitude (G(x, y)) and the gradient direction of the edge pixel: gradient direction ( (x, y)), form a boundary set B, mark the edge according to the boundary set B, and then use high and low thresholds to perform double threshold detection on the gradient amplitude in the boundary set B, mark the edge and non-edge, and then use non-maximum suppression to retain the local maximum value and remove non-edge points to form the detected boundary set B.

7. The collaborative mapping control system based on unmanned boat and unmanned aerial vehicle according to claim 6, Features: The gradient magnitude (G(x, y)) and gradient direction ( (x, y)) is obtained by the following calculation formula: ； ； In the formula, Represents the gradient magnitude at the edge pixel coordinates (x, y), Represents the gradient direction at the edge pixel coordinate (x, y), where and The Sobel operator is used to calculate the gradient of the image in the x and y directions.

8. The collaborative mapping control system based on unmanned boat and unmanned aerial vehicle according to claim 6, Features: The feature point matching module includes a matching unit; The matching unit extracts feature data set F according to the boundary set B i Use the RANSAC algorithm to match feature points and find the best matching feature point set {(f i , f j )}, and through the feature point set {(f i , f j )}Composition of drone graphics set and unmanned boat image collection , and fuse it with the surveying and mapping data set P of the unmanned boat and the unmanned aerial vehicle to obtain the fused image I fused , and then use the BFGS algorithm to iteratively adjust the fused image I fused The fusion and intersection area; The feature point set {(f i , f j )} is obtained through the following calculation formula: ； In the formula, and denote the vth pair of feature points A and B respectively, H denotes the fitted model, including the homography matrix fitting model, Represents the loss function, specifically, it is used to calculate the error term; The fused image I fused Obtained through the following calculation formula: ； In the formula, Represents a drone graphics set, represents the image set of the unmanned boat, warpPerspective represents the perspective transformation function, Indicates the fusion preset weight coefficient.

9. The collaborative mapping control system based on unmanned boat and unmanned aerial vehicle according to claim 8, Features: in, Use BFGS algorithm to iteratively adjust the fused image I fused Iterate in the following way: ； In the formula, Indicates that the fused image is The edge pixel coordinates of the position specifically represent the pixel brightness of the current fused image. Indicates that the target image is The edge pixel coordinates of the position specifically represent the pixel brightness of the unfused target image. Represents the accumulation operation of all pixels. Represents the fused image I to be iteratively adjusted fused gather.

10. A method for collaborative mapping control based on an unmanned boat and an unmanned aerial vehicle, comprising the control system for collaborative mapping based on an unmanned boat and an unmanned aerial vehicle as claimed in any one of claims 1 to 9, Features: The following steps are involved: Step 1: The data acquisition and preprocessing module collects real-time mapping 3D image data from unmanned boats and drones by using LiDAR, sonar and camera sensors, and performs denoising and calibration preprocessing to form a mapping data group P; Step 2: The feature point extraction module uses the ORB algorithm to extract feature points from the surveying and mapping data set P, including extracting gradient direction feature points and extracting texture feature points in the image, and integrates them into the extracted feature data set F i ; Step 3: The edge detection module extracts the feature data set F i Use edge detection algorithm to identify edges in the image, obtain boundary set B, and mark edges according to boundary set B simultaneously; Step 4: The feature point matching module extracts feature data set F based on boundary set B i The RANSAC algorithm is used to match feature points, and based on the feature point matching results, the mapping data set P of the unmanned boat and the drone is fused to obtain the fused image I. fused .

Images