A point cloud correction method and system in unmanned aerial vehicle surveying and mapping based on artificial intelligence

Abstract

The application discloses a kind of point cloud correction method and system in unmanned aerial vehicle surveying and mapping based on artificial intelligence, it is related to unmanned aerial vehicle surveying and mapping field;Optimal frame image is selected by image overlapping area matching, can be well filtered image with jelly effect;The precision sparse point cloud is corrected or completed using the texture information of RGB image, makes up the image defect caused by resonance.It specifically includes: obtaining first image, second image and first depth image;Obtain the region of interest in first image and second image;Obtain the difference degree index of the region of interest in first image and the region of interest in second image and the image pair to be selected;If three-dimensional angular velocity and three-dimensional acceleration are not changed, then the image pair to be selected is selected image pair;According to the gray gradient of selected image pair, the first depth image is corrected.The specific application scenario of the application is: when unmanned aerial vehicle city surveying and mapping, point cloud data is corrected by RGB image.

Description

Technical Field

[0001] This application relates to the field of UAV mapping, specifically to a point cloud correction method and system for UAV mapping based on artificial intelligence. Background Technology

[0002] The field of modern drone mapping is developing rapidly. The lidar built into the drone can obtain point cloud data with high precision and export point cloud images of the scene. Each image has geographic information tags, including the three-dimensional coordinates of the image center point and the dividing lines and texture information of buildings.

[0003] During the flight of UAVs for urban mapping, resonance and shaking are inevitable, which can cause inaccurate or even missing point cloud information acquired by lidar. At the same time, the point cloud data of urban buildings may be missing or have low point cloud accuracy due to the long shooting distance, resulting in sparse point cloud data and affecting the mapping accuracy and image quality. Therefore, it is necessary to correct the point cloud data to obtain high-precision point cloud images. Summary of the Invention

[0004] This invention provides a point cloud correction method and system for UAV mapping based on artificial intelligence, comprising: acquiring a first image, a second image, and a first depth image; acquiring a region of interest (ROI) in the overlapping region of the first and second images; acquiring a difference index between the ROI in the first image and the ROI in the second image; determining candidate image pairs based on the difference index and a preset difference threshold; if the three-dimensional angular velocity and three-dimensional acceleration of the candidate image pair at the corresponding time do not change, then the candidate image pair is selected; correcting the first depth image based on the grayscale gradient of the selected image pair. Compared with existing technologies, by selecting the optimal frame image through image overlapping region matching, the best comparison image can be obtained for point cloud correction, resulting in a more stable image with better robustness; utilizing the gradient distribution and color distribution of the ROI in the two images can obtain a more accurate image difference, which can effectively filter out images with rolling shutter effect.

[0005] To address the aforementioned technical problems, this invention proposes a point cloud correction method and system for UAV mapping based on artificial intelligence.

[0006] Firstly, this paper proposes a point cloud correction method for UAV mapping based on artificial intelligence, including:

[0007] Acquire the first image, the second image, and the first depth image.

[0008] Obtain the region of interest in the overlapping region of the first image and the second image; the region of interest is the part of the object to be tested in the overlapping region.

[0009] Obtain the difference index between the region of interest in the first image and the region of interest in the second image.

[0010] The candidate image pairs are determined based on the difference index and the preset difference threshold.

[0011] Determine whether the three-dimensional angular velocity and three-dimensional acceleration of the candidate image pair at the corresponding time have not changed: if the determination result is yes, then the candidate image pair is the selected image pair, and the second image in the selected image pair is the optimal frame image.

[0012] The first depth image is corrected based on the grayscale gradient of the selected image pair.

[0013] Furthermore, in the point cloud correction method for UAV mapping based on artificial intelligence, the difference index includes color difference, grayscale gradient difference, and edge difference.

[0014] Furthermore, in the aforementioned point cloud correction method for UAV mapping based on artificial intelligence, the step of obtaining the color difference degree includes:

[0015] Divide the region of interest into different regions based on hue, within a preset distance threshold range.

[0016] Calculate the hue, saturation, and brightness of each region, as well as the average hue, saturation, and brightness of all regions.

[0017] The color difference is obtained based on the hue, saturation, and brightness of each region, as well as the average hue, saturation, and brightness of all regions.

[0018] Furthermore, in the aforementioned point cloud correction method for UAV mapping based on artificial intelligence, the step of obtaining the grayscale gradient difference includes:

[0019] The first image and the second image are processed to obtain a first grayscale image and a second grayscale image, respectively.

[0020] The gradient information of the first grayscale image is obtained based on the gradient values ​​of the pixels in the region of interest of the first grayscale image.

[0021] The gradient information of the second grayscale image is obtained based on the gradient values ​​of the pixels in the region of interest of the second grayscale image.

[0022] The grayscale gradient difference is obtained based on the gradient information of the first grayscale image and the gradient information of the second grayscale image.

[0023] Furthermore, in the aforementioned point cloud correction method for UAV mapping based on artificial intelligence, the step of obtaining the edge difference degree includes:

[0024] Obtain a first image edge pixel set and a second image edge pixel set. The first image edge pixel set is a set of pixels representing the edges of the region of interest in the first image, and the second image edge pixel set is a set of pixels representing the edges of the region of interest in the second image.

[0025] The edge difference is obtained based on the first image edge pixel set and the second image edge pixel set.

[0026] Furthermore, the point cloud correction method in the AI-based UAV mapping further includes the following steps before acquiring the first image, second image, and first depth image during the mapping process:

[0027] The damping of the drone gimbal is adaptively adjusted according to the preset adjustment range.

[0028] Secondly, this invention proposes a point cloud correction system for UAV mapping based on artificial intelligence, comprising: an image acquisition module, a region of interest acquisition module, a difference calculation module, a candidate image pair acquisition module, a selected image pair acquisition module, and an image correction module.

[0029] The image acquisition module is used to acquire a first image, a second image, and a first depth image.

[0030] The region of interest acquisition module is used to acquire the region of interest in the overlapping region of the first image and the second image; the region of interest is the part to be corrected in the overlapping region.

[0031] The difference calculation module is used to obtain the difference index between the region of interest in the first image and the region of interest in the second image.

[0032] The candidate image pair acquisition module is used to determine candidate image pairs based on the difference index and a preset difference threshold.

[0033] The selected image pair acquisition module is used to determine whether the three-dimensional angular velocity and three-dimensional acceleration corresponding to the time of the candidate image pair have not changed: if the determination result is yes, then the candidate image pair is the selected image pair.

[0034] The image correction module is used to correct the first depth image based on the grayscale gradient of the selected image pair.

[0035] This invention provides a point cloud correction method and system for UAV mapping based on artificial intelligence, comprising: acquiring a first image, a second image, and a first depth image; acquiring a region of interest (ROI) in the overlapping region of the first and second images; acquiring a difference index between the ROI in the first image and the ROI in the second image; determining candidate image pairs based on the difference index and a preset difference threshold; if the three-dimensional angular velocity and three-dimensional acceleration of the candidate image pairs at the corresponding time do not change, then the selected image pairs are selected; and correcting the first depth image based on the grayscale gradient of the selected image pairs.

[0036] Compared to existing technologies, selecting the optimal frame image by matching overlapping regions of images can yield the best comparison image for point cloud correction, resulting in a more stable image with better robustness. Utilizing the gradient and color distribution of the ROI (Region of Interest) in two images can provide a more accurate image difference, effectively filtering out images with jelly effect. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating a point cloud correction method for UAV mapping based on artificial intelligence, provided in an embodiment of the present invention.

[0039] Figure 2 This is a flowchart illustrating another point cloud correction method for UAV mapping based on artificial intelligence, provided in an embodiment of the present invention.

[0040] Figure 3 This is a flowchart illustrating a point cloud correction system for UAV mapping based on artificial intelligence, provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0042] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0043] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature; in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0044] Example 1

[0045] This invention provides a point cloud correction method for UAV mapping based on artificial intelligence, such as... Figure 1 As shown, it includes:

[0046] S101. Acquire the first image, the second image, and the first depth image.

[0047] In this embodiment, a first image is obtained by a global shutter camera deployed on the drone, and a second image is obtained by a rolling shutter camera. The images obtained by both the global shutter camera and the rolling shutter camera are RGB format images. Due to its own exposure method, the first image obtained by the global shutter camera will not have a rolling shutter effect caused by the drone's resonance. The first image and the second image correspond one-to-one.

[0048] Global shutter achieves this by exposing the entire scene simultaneously. All pixels on the image sensor collect light and are exposed at the same time; that is, the image sensor begins collecting light at the start of exposure and the light-collecting circuit is cut off at the end of exposure. The image sensor readings then represent a photograph.

[0049] Unlike a global shutter, a rolling shutter achieves exposure line by line through a photosensitive element. At the start of exposure, the photosensitive element scans and exposes line by line until all pixels are exposed. All of this is completed in a very short time.

[0050] If the subject is moving at high speed relative to the camera, using global shutter mode will result in a blurry image if the exposure time is too long. Using rolling shutter mode, the progressive scan speed is insufficient, which may lead to issues such as "tilt," "wobbly," or "partial exposure." This phenomenon observed with rolling shutter mode is called the rolling shutter effect.

[0051] In this embodiment, the first depth image is a TOF (Time of Flight) point cloud image obtained by high-frequency scanning using a lidar.

[0052] A lidar (Light Detection and Ranging) system is an active optical sensor that emits a laser beam toward a target as it moves along a specific measurement path. Receivers within the lidar sensor detect and analyze the laser light reflected back from the target. These receivers record the precise time it takes for the laser pulse to travel from leaving the system to returning, thereby calculating the range distance between the sensor and the target. These distance measurements, along with position information, are then converted into measurements of the actual three-dimensional point of the reflected target in object space.

[0053] S102, Obtain the region of interest in the overlapping region of the first image and the second image; the region of interest is the part to be corrected in the overlapping region.

[0054] In the field of image processing, a region of interest (ROI) is a selected area in an image that is the focus of the image analysis. This region is delineated for further processing. Using ROIs to define the target area can reduce processing time and increase accuracy. In this embodiment, the ROI is the portion to be corrected within the overlapping region; in this embodiment, the portion to be corrected is a building section.

[0055] S103. Obtain the difference index between the region of interest in the first image and the region of interest in the second image.

[0056] The difference indicators mentioned in this embodiment include color difference, grayscale gradient difference, and edge difference.

[0057] In this embodiment, color refers to the hue (H), saturation (S), and value (V) of the image after converting the RGB image to an HSV image. HSV represents a color mode: in the HSV mode, H (hue) represents hue, S (saturation) represents saturation, and V (value) represents value.

[0058] Hue: On a standard color wheel ranging from 0 to 360 degrees, hue is measured by its position. In common usage, hue is identified by color names, such as red, green, or orange. Black and white have no hue.

[0059] Saturation (S): Represents the purity of a color; 0 represents gray. White, black, and other gray colors have no saturation. At maximum saturation, each hue has the purest color light. Values ​​range from 0 to 100%.

[0060] Brightness (V.value): This is the lightness or darkness of a color. A value of 0 represents black. Maximum brightness is the most vivid state of the color. Values ​​range from 0 to 100%.

[0061] The color difference refers to the difference in hue (H), saturation (S), and brightness (V).

[0062] Viewing an image as a two-dimensional discrete function, the gray-level gradient is actually the derivative of this two-dimensional discrete function. We use the difference to replace the differential to obtain the gray-level gradient of the image. Some commonly used gray-level gradient templates include: Roberts gradient, Sobel gradient, Prewitt gradient, and Laplacian gradient.

[0063] In this embodiment, edge difference refers to the difference between pixels on the edge lines of buildings in the ROI in the first and second images.

[0064] S104. Determine the candidate image pairs based on the difference index and the preset difference threshold.

[0065] In this embodiment, since the first image and the second image are acquired at the same time, all the first images and the second images are in a one-to-one correspondence at the same time. Therefore, the candidate image pair includes the first image and the second image corresponding to the first image.

[0066] S105. If the three-dimensional angular velocity and three-dimensional acceleration of the candidate image pair at the corresponding time do not change, then the candidate image pair is the selected image pair.

[0067] In this embodiment, the influence of changes in the three-dimensional angular velocity and three-dimensional acceleration of the UAV on the image imaging effect is considered. If the three-dimensional angular velocity and three-dimensional acceleration of the UAV change, the second image in the candidate image pair may have a jelly effect. When the three-dimensional angular velocity and three-dimensional acceleration at the time corresponding to the candidate image pair do not change, the candidate image is determined to be the selected image pair, and the second image in the selected image pair is the optimal frame image.

[0068] S106. Correct the first depth image according to the grayscale gradient of the selected image pair.

[0069] Image correction refers to the restorative processing of distorted images. The causes of image distortion include: image distortion due to aberrations, distortions, and limited bandwidth of the imaging system; geometric distortion due to the shooting posture and scanning nonlinearity of the imaging device; and image distortion due to motion blur, radiation distortion, and introduced noise. The basic idea of ​​image correction is to establish a corresponding mathematical model based on the cause of image distortion, extract the necessary information from the contaminated or distorted image signal, and restore the original image by reversing the process that caused the distortion. The actual restoration process involves designing a filter that can calculate an estimate of the true image from the distorted image, approximating the true image as closely as possible according to a predefined error criterion.

[0070] In this embodiment, image correction refers to supplementing missing point cloud data and correcting point cloud data with abnormalities.

[0071] Compared with traditional technical solutions, the beneficial effects of this invention are:

[0072] 1. By matching the overlapping regions of images to select the optimal frame image, the best comparison image can be obtained for point cloud correction, resulting in a more stable image with better robustness.

[0073] 2. By utilizing the gradient and color distribution of the ROI (Region of Interest) in two images, a more accurate image difference can be obtained, which can effectively filter out images with jelly effect.

[0074] 3. Point cloud data is corrected by using the optimal frame image. The rich texture information in the RGB image can effectively correct or complete the sparse point cloud, which can make up for the shortcomings of LiDAR scanning and is simple to implement.

[0075] Example 2

[0076] This invention provides a point cloud correction method for UAV mapping based on artificial intelligence, such as... Figure 2 As shown, it includes:

[0077] S201. Acquire the first image, the second image, and the first depth image.

[0078] In this embodiment, a first image is obtained by a global shutter camera deployed on the drone, and a second image is obtained by a rolling shutter camera. In this embodiment, the first depth image refers to a TOF point cloud image obtained by a high-frequency scanning strategy.

[0079] The purpose of this step is to deploy a global shutter camera and a rolling shutter camera on a drone to acquire corresponding images, and to obtain TOF point cloud images through a high-frequency scanning strategy.

[0080] Because global shutter cameras use global exposure during imaging, the image clarity will be poor when the drone is flying at a higher altitude. However, global shutter cameras will not experience the rolling shutter effect due to the resonance and shaking of the drone. On the other hand, rolling shutter cameras will experience the rolling shutter effect due to the resonance of the drone because of their line-by-line exposure characteristics.

[0081] When deploying a global shutter camera and a rolling shutter camera on a drone, the viewing angles of the two cameras should be kept as close as possible to avoid large offsets, so that corresponding areas of the images can be matched later. At the same time, the damping of the drone gimbal should be adaptively adjusted according to the preset adjustment range threshold to obtain the first and second images.

[0082] S202. Obtain the region of interest in the overlapping region of the first image and the second image.

[0083] In this embodiment, the region of interest is the building portion of the overlapping region. The purpose of this step is to obtain the overlapping region of the first image and the second image captured by different cameras, further obtain the ROI (Region of Interest) from the overlapping region through the regional image features, and register the overlapping region so that the image features of the ROI in the first image and the second image are the same.

[0084] The overlapping region matching of the obtained first and second images includes the following:

[0085] First, given the prior information indicating the UAV's flight direction, the global shutter camera is to the left of the rolling shutter camera. The target image in the first image should appear to the left of the second image; this area is the overlapping area.

[0086] Secondly, the left region of the second image and the center region of the first image are overlapped and matched using the prior region offset α. In this embodiment, the region offset α = 100. α can be adjusted according to the specific situation and the camera deployment position. Setting the region offset can reduce the amount of calculation for subsequent image matching.

[0087] In this embodiment, buildings are used as objects to be corrected. In this embodiment, edge detection is performed on all buildings in the overlapping area to obtain the edge contour of the buildings. Then, the pixel coordinates (a, b) of a corner point of the edge contour of the building in the first image are taken. Then, the pixel coordinates of the corresponding corner point in the second image should be (a, b-α).

[0088] The edge detection method used in this embodiment is the Canny edge detection algorithm. The Canny algorithm can better preserve the edge information of the image to obtain accurate edge contours; then, image denoising processing is performed on the image after edge contour detection is completed.

[0089] Furthermore, when the corner point matching fails, a 3×3 corner point search box is constructed in this embodiment to perform a horizontal bidirectional search on the corner point, with the corresponding corner point pixel coordinates that should appear in the second image as the center point. The step size of the search box is 1. The search stops after the corresponding corner point is matched, and then the region offset value α is updated to obtain a new region offset value, thus obtaining the approximate overlapping region, i.e., ROI.

[0090] S203. Obtain the difference index between the region of interest in the first image and the region of interest in the second image.

[0091] The difference indicators include color difference, grayscale gradient difference, and edge difference.

[0092] The purpose of this step is to perform pixel matching on the ROIs in the first image and the second image. Due to the exposure method, the first image obtained by the global shutter camera will not have a rolling shutter effect caused by drone resonance. Therefore, this embodiment uses the first image to judge the second image and determines whether the difference between the first image and the second image is within the preset difference threshold range.

[0093] The steps for obtaining color difference, grayscale gradient difference, and edge difference include the following:

[0094] S2031. Obtain the color difference between the region of interest in the first image and the region of interest in the second image.

[0095] The purpose of this step is to obtain the color difference C between the ROIs in the first image and the ROIs in the second image by performing color analysis on the ROIs in the first image and the ROIs in the second image. diff .

[0096] First, the RGB image of the ROI is converted to HSV color space to obtain the hue (H), saturation (S), and value (V) information of the ROI. The color space conversion method specifically includes the following:

[0097] max=max(R,G,B), min=min(R,G,B), V=max(R,G,B), Where R is the pixel value of the red channel in the image, G is the pixel value of the green channel in the image, and B is the pixel value of the blue channel in the image.

[0098]

[0099] This embodiment focuses on the color information of the ROI edge region. Since the two cameras obtained the image sequence under almost the same viewing angle and lighting conditions, the color information matching does not require global matching. It is only necessary to match the color information of the region edge and the building area. The color information of the middle area can be matched naturally. This invention focuses on the color difference information at the left edge of the ROI.

[0100] In this embodiment, the region from the left edge of the ROI to the right offset by ω columns, where ω is a preset distance threshold (ω = 200 in this embodiment), is divided into the same region block for areas with the same hue, and then the region from the left edge of the ROI to the right offset by 200 columns is divided into different region blocks.

[0101] Calculate the hue (H) of all regions. A1 V A2 H Av ), Saturation (S) A1 S A2 S Av ) and brightness (V) A1 V A2 V Ac Simultaneously, the average hue of all regions within the corresponding ROI in the second image is calculated. Average saturation and average brightness

[0102]

[0103]

[0104] in, This represents the average hue of all regions in the second image corresponding to the c-th region in the first image. This represents the average saturation of all regions in the second image corresponding to the c-th region in the first image. This represents the average brightness of all regions in the second image corresponding to the c-th region in the first image.

[0105] c is a positive integer in the range [1, v], d is a positive integer in the range [1, m], and H Bcd S represents the hue of the d-th region in the second image corresponding to the c-th region in the first image. Bcd V represents the saturation of the d-th region in the second image corresponding to the c-th region in the first image. Bcd This represents the brightness of the d-th region in the second image, which corresponds to the c-th region in the first image.

[0106] Since the range of hue H is 0≤H≤360, the hue needs to be normalized before calculating the color difference.

[0107] When color difference When the value approaches 0, it indicates that the color distribution of the ROI edge region image is normally matched, and the region offset α is reasonable and accurate.

[0108] When color difference When this occurs, it indicates that the region offset α needs to be further adjusted.

[0109] The calculation of color information matching for building areas is the same as that for edge color difference, ultimately yielding the color difference of the building area. When the color difference within a building area reflects the presence of a rolling shutter effect in image sequence B, then the color difference... The more severe the jelly effect, the greater the color difference.

[0110] S2032. Obtain the grayscale gradient difference between the region of interest in the first image and the region of interest in the second image.

[0111] By converting the first and second images to grayscale, and matching the grayscale gradients of the building regions within the ROIs in the first and second images, the grayscale gradient difference G is obtained. diff .

[0112] The gray-level gradient of pixels in an image is obtained by calculating the gray-level gradients in the x-axis, y-axis, and diagonal directions of the edge region in the building's edge contour. The specific methods for calculating the gray-level gradient of pixels in an image include the following:

[0113]

[0114] G x (x,y)=I(x+1,y)-I(x,y),

[0115] G y (x, y)=I(x, y+1)-I(x, y),

[0116]

[0117] Among them G x (x, y) represents the gray-level gradient along the x-axis, G y (x, y) represents the gray-level gradient along the y-axis. Represents the gray-level gradient along the diagonal direction, where I(x+1, y) is the gray-level value of the pixel at coordinates (x+1, y), (x, y) is the gray-level value of the pixel at coordinates (x, y), and (x, y+1) is the gray-level value of the pixel at coordinates (x, y+1).

[0118] Based on the gray-level gradient calculation method for pixels in the image described above, the gray-level gradient difference between the first image and the second image is obtained. Among them G k G′ represents the gradient information of the k-th pixel in the first image. k This represents the gradient information of the k-th pixel in the second image.

[0119] S2033. Obtain the edge difference between the region of interest in the first image and the region of interest in the second image.

[0120] Based on the set of pixel coordinates e of the building edges within the ROI in the first image A The set of pixels e corresponding to the edge of the ROI in the second image B The edge difference E is obtained. diff ...

[0121] To improve computational efficiency while enhancing the validity and accuracy of the results, this embodiment sets a 10×20 window to slide within the ROI of the image sequence pair. The sliding direction is along the column direction of the pixels in the image, with the starting position at the top of the detected edge region. The sliding step size is variable and adjusted based on the edge difference detected previously. The specific activity process of the window includes the following:

[0122] 1. In this embodiment, the initial step size of the sliding window is 10. When the edge difference detected at the initial position... When the value approaches 0, in this embodiment, the step size is increased by 1 time to continue detecting the edge difference of the next sliding window region.

[0123] 2. When the edge difference of the window At this time, the step size remains unchanged and the detection continues. D=1 is the set empirical threshold, which can be adjusted according to the actual accuracy requirements.

[0124] 3. The sliding window will automatically stop after sliding 3 times. It can also stop sliding earlier if the edge disappears before reaching the third time.

[0125] in This represents the edge difference degree of the i-th sliding window, where i = 1, 2, 3. This represents the x-coordinate of the j-th pixel on the edge of the building in the first image. This represents the x-coordinate of the j-th pixel on the edge of the building in the second image. This represents the ordinate of the j-th pixel on the edge of the building in the first image. This represents the ordinate of the j-th pixel on the building edge line in the second image. The coordinates of the corresponding points in the first image and the corresponding points in the second image should satisfy the rules for corner coordinates in step S2.

[0126] Finally, the degree of difference was constructed using the three indicators obtained. The three indicators are normalized to eliminate the influence of the dimensions between the indicators, and the range of Z is normalized to [0, 1].

[0127] S204. Determine the candidate image pairs based on the difference index and the preset difference threshold.

[0128] The closer the difference Z value is to 1, the higher the degree of jelly effect in the second image; the closer the difference Z value is to 0, the higher the degree of matching between the first and second images, and the smaller the jelly effect in the second image. The candidate image pair includes the first image and the second image corresponding to the first image.

[0129] A difference threshold z is set. When the difference Z approaches 0, i.e., Z∈[0, z], the image pair formed by the first image and the second image is taken as the candidate image pair. At the same time, gradient information, edge information and color information in the image are important bases for reflecting the difference between the images. The lack of any one of them will cause the matching between the images to be incorrect.

[0130] S205. If the three-dimensional angular velocity and three-dimensional acceleration of the UAV at the time corresponding to the candidate image pair do not change, then the candidate image pair is the selected image pair.

[0131] After obtaining the candidate image pairs, the selection of the candidate image pairs is determined based on whether the three-dimensional angular velocity and three-dimensional acceleration of the UAV remain unchanged. If the three-dimensional angular velocity and three-dimensional acceleration of the UAV do not change within the time period from the previous moment to the current moment, then the candidate image pairs are selected. In this embodiment, an IMU (Inertial Measurement Unit) is used to measure and record the three-dimensional angular velocity and three-dimensional acceleration of the UAV.

[0132] When acquiring the first and second images, the IMU readings are read from the IMU readings when acquiring the previous frame image. The IMU readings are checked to see if there are any changes in the IMU readings during this period. If there are changes in the IMU readings, the second image in the candidate image pair may have a rolling shutter effect. The selection of image pairs continues until the IMU readings do not change. The candidate image pair is then the selected image pair, and the second image in the selected image pair is the optimal frame image.

[0133] S206. Correct the first depth image according to the grayscale gradient of the selected image pair.

[0134] The purpose of this step is to use the selected image obtained in S205 to align the pixel grayscale gradient in the second image obtained by the rolling shutter camera to correct the point cloud distribution in the corresponding area, correct any abnormal point clouds, and complete any missing point clouds.

[0135] Since the grayscale gradient of a building area generally does not change significantly except at the edges, and anomalies or missing points in the point cloud information usually exist at the edges, the grayscale gradient information of the grayscale image at the edges can be used to correct the points in the edge area effectively.

[0136] In this embodiment, the first depth image is a TOF (Time of Flight) point cloud image obtained using a high-frequency scanning strategy. At the edge of the building, the depth information of the point cloud near the edge is corrected based on the grayscale gradient of the grayscale image of the second image in the selected image pair. For edge regions with consistent grayscale gradients, the depth gradient of the point cloud should also be consistent. In this embodiment, points with consistent grayscale gradients but inconsistent depth values ​​are abnormal point clouds. Conversely, points with consistent grayscale gradients and consistent depth values ​​are grayscale consistent point clouds, i.e., normal point clouds.

[0137] Abnormal point clouds are corrected by using the depth values ​​of point clouds with the same gray level as their neighbors; missing point cloud data at the edges are filled by nearest neighbor interpolation, and finally the depth image of the building area is obtained by coordinate system transformation.

[0138] Example 3

[0139] This invention provides a point cloud correction system for UAV mapping based on artificial intelligence, characterized in that, as Figure 3 The module includes: an image acquisition module 301, a region of interest acquisition module 302, a difference calculation module 303, a candidate image pair acquisition module 304, a selected image pair acquisition module 305, and an image correction module 306.

[0140] The image acquisition module 301 is used to acquire a first image, a second image, and a first depth image.

[0141] The region of interest acquisition module 302 is used to acquire the region of interest in the overlapping region of the first image and the second image; the region of interest is the part to be corrected in the overlapping region.

[0142] The difference calculation module 303 is used to obtain the difference index between the region of interest in the first image and the region of interest in the second image.

[0143] The candidate image pair acquisition module 304 is used to determine candidate image pairs based on the difference index and the preset difference threshold.

[0144] The selected image pair acquisition module 305 is used to determine whether the three-dimensional angular velocity and three-dimensional acceleration corresponding to the time of the candidate image pair have not changed: if the determination result is yes, then the candidate image pair is the selected image pair.

[0145] The image correction module 306 is used to correct the first depth image according to the grayscale gradient of the selected image pair.

[0146] In summary, this invention proposes an AI-based point cloud correction method for UAV mapping. Images are acquired using a dual-camera system with both global shutter and rolling shutter on the UAV. Based on camera imaging rules, texture, edge, and color information from the optimal Region of Interest (ROI) are matched, and the image with the least fluctuation and optimal imaging is selected as the reference image. This filters out any rolling shutter effect that may occur in the acquired images. The Time-of-Flight (TOF) point cloud data is corrected based on the gradient information at the edges in the optimal frame image, thereby obtaining accurate point cloud data and generating a depth image that meets the mapping accuracy requirements.

[0147] In this invention, terms such as “including,” “comprising,” and “having” are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or” and are used interchangeably with them unless the context explicitly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0148] It should also be noted that in the methods and systems of the present invention, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0149] The above embodiments are merely illustrative examples for clear explanation and do not constitute a limitation on the scope of protection of this invention. Those skilled in the art can make other variations or modifications based on the above description, and it is neither necessary nor possible to exhaustively list all possible implementations. All designs that are the same as or similar to this invention fall within the scope of protection of this invention.

Claims

1. A point cloud correction method for UAV mapping based on artificial intelligence, characterized in that, include: Acquire the first image, the second image, and the first depth image; The first image was obtained using a global shutter camera deployed on the UAV, and the second image was obtained using a rolling shutter camera; the first depth image was a time-of-flight point cloud image obtained by high-frequency scanning using LiDAR. Obtain the region of interest in the overlapping region of the first image and the second image; the region of interest is the part of the overlapping region to be corrected. Obtain the difference index between the region of interest in the first image and the region of interest in the second image; The candidate image pairs are determined based on the difference index and the preset difference threshold; Determine whether the three-dimensional angular velocity and three-dimensional acceleration of the candidate image remain unchanged at the corresponding time: If the judgment result is yes, then the candidate image pair is the selected image pair; The first depth image is corrected based on the grayscale gradient of the selected image pair. At the edge of the building, the depth information of the point cloud near the edge is corrected based on the grayscale gradient of the grayscale image of the second image in the selected image pair. The depth gradient of the point cloud should also be consistent in edge regions with consistent grayscale gradients. Points with consistent gray-level gradients but inconsistent depth values ​​are abnormal point clouds; conversely, points with consistent gray-level gradients and consistent depth values ​​are point clouds with consistent gray levels, i.e., normal point clouds. Abnormal point clouds are corrected by using the depth values ​​of point clouds with the same gray level as their neighbors; missing point cloud data at the edges are filled by nearest neighbor interpolation, and finally the depth image of the building area is obtained by coordinate system transformation.

2. The point cloud correction method for UAV mapping based on artificial intelligence according to claim 1, characterized in that, The difference indicators include color difference, grayscale gradient difference, and edge difference.

3. The point cloud correction method for UAV mapping based on artificial intelligence according to claim 2, characterized in that, The steps for obtaining the color difference include: Divide the region of interest into different regions based on hue, within a preset distance threshold range of the region of interest. Calculate the hue, saturation, and brightness of each region, as well as the average hue, saturation, and brightness of all regions; The color difference is obtained based on the hue, saturation, and brightness of each region, as well as the average hue, saturation, and brightness of all regions.

4. The point cloud correction method for UAV mapping based on artificial intelligence according to claim 2, characterized in that, The steps for obtaining the grayscale gradient difference include: The first image and the second image are processed to obtain a first grayscale image and a second grayscale image, respectively; The gradient information of the first grayscale image is obtained based on the gradient values ​​of the pixels in the region of interest of the first grayscale image; The gradient information of the second grayscale image is obtained based on the gradient values ​​of the pixels in the region of interest in the second grayscale image; The grayscale gradient difference is obtained based on the gradient information of the first grayscale image and the gradient information of the second grayscale image.

5. The point cloud correction method for UAV mapping based on artificial intelligence according to claim 2, characterized in that, The steps for obtaining the edge difference include: Obtain a first image edge pixel set and a second image edge pixel set, wherein the first image edge pixel set is a set of pixels representing the edges of the region of interest in the first image, and the second image edge pixel set is a set of pixels representing the edges of the region of interest in the second image; The edge difference is obtained based on the first image edge pixel set and the second image edge pixel set.

6. The point cloud correction method for UAV mapping based on artificial intelligence according to claim 1, characterized in that, Before acquiring the first image, second image, and first depth image during the mapping process, the process also includes: The damping of the drone gimbal is adaptively adjusted according to the preset adjustment range.

7. A point cloud correction system for UAV mapping based on artificial intelligence, characterized in that, include: The system includes an image acquisition module, a region of interest acquisition module, a difference calculation module, a candidate image pair acquisition module, a selected image pair acquisition module, and an image correction module. The image acquisition module is used to acquire a first image, a second image, and a first depth image; the first image is acquired by a global shutter camera deployed on the UAV, the second image is acquired by a rolling shutter camera; the first depth image is a time-of-flight point cloud image obtained by high-frequency scanning using a lidar. The region of interest acquisition module is used to acquire the region of interest in the overlapping region of the first image and the second image; the region of interest is the part to be corrected in the overlapping region; The difference calculation module is used to obtain the difference index between the region of interest in the first image and the region of interest in the second image; The candidate image pair acquisition module is used to determine candidate image pairs based on the difference index and a preset difference threshold. The selected image pair acquisition module is used to determine whether the three-dimensional angular velocity and three-dimensional acceleration at the time corresponding to the candidate image pair have not changed: if the determination result is yes, then the candidate image pair is the selected image pair; The image correction module is used to correct the first depth image according to the gray level gradient of the selected image pair, and to correct the depth information of the point cloud near the edge of the building according to the gray level gradient of the gray level image of the second image in the selected image pair. The depth gradient of the point cloud should also be consistent in edge regions with consistent gray level gradients. Points with consistent gray-level gradients but inconsistent depth values ​​are abnormal point clouds; conversely, points with consistent gray-level gradients and consistent depth values ​​are point clouds with consistent gray levels, i.e., normal point clouds. Abnormal point clouds are corrected by using the depth values ​​of point clouds with the same gray level as their neighbors; missing point cloud data at the edges are filled by nearest neighbor interpolation, and finally the depth image of the building area is obtained by coordinate system transformation.

Images