An AI model construction system and method for drone image recognition

Abstract

The present invention relates to the field of drone control, and specifically to an AI model construction system and method for drone image recognition. The system includes: a positioning entity module, a corner training module, a distortion correction module, an image fusion module, and a color restoration module. The positioning entity module is used to set a positioning plumb bob and determine the field of view. The corner training module is used to train the AI model and input the original calibration parameters. The distortion correction module is used to establish a calibration model and perform preliminary image calibration. The image fusion module is used to capture a set of pictures and generate a fused image. The color restoration module is used to perform color correction on the original image. The present invention can simplify the AI training process, optimize the spatial accuracy of the image, improve the stability and usability of the image under different environmental conditions, eliminate the overlapping error between images, ensure the accuracy of the image data, and facilitate the subsequent analysis and application of drone images.

Description

Technical Field

[0001] The present invention relates to the field of UAV control, and specifically to an AI model construction system and method for UAV image recognition. Background Art

[0002] UAV images refer to image or video data captured from the air by a camera or other sensors carried by an unmanned aerial vehicle (UAV), which can be used for functions such as agricultural land analysis, environmental monitoring, real-time warning, etc. To accurately capture images, the UAV must hover above the target and capture images downward through a camera. During this process, devices such as gyroscopes are required for shooting calibration.

[0003] Since the UAV operates at high altitude, the pictures taken by the digital camera carried on it are prone to non-linear optical distortion at high altitude, causing the position of some pixel points to shift, resulting in problems such as imaging distortion and fisheye effect, making the image unable to well reflect the actual state of the object.

[0004] One of the means to solve this problem is to construct an AI model for image restoration. However, since the only source of information obtained by the high-altitude UAV is the camera itself, there is a lack of objective positioning points during the construction of the AI model, causing difficulties in model construction.

[0005] In addition, due to shooting problems such as exposure and focal length, the color of UAV images will also have offsets. The color restoration of pictures requires multiple focal length adjustments and aperture adjustments, which affect the shooting efficiency of the UAV and cannot well reflect the true color of the image. Summary of the Invention

[0006] The purpose of the present invention is to provide an AI model construction system and method for UAV image recognition to solve the problems raised in the above background art.

[0007] To solve the above technical problems, the present invention provides the following technical solution: An AI model construction system for UAV image recognition, including: a positioning entity module, a corner point training module, a distortion correction module, an image fusion module, and a color restoration module;

[0008] The positioning entity module is used to fixedly install a retractable positioning plumb on the UAV. Before starting shooting, the UAV is made to hover above the shooting object and the plumb is released. The camera captures an image and records the position of the plumb in the captured image. The visual field range of the picture is determined based on the flight altitude of the UAV, the hoisting length of the plumb, and the position of the plumb in the image;

[0009] The corner training module uses the pixel where the plumb center is located as the corner, extracts the pixels within a fixed range outside the corner, making the extracted pixel range larger than the plumb range. After smoothing the pixels outside the plumb range by the mean filtering method, they are stored in the training library together with the plumb pixels. The AI model learns the content in the training library, and takes the original data of the plumb, the field of view of the image, and the flight altitude of the drone as external parameters and inputs them into the AI model. The trained AI model outputs the principal point offset value of the camera, and the offset value is entered into the database as the original calibration parameter;

[0010] The distortion correction module is used to take another picture with the camera after the plumb is retracted. It uses a weighted algorithm to identify the corners in the image. Taking the corners as the center, it introduces radial distortion and tangential distortion into the bundle adjustment of the regional network, establishes a distortion correction mathematical model, calculates the adjustment value of the corners using the distortion correction mathematical model, establishes a calibration model according to the adjusted corners, and maps the transitional pixels between the corners in the image into the calibration model. After mapping, the preliminary calibration of the image is completed;

[0011] The image fusion module is used to control the drone to fly in a preset direction at the same altitude and continuously take pictures to generate an image set until the images taken in the image set contain all the features of the original images. It uses the trained AI model to fuse the image set to obtain a fused image, and uses the nearest neighbor method to determine the position of the original images in the fused image;

[0012] The color restoration module is used to calculate the contrast between the original image and the edge image in different regions, calculate the adjustment value according to the contrast of adjacent regions, and use the adjustment value as a parameter to adjust the color gradient vector between the pixels in the original image, so as to perform color correction on the original image and output the image after shape calibration and color correction processing.

[0013] Further, the positioning entity module includes: a plumb hoisting unit and a camera unit;

[0014] The plumb hoisting unit is used to set a plumb and a hoisting device at a fixed position under the drone camera, so that the plumb can be raised or lowered according to the control instruction, and the camera is not blocked when the plumb is raised;

[0015] The camera unit is used to take pictures of the scenery below the drone to obtain the original image and transmit the original image to the data processing device.

[0016] Further, the corner training module includes: an image processing unit, an AI modeling unit, and a parameter input unit;

[0017] The image processing unit is used for pixel recognition, pixel extraction, and mean filtering and smoothing of pixels to remove the noise in the image;

[0018] The AI modeling unit is used to introduce an AI model to learn the original image and provide external parameters. The AI models include: SSD model, DCGAN model, Mask R-CNN model, and SimCLR self-supervised learning model;

[0019] The parameter input unit is used to calculate the principal point offset value of the camera, generate and input image calibration parameters.

[0020] Further, the distortion correction module includes: a feature extraction unit, a model adjustment unit, and a pixel mapping unit;

[0021] The feature extraction unit is used to identify corner points in the image using a weighted algorithm. The weighted algorithms include: SUSAN feature extraction algorithm, Harris feature extraction algorithm, and SIFT transformation algorithm;

[0022] The model adjustment unit is used to construct a bundle adjustment area network centered on the corner points, calculate the calibration amount, and generate a calibration model;

[0023] The pixel mapping unit is used to map non-corner pixels in the original image to the calibration model to correct the image deviation.

[0024] Further, the image fusion module includes: an image storage unit and an intelligent fusion unit;

[0025] The image storage unit is used to control the drone to take images, store all the captured images, and analyze the coverage of the images;

[0026] The intelligent fusion unit is used to identify and stitch all the images in the image set to form a fused image with a higher number of pixel points than the original image.

[0027] Further, the color restoration module includes: a superposition comparison unit and a gradient restoration unit;

[0028] The superposition comparison unit is used to divide the pixel regions of the fused image and the original image and calculate the contrast of each region image;

[0029] The gradient restoration unit is used to calculate the contrast adjustment amount, adjust the pixel colors in the original image, and output the restored image.

[0030] An AI model construction method for drone image recognition includes the following steps:

[0031] Step S1. Set a plumb bob and a hoisting device under the drone camera, make the drone hover above the shooting object and release the plumb bob, take an image of the scene below the drone to obtain a calibration image, and determine the shaking parameters and the maximum field of view range of the drone according to the positions of the plumb bobs in adjacent calibration images and the drone height;

[0032] Step S2. Using the pixel where the plumb center is located as a corner point, extract the pixels outside the corner point. After gray-scale smoothing, store them in the training library. Using the training library as the internal source data and the plumb data, the UAV height, and the field of view range as the external source parameters, train the AI model so that the AI model outputs the radial deformation and tangential deformation of the image.

[0033] Step S3. Retract the plumb. The camera takes the original image. Use the weighted algorithm to identify the corner points in the original image. Construct a bundle adjustment area network with the corner points as the center, calculate the calibration amount and generate a calibration model. Provide the non-corner point pixels in the original image to the calibration model for adjustment to correct the non-linear distortion of the digital camera lens when the UAV flight is unstable.

[0034] Step S4. Command the UAV to fly in a preset direction and take an image set until the image set contains all the features of the original images. Identify and stitch all the images in the image set to obtain a fused image.

[0035] Step S5. Calculate the contrast of each image area in the fused image and the original images, adjust the contrast between the pixels of each area in the original image to make it consistent with the fused image, and repair the lens color distortion caused by water droplets or clouds.

[0036] Further, step S1 includes:

[0037] Step S11. Set a plumb and a hoisting device below the UAV camera so that the plumb can be raised or lowered according to the control command, and ensure that the plumb does not block the camera when it is raised or completely lowered. The plumb has a regular circular or elliptical upper surface and lower surface. The upper surface is made of a reflective material or an absorbent material, and the length of the hoisting line is greater than the vertical distance between the plumb and the camera.

[0038] Step S12. The UAV reaches above the shooting point and hovers at the shooting point at a fixed height. After completely releasing the plumb, take a calibration image, obtain the upper surface projection of the plumb in the calibration image, continuously take images, determine the swaying direction and amplitude of the UAV according to the change of the plumb center point in adjacent images, and input the UAV swaying parameters into the digital camera to obtain the non-orthogonality distortion coefficient and the non-square ratio coefficient.

[0039] Step S13. Calculate the field of view range. The calculation method is:

[0040] ;

[0041] where α represents the angle between the image edge and the image central axis, D α represents the widest field of view of the image in the α direction, S 1 and S 2respectively represent the area of the upper surface of the plumb bob on the calibration image and the actual one, P represents the proportion of the area of the upper surface of the plumb bob in the calibration image to the overall image, H 1 and H 2 respectively represent the length of the lifting line of the plumb bob and the flight height of the drone, and cosθ represents the focal point offset angle of the upper surface of the plumb bob in the α direction.

[0042] Furthermore, step S2 includes:

[0043] Step S21. Identify the center point of the upper surface of the plumb bob. Taking the center point as a corner point, extract the pixels within the range of n·n outside the corner point, where n is the side length of the pixel extraction area, and ensure that the extracted pixel range is larger than the range of the plumb bob. Smooth the pixels outside the range of the plumb bob by the mean filtering method to correct the pixel color.

[0044] Step S22. Store the extracted pixels in the training library, introduce an AI model to learn the data in the training library, and input the plumb bob data, drone height, and field of view range as external parameters into the AI model for combined training to obtain a trained AI model. The selected AI models include: SSD model, DCGAN model, Mask R-CNN model, and SimCLR self-supervised learning model;

[0045] Step S23. Let the trained AI model analyze the offset value of the principal point of the camera and record the offset value as the original calibration parameter in the database.

[0046] Furthermore, step S3 includes:

[0047] Step S31. After the plumb bob is recovered, the camera takes a new image and stores it as the original image. Use a weighted algorithm to identify the corner points in the original image. The corner points represent the pixels in the image with a color or texture difference greater than the threshold from adjacent pixel points. The weighted algorithms include: SUSAN feature extraction algorithm, Harris feature extraction algorithm, and SIFT transformation algorithm;

[0048] Step S32. Construct a bundle adjustment area network with the corner points as the center, calculate the calibration quantity, and generate a calibration model. The calculation method of the calibration quantity is:

[0049] ;

[0050] where xe and ye respectively represent the horizontal and vertical calibration quantities, x0 and y0 are the horizontal and vertical principal point offset values respectively, Dx and Dy respectively represent the widest fields of view in the horizontal and vertical directions, x and y represent the pixel point coordinates, a represents the non-square ratio coefficient, and b represents the non-orthogonal distortion coefficient, which are obtained from the camera parameters;

[0051] Step S33. Adjust each corner point according to the calibration amount, generate a calibration model, map the non-corner pixels in the original image to the calibration model, and output the calibration result.

[0052] Further, step S4 includes:

[0053] Step S41. Keep the drone at the same altitude, fly in each preset direction, and continuously capture the scenery below, so that the captured images contain at least a part of the original image, store the captured images and form an image set until the image set contains all the features of the original image;

[0054] Step S42. Use the trained AI model to fuse the image set to form a fused image with a pixel count higher than that of the original image, and use the nearest neighbor method to determine the position of the original image within the fused image.

[0055] Further, step S5 includes:

[0056] Step S51. Divide the part of the edge image that contains the original image into a preset number of sub-regions, and use image recognition software to calculate the average contrast within each sub-region;

[0057] Step S52. Calculate the average contrast of the adjacent regions of each sub-region, subtract the average contrast within the sub-region from the average contrast of the adjacent regions to obtain an adjustment value, process the color gradient of each pixel within the sub-region according to the adjustment value, complete color calibration, and output the image after shape calibration and color correction processing.

[0058] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0059] By setting a retractable positioning plumb on the drone, recording the position of the plumb in the captured image, thereby determining the field of view of the picture, and training the AI model with the plumb as the corner point, entering the principal point offset value into the original calibration parameters, adding objective positioning points, simplifying the AI training process, helping to adjust the position of the drone, and improving the spatial accuracy of the image.

[0060] By continuously capturing images, using a weighted algorithm to identify the corner points in the images, taking the corner points as the center, introducing radial distortion and tangential distortion into the bundle adjustment of the regional network, establishing a distortion correction mathematical model, and mapping each pixel in the image into the model, thereby completing the preliminary calibration of the image, which can help eliminate image distortion, ensure the consistency between different sensor data, facilitate subsequent analysis and application of the image, and improve the stability and usability of the image under different environmental conditions.

[0061] The present invention constructs an image set by taking images at different positions, so that the image set contains the features of all the original images. The pixel gradient vectors in the original images are adjusted according to the contrast, the images are color-corrected, and the overlapping errors between the images are eliminated, making them closer to the real scene. This can improve the spatial accuracy of the images and ensure the accuracy of the image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0063] Figure 1 is a schematic structural diagram of an AI model construction system for drone image recognition according to the present invention;

[0064] Figure 2 is a schematic diagram of the steps of an AI model construction method for drone image recognition according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0066] Please refer to Figure 1 , the present invention provides a technical solution: an AI model construction system for drone image recognition, including: a positioning entity module, a corner point training module, a distortion correction module, an image fusion module, and a color restoration module;

[0067] The positioning entity module is used to set a retractable positioning plumb at a fixed position on the drone. Before starting to take pictures, the drone is made to hover above the object to be photographed and the plumb is released. The camera takes pictures and records the position of the plumb in the taken pictures. The viewing range of the picture is determined according to the flight height of the drone, the hoisting length of the plumb, and the position of the plumb in the image;

[0068] The positioning entity module includes: a plumb hoisting unit and a camera unit;

[0069] The plumb hoisting unit is used to set a plumb and a hoisting device at a fixed position under the drone camera, so that the plumb can be raised or lowered according to a control command, and the camera is not blocked when the plumb is raised;

[0070] The camera unit is used to take pictures of the scenery below the drone to obtain the original images and transmit the original images to the data processing device.

[0071] The corner training module is used to take the pixel where the plumb center is located as the corner, extract the pixels within a fixed range outside the corner, so that the extracted pixel range is larger than the plumb range. After smoothing the pixels outside the plumb range by the mean filter method, the pixels are stored in the training library together with the plumb pixels. The AI model learns the content in the training library, and takes the original data of the plumb, the field of view range of the image, and the flight altitude of the drone as external parameters and inputs them into the AI model. The trained AI model outputs the principal point offset value of the camera, and enters the offset value into the database as the original calibration parameter;

[0072] The corner training module includes: an image processing unit, an AI modeling unit, and a parameter entry unit;

[0073] The image processing unit is used for pixel recognition, pixel extraction, and mean filter smoothing of pixels to remove noise in the image;

[0074] The AI modeling unit is used to introduce the AI model to learn the original image and provide external parameters. The AI model includes: SSD model, DCGAN model, Mask R-CNN model, and SimCLR self-supervised learning model;

[0075] The parameter entry unit is used to calculate the principal point offset value of the camera, generate and enter the image calibration parameter.

[0076] The distortion correction module is used to take another picture with the camera after the plumb is retracted. The weighted algorithm is used to identify the corners in the image. Taking the corners as the center, the radial distortion and tangential distortion are introduced into the bundle adjustment of the regional network, and a distortion correction mathematical model is established. The adjustment value of the corners is calculated by using the distortion correction mathematical model, and a calibration model is established according to the adjusted corners. The transitional pixels between the corners in the image are mapped into the calibration model, and the image is preliminarily calibrated after mapping;

[0077] The distortion correction module includes: a feature extraction unit, a model adjustment unit, and a pixel mapping unit;

[0078] The feature extraction unit is used to identify the corners in the image by using the weighted algorithm. The weighted algorithm includes: SUSAN feature extraction algorithm, Harris feature extraction algorithm, and SIFT transformation algorithm;

[0079] The model adjustment unit is used to construct a bundle adjustment regional network with the corners as the center, calculate the calibration amount and generate a calibration model;

[0080] The pixel mapping unit is used to map the non-corner pixels in the original image into the calibration model to correct the picture deviation.

[0081] The image fusion module is used to control the drone to fly in a preset direction at the same altitude and continuously capture images to generate an image set until the images captured in the image set contain the features of all the original images. Then, it uses the trained AI model to fuse the image set to obtain a fused image, and determines the positions of the original images in the fused image by using the nearest neighbor method.

[0082] The image fusion module includes: an image storage unit and an intelligent fusion unit;

[0083] The image storage unit is used to control the drone to capture images, store all the captured images, and analyze the coverage range of the images;

[0084] The intelligent fusion unit is used to identify and splice all the images in the image set to form a fused image with a higher number of pixel points than the original images.

[0085] The color restoration module is used to calculate the contrast between the original image and the edge image in different regions, calculate the adjustment value according to the contrast of adjacent regions, and use the adjustment value as a parameter to adjust the color gradient vectors between the pixels in the original image, so as to perform color correction on the original image and output the image after shape calibration and color correction processing.

[0086] The color restoration module includes: a superposition comparison unit and a gradient restoration unit;

[0087] The superposition comparison unit is used to divide the pixel regions of the fused image and the original image and calculate the contrast of the images in each region;

[0088] The gradient restoration unit is used to calculate the contrast adjustment amount, adjust the pixel colors in the original image, and output the restored image.

[0089] As Figure 2 shown, an AI model construction method for drone image recognition includes the following steps:

[0090] Step S1. Set a plumb bob and a hoisting device at a fixed position under the drone camera. Before shooting, make the drone hover above the shooting object and release the plumb bob, and capture an image of the scene below the drone to obtain a calibration image. Determine the field of view of the camera according to the position of the plumb bob, the height of the drone, and the plumb bob data in the calibration image.

[0091] Step S1 includes:

[0092] Step S11. Set a plumb bob and a hoisting device below the drone camera, enabling the plumb bob to rise or fall according to control instructions, and ensuring that the plumb bob does not block the camera when it rises or fully descends. The plumb bob has upper and lower surfaces with regular circular or elliptical shapes. The upper surface is made of a reflective or light-absorbing material, and the length of the hoisting line is greater than the vertical distance between the plumb bob and the camera.

[0093] Step S12. The drone reaches above the shooting point and hovers at a fixed height at the shooting point. After fully releasing the plumb bob, it takes a calibration image, obtains the upper surface projection of the plumb bob in the calibration image, and calculates the field of view according to the position of the plumb bob, the height of the drone, and the plumb bob data. The plumb bob data includes the upper surface area of the plumb bob, the length of the hoisting line, and the upper surface focus. The calculation method is as follows:

[0094] ;

[0095] where α represents the angle between the image edge and the image central axis, D α represents the widest field of view of the image in the α direction, S 1 and S 2 respectively represent the upper surface area of the plumb bob in the calibration image and the actual upper surface area of the plumb bob, P represents the proportion of the upper surface area of the plumb bob in the calibration image to the overall image, H 1 and H 2 respectively represent the length of the hoisting line of the plumb bob and the flight height of the drone, and cosθ represents the focus offset angle of the upper surface of the plumb bob in the α direction.

[0096] Step S2. Take the pixel where the center of the plumb bob is located as a corner point, extract the pixels within a fixed range outside the corner point, perform gray-scale smoothing by the mean filtering method and store them in the training library. Use the training library as the internal source data, and use the plumb bob data, the drone height, and the field of view as external source parameters to train the AI model, so that the AI model outputs the calibration parameters of the image.

[0097] Step S2 includes:

[0098] Step S21. Identify the center point of the upper surface of the plumb bob. Take the center point as a corner point and extract the pixels within an n·n range outside the corner point, where n is the side length of the pixel extraction area, and ensure that the extracted pixel range is larger than the plumb bob range. Smooth the pixels outside the plumb bob range by the mean filtering method to correct the pixel color.

[0099] Step S22. Store the extracted pixels in the training library, introduce the AI model to learn the data in the training library, and use the plumb bob data, the drone height, and the field of view as external source parameters to input into the AI model for combined training to obtain the trained AI model. The selected AI models include: SSD model, DCGAN model, Mask R-CNN model, and SimCLR self-supervised learning model.

[0100] Step S23. Let the trained AI model analyze the principal point offset value of the camera, and input the offset value into the database as the original calibration parameter.

[0101] Step S3. Retract the plumb bob, and the camera captures the original image. Use the weighted algorithm to identify the corner points in the original image, construct a bundle adjustment network with the corner points as the center, calculate the calibration amount and generate a calibration model, and map the non-corner pixels in the original image to the calibration model to correct the shape deviation.

[0102] Step S3 includes:

[0103] Step S31. After the plumb bob is retracted, the camera captures a new image and stores it as the original image. Use the weighted algorithm to identify the corner points in the original image. The corner points represent the pixels in the image whose color or texture difference from adjacent pixel points is greater than the threshold. The weighted algorithm includes: SUSAN feature extraction algorithm, Harris feature extraction algorithm, and SIFT transformation algorithm.

[0104] Step S32. Construct a bundle adjustment network with the corner points as the center, calculate the calibration amount and generate a calibration model. The calculation method of the calibration amount is:

[0105] ;

[0106] where xe and ye represent the horizontal and vertical calibration amounts respectively, x0 and y0 are the horizontal and vertical principal point offset values respectively, Dx and Dy represent the widest fields of view in the horizontal and vertical directions respectively, x and y represent the pixel coordinates, a represents the non-square ratio coefficient, and b represents the non-orthogonal distortion coefficient, which are obtained from the camera parameters.

[0107] Step S33. Adjust each corner point according to the calibration amount, generate a calibration model, map the non-corner pixels in the original image to the calibration model, and output the calibration result.

[0108] Step S4. Let the drone fly in all directions and capture an image set until the image set contains all the features of the original images, identify and stitch all the images in the image set to obtain a fused image.

[0109] Step S4 includes:

[0110] Step S41. Keep the drone at the same altitude, fly in all preset directions, and continuously capture the scenery below, so that the captured images contain at least a part of the original images, store the captured images and form an image set until the image set contains all the features of the original images.

[0111] Step S42. Use the trained AI model to fuse the image set to form a fused image with a higher number of pixels than the original image, and use the nearest neighbor method to determine the position of the original image within the fused image.

[0112] Step S5. Calculate the contrast of each image region in the fused image and the original image, calculate the difference between the contrast of each region and that of all adjacent regions to obtain an adjustment value, adjust the color gradient between pixels in each region of the original image, and output the original image after color restoration is completed.

[0113] Step S5 includes:

[0114] Step S51. Divide the part of the edge image containing the original image into a preset number of sub-regions, and use image recognition software to calculate the average contrast within each sub-region;

[0115] Step S52. Calculate the average contrast of adjacent regions of each sub-region, subtract the average contrast within the sub-region from the average contrast of the adjacent regions to obtain an adjustment value, process the color gradient of each pixel point within the sub-region according to the adjustment value, complete color calibration, and output the image after shape calibration and color correction processing.

[0116] Example: Release a circular plumb bob with a radius of 5 cm from the drone. The length of the hoisting line is 10 m, and the hovering height of the drone is 100 m. In the calibrated image, the radius of the plumb bob is 1 cm, accounting for 20% of the overall image area. The offset angles in the horizontal and vertical directions are 2° and 3° respectively, and the picture resolution is 1000*1000. Then the field of view is approximately 49 cm 2 , after recording the data, retract the plumb bob. Calculate that the horizontal principal point offset value per pixel is 0.007sin2° cm, and the vertical principal point offset value per pixel is 0.007sin3° cm;

[0117] The drone takes pictures. Given that the non-orthogonal distortion coefficient of the digital camera caused by the shaking of the drone is 0.05 and the non-square ratio coefficient is 0.01, then take the pixel at the center of the image as the corner point and correct the non-linear distortion in the original image. Taking the pixel point (-5,10) as an example, the horizontal and vertical calibration amounts in the lens are -0.014 cm and 0.026 cm respectively;

[0118] Keep the drone at the same height for extended shooting, fuse the captured images, and use the fused image for color restoration. Taking the pixel points (-5,10) and (-5,9) as an example, the contrast between the pixel points (-5,10) and (-5,9) in the fused image is -0.01, and the contrast in the captured image is -0.008. Then adjust the color gradient between the pixel points by -0.002, process all pixel points, and output the image to complete the distortion correction and color correction of the camera.

[0119] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device.

[0120] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for constructing an AI model for drone image recognition, characterized in that: The method comprises the following steps: Step S1. A plumb bob and a hoisting device are set under the drone camera, the drone is hovered above the object and the plumb bob is released, an image of the scene below the drone is captured, a calibration image is obtained, and the sway parameter and maximum field of view of the drone are determined according to the plumb bob position in the adjacent calibration image and the drone height; Step S2. Take the pixel at the center of the plumb bob as the corner point, extract the pixels outside the corner point, smooth the grayscale and store them in the training library, use the training library as the endogenous data, and the plumb bob data, the height of the drone and the field of view as the exogenous parameters to train the AI ​​model, so that the AI ​​model outputs the radial deformation and tangential deformation of the image; Step S3. put away the plumb bob, shoot the original image with the camera, construct the bundle method regional block adjustment with the center of the image as the corner point, calculate the calibration amount and generate the calibration model, provide the non-corner pixels in the original image to the calibration model for adjustment, and correct the nonlinear distortion of the digital camera lens when the drone is unstable in flight; Step S4. Make the drone fly in a preset direction and shoot an image set until the image set contains the features of all the original images, identify and stitch all the images in the image set to obtain a fused image; Step S5. Calculate the contrast between the fused image and each image area in the original image, adjust the contrast between pixels in each area of ​​the original image to make it consistent with the fused image, and repair the lens color distortion caused by water droplets or fog.

2. The AI ​​model construction method for drone image recognition according to claim 1, characterized in that: Step S1 includes: Step S11. A plumb bob and a hoisting device are arranged below the camera of the drone, so that the plumb bob is raised or lowered according to the control command, and the camera is not blocked when the plumb bob is raised or completely lowered, the plumb bob has an upper surface and a lower surface in a regular circular or elliptical shape, the upper surface is made of a reflective material or a light-absorbing material, and the length of the hoisting line is greater than the vertical distance between the plumb bob and the camera; Step S12. The UAV arrives above the shooting point and hovers at a fixed height at the shooting point. After the plumb bob is completely released, a calibration image is taken, and the upper surface projection of the plumb bob in the calibration image is obtained. Images are taken continuously, and the shaking direction and amplitude of the UAV are determined according to the change of the plumb bob center point in adjacent images. The UAV shaking parameters are input into a digital camera to obtain non-orthogonal distortion coefficients and non-square ratio coefficients. Step S13: Calculate the visual field range by: ； Among them, α represents the angle between the edge of the image and the central axis of the image, D α represents the widest field of view of the image in the α direction, S1 and S2 represent the surface area of ​​the plumb bob in the calibration image and the actual plumb bob respectively, P represents the proportion of the plumb bob surface area in the calibration image to the overall image, H1 and H2 represent the length of the plumb bob’s hoisting line and the UAV’s flight altitude respectively, and cosθ represents the focus offset angle of the plumb bob’s surface in the α direction.

3. The AI ​​model construction method for drone image recognition according to claim 2 is characterized in that: Step S2 includes: Step S21. Identify the center point of the upper surface of the plumb bob, take the center point as the corner point, extract pixels within a range of n·n outside the corner point, where n is the side length of the extracted pixel area, and keep the extracted pixel range larger than the plumb bob range. Smooth the pixels beyond the plumb bob range by a mean filter method to correct the pixel color. Step S22. The extracted pixels are stored in the training library, the data in the AI ​​model learning training library is introduced, the plumb bob data, the height of the drone and the field of view are input as exogenous parameters into the AI ​​model for combined training, and the trained AI model is obtained. The selected AI models include: SSD model, DCGAN model, Mask R-CNN model and SimCLR self-supervised learning model; Step S23: Instruct the trained AI model to analyze the principal point offset value of the camera, and enter the offset value into the database as the original calibration parameter.

4. The AI ​​model construction method for drone image recognition according to claim 3 is characterized in that: Step S3 includes: Step S31. After the plumb bob is recovered, the camera retakes the image and stores it as the original image. A weighted algorithm is used to identify corner points in the original image. The corner points represent pixels in the image whose color or texture difference with adjacent pixels is greater than a threshold. The weighted algorithm includes: SUSAN feature extraction algorithm, Harris feature extraction algorithm and SIFT transformation algorithm. Step S32. Construct a bundle method block adjustment with the corner point as the center, calculate the calibration amount and generate a calibration model. The calibration amount calculation method is: ； Where xe and ye represent the horizontal and vertical calibration values, respectively; x0 and y0 represent the horizontal and vertical principal point offset values, respectively; Dx and Dy represent the widest field of view in the horizontal and vertical directions, respectively; x and y represent the pixel coordinates; a represents the non-square scale coefficient; and b represents the non-orthogonal distortion coefficient, which is obtained from the camera parameters; Step S33. Adjust each corner point according to the calibration amount, generate a calibration model, map the non-corner point pixels in the original image to the calibration model, and output the calibration result.

5. The AI ​​model construction method for drone image recognition according to claim 4 is characterized in that: Step S4 includes: Step S41. Keep the drone at the same altitude, fly in each preset direction, and continue to shoot the scene below, so that the captured image contains at least a part of the original image, store the captured image and form an image set, until the image set contains all the features of the original image; Step S42. Using the trained AI model to fuse the image set, a fused image with a higher number of pixels than the original image is formed, and the position of the original image in the fused image is determined by using the nearest neighbor method; Step S5 includes: Step S51. Divide the portion of the edge image containing the original image into a preset number of sub-regions, and use image recognition software to calculate the average contrast in each sub-region; Step S52. Calculate the average contrast of the adjacent areas of each sub-area, subtract the average contrast in the sub-area from the average contrast of the adjacent areas to obtain an adjustment value, process the color gradient of each pixel in the sub-area according to the adjustment value, complete color calibration, and output the image after shape calibration and color correction.

6. An AI model building system for drone image recognition, characterized in that: The system includes the following modules: entity positioning module, corner point training module, distortion correction module, image fusion module and color restoration module; The positioning entity module is used to set a retractable positioning plumb bob at a fixed position on the drone, and before starting to shoot, the drone is hovered above the object and the plumb bob is released. The camera shoots an image and records the position of the plumb bob in the captured image. The field of view of the image is determined according to the flight altitude of the drone, the plumb bob hoisting length and the position of the plumb bob in the image. The corner point training module is used to take the pixel at the center of the plumb bob as the corner point, extract pixels within a fixed range outside the corner point, make the extracted pixel range larger than the plumb bob range, smooth the pixels beyond the plumb bob range by a mean filter method, and store them in a training library together with the plumb bob pixels. The AI ​​model learns the content in the training library, and inputs the original data of the plumb bob, the field of view of the picture, and the flight altitude of the drone as exogenous parameters into the AI ​​model. The trained AI model outputs the image principal point offset value of the camera, and enters the offset value into the database as the original calibration parameter; The distortion correction module is used for, after the plumb bob is put away, the camera takes an image again, adopts a weighted algorithm to identify the corner point in the image, takes the corner point as the center, introduces radial deformation and tangential deformation into the bundle method regional block adjustment, establishes a distortion correction mathematical model, calculates the adjustment value of the corner point by using the distortion correction mathematical model, establishes a calibration model according to the adjusted corner point, maps the transition pixels between the corner points in the image into the calibration model, and completes the preliminary calibration of the image after mapping; The image fusion module is used to control the drone to fly in a preset direction at the same altitude and continuously take images to generate an image set until the images taken in the image set contain the features of all the original images, fuse the image set using the trained AI model to obtain a fused image, and use the nearest neighbor method to determine the position of the original image in the fused image; The color restoration module is used to calculate the contrast between the original image and the edge image in different areas, calculate the adjustment value according to the contrast of adjacent areas, and use the adjustment value as a parameter to adjust the color gradient vector between pixels in the original image, thereby performing color correction on the original image and outputting an image after shape calibration and color correction processing.

7. The AI ​​model building system for drone image recognition according to claim 6, characterized in that: The positioning entity module includes: a plumb bob hoisting unit and a camera unit; The plumb bob hoisting unit is used to set a plumb bob and a hoisting device at a fixed position under the camera of the drone, so that the plumb bob is raised or lowered according to the control command, and the camera is kept from being blocked when the plumb bob is raised; The camera unit is used to capture images of the scenery below the drone, obtain original images, and transmit the original images to a data processing device.

8. The AI ​​model building system for drone image recognition according to claim 7, characterized in that: The corner point training module includes: an image processing unit, an AI modeling unit and a parameter input unit; The image processing unit is used for pixel recognition, pixel extraction and pixel mean filtering and smoothing to remove noise in the image; The AI ​​modeling unit is used to introduce an AI model to learn the original image and provide exogenous parameters, and the AI ​​model includes: an SSD model, a DCGAN model, a Mask R-CNN model and a SimCLR self-supervised learning model; The parameter input unit is used to calculate the principal point offset value of the camera, and generate and input the image calibration parameters.

9. The AI ​​model building system for drone image recognition according to claim 8, characterized in that: The distortion correction module includes: a feature extraction unit, a model adjustment unit and a pixel mapping unit; The feature extraction unit is used to identify corner points in an image by using a weighted algorithm, wherein the weighted algorithm includes: a SUSAN feature extraction algorithm, a Harris feature extraction algorithm, and a SIFT transformation algorithm; The model adjustment unit is used to construct a beam method regional network with the corner point as the center, calculate the calibration amount and generate a calibration model; The pixel mapping unit is used to map non-corner pixels in the original image to the calibration model to correct image deviation.

10. The AI ​​model building system for drone image recognition according to claim 9, characterized in that: The image fusion module includes: an image storage unit and an intelligent fusion unit; The image storage unit is used to control the drone to capture images, store all captured images, and analyze the coverage of the images; The intelligent fusion unit is used to identify and stitch all images in the image set to form a fused image with a higher number of pixels than the original image; The color restoration module includes: a superposition contrast unit and a gradient restoration unit; The superposition contrast unit is used to divide the pixel areas of the fused image and the original image, and calculate the contrast of the image in each area; The gradient restoration unit is used to calculate the contrast adjustment amount, adjust the pixel colors in the original image, and output the restored image.

Images