Unmanned aerial vehicle view distortion correction method based on high-order polynomial and deep network fusion

Abstract

The invention relates to the field of image processing, in particular to a high-order polynomial and deep network fused unmanned aerial vehicle view distortion correction method, which comprises the following steps of: analyzing a distorted image by using a geometric encoder and a space encoder respectively to obtain a radial distortion parameter and a tangential distortion parameter; respectively inputting radial distortion parameters and tangential distortion parameters output by the two encoders to a high-order polynomial model to generate a specific correction flow, and calculating a confidence coefficient corresponding to each pixel of the correction flow through a full connection layer; and fusing the correction streams output by the two encoders through an exponential decay function based on confidence to obtain a final correction stream, and applying the final correction stream to the distorted image to obtain a corrected image. According to the method, distortion recovery can be stably carried out on the image with radial or tangential distortion under the condition of not depending on special hardware such as a calibration board.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for correcting unmanned aerial vehicle (UAV) view distortion by fusing a high-order polynomial with a deep network. Background Art

[0002] As an important tool for data collection, drones are widely used in scenarios such as traffic flow monitoring, disaster assessment, and urban planning. However, due to the flight altitude and viewing angle changes of drones, as well as the wide-angle or telephoto design of the lens, radial or tangential distortion is easily introduced, especially in large areas or complex scenes. The distortion problem is more significant. This image distortion will affect the accuracy and reliability of subsequent visual analysis tasks, such as target recognition and trajectory tracking. The present invention is committed to proposing a correction method for restoring distorted images caused by the lens, generating a confidence-based correction flow, effectively solving the image distortion problem, and improving the processing accuracy and robustness of traffic monitoring data. Summary of the invention

[0003] The present application provides a drone view distortion correction method that integrates high-order polynomials and deep networks, which can stably restore the distortion of images with radial or tangential distortion without relying on special hardware such as calibration plates.

[0004] To solve the above problems, this application provides the following solutions: R1. Use geometric encoders respectively and spatial encoder Distorted image Analyze and find out the radial distortion parameters and and tangential distortion parameters and ,in and It is a geometric encoder The extracted radial distortion parameters and tangential distortion parameters are and It is a geometric encoder The extracted radial distortion parameters and tangential distortion parameters; R2. Input the radial distortion parameters and tangential distortion parameters output by the two encoders into the high-order polynomial model to generate a specific correction flow and And the confidence of the correction flow corresponding to each pixel is calculated through the fully connected layer and ; R3. The correction streams output by the two encoders are fused through an exponential decay function based on confidence to obtain the final correction stream , and the correction flow Applied to distorted images The corrected image is obtained .

[0005] Wherein, the geometric encoder The processing process is divided into two stages: edge feature extraction and geometric feature encoding; The geometric encoder Edge feature extraction refers to using the Canny edge detector to extract the edge features of the input image. Processing to generate edge map ; The geometric encoder The geometric feature encoding refers to the use of ResNet50 to encode the edge graph Encode and generate radial distortion parameters and tangential distortion parameters ; The spatial encoder The processing process is realized by encoding and decoding the input image The encoding of The spatial encoder The encoding process refers to using MobileNet to encode the input image Extract The spatial encoding feature map of the layer ,in For passing Multi-scale feature maps generated by layer convolutional neural networks, feature maps With different resolutions and feature dimensions ; The spatial encoder The decoding process refers to gradually restoring the spatial resolution of the feature map by upsampling layer by layer through transposed convolution, introducing skip connections, and combining with the spatial encoder The low-level detail features of the encoding process and the high-level semantic features of the decoder are combined through the fully connected layer Spatial decoding feature map of the layer Calculate the radial distortion parameters and tangential distortion parameters ; The spatial encoder The decoding process of is to gradually restore the feature map by upsampling layer by layer through transposed convolution. The process is described as: Where ConvTranspose represents the transposed convolution operation. For the Spatial decoding feature map of the layer After The layer transposed convolutional neural network and passed Multi-scale feature maps generated by layer convolutional neural network The spliced ​​generated The spatial decoded feature map of the layer; The correction flow represents a mapping from the image coordinate system to the correction coordinate system, where is the pixel coordinate in the distorted image. After correction, the pixel will be mapped to the new position ,Right now: , , in, is the corrected pixel coordinate, and Represents the pixel width and pixel height of the image respectively. The radial distortion parameters and tangential distortion parameters output by the input encoder are used to generate a specific correction flow using a high-order polynomial model. The process is as follows: Each pixel coordinate on , according to the radial distortion parameter Calculate the radial distortion factor , and then using the radial distortion factor , tangential distortion parameters and pixel coordinates , calculate the new position of the pixel in the corrected coordinate system ; The radial distortion factor The calculation method is as follows: in, Indicates that it contains A set of radial distortion parameters, Represents pixels To the center of the image The square distance of The pixel coordinates New position in the corrected coordinate system The calculation method is: The confidence corresponding to each pixel of the correction flow is calculated through the fully connected layer The specific steps are: in, and To learn the parameters, The Sigmoid activation function ensures that the confidence level is between 0 and 1. The confidence-based exponential decay function is used to fuse the final correction flow The process is as follows: in, Reason and The confidence level of the calculated horizontal axis, Reason and The confidence level of the calculated ordinate, yes and The calculated correction flow of the abscissa, yes and The calculated correction flow of the ordinate, Reason and The confidence level of the calculated horizontal axis, Reason and The confidence level of the calculated ordinate, yes and The calculated correction flow of the abscissa, yes and The calculated correction flow of the ordinate, is the attenuation coefficient, which is used to control the degree of confidence attenuation. The will correct flow Applied to distorted images The corrected image is obtained Refers to the correction flow New position mapped to the corrected coordinate system .

[0006] Compared with the prior art, the above technical solution provides a method for correcting drone view distortion by integrating high-order polynomials and deep networks, which has the following beneficial effects: 1. The geometric encoder and spatial encoder are used to extract the geometric features and spatial features of the input image respectively, so that the method can adapt to different scenes and various types of surveillance image distortion, especially in complex scenes with significant depth changes such as intersections and high-altitude surveillance, showing stronger robustness and adaptability; 2. The high-order polynomial model is introduced as prior knowledge, which effectively ensures the stability and accuracy of the distortion correction process. It can handle radial distortion and tangential distortion at the same time and meet the requirements of high-precision correction. 3. A confidence-based exponential decay function is introduced to dynamically weighted fuse the correction streams generated by the geometric encoder and the spatial encoder, thereby significantly improving the accuracy and stability of the correction streams and ensuring the accuracy and consistency of the final correction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 The present invention is a flowchart of a method for correcting drone view distortion by integrating high-order polynomials and deep networks in an embodiment of the present invention.

[0009] Figure 2 The distorted image in the present invention Holistic model framework for radial and tangential distortion correction. DETAILED DESCRIPTION

[0010] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0011] An embodiment of the present application provides a method for correcting drone view distortion by fusing high-order polynomials and deep networks. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0012] like Figure 1 As shown, an embodiment of a method for correcting drone view distortion by integrating a high-order polynomial and a deep network in an embodiment of the present application, the specific steps are as follows: R1. The embodiments of the present invention use geometric encoders respectively and spatial encoder Distorted image Analyze and find out the radial distortion parameters and and tangential distortion parameters and ,in and It is a geometric encoder The extracted radial distortion parameters and tangential distortion parameters are and It is a geometric encoder The extracted radial distortion parameters and tangential distortion parameters; R2. In the embodiment of the present invention, the radial distortion parameters and tangential distortion parameters output by the two encoders are respectively input into the high-order polynomial model to generate a specific correction flow and And the confidence of the correction flow corresponding to each pixel is calculated through the fully connected layer and ; R3. In the embodiment of the present invention, the correction streams output by the two encoders are merged through an exponential decay function based on confidence to obtain the final correction stream , and the correction flow Applied to distorted images The corrected image is obtained .

[0013] Among them, Figure 2 As shown, the geometric encoder described in the embodiment of the present invention The processing process is divided into two stages: edge feature extraction and geometric feature encoding; The geometric encoder Edge feature extraction refers to using the Canny edge detector to extract the input image Processing to generate edge map ; The geometric encoder The geometric feature encoding refers to the use of ResNet50 to encode the edge graph Encode and generate radial distortion parameters and tangential distortion parameters ; The spatial encoder The processing process is realized by encoding and decoding the input image The encoding of The spatial encoder The encoding process refers to using MobileNet to encode the input image Extract The spatial encoding feature map of the layer ,in For passing Multi-scale feature maps generated by layer convolutional neural networks, feature maps With different resolutions and feature dimensions ; The spatial encoder The decoding process refers to gradually restoring the spatial resolution of the feature map by upsampling layer by layer through transposed convolution, introducing skip connections, and combining with the spatial encoder The low-level detail features of the encoding process and the high-level semantic features of the decoder are combined through the fully connected layer Spatial decoding feature map of the layer Calculate the radial distortion parameters and tangential distortion parameters ; The spatial encoder The decoding process of is to gradually restore the feature map by upsampling layer by layer through transposed convolution. The process is described as: Where ConvTranspose represents the transposed convolution operation. For the Spatial decoding feature map of the layer After The layer transposed convolutional neural network and passed Multi-scale feature maps generated by layer convolutional neural network The spliced ​​generated The spatial decoded feature map of the layer; The correction flow represents a mapping from the image coordinate system to the correction coordinate system, where is the pixel coordinate in the distorted image. After correction, the pixel will be mapped to the new position ,Right now: , , in, is the corrected pixel coordinate, and Represents the pixel width and pixel height of the image respectively. The radial distortion parameters and tangential distortion parameters output by the input encoder are used to generate a specific correction flow using a high-order polynomial model. The process is as follows: Each pixel coordinate on , according to the radial distortion parameter Calculate the radial distortion factor , and then using the radial distortion factor , tangential distortion parameters and pixel coordinates , calculate the new position of the pixel in the corrected coordinate system ; The radial distortion factor The calculation method is as follows: in, Indicates that it contains A set of radial distortion parameters, Represents pixels To the center of the image The square distance of The pixel coordinates New position in the corrected coordinate system The calculation method is: The confidence corresponding to each pixel of the correction flow is calculated through the fully connected layer The specific steps are: in, and To learn the parameters, The Sigmoid activation function ensures that the confidence level is between 0 and 1. The confidence-based exponential decay function is used to fuse the final correction flow The process is as follows: in, Reason and The confidence level of the calculated horizontal axis, Reason and The confidence level of the calculated ordinate, yes and The calculated correction flow of the abscissa, yes and The calculated correction flow of the ordinate, Reason and The confidence level of the calculated horizontal axis, Reason and The confidence level of the calculated ordinate, yes and The calculated correction flow of the abscissa, yes and The calculated correction flow of the ordinate, is the attenuation coefficient, which is used to control the degree of confidence attenuation. The will correct flow Applied to distorted images The corrected image is obtained Refers to the correction flow New position mapped to the corrected coordinate system .

[0014] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. For ordinary technicians in this technical field, after knowing the contents recorded in the present invention, they can make several equivalent changes and substitutions without departing from the principle of the present invention. These equivalent changes and substitutions should also be regarded as belonging to the protection scope of the present invention.

Claims

1. A drone view distortion correction method based on high-order polynomial and deep network fusion, characterized in that: The method for correcting the distortion of drone view by integrating high-order polynomial and deep network includes: Use the geometry encoder and spatial encoder Distorted image Analyze and find out the radial distortion parameters and and tangential distortion parameters and ,in and It is a geometric encoder The extracted radial distortion parameters and tangential distortion parameters are and It is a geometric encoder The extracted radial distortion parameters and tangential distortion parameters; Input the radial distortion parameters and tangential distortion parameters output by the two encoders into the high-order polynomial model to generate a specific correction flow and And the confidence of the correction flow corresponding to each pixel is calculated through the fully connected layer and ; The correction streams output by the two encoders are fused through an exponential decay function based on confidence to obtain the final correction stream , and the correction flow Applied to distorted images The corrected image is obtained .

2. The method for correcting drone view distortion by integrating high-order polynomial and deep network according to claim 1, Features: Wherein, the geometric encoder The processing process is divided into two stages: edge feature extraction and geometric feature encoding; The geometric encoder Edge feature extraction refers to using the Canny edge detector to extract the edge features of the input image. Processing to generate edge map ; The geometric encoder The geometric feature encoding refers to the use of ResNet50 to encode the edge graph Encode and generate radial distortion parameters and tangential distortion parameters ; The spatial encoder The processing process is realized by encoding and decoding the input image The encoding of The spatial encoder The encoding process refers to using MobileNet to encode the input image Extract The spatial encoding feature map of the layer ,in For passing Multi-scale feature maps generated by layer convolutional neural networks, feature maps With different resolutions and feature dimensions ; The spatial encoder The decoding process refers to gradually restoring the spatial resolution of the feature map by upsampling layer by layer through transposed convolution, introducing skip connections, and combining with the spatial encoder The low-level detail features of the encoding process and the high-level semantic features of the decoder are combined through the fully connected layer. Spatial decoding feature map of the layer Calculate the radial distortion parameters and tangential distortion parameters ; The spatial encoder The decoding process of is to gradually restore the feature map by upsampling layer by layer through transposed convolution. The process is described as: Where ConvTranspose represents the transposed convolution operation. For the Spatial decoding feature map of the layer After The layer transposed convolutional neural network and passed Multi-scale feature maps generated by layer convolutional neural network The spliced ​​generated The spatial decoded feature map of the layer; The correction flow represents a mapping from the image coordinate system to the correction coordinate system, where is the pixel coordinate in the distorted image. After correction, the pixel will be mapped to the new position ,Right now: , , in, is the corrected pixel coordinate, and Represent the pixel width and pixel height of the image respectively; The radial distortion parameters and tangential distortion parameters output by the input encoder are used to generate a specific correction flow using a high-order polynomial model. The process is as follows: Each pixel coordinate on , according to the radial distortion parameter Calculate the radial distortion factor , and then using the radial distortion factor , tangential distortion parameters and pixel coordinates , calculate the new position of the pixel in the corrected coordinate system ; The radial distortion factor The calculation method is as follows: in, Indicates that it contains A set of radial distortion parameters, Represents pixels To the center of the image The square distance of The pixel coordinates New position in the corrected coordinate system The calculation method is: The confidence corresponding to each pixel of the correction flow is calculated through the fully connected layer The specific steps are: in, and To learn the parameters, Sigmoid activation function ensures that the confidence level is between 0 and 1; The confidence-based exponential decay function is used to fuse the final correction flow The process is as follows: in, Reason and The confidence level of the calculated horizontal axis, Reason and The confidence level of the calculated ordinate, yes and The calculated correction flow of the abscissa, yes and The calculated correction flow of the ordinate, Reason and The confidence level of the calculated horizontal axis, Reason and The confidence level of the calculated ordinate, yes and The calculated correction flow of the abscissa, yes and The calculated correction flow of the ordinate, is the attenuation coefficient, which is used to control the degree of attenuation of confidence; The will correct flow Applied to distorted images The corrected image is obtained Refers to the correction flow New position mapped to the corrected coordinate system .

Images