Unmanned aerial vehicle low-altitude photography image enhancement method and system

Abstract

The invention discloses an unmanned aerial vehicle low-altitude photography image enhancement method and system. The method comprises the steps of data preparation, motion joint deblurring optimization, multi-source data registration, scene adaptive enhancement and image comprehensive enhancement. The invention relates to the technical field of unmanned aerial vehicle image enhancement, and realizes high-precision deblurring, precise registration and adaptive visual enhancement of an unmanned aerial vehicle image by fusing inertial measurement unit guided blurring kernel prediction, cross-modal graph structure matching and elastic transformation optimization and an enhancement strategy dynamic adjustment mechanism based on scene analysis. The image definition, the structure consistency and the scene readability are remarkably improved, and the method is suitable for image processing tasks in a complex low-altitude flight environment.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) image enhancement, and in particular to a method and system for enhancing low-altitude photography images of unmanned aerial vehicles (UAVs). Background Art

[0002] The image enhancement method for UAV low-altitude photography refers to the use of image processing and multi-source data fusion technology to optimize the clarity, contrast, color restoration and other aspects of the images collected by the UAV during low-altitude flight, so as to improve the visual quality and information availability of the image; this method can effectively deal with image degradation problems caused by factors such as flight jitter, lighting changes, motion blur, etc., and is widely used in agricultural monitoring, urban inspections, disaster assessment and other fields, helping to improve the efficiency of image interpretation and the accuracy of intelligent analysis.

[0003] However, in the existing UAV low-altitude photography image enhancement methods, there are technical problems such as jitter, dynamic change, blur and weak multi-source heterogeneous information fusion ability in the image acquisition process, which lead to unstable quality of the final enhanced image, missing details and obvious noise residue; in the existing image deblurring optimization methods, there are technical problems such as inaccurate modeling of complex motion blur in the unstable flight state of the UAV, weak edge structure recovery ability of the traditional degradation model, and lack of physical rationality of end-to-end network training; in the existing multi-source data registration methods, there are technical problems such as insufficient cross-modal information fusion ability, low registration accuracy between images of different resolutions, and difficulty in aligning semantic structures by traditional geometric registration methods; in the existing scene enhancement methods, there are technical problems such as poor generalization of enhancement strategies, rough scene content recognition, and insensitivity of enhancement operations to edge structures and regional brightness control. Summary of the invention

[0004] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides a method and system for enhancing low-altitude photography of unmanned aerial vehicles. In view of the technical problems that jitter, dynamic change, blur and weak multi-source heterogeneous information fusion ability in the existing low-altitude photography image enhancement methods of unmanned aerial vehicles lead to unstable quality of the final enhanced image, missing details and obvious residual noise due to the jitter, dynamic change, blur and weak multi-source heterogeneous information fusion ability in the image acquisition process, this solution creatively adopts a comprehensive image enhancement method combining deblurring optimization, multi-source data registration and scene enhancement, realizes a closed-loop processing chain from image acquisition, deblurring, registration to intelligent enhancement, and effectively separates various interference factors through modular design. The interference factors are eliminated, which improves the robustness of the enhancement system and significantly improves the clarity, structural consistency and scene readability of low-altitude photographic images. It is suitable for a variety of low-altitude flight environments such as complex lighting and dynamic changes in viewing angles. In view of the technical problems in the existing image deblurring optimization methods, there are inaccurate modeling of complex motion blur in the unstable flight state of drones, weak recovery ability of traditional degradation models for edge structures, and lack of physical rationality in end-to-end network training. This solution creatively adopts a neural fuzzy kernel prediction method guided by physical constraints and inertial measurement units to perform motion joint deblurring optimization, realizing the physical integration of blur kernel prediction and image reconstruction process. Consistency modeling, inverting the six-degree-of-freedom flight trajectory through IMU data, combining Kalman filter smoothing, blur kernel network prediction and improved U-shaped network reconstruction, effectively alleviates the spatial distortion and edge smearing problems caused by high dynamic scenes, and enhances the image edge sharpness and texture detail fidelity; In view of the technical problems of insufficient cross-modal information fusion capability, low registration accuracy between images of different resolutions, and difficulty in aligning semantic structures in traditional geometric registration methods in existing multi-source data registration methods, this scheme creatively adopts elastic transformation improved registration method combined with cross-spectral feature attention matching to perform multi-source data registration, realizing the process from point cloud generation to image structure. The cross-modal alignment process with structure matching effectively improves the spatial consistency and structural integrity of the enhanced multi-source images after alignment. In view of the technical problems in the existing scene enhancement methods, such as poor generalization of enhancement strategies, rough scene content recognition, and insensitivity of enhancement operations to edge structures and regional brightness control, this scheme creatively adopts a dynamic physical model coupling enhancement method combined with scene perception enhancement to perform scene adaptive enhancement, obtain scene adaptive enhanced image data, realize on-demand dynamic adaptation of enhancement strategies, effectively avoids the problems of "over-enhancement" and "introduction of artifacts" in traditional enhancement methods, and comprehensively improves the visual realism and expression efficiency of images.

[0005] The technical solution adopted by the present invention is as follows: The present invention provides a method for enhancing an image of a drone low-altitude photography, the method comprising the following steps:

[0006] Step S1: data preparation;

[0007] Step S2: motion joint deblurring optimization;

[0008] Step S3: multi-source data registration;

[0009] Step S4: scene adaptive enhancement;

[0010] Step S5: low-altitude photography image enhancement.

[0011] Furthermore, in step S1, the data preparation is used to collect low-altitude photography original images and perform preliminary preprocessing, specifically, obtaining low-altitude photography original image data through multi-source image acquisition by unmanned aerial vehicles, and performing preliminary data preprocessing by sequentially performing time synchronization, spatial alignment, radiation correction and data index structuring to obtain optimized data for unmanned aerial vehicle low-altitude photography images.

[0012] Further, in step S2, the motion joint deblurring optimization is used to eliminate motion blur caused by the jitter and high-speed motion of the drone. Specifically, based on the drone low-altitude photography image optimization data, a neural fuzzy kernel prediction method combined with physical constraints and inertial measurement unit guidance is used to perform motion joint deblurring optimization to obtain deblurred optimized image data, including the following steps:

[0013] Step S21: motion trajectory modeling, specifically, collecting the angular velocity and linear acceleration data of the UAV through an inertial measurement unit, eliminating velocity data noise through Kalman filtering, constructing a six-degree-of-freedom motion equation, modeling the UAV motion differential trajectory, and obtaining smoothed and optimized motion state characteristic data through parameterized fitting;

[0014] Step S22: fuzzy kernel prediction, specifically, constructing a long short-term memory fuzzy kernel network fusion model based on the smoothed optimized motion state feature data, and performing fuzzy kernel prediction in combination with physical constraint loss to obtain physical constraint defuzzification prediction data;

[0015] Step S23: physical constraint reconstruction, specifically, constructing a non-uniform blur degradation model based on the physical constraint deblurring prediction data, performing physical constraint reconstruction, obtaining physical reconstruction feature intervention data, and constructing an improved U-shaped network, performing physical constraint image reconstruction model training based on a joint loss function, and combining the physical reconstruction feature intervention data, using the physical constraint image reconstruction model, reconstructing a deblurred optimized image, and obtaining a deblurred optimized image output;

[0016] Step S24: motion joint deblurring, specifically performing edge sharpening on the deblurred optimized image output to obtain deblurred optimized image data.

[0017] Further, in step S3, the multi-source data registration is used to achieve pixel-level alignment of data, specifically, based on the deblurred optimized image data, an elastic transformation improved registration method combined with cross-spectral feature attention matching is used to perform multi-source data registration to obtain pixel-aligned enhanced data, including the following steps:

[0018] Step S31: cross-modal feature extraction, specifically generating feature depth map and feature intensity map reference data through laser radar point cloud projection based on the visible light image data in the deblurred optimized image data and the multispectral data in the drone low-altitude photography image optimization data, and performing cross-modal feature extraction by constructing a three-branch feature extraction network, and obtaining multi-source data registration feature data through feature fusion;

[0019] Step S32: graph structure matching, specifically, based on the multi-source data registration feature data, using an improved linear iterative clustering algorithm to perform pixel segmentation, and constructing pixel node graph structure data through graph nodes and node features, and using a three-layer graph attention network to perform node similarity calculation to obtain cross-modal registration matching pair set data;

[0020] The improved linear iterative clustering algorithm introduces a multi-spectral feature + cross-modal feature dual consistency optimization item on the basis of a standard simple linear iterative clustering algorithm to perform image pixel structure matching clustering optimization;

[0021] Step S33: elastic transformation optimization, specifically, based on the cross-modal registration matching pair set data, an improved thin plate spline transformation solution algorithm is constructed to perform elastic transformation optimization to obtain resampled registration feature data;

[0022] Step S34: multi-source data registration, specifically, obtaining pixel alignment enhanced data by sequentially performing local alignment mutual information check and adaptive iterative local alignment optimization based on the resampled alignment feature data.

[0023] Further, in step S4, the scene adaptive enhancement is used to intelligently adjust the enhancement strategy according to the image scene content, specifically, based on the pixel alignment enhancement data, a dynamic physical model coupling enhancement method combined with scene perception enhancement is adopted to perform scene adaptive enhancement to obtain scene adaptive enhanced image data, including the following steps:

[0024] Step S41: scene parsing improvement, specifically, performing scene parsing by constructing a lightweight segmentation pre-training network and obtaining scene parsing reference data;

[0025] Step S42: Quantitative analysis of regional characteristics, specifically, constructing feature indicators according to the specific scene type in the scene analysis reference data, and constructing a multi-scale feature fusion pyramid structure according to the feature indicators to perform quantitative analysis of regional characteristics, so as to obtain quantitative feature data of the scene region;

[0026] Step S43: dynamic scene enhancement, specifically, constructing a classification enhancement sub-library based on the scene area quantitative feature data, and performing dynamic scene enhancement based on the classification enhancement sub-library and the scene type and scene feature quantization value in the scene area quantitative feature data to obtain scene dynamic preliminary enhancement data;

[0027] Step S44: scene-adaptive image enhancement, specifically, based on the scene dynamic preliminary enhancement data, edge consistency processing and bilateral filtering smooth transition operations are used to perform comprehensive image adaptive enhancement to obtain scene-adaptive enhanced image data.

[0028] Furthermore, in step S5, the low-altitude photography image enhancement is used to integrate the previous processing results to output the final enhanced image, specifically, based on the deblurred optimized image data, the pixel alignment enhancement data and the scene adaptive enhancement image data, to perform comprehensive enhancement of the low-altitude photography image to obtain the comprehensive enhanced image data of the UAV low-altitude photography.

[0029] The present invention provides an unmanned aerial vehicle low-altitude photography image enhancement system, comprising a data preparation module, a deblurring optimization module, a multi-source data registration module, a scene adaptive enhancement module and a low-altitude photography image enhancement module;

[0030] The data preparation module is used for data preparation, and obtains the optimized data of the UAV low-altitude photography image through data preparation, and sends the optimized data of the UAV low-altitude photography image to the deblurring optimization module;

[0031] The deblurring optimization module is used for motion joint deblurring optimization, obtains deblurred optimized image data through motion joint deblurring optimization, and sends the deblurred optimized image data to the multi-source data registration module and the low-altitude photography image enhancement module;

[0032] The multi-source data registration module is used for multi-source data registration, obtains pixel alignment enhancement data through multi-source data registration, and sends the pixel alignment enhancement data to the scene adaptive enhancement module and the low-altitude photography image enhancement module;

[0033] The scene adaptive enhancement module is used for scene adaptive enhancement, obtains scene adaptive enhanced image data through scene adaptive enhancement, and sends the scene adaptive enhanced image data to the low-altitude photography image enhancement module;

[0034] The low-altitude photography image enhancement module is used for low-altitude photography image enhancement, and obtains comprehensive enhanced image data of UAV low-altitude photography through low-altitude photography image enhancement.

[0035] The beneficial effects achieved by the present invention using the above scheme are as follows:

[0036] (1) In view of the technical problems in the existing UAV low-altitude photography image enhancement methods, there are problems such as jitter, dynamic changes, blurring and weak multi-source heterogeneous information fusion capabilities during the image acquisition process, which lead to unstable quality, missing details and obvious noise residue in the final enhanced image. This solution creatively adopts a comprehensive image enhancement method that combines deblurring optimization, multi-source data registration and scene enhancement, and realizes a closed-loop processing chain from image acquisition, deblurring, registration to intelligent enhancement. Through modular design, various interference factors are effectively separated, the robustness of the enhancement system is improved, and the clarity, structural consistency and scene readability of low-altitude photography images are significantly improved. It is suitable for a variety of low-altitude flight environments such as complex lighting and dynamic changes in perspective;

[0037] (2) In view of the technical problems in existing image deblurring optimization methods, such as inaccurate modeling of complex motion blur in unstable flight states of drones, weak edge structure recovery capabilities of traditional degradation models, and lack of physical rationality in end-to-end network training, this solution creatively adopts a neural fuzzy kernel prediction method that combines physical constraints and inertial measurement unit guidance to perform joint motion deblurring optimization, thus achieving physical consistency modeling of blur kernel prediction and image reconstruction processes. By inverting the six-degree-of-freedom flight trajectory through IMU data, combined with Kalman filter smoothing, blur kernel network prediction, and improved U-shaped network reconstruction, the spatial distortion and edge smearing problems caused by high-dynamic scenes are effectively alleviated, and the image edge sharpness and texture detail fidelity are enhanced;

[0038] (3) In view of the technical problems of insufficient cross-modal information fusion capability, low registration accuracy between images of different resolutions, and difficulty in aligning semantic structures in traditional geometric registration methods in existing multi-source data registration methods, this scheme creatively adopts elastic transformation and cross-spectral feature attention matching to improve the registration method for multi-source data registration, realizing a cross-modal alignment process from point cloud generation to graph structure matching, effectively improving the spatial consistency and structural integrity of the multi-source images after alignment;

[0039] (4) In view of the technical problems in existing scene enhancement methods, such as poor generalization of enhancement strategies, rough scene content recognition, and insensitivity of enhancement operations to edge structures and regional brightness control, this scheme creatively adopts a dynamic physical model coupling enhancement method combined with scene perception enhancement to perform scene adaptive enhancement and obtain scene adaptive enhanced image data, thus achieving on-demand dynamic adaptation of enhancement strategies, effectively avoiding the problems of "over-enhancement" and "introduction of artifacts" in traditional enhancement methods, and comprehensively improving the visual realism and expression efficiency of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a flow chart of a method for enhancing low-altitude photography images of a drone provided by the present invention;

[0041] Figure 2 A schematic diagram of an unmanned aerial vehicle low-altitude photography image enhancement system provided by the present invention;

[0042] Figure 3 It is a schematic diagram of the process of motion joint deblurring optimization in step S2;

[0043] Figure 4 A schematic diagram of the process of multi-source data registration in step S3;

[0044] Figure 5 This is a schematic diagram of the process of scene adaptive enhancement in step S4.

[0045] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0047] In the description of the present invention, it should be understood that terms such as “upper”, “lower”, “front”, “back”, “left”, “right”, “top”, “bottom”, “inside” and “outside” indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0048] Example 1, see Figure 1The present invention provides a method for enhancing an image of a drone low-altitude photography, the method comprising the following steps:

[0049] Step S1: data preparation;

[0050] Step S2: motion joint deblurring optimization;

[0051] Step S3: multi-source data registration;

[0052] Step S4: scene adaptive enhancement;

[0053] Step S5: low-altitude photography image enhancement.

[0054] By performing the above operations, in view of the technical problems in the existing UAV low-altitude photography image enhancement methods, such as jitter, dynamic changes, blur and weak multi-source heterogeneous information fusion capabilities during image acquisition, which lead to unstable quality, missing details and obvious noise residue in the final enhanced image, this solution creatively adopts a comprehensive image enhancement method that combines deblurring optimization, multi-source data registration and scene enhancement, and realizes a closed-loop processing chain from image acquisition, deblurring, registration to intelligent enhancement. Through modular design, various interference factors are effectively separated, the robustness of the enhancement system is improved, and the clarity, structural consistency and scene readability of low-altitude photography images are significantly improved, which is suitable for a variety of low-altitude flight environments such as complex lighting and dynamic changes in perspective.

[0055] Example 2, see Figure 1 and Figure 2 , This embodiment is based on the above embodiment. In step S1, the data preparation is used to collect low-altitude photography original images and perform preliminary preprocessing, specifically, obtaining low-altitude photography original image data through multi-source image acquisition by unmanned aerial vehicles, and performing preliminary data preprocessing by sequentially performing time synchronization, spatial alignment, radiation correction and data index structuring to obtain optimized data of unmanned aerial vehicle low-altitude photography images;

[0056] The low-altitude photography original image data specifically includes the original image and the original image parameters. The original image includes visible light image data, infrared image data, multi-spectral image data, laser radar point cloud data and thermal imaging image data; the original image parameters specifically include acquisition timestamp, positioning information data, attitude information data, sensor parameters and image resolution parameters;

[0057] The UAV low-altitude photography image optimization data specifically includes format-unified image data, spatiotemporal consistency parameter data, radiation characteristic correction data, and data index annotation structured data.

[0058] Example 3, see Figure 1 , Figure 2 and Figure 3This embodiment is based on the above embodiment. In step S2, the motion joint deblurring optimization is used to eliminate motion blur caused by the jitter and high-speed motion of the drone. Specifically, based on the drone low-altitude photography image optimization data, a neural fuzzy kernel prediction method combined with physical constraints and inertial measurement unit guidance is used to perform motion joint deblurring optimization to obtain deblurred optimized image data, including the following steps:

[0059] Step S21: motion trajectory modeling, specifically, collecting the angular velocity and linear acceleration data of the UAV through an inertial measurement unit, eliminating velocity data noise through Kalman filtering, constructing a six-degree-of-freedom motion equation, modeling the UAV motion differential trajectory, and obtaining smoothed and optimized motion state characteristic data through parameterized fitting;

[0060] The six-degree-of-freedom motion equation is used to model the differential trajectory of UAV motion, and the calculation formula is:

[0061] ;

[0062] Where R is the attitude rotation matrix, which is used to describe the transformation from the drone body coordinate system to the geographic coordinate system. is the antisymmetric matrix of angular velocity, is the matrix cross product operator, v is the UAV velocity vector, t is the time index, and a is the linear acceleration vector, which is collected by the inertial measurement unit. is the earth's rotation angular velocity vector, and p is the drone's position vector;

[0063] The parameterized fitting specifically adopts a cubic B-spline curve parameterized trajectory, and the calculation formula is:

[0064] ;

[0065] Where T(·) is the smooth trajectory point at time t, which is used to represent the trajectory data point in the smooth optimization motion state feature data, n is the total number of control points, i is the control point index, and N i,3 (·) is the cubic B-spline basis function corresponding to the ith control point, P i is the spline control point entity;

[0066] Step S22: fuzzy kernel prediction, specifically, constructing a long short-term memory fuzzy kernel network fusion model based on the smoothed optimized motion state feature data, and performing fuzzy kernel prediction in combination with physical constraint loss to obtain physical constraint defuzzification prediction data;

[0067] The long short-term memory fuzzy kernel network fusion model includes a motion encoder, a spatial attention mechanism block and a fuzzy physical constraint loss function;

[0068] The motion encoder specifically adopts a standard bidirectional long short-term memory neural network to extract the time series of UAV motion features based on the smoothed optimized motion state feature data. The calculation formula is:

[0069] ;

[0070] In the formula, h t is the hidden state of the motion encoder output, LSTM(·) is the standard bidirectional long short-term memory neural network function, is the characteristic change of the motion state corresponding to time t;

[0071] The fuzzy physical constraint loss function introduces the motion energy conservation loss term as the model loss function, and the calculation formula is:

[0072] ;

[0073] In the formula, is the fuzzy physical constraint loss function, x is the horizontal index of the pixel, y is the vertical index of the pixel, K(·) is the predicted pixel value, T a is the total length of time, t is the time index, is the characteristic change of the motion state corresponding to time t;

[0074] Step S23: physical constraint reconstruction, specifically, constructing a non-uniform blur degradation model based on the physical constraint deblurring prediction data, performing physical constraint reconstruction, obtaining physical reconstruction feature intervention data, and constructing an improved U-shaped network, performing physical constraint image reconstruction model training based on a joint loss function, and combining the physical reconstruction feature intervention data, using the physical constraint image reconstruction model, reconstructing a deblurred optimized image, and obtaining a deblurred optimized image output;

[0075] The non-uniform blur degradation model is used to reconstruct an image that is locally uniform but overall non-uniformly blurred in space. The calculation formula is:

[0076] ;

[0077] In the formula, I b is the blurred image data after being processed by the non-uniform blur degradation model, M is the total number of local blurred regions, m is the local blurred region index, and w m is the weight of the mth local fuzzy region, I s is the reconstructed target image, K m is the mth local fuzzy kernel function, is the noise bias term;

[0078] The improved U-shaped network includes a multi-stage downsampling encoder, a decoder and a joint loss function;

[0079] The multi-level downsampling encoder specifically adopts a five-level downsampling structure, receives and fuses the features in the physical constraint deblurring prediction data through a 3×3 convolution and motion feature concatenation layer;

[0080] The decoder introduces variable convolution to adapt to motion distortion and optimizes attention feature perception through standard skip connection design;

[0081] The calculation formula of the improved U-type network is:

[0082] ;

[0083] In the formula, E l is the output of the encoder at layer l, E l (·) is the encoder operation function of the lth layer, Conv 3×3 is the 3×3 convolution operation function, F l-1 is the output feature of the l-1th layer encoder, M t is the motion feature data in the physical constraint deblurring prediction data, is the motion feature mapping operation, is the feature fusion operator, is the reconstructed image output by the decoder, D 1 (·) is the upsampling function in the first layer decoder, D 5 (·) is the upsampling function in the 5th layer decoder, DCN(·) is the deformable convolutional network function, E 5 is the 5th layer encoder output, A(·) is the attention enhancement operation function, E 4 is the output of the 4th layer encoder, E 1 is the output of the first layer encoder;

[0084] The joint loss function specifically adopts a weighted combination of L1 reconstruction loss and motion consistency loss;

[0085] Step S24: motion joint deblurring, specifically performing edge sharpening on the deblurred optimized image output to obtain deblurred optimized image data.

[0086] By performing the above operations, in order to address the technical problems in the existing image deblurring optimization methods, such as inaccurate modeling of complex motion blur in unstable flight states of drones, weak edge structure recovery capabilities of traditional degradation models, and lack of physical rationality in end-to-end network training, this solution creatively adopts a neural fuzzy kernel prediction method guided by physical constraints and inertial measurement units to perform motion joint deblurring optimization, and realizes the physical consistency modeling of blur kernel prediction and image reconstruction processes. By inverting the six-degree-of-freedom flight trajectory through IMU data, combined with Kalman filter smoothing, blur kernel network prediction and improved U-shaped network reconstruction, it effectively alleviates the spatial distortion and edge smearing problems caused by high dynamic scenes, and enhances the image edge sharpness and texture detail fidelity.

[0087] Example 4, see Figure 1 , Figure 2 and Figure 4 This embodiment is based on the above embodiment. In step S3, the multi-source data registration is used to achieve pixel-level alignment of data. Specifically, based on the deblurred optimized image data, an elastic transformation improved registration method combined with cross-spectral feature attention matching is used to perform multi-source data registration to obtain pixel-aligned enhanced data, including the following steps:

[0088] Step S31: cross-modal feature extraction, specifically generating feature depth map and feature intensity map reference data through laser radar point cloud projection based on the visible light image data in the deblurred optimized image data and the multispectral data in the drone low-altitude photography image optimization data, and performing cross-modal feature extraction by constructing a three-branch feature extraction network, and obtaining multi-source data registration feature data through feature fusion;

[0089] The three-branch feature extraction network includes a visible light branch, a multispectral branch and a point cloud feature extraction branch;

[0090] The feature fusion specifically uses a multi-layer perceptron to perform feature fusion of three types of modalities;

[0091] Step S32: graph structure matching, specifically, based on the multi-source data registration feature data, using an improved linear iterative clustering algorithm to perform pixel segmentation, and constructing pixel node graph structure data through graph nodes and node features, and using a three-layer graph attention network to perform node similarity calculation to obtain cross-modal registration matching pair set data;

[0092] The improved linear iterative clustering algorithm introduces dual consistency optimization items of multispectral features and cross-modal features on the basis of the standard simple linear iterative clustering algorithm to perform clustering optimization of image pixel structure matching. The calculation formula is:

[0093] ;

[0094] In the formula, F tar is the cluster matching target graph structure data, which is used to represent the result of pixel segmentation. S is the target segmentation pixel. is the total number of segmented pixels, used to represent the total number of graph nodes, k is the segmented pixel index, P is the original pixel index of the image, S k is the pixel value corresponding to the kth segmented pixel, I vis (·) is the visible light eigenvector function, is the feature center of the visible light image, is the registration trade-off weight parameter, I ms (·) is the multispectral feature vector function, is the characteristic center of the multispectral image;

[0095] Step S33: elastic transformation optimization, specifically, based on the cross-modal registration matching pair set data, an improved thin plate spline transformation solution algorithm is constructed to perform elastic transformation optimization to obtain resampled registration feature data;

[0096] The improved thin plate spline transform solution algorithm specifically adopts an energy function that is a weighted combination of regularization terms, spectral consistency terms, and geometric constraint terms to improve the thin plate spline transform. The calculation formula is:

[0097] ;

[0098] Where E is the total energy function value. By minimizing the total energy function value, the optimal transformation solution is performed. reg is a regularization term used to control the smoothness of the transformation, and is specifically calculated using the Laplace smoothing function. is the spectral consistency weight, E spec is the spectral consistency term, which is used to minimize the difference in reflectance between visible light and near-infrared bands. It is calculated using the sum of squared characteristic errors. is the geometric constraint weight, E geom It is a geometric constraint term used to stabilize the structural relationship between adjacent pixels, and is specifically calculated using the difference between adjacent pixels;

[0099] Step S34: multi-source data registration, specifically, obtaining pixel alignment enhanced data by sequentially performing local alignment mutual information check and adaptive iterative local alignment optimization based on the resampled alignment feature data.

[0100] By performing the above operations, in order to address the technical problems in the existing multi-source data registration methods, such as insufficient cross-modal information fusion capability, low registration accuracy between images of different resolutions, and difficulty in aligning semantic structures using traditional geometric registration methods, this scheme creatively adopts an elastic transformation improved registration method combined with cross-spectral feature attention matching to perform multi-source data registration, realizing a cross-modal alignment process from point cloud generation to graph structure matching, effectively improving the enhanced spatial consistency and structural integrity after multi-source image alignment.

[0101] Example 5, see Figure 1 , Figure 2 and Figure 5 This embodiment is based on the above embodiment. In step S4, the scene adaptive enhancement is used to intelligently adjust the enhancement strategy according to the image scene content. Specifically, based on the pixel alignment enhancement data, a dynamic physical model coupling enhancement method combined with scene perception enhancement is used to perform scene adaptive enhancement to obtain scene adaptive enhanced image data, including the following steps:

[0102] Step S41: scene parsing improvement, specifically, performing scene parsing by constructing a lightweight segmentation pre-training network and obtaining scene parsing reference data;

[0103] The lightweight segmentation pre-training network specifically adopts an improved BiSeNetV2+ lightweight model for scene parsing. The improved BiSeNetV2+ lightweight model uses parallel depth-separable convolution and global enhancement modules to extract branch features based on the pixel alignment enhancement data, and obtains scene parsing features through attention feature fusion, and performs scene type parsing based on the scene parsing features;

[0104] The scene types in the scene parsing reference data specifically include vegetation, buildings, water areas and shadows;

[0105] Step S42: Quantitative analysis of regional characteristics, specifically, constructing feature indicators according to the specific scene type in the scene analysis reference data, and constructing a multi-scale feature fusion pyramid structure according to the feature indicators to perform quantitative analysis of regional characteristics, so as to obtain quantitative feature data of the scene region;

[0106] The multi-scale feature fusion pyramid structure specifically uses a Gaussian pyramid to calculate hierarchical features;

[0107] Preferably, Table 1 is a reference example table for the quantitative analysis of regional characteristics. For example, when the scene type is vegetation, the characteristic index is set to the red edge band ratio index, and is calculated according to the calculation formula. In the calculation formula of the red edge band ratio index, R nir is the near-infrared reflectivity, R redis the reflectivity of the red light band;

[0108] When the scene type is a building, the characteristic index is set to edge structure complexity, and is calculated according to a calculation formula. In the calculation formula of edge structure complexity, is the image gradient, Area is the pixel area of ​​the region, and SSIM is the image local structure similarity index;

[0109] When the scene type is water area, the characteristic index is set to polarization reflection variance, and the calculation is performed according to the calculation formula. In the calculation formula of polarization reflection variance, Var(·) is the variance operation function, I vis is the visible light image data, M polar is the polarization mask image data;

[0110] When the scene type is shadow, the characteristic index is set to the light attenuation coefficient, and is calculated according to the calculation formula. In the calculation formula of the light attenuation coefficient, is the average gray value of the shadow, is the average gray value of the unobstructed area;

[0111] Table 1 Reference example table for regional characteristics quantitative analysis

[0112]

[0113] Step S43: dynamic scene enhancement, specifically, constructing a classification enhancement sub-library based on the scene area quantitative feature data, and performing dynamic scene enhancement based on the classification enhancement sub-library and the scene type and scene feature quantization value in the scene area quantitative feature data to obtain scene dynamic preliminary enhancement data;

[0114] Preferably, Table 2 is a reference example table of the classification enhancement sub-library. For example, when the scene type is vegetation, red edge band enhancement and chlorophyll inversion are used for scene enhancement; the red edge band enhancement specifically enhances the red edge band (705nm-740nm) in the multispectral image, and the chlorophyll inversion specifically uses an index-based method for chlorophyll inversion;

[0115] When the scene type is a building, a guided filter structure enhancement is used for scene enhancement, wherein the guided filter structure enhancement specifically uses an edge-preserving filter algorithm based on a local linear model to perform the guided filter structure enhancement;

[0116] When the scene type is water area, polarization feature enhancement and specular reflection suppression are used for scene enhancement. The polarization feature enhancement is specifically performed by collecting polarization images for contrast enhancement. The specular reflection suppression is specifically performed by using a brightness reflected light separation method based on a Retinex filter to suppress specular reflection.

[0117] When the scene type is shadow, physical illumination compensation and adversarial generative restoration are used to enhance the scene. The physical illumination compensation specifically uses a brightness normalization method for enhancement, and the adversarial generative restoration uses a pre-trained CycleGAN architecture for shadow restoration.

[0118] Table 2 Classification enhancer library reference example table

[0119]

[0120] Step S44: scene-adaptive image enhancement, specifically, based on the scene dynamic preliminary enhancement data, edge consistency processing and bilateral filtering smooth transition operations are used to perform comprehensive image adaptive enhancement to obtain scene-adaptive enhanced image data.

[0121] By performing the above operations, in order to address the technical problems in existing scene enhancement methods, such as poor generalization of enhancement strategies, rough scene content recognition, and insensitivity of enhancement operations to edge structures and regional brightness control, this solution creatively adopts a dynamic physical model coupling enhancement method combined with scene perception enhancement to perform scene adaptive enhancement, obtain scene adaptive enhanced image data, and realize on-demand dynamic adaptation of enhancement strategies, effectively avoiding the problems of "over-enhancement" and "introduction of artifacts" in traditional enhancement methods, and comprehensively improving the visual realism and expression efficiency of images.

[0122] Example 6, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In step S5, the low-altitude photography image enhancement is used to integrate the previous processing results to output the final enhanced image. Specifically, the low-altitude photography image is comprehensively enhanced based on the deblurred optimized image data, the pixel alignment enhancement data and the scene adaptive enhancement image data to obtain the comprehensive enhanced image data of the drone low-altitude photography.

[0123] Embodiment 7, see Figure 1 and Figure 2 , this embodiment is based on the above embodiment, and the present invention provides a UAV low-altitude photography image enhancement system, including a data preparation module, a deblurring optimization module, a multi-source data registration module, a scene adaptive enhancement module and a low-altitude photography image enhancement module;

[0124] The data preparation module is used for data preparation, and obtains the optimized data of the UAV low-altitude photography image through data preparation, and sends the optimized data of the UAV low-altitude photography image to the deblurring optimization module;

[0125] The deblurring optimization module is used for motion joint deblurring optimization, obtains deblurred optimized image data through motion joint deblurring optimization, and sends the deblurred optimized image data to the multi-source data registration module and the low-altitude photography image enhancement module;

[0126] The multi-source data registration module is used for multi-source data registration, obtains pixel alignment enhancement data through multi-source data registration, and sends the pixel alignment enhancement data to the scene adaptive enhancement module and the low-altitude photography image enhancement module;

[0127] The scene adaptive enhancement module is used for scene adaptive enhancement, obtains scene adaptive enhanced image data through scene adaptive enhancement, and sends the scene adaptive enhanced image data to the low-altitude photography image enhancement module;

[0128] The low-altitude photography image enhancement module is used for low-altitude photography image enhancement, and obtains comprehensive enhanced image data of UAV low-altitude photography through low-altitude photography image enhancement.

[0129] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0130] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.

[0131] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A method for enhancing an image of a drone in low altitude photography, characterized in that: The method comprises the following steps: Step S1: Data preparation, obtaining optimized data of UAV low-altitude photography images; Step S2: motion joint deblurring optimization, using a neural fuzzy kernel prediction method combined with physical constraints and guided by an inertial measurement unit to perform motion joint deblurring optimization to obtain deblurred optimized image data, including the following steps: step S21: motion trajectory modeling; step S22: fuzzy kernel prediction; step S23: physical constraint reconstruction; step S24: motion joint deblurring; Step S3: multi-source data registration, using the elastic transformation improved registration method combined with cross-spectral feature attention matching to perform multi-source data registration to obtain pixel alignment enhanced data, including the following steps: step S31: cross-modal feature extraction; step S32: graph structure matching; step S33: elastic transformation optimization; step S34: multi-source data registration; Step S4: scene adaptive enhancement, using a dynamic physical model coupling enhancement method combined with scene perception enhancement to perform scene adaptive enhancement to obtain scene adaptive enhanced image data, including the following steps: step S41: scene analysis improvement; step S42: regional characteristic quantitative analysis; step S43: dynamic scene enhancement; step S44: scene adaptive image enhancement; Step S5: low-altitude photography image enhancement to obtain comprehensive enhanced image data of UAV low-altitude photography.

2. The method for enhancing the image of a drone low-altitude photography according to claim 1, characterized in that: In step S1, the data preparation is used to collect low-altitude photography original images and perform preliminary preprocessing, specifically, obtaining low-altitude photography original image data through drone multi-source image acquisition, and performing preliminary data preprocessing by sequentially performing time synchronization, spatial alignment, radiation correction and data index structuring to obtain drone low-altitude photography image optimization data.

3. The method for enhancing the image of a drone low-altitude photography according to claim 2, characterized in that: In step S2, the motion joint deblurring optimization is used to eliminate motion blur caused by the jitter and high-speed motion of the drone. Specifically, based on the drone low-altitude photography image optimization data, a neural fuzzy kernel prediction method combined with physical constraints and inertial measurement unit guidance is used to perform motion joint deblurring optimization to obtain deblurred optimized image data, including the following steps: Step S21: motion trajectory modeling, specifically, collecting the angular velocity and linear acceleration data of the UAV through an inertial measurement unit, eliminating velocity data noise through Kalman filtering, constructing a six-degree-of-freedom motion equation, modeling the UAV motion differential trajectory, and obtaining smoothed and optimized motion state characteristic data through parameterized fitting; Step S22: fuzzy kernel prediction, specifically, constructing a long short-term memory fuzzy kernel network fusion model based on the smoothed optimized motion state feature data, and performing fuzzy kernel prediction in combination with physical constraint loss to obtain physical constraint defuzzification prediction data; The long short-term memory fuzzy kernel network fusion model includes a motion encoder, a spatial attention mechanism block and a fuzzy physical constraint loss function; The motion encoder specifically adopts a standard bidirectional long short-term memory neural network to extract the time series of UAV motion features based on the smoothed optimized motion state feature data; The fuzzy physical constraint loss function introduces the motion energy conservation loss term as the model loss function; Step S23: physical constraint reconstruction, specifically, constructing a non-uniform blur degradation model based on the physical constraint deblurring prediction data, performing physical constraint reconstruction, obtaining physical reconstruction feature intervention data, and constructing an improved U-shaped network, performing physical constraint image reconstruction model training based on a joint loss function, and combining the physical reconstruction feature intervention data, using the physical constraint image reconstruction model, reconstructing a deblurred optimized image, and obtaining a deblurred optimized image output; The non-uniform blur degradation model is used to reconstruct an image that is locally uniform but globally non-uniformly blurred in space; The improved U-shaped network includes a multi-stage downsampling encoder, a decoder and a joint loss function; The multi-level downsampling encoder specifically adopts a five-level downsampling structure, receives and fuses the features in the physical constraint deblurring prediction data through a 3×3 convolution and motion feature concatenation layer; The decoder introduces variable convolution to adapt to motion distortion and optimizes attention feature perception through standard skip connection design; Step S24: motion joint deblurring, specifically performing edge sharpening on the deblurred optimized image output to obtain deblurred optimized image data.

4. The method for enhancing the image of a drone low-altitude photography according to claim 3, characterized in that: In step S3, the multi-source data registration is used to achieve pixel-level alignment of data. Specifically, based on the deblurred optimized image data, an elastic transformation improved registration method combined with cross-spectral feature attention matching is used to perform multi-source data registration to obtain pixel-aligned enhanced data, including the following steps: Step S31: cross-modal feature extraction, specifically generating feature depth map and feature intensity map reference data through laser radar point cloud projection based on the visible light image data in the deblurred optimized image data and the multispectral data in the drone low-altitude photography image optimization data, and performing cross-modal feature extraction by constructing a three-branch feature extraction network, and obtaining multi-source data registration feature data through feature fusion; Step S32: graph structure matching, specifically, based on the multi-source data registration feature data, using an improved linear iterative clustering algorithm to perform pixel segmentation, and constructing pixel node graph structure data through graph nodes and node features, and using a three-layer graph attention network to perform node similarity calculation to obtain cross-modal registration matching pair set data; The improved linear iterative clustering algorithm introduces a multi-spectral feature + cross-modal feature dual consistency optimization item on the basis of a standard simple linear iterative clustering algorithm to perform image pixel structure matching clustering optimization; Step S33: elastic transformation optimization, specifically, based on the cross-modal registration matching pair set data, an improved thin plate spline transformation solution algorithm is constructed to perform elastic transformation optimization to obtain resampled registration feature data; Step S34: multi-source data registration, specifically, obtaining pixel alignment enhanced data by sequentially performing local alignment mutual information check and adaptive iterative local alignment optimization based on the resampled alignment feature data.

5. The method for enhancing the image of a drone low-altitude photography according to claim 4, characterized in that: In step S4, the scene adaptive enhancement is used to intelligently adjust the enhancement strategy according to the image scene content, specifically, based on the pixel alignment enhancement data, a dynamic physical model coupling enhancement method combined with scene perception enhancement is used to perform scene adaptive enhancement to obtain scene adaptive enhanced image data, including the following steps: Step S41: scene parsing improvement, specifically, performing scene parsing by constructing a lightweight segmentation pre-training network and obtaining scene parsing reference data; Step S42: Quantitative analysis of regional characteristics, specifically, constructing feature indicators according to the specific scene type in the scene analysis reference data, and constructing a multi-scale feature fusion pyramid structure according to the feature indicators to perform quantitative analysis of regional characteristics, so as to obtain quantitative feature data of the scene region; Step S43: dynamic scene enhancement, specifically, constructing a classification enhancement sub-library based on the scene area quantitative feature data, and performing dynamic scene enhancement based on the classification enhancement sub-library and the scene type and scene feature quantization value in the scene area quantitative feature data to obtain scene dynamic preliminary enhancement data; Step S44: scene-adaptive image enhancement, specifically, based on the scene dynamic preliminary enhancement data, edge consistency processing and bilateral filtering smooth transition operations are used to perform comprehensive image adaptive enhancement to obtain scene-adaptive enhanced image data.

6. The method for enhancing the image of a drone low-altitude photography according to claim 5, characterized in that: In step S5, the low-altitude photography image enhancement is used to integrate the previous processing results to output the final enhanced image. Specifically, the low-altitude photography image is comprehensively enhanced based on the deblurred optimized image data, the pixel alignment enhancement data and the scene adaptive enhancement image data to obtain the comprehensive enhanced image data of the drone low-altitude photography.

7. A drone low-altitude photography image enhancement system, used to implement a drone low-altitude photography image enhancement method as claimed in any one of claims 1 to 6, characterized in that: It includes data preparation module, deblurring optimization module, multi-source data registration module, scene adaptive enhancement module and low-altitude photography image enhancement module.

8. The UAV low-altitude photography image enhancement system according to claim 7, characterized in that: The data preparation module is used for data preparation, and obtains the optimized data of the UAV low-altitude photography image through data preparation, and sends the optimized data of the UAV low-altitude photography image to the deblurring optimization module; The deblurring optimization module is used for motion joint deblurring optimization, obtains deblurred optimized image data through motion joint deblurring optimization, and sends the deblurred optimized image data to the multi-source data registration module and the low-altitude photography image enhancement module; The multi-source data registration module is used for multi-source data registration, obtains pixel alignment enhancement data through multi-source data registration, and sends the pixel alignment enhancement data to the scene adaptive enhancement module and the low-altitude photography image enhancement module; The scene adaptive enhancement module is used for scene adaptive enhancement, obtains scene adaptive enhanced image data through scene adaptive enhancement, and sends the scene adaptive enhanced image data to the low-altitude photography image enhancement module; The low-altitude photography image enhancement module is used for low-altitude photography image enhancement, and obtains comprehensive enhanced image data of UAV low-altitude photography through low-altitude photography image enhancement.

Images