An unmanned aerial vehicle imaging control method and device based on AI and flight control parameter fusion

Abstract

The application relates to an unmanned aerial vehicle imaging control method and device based on AI and flight control parameter fusion, which comprises the following steps: scene feature extraction under multiple scales is carried out by using historical imaging data to obtain image features, an AI scene adaptation strategy is constructed according to the image features, a light intensity prediction model is solved, and an optimal imaging parameter set corresponding to a camera is obtained; original image data photographed by the camera under the optimal imaging parameter set is acquired, the original image data is optimized based on the AI scene adaptation strategy, the flight state of the unmanned aerial vehicle is predicted based on flight control parameters to generate imaging pre-adjustment parameters, and the imaging pre-adjustment parameters are used to control the photographing of the unmanned aerial vehicle. By using AI to construct an AI scene adaptation strategy in combination with historical imaging data, the problem of poor scene adaptability is solved, and in addition, a light intensity prediction model is constructed in combination with flight control parameters, the light intensity prediction model is solved, and an optimal imaging parameter set corresponding to the camera is obtained, so that the defect of weak light adaptation capability is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a UAV imaging control method and device based on AI and flight control parameter fusion, an electronic device and a computer readable storage medium. BACKGROUND

[0002] Taking pictures using a camera on a UAV faces two core challenges in actual operation: one is the complex lighting environment (such as high-altitude strong light, low-altitude shadow, backlight, low light, etc.), and the other is the dynamic change of flight state (such as high-speed movement, hovering, attitude adjustment, etc.).

[0003] The traditional ISP system has the following key defects:

[0004] 1. Weak light adaptation: fixed aperture lens cannot cope with large dynamic range scenes, and aperture, exposure, and gain adjustment are independent, making it difficult to match the optimal image quality.

[0005] 2. Poor scene adaptability: relying on preset parameter templates, unable to identify scene types in real time (such as daytime HDR, low light, backlight). SUMMARY

[0006] The embodiments of the present application provide a UAV imaging control method and device based on AI and flight control parameter fusion to at least solve the problems of weak light adaptation and poor scene adaptability in related technologies.

[0007] In a first aspect, the embodiments of the present application provide a UAV imaging control method based on AI and flight control parameter fusion, comprising:

[0008] Obtaining historical imaging data of a camera, performing scene feature extraction on the historical imaging data based on an attention mechanism to obtain image features at multiple scales, and constructing an AI scene adaptation strategy according to the image features;

[0009] Obtaining real-time imaging data of the camera, constructing a lighting intensity prediction model in combination with the flight control parameters, solving the lighting intensity prediction model, and obtaining an optimal imaging parameter set corresponding to the camera;

[0010] Obtaining original image data taken by the camera under the optimal imaging parameter set, optimizing the original image data based on the AI scene adaptation strategy, and extracting optimization parameters in the optimization process;

[0011] Predicting the flight state of the UAV based on the flight control parameters, generating imaging pre-adjustment parameters for the UAV in combination with the optimization parameters, and controlling the shooting of the UAV using the imaging pre-adjustment parameters.

[0012] In an embodiment, the historical imaging data of the camera is acquired, scene features are extracted from the historical imaging data based on an attention mechanism to obtain image features at multiple scales, and an AI scene adaptation strategy is constructed according to the image features, including:

[0013] The historical imaging data is acquired, scene features are extracted from the historical imaging data using a convolution block attention module, and the image features are obtained by fusing the scene features at multiple scales;

[0014] The image processing parameters in the pre-constructed basic strategy library are adjusted based on the image features using AI, and the AI scene adaptation strategy corresponding to different scenes is obtained.

[0015] In an embodiment, the real-time imaging data of the camera is acquired, a light intensity prediction model is constructed in combination with the flight control parameters, and the optimal imaging parameter set corresponding to the camera is obtained by solving the light intensity prediction model, including:

[0016] The real-time imaging data is acquired from the camera, and real-time image brightness mean and ambient light sensor data are extracted from the real-time imaging data;

[0017] The flight height and current flight time of the unmanned aerial vehicle are extracted from the flight control parameters, and the light intensity prediction model is constructed in combination with the real-time image brightness mean and the ambient light sensor data;

[0018] A target function is constructed from three dimensions of picture signal-to-noise ratio, dynamic range, and color deviation, the light intensity prediction model is solved, and the optimal imaging parameter set corresponding to the optimal implementation imaging data is obtained.

[0019] In an embodiment, the original image data captured by the camera under the optimal imaging parameter set is acquired, and the original image data is optimized based on the AI scene adaptation strategy, and the optimization parameters in the optimization process are extracted, including:

[0020] The original image data captured by the camera under the optimal imaging parameter set is acquired, image features in the original image data are extracted, and the original scene corresponding to the original image data is determined according to the image features;

[0021] The optimization algorithm corresponding to the original scene is selected based on the AI scene adaptation strategy, and the algorithm parameters adapted to the current scene are generated through the optimization algorithm;

[0022] The optimization effect of the algorithm parameters is verified through a lightweight quality evaluation model, if the expected effect is not achieved, the algorithm parameters are reselected, and the algorithm parameters when the optimization effect of each frame of image meets the scene requirements are selected as the optimization parameters.

[0023] In an embodiment, the AI scene adaptation strategy selects an optimization algorithm corresponding to the original scene, generates algorithm parameters adapted to the current scene through the optimization algorithm, including:

[0024] Determining an optimization target according to the original scene;

[0025] Analyzing the optimization target based on the AI scene adaptation strategy, determining the image optimization step corresponding to the optimization target, and selecting the optimization algorithm corresponding to the original scene according to the image optimization step;

[0026] Generating algorithm parameters adapted to the current scene through the optimization algorithm.

[0027] In an embodiment, the flight state of the UAV is predicted based on the flight control parameters, and imaging pre-adjustment parameters for the UAV are generated in combination with the optimization parameters, and the imaging pre-adjustment parameters are used to control the shooting of the UAV, including:

[0028] Extracting GPS data, IMU data, and trajectory planning data of the UAV from the flight control parameters;

[0029] Predicting the flight state of the UAV based on the GPS data, the IMU data, and the trajectory planning data;

[0030] If the prediction result exceeds the flight state change threshold, the imaging pre-adjustment parameters are used to control the shooting of the UAV.

[0031] In an embodiment, the flight state of the UAV is predicted based on the GPS data, the IMU data, and the trajectory planning data, including:

[0032] Respectively pre-processing the GPS data, the IMU data, and the trajectory planning data;

[0033] Based on the UAV motion model, a state equation and an observation equation describing the dynamic change of the flight state and the observation relationship are constructed;

[0034] Using the state equation to predict the future position of the UAV, and using the observation residual obtained by the observation equation to correct the future position to obtain the flight state of the UAV at the future time.

[0035] In a second aspect, the embodiments of the present application provide a UAV imaging control device based on AI and flight control parameter fusion, including:

[0036] An adaptation strategy construction module is configured to obtain historical imaging data of a camera, perform scene feature extraction on the historical imaging data based on an attention mechanism to obtain image features at multiple scales, and construct an AI scene adaptation strategy based on the image features;

[0037] An imaging parameter set acquisition module is configured to obtain real-time imaging data of the camera, construct a light intensity prediction model in combination with the flight control parameters, solve the light intensity prediction model, and obtain an optimal imaging parameter set corresponding to the camera;

[0038] An optimization parameter extraction module is configured to obtain original image data captured by the camera under the optimal imaging parameter set, perform optimization on the original image data based on the AI scene adaptation strategy, and extract optimization parameters in the optimization process;

[0039] An imaging control module is configured to predict a flight state of the unmanned aerial vehicle based on the flight control parameters, generate imaging pre-adjustment parameters for the unmanned aerial vehicle in combination with the optimization parameters, and control photographing of the unmanned aerial vehicle using the imaging pre-adjustment parameters.

[0040] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the unmanned aerial vehicle imaging control method based on AI and flight control parameter fusion according to the first aspect.

[0041] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program executable by a processor, and the program implements the unmanned aerial vehicle imaging control method based on AI and flight control parameter fusion according to the second aspect.

[0042] The unmanned aerial vehicle imaging control method and device based on AI and flight control parameter fusion provided by the embodiments of the present application have at least the following technical effects.

[0043] By using AI to construct an AI scene adaptation strategy in combination with historical imaging data, the problem of poor scene adaptability in the prior art is solved, and by constructing a light intensity prediction model in combination with flight control parameters, solving the light intensity prediction model, and obtaining an optimal imaging parameter set corresponding to the camera, the defect of weak light adaptation capability in the prior art is solved.

[0044] Details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more clear and easy to understand. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0046] Figure 1 is a flowchart of a UAV imaging control method based on AI and flight control parameter fusion provided by the present application;

[0047] Figure 2 is a flowchart of step S10 according to the related art;

[0048] Figure 3 is a flowchart of step S20 according to the related art;

[0049] Figure 4 is a flowchart of step S30 according to the related art;

[0050] Figure 5 is a flowchart of step S40 according to the related art;

[0051] Figure 6 is a structural block diagram of a UAV imaging control device based on AI and flight control parameter fusion provided by the present application;

[0052] Figure 7 is a structural diagram of an electronic device according to the related art. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is described and explained below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. Based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0054] In a first aspect, the embodiments of the present application provide a UAV imaging control method based on AI and flight control parameter fusion, Figure 1 is a flowchart of the method, as Figure 1 shown, specifically comprising:

[0055] Step S10, obtaining historical imaging data of the camera, performing scene feature extraction on the historical imaging data based on an attention mechanism to obtain image features at multiple scales, and constructing an AI scene adaptation strategy according to the image features.

[0056] Step S20, real-time imaging data of the camera is obtained, a light intensity prediction model is constructed in combination with the flight control parameters, the light intensity prediction model is solved, and an optimal imaging parameter set corresponding to the camera is obtained.

[0057] Step S30, the original image data photographed by the camera under the optimal imaging parameter set is obtained, and the original image data is optimized based on an AI scene adaptation strategy, and an optimization parameter in the optimization process is extracted.

[0058] Step S40, the flight state of the unmanned aerial vehicle is predicted based on the flight control parameters, and an imaging pre-adjustment parameter for the unmanned aerial vehicle is generated in combination with the optimization parameter, and the imaging pre-adjustment parameter is used to control the photographing of the unmanned aerial vehicle.

[0059] By using AI to construct an AI scene adaptation strategy in combination with historical imaging data, the problem of poor scene adaptability in the prior art is solved. In addition, by constructing a light intensity prediction model in combination with the flight control parameters, solving the light intensity prediction model, and obtaining an optimal imaging parameter set corresponding to the camera, the defect of weak light adaptability in the prior art is solved.

[0060] Figure 2 is a flowchart of step S10 according to an exemplary embodiment, as shown in Figure 2 , step S10 includes:

[0061] Step S101, historical imaging data is obtained, scene features are extracted from the historical imaging data using a convolution block attention module, and image features are obtained by fusing the scene features at multiple scales.

[0062] The traditional CNN model is easily disturbed by complex backgrounds (such as clouds in high-altitude scenes and tree blocks in low-altitude scenes) when extracting unmanned aerial vehicle scene features, which causes the key features of the scene to be submerged. The present technology adopts a convolution block attention module (CBAM module), which realizes accurate capture of key scene features through the synergistic effect of channel attention and spatial attention. The specific process is as follows:

[0063] (1) Channel attention mechanism: screening scene core feature channels

[0064] The channel attention module learns the importance weight of different channels, strengthens the feature channels related to scene recognition (such as the blue channel of a water surface scene and the brightness channel of a snow-covered scene), and suppresses irrelevant channel interference. The specific implementation includes three steps:

[0065] Global Average Pooling: Global average pooling is performed on the feature map output by the convolutional layer (dimension CxHxW, C is the number of channels, H is the height, and W is the width), resulting in a channel feature vector with dimension Cx1x1, which retains the global information of each channel.

[0066] Multi-Layer Perceptron (MLP) Mapping: The channel feature vector is input into an MLP composed of two fully connected layers. The first layer compresses the dimension to C / 4 (reduces computational load), and the second layer restores it to C dimension. The sigmoid function outputs the weight of each channel (range 0-1).

[0067] Channel Weight Assignment: The learned channel weight is multiplied with the original feature map channel by channel to strengthen the high-weight channel.

[0068] In view of the particularity of the unmanned aerial vehicle scene, the channel attention is optimized in two aspects:

[0069] Dynamic Channel Pruning: By setting a weight threshold (such as 0.3), channels with weights below the threshold are automatically shielded, reducing invalid channel calculation in high-altitude cloud and fog scenes.

[0070] Scene Adaptive Weight Initialization: Based on the preliminary flight state transmitted by the flight control (such as height > 500m to determine high-altitude scene), the initial values of channel weights are set for typical scenes such as high-altitude, low-altitude, and high-speed flight, shortening the MLP convergence time and reducing the delay of feature extraction during scene switching.

[0071] (2) Spatial Attention Mechanism: Locating key areas in the scene

[0072] The spatial attention module focuses on key spatial positions in the feature map (such as moving objects in high-speed flight scenes and target subjects in hovering scenes). Spatial feature enhancement is achieved through the following steps:

[0073] Channel Pooling Fusion: After processing the feature map with channel attention, global average pooling and global maximum pooling are performed respectively to obtain two HxWx1 feature maps, which are then concatenated into an HxWx2 feature map.

[0074] Convolution Compression and Activation: The concatenated feature map is compressed by a 7x7 convolution kernel (compared to the traditional 3x3 convolution kernel, which is more suitable for capturing large-scale features in unmanned aerial vehicle scenes such as tree distribution in forest scenes), reducing the number of channels to 1, and then outputting a spatial attention weight map through the sigmoid function.

[0075] Spatial Weight Assignment: The spatial weight map is multiplied with the feature map pixel by pixel to strengthen the regions with weight higher than 0.6 (such as the reflective area of water surface scenes and the outline area of urban building scenes).

[0076] Actual application effect: In the backlight flying scene, the spatial attention module can increase the feature response value of the ground target area by 3-4 times, effectively solving the ground target missing detection problem caused by strong light interference of traditional models; in the low-altitude penetration scene, the feature capture accuracy of obstacles (such as power poles) is increased to 98.5%, which is 12% higher than the traditional CBAM algorithm.

[0077] (3) Multi-scale feature fusion: Adapt to scene details at different flight altitudes

[0078] At different flight altitudes, the scale of scene features varies greatly (e.g. in high-altitude scenes, buildings appear as small-scale pixel blocks; in low-altitude scenes, building details are clear). To solve this problem, the algorithm introduces a multi-scale feature fusion module, which is implemented as follows:

[0079] Feature pyramid construction: Extract feature maps at different stages of the MobileNetV3 backbone network (e.g. shallow, middle, and deep layers). Shallow feature maps (high resolution) retain scene details (e.g. window textures in low-altitude scenes), and deep feature maps (low resolution) retain global scene information (e.g. terrain contours in high-altitude scenes);

[0080] Cross-layer attention fusion: After applying the CBAM attention mechanism to feature maps of different scales, upsample (shallow feature maps) or downsample (deep feature maps) to the same resolution, and then implement fusion through weighted summation;

[0081] Edge feature enhancement: In the fused feature map, extract edge features using the Sobel operator, and overlay them with the fused feature map. In high-speed flying scenes, this can improve the edge sharpness of moving objects.

[0082] Step S102, using AI to adjust the image processing parameters in the pre-constructed basic strategy library based on image features, to obtain AI scene adaptation strategies corresponding to different scenes.

[0083] Multi-label classification algorithm is used to identify complex scenes and output image processing strategies for scene adaptation. For example, when identifying "snow scene", the model will output the optimization instruction "reduce contrast, increase blue channel gain to restore the pure white quality of snow"; when identifying "high-speed flying scene", output the instruction "enhance edge sharpening, increase dynamic blur suppression strength", the specific process is as follows:

[0084] (1) Multi-label classification model: Composite scene recognition based on BCEWithLogitsLoss

[0085] Traditional multi-label classification algorithms are prone to label redundancy (e.g., "snow" and "high brightness" labels are highly related) when dealing with unmanned aerial vehicle (UAV) scenes, leading to decreased classification accuracy. This algorithm addresses this issue through the following optimizations:

[0086] Label system construction: 12 basic scenes are divided into 4 dimensions (flight state: high speed / hover / low-altitude traversal; lighting condition: backlight / weak light / strong light; environment type: snow / water / forest / urban building; weather condition: cloud / fog / sand), each dimension contains multiple mutually exclusive labels, reducing label redundancy;

[0087] Loss function optimization: binary cross-entropy loss combined with label correlation penalty term, formula as follows:

[0088] Total loss = Loss (predicted label, true label) + λ × ∑ (label correlation matrix × predicted label error²)

[0089] Where λ is the penalty coefficient (value 0.3), the label correlation matrix is obtained by calculating the label co-occurrence probability of 1 million + samples, reducing the misjudgment rate of related labels;

[0090] Flight control data assisted classification: real-time flight control data (flight speed, height, attitude angle) are used as additional input, converted to a 16-dimensional feature vector through a fully connected layer, and then input into the classifier after being concatenated with image features. For example, when the flight speed is > 50 km / h, the prediction probability of the "high-speed flight" label is automatically increased by 0.3, reducing classification errors caused by image blur.

[0091] (2) Dynamic adaptation strategy generation: based on rule base and AI inference

[0092] To avoid the disconnection between classification results and image processing strategies, a scene-strategy mapping rule base is constructed, and an accurate adaptation strategy is generated through AI inference. The specific implementation is divided into two steps:

[0093] Basic strategy library construction: for each basic scene and composite scene, predefine the image processing parameter adjustment range (e.g., snow scene: contrast reduction 10%-20%, blue channel gain increase 5%-15%; high-speed flight scene: edge sharpening intensity increase 20%-30%, dynamic blur suppression coefficient set to 0.7-0.9). The parameter range is determined through 5000+ comparative experiments to ensure optimization effect;

[0094] AI dynamically adjusts inference: Based on the image features of the current image (such as noise intensity, blur level, and color deviation), the basic policy is fine-tuned through a reinforcement learning model. The reward function for reinforcement learning is set to "maximize the image quality score (PSNR + SSIM + ΔE)," where PSNR (Peak Signal-to-Noise Ratio) has a weight of 0.4, SSIM (Structural Similarity) has a weight of 0.4, and ΔE (Color Deviation) has a weight of 0.2. For example, when there are local shadows in a snow scene, the model automatically increases the contrast of the shadow area to avoid losing details in the shadow area.

[0095] Figure 3 This is a flowchart illustrating step S20 according to an exemplary embodiment, such as... Figure 3 As shown, step S20 includes:

[0096] Step S201: Obtain real-time imaging data from the camera, and extract the average real-time image brightness and ambient light sensor data from the real-time imaging data.

[0097] Step S202: Extract the UAV's flight altitude and current flight time from the flight control parameters, and construct a light intensity prediction model by combining the real-time image brightness average and ambient light sensor data.

[0098] Light intensity prediction model: This model employs an algorithm combining linear regression and environmental factor compensation. Input parameters include real-time image brightness values ​​(mean value of the Y channel), ambient light sensor data, flight altitude, and time (e.g., noon / dusk). The output is the corrected actual light intensity. The formula is as follows:

[0099] Actual illumination intensity = α × mean image brightness + β × ambient light sensor data + γ × flight altitude compensation + δ × time compensation

[0100] α, β, γ, and δ are weight coefficients obtained through training with massive amounts of scene data, with the error controlled within 5%.

[0101] Step S203: Construct an objective function from three dimensions: image signal-to-noise ratio, dynamic range, and color deviation, solve the illumination intensity prediction model, and obtain the optimal imaging parameter set corresponding to the optimal implementation imaging data.

[0102] By establishing a mapping model of illumination intensity and imaging parameters, the limitations of traditional ISP aperture-exposure-gain independent adjustment are broken. The algorithm first obtains real-time illumination data (such as pixel brightness average, dynamic range proportion) through the image sensor, then combines the current flight height of the UAV (GPS data from the flight control parameters) to correct the illumination intensity (for example, high altitude ultraviolet is strong, the gain needs to be reduced to avoid picture whitening), and finally outputs the optimal imaging parameter set containing the optimal aperture, exposure time (1 / 100s-1 / 10000s adjustable) and ISO gain (100-6400 adjustable).

[0103] Three-dimensional parameter optimization algorithm: based on genetic algorithm to realize multi-objective optimization, taking "the highest picture signal-to-noise ratio, the largest dynamic range, and the smallest color deviation" as the objective function, quickly searching for the optimal solution in the parameter space of aperture, exposure, and gain. The iteration number of the algorithm is controlled within 10 times, and the time consumption of single calculation is ≤10 ms, meeting the real-time adjustment requirement.

[0104] The "optimal imaging parameter initial value" (aperture, exposure time, ISO gain) output in this step will be used as the "basic parameter input" in the subsequent imaging step, reducing the scene misjudgment caused by poor original image quality.

[0105] Figure 4 is a flowchart of step S30 according to an exemplary embodiment, as shown in Figure 4 as shown, step S30 includes:

[0106] Step S301, acquiring the original image data shot by the camera under the optimal imaging parameter set, extracting the image features in the original image data, and determining the original scene corresponding to the original image data according to the image features.

[0107] Bad point repair (based on neighborhood interpolation algorithm) and dark corner correction (based on lens distortion model) are performed on the image to eliminate image defects caused by hardware;

[0108] Step S302, selecting the optimization algorithm corresponding to the original scene based on the AI scene adaptation strategy, and generating the algorithm parameters adapted to the current scene through the optimization algorithm.

[0109] Specifically includes:

[0110] 1) Determine the optimization target according to the original scene.

[0111] 2) Analyze the optimization target based on the AI scene adaptation strategy, determine the image optimization step corresponding to the optimization target, and select the optimization algorithm corresponding to the original scene according to the image optimization step.

[0112] 3) Generate algorithm parameters adapted to the current scene through the optimization algorithm.

[0113] Weak light scene noise suppression: BM3D (block matching 3D filter) algorithm is used, combined with AI-generated noise model (based on current ISO gain and light intensity), to perform block matching and 3D transform filtering on the image, while suppressing noise and preserving details. Compared with traditional Gaussian filtering, the peak signal-to-noise ratio (PSNR) is improved by 2-3 dB.

[0114] High-speed flight scene blur suppression: A motion blur recovery algorithm based on optical flow estimation is used. The LK optical flow algorithm is used to calculate the motion vector of adjacent frames to determine the blur kernel size and direction. Then, Wiener filtering is used to recover the blur, which can reduce the image blur caused by high-speed flight (speed > 50 km / h) by more than 60%.

[0115] Color distortion correction: A scene-based white balance algorithm is used. For example, in a snow scene, the RGB ratio of the white area is identified to adjust the red and blue channel gain to avoid blue bias in snow images. In a city building scene, the green channel is enhanced to restore the color of vegetation, and the color deviation (ΔE) is controlled within 3, meeting professional image standards.

[0116] Step S303: Verify the optimization effect of the algorithm parameters through the lightweight quality evaluation model. If the expected result is not achieved, reselect the algorithm parameters. Select the algorithm parameters that meet the scene requirements for each frame of image as the optimization parameters.

[0117] Based on the AI-driven scene recognition output "scene adaptation strategy", different scene image problems (such as weak light noise, high-speed blur, color distortion) are addressed using scene-based image processing algorithms, achieving an upgrade from "general optimization" to "precise optimization". The processing flow is divided into three stages: preprocessing, core optimization, and post-processing.

[0118] The core optimization stage is the core of intelligent image processing technology, which takes over the "hardware defect repair" in the preprocessing stage, and according to the "scene type + adaptation strategy instruction" output by the AI-driven scene recognition technology, realizes the key transformation from "general image processing" to "scene customized optimization". Its working logic follows the closed-loop process of "strategy analysis-algorithm matching-parameter optimization-effect verification": First, the scene adaptation strategy output by AI is structurally analyzed to extract the "core optimization target" (such as "noise reduction priority" in weak light scenes, "color restoration priority" in snow scenes) and "algorithm call list"; then, according to the real-time features of the image (such as noise intensity, color deviation value), the corresponding optimization algorithm module is matched; then, the algorithm parameters suitable for the current scene are generated through dynamic parameter optimization algorithm; finally, the optimization effect is verified through the lightweight quality evaluation model (such as the simplified SSIM calculation), and if the expected result is not achieved, the parameters are adjusted again to ensure that the optimization effect of each frame of image meets the scene requirements.

[0119] Core optimization implementation of typical scenarios:

[0120] 1. Weak light scene (illumination intensity <10 lux, such as night aerial photography, tunnel inspection)

[0121] Scene adaptation strategy analysis: AI scene recognition outputs "core goal: suppress noise + improve brightness + retain details; recommended algorithm: hierarchical noise reduction algorithm + adaptive brightness enhancement algorithm + detail compensation algorithm".

[0122] Algorithm invocation and parameter tuning:

[0123] Step 1: Hierarchical noise reduction algorithm:

[0124] Principle: According to the characteristics of weak light scenes, "dense noise in dark areas, easy to lose details in bright areas", the image is divided into dark area (pixel brightness value <50), medium bright area (50 ≤ brightness value <150), and bright area (brightness value ≥150) according to brightness, and different noise reduction strategies are adopted.

[0125] Operation details: Dark area uses 3D block matching filter (BM3D), sets block size to 8x8 pixels and search window to 24x24 pixels, suppresses Gaussian noise and salt and pepper noise by stacking similar blocks from multiple frames, and noise suppression intensity is improved by 40% compared to traditional Gaussian filter; medium bright area uses bilateral filter to smooth particle noise while retaining edge details, with spatial domain standard deviation set to 1.5 and gray domain standard deviation set to 20; bright area uses guided filter to avoid excessive filtering that causes detail blur.

[0126] Dynamic parameter tuning: Adjust filter parameters according to current ISO gain value (e.g. ISO=6400, dark noise reduction intensity increases by 20%), to ensure the balance between noise reduction effect and detail retention.

[0127] Step 2: Adaptive brightness enhancement algorithm:

[0128] Principle: Multi-scale brightness enhancement based on Retinex theory is used to separate the "brightness component" and "color component" of the image, and only the brightness component is enhanced to avoid color distortion.

[0129] Operation details: Decompose the brightness component into 5 scales through Gaussian pyramid, perform non-linear stretching on the dark area of each scale (stretching coefficient = (255-current brightness) / 255), and then reconstruct the enhanced brightness component through pyramid reconstruction; meanwhile, introduce a brightness constraint factor to avoid local overexposure (such as light area), when the enhanced brightness value >230, automatically reduce the stretching coefficient to 0.5.

[0130] Effect verification: The average brightness value of the enhanced image is increased from 30 to 80-100, and the overexposure area ratio is controlled within 3%.

[0131] Step 3: Detail compensation algorithm:

[0132] Principle: By extracting the high-frequency details of the original image (such as building outlines, tree textures), it is superimposed on the enhanced image to make up for the loss of details in the noise reduction and brightness enhancement process.

[0133] Operation details: Use Laplace operator to extract high-frequency detail map of the original image, set operator coefficient to [0,-1,0;-1,5,-1;0,-1,0] to enhance edge contrast; fuse the high-frequency detail map with the enhanced image according to the weight of 0.3:0.7, so that the dark details (such as night building windows) improve the clarity by 30%.

[0134] 2. Snow scene

[0135] Scene adaptation strategy analysis: AI scene recognition outputs "core goal: correct snow white / brown + improve contrast + suppress reflection; recommended algorithm: adaptive white balance algorithm + regional contrast enhancement algorithm + reflection suppression algorithm".

[0136] Algorithm calling and parameter tuning:

[0137] Step 1: Adaptive white balance algorithm:

[0138] Principle: For the problem of "white area blue bias (due to snow reflection of blue light)" in snow scene, by identifying the "pure white reference point" in the image, adjust the RGB channel gain to restore the true white of the snow.

[0139] Operation details: Use the combination of "gray world method + white point detection", first calculate the RGB channel mean value of the image, if the blue channel mean value is higher than the red channel by 20 or more (judged as blue bias), start white point detection; determine the pure white reference point through threshold screening (brightness value>200, RGB difference<15), calculate the RGB mean value of the reference point (such as R=220, G=225, B=235), adjust the blue channel gain=R mean / B mean=220 / 235≈0.93, red channel gain=1.05, green channel gain=1.0, so that the RGB value of the pure white area tends to be consistent (such as R=225, G=225, B=225).

[0140] Step 2: Regional contrast enhancement algorithm:

[0141] Principle: Snow scene is prone to "low overall contrast, fuzzy boundary between ground objects and snow", so it is necessary to enhance the local details by partitioning the contrast.

[0142] Operation details: Divide the image into 16x16 sub-blocks, calculate the contrast of each sub-block (contrast = (max-min) / (max+min)), linear stretch for sub-blocks with contrast <0.1 (such as large area of snow), the stretching range from [50,200] to [30,220]; for sub-blocks with contrast ≥0.1 (such as rocks in the snow, vegetation), keep the original contrast, avoid distortion caused by over-enhancement.

[0143] Third step: Anti-reflection algorithm:

[0144] Principle: For the strong reflection area of snow surface (such as snow under sunlight, brightness value >240), through dynamic adjustment of exposure parameters and pixel replacement, reduce the reflection interference.

[0145] Operation details: Threshold segmentation (brightness value >240) is used to determine the reflection area, and the pixel value of the reflection area is replaced, the replacement formula =240-(original brightness-240)×0.5; at the same time, feedback to the automatic aperture control module, fine-tune the aperture position (such as from F2.8 to F4.0), reduce the reflection area of subsequent frames.

[0146] 3. Urban building scene (such as urban planning aerial photography, high-rise inspection)

[0147] Scene adaptation strategy analysis: AI scene recognition outputs "core target: strengthen building edge + correct perspective distortion + optimize color restoration; recommended algorithm: edge enhancement algorithm + perspective distortion correction algorithm + color saturation optimization algorithm".

[0148] Algorithm calling and parameter tuning:

[0149] First step: Edge enhancement algorithm:

[0150] Principle: Multi-threshold edge detection and enhancement based on Canny operator is used to highlight the straight edges of buildings (such as wall outlines, window frames), and to improve the structural clarity of the image.

[0151] Operation details: First, smooth the image noise through Gaussian filtering (standard deviation =1.0), then calculate the gradient amplitude and direction of the image; set double threshold (high threshold =80, low threshold =40) to filter edge pixels, high threshold edge is directly retained, low threshold edge is retained if it is connected with high threshold edge; finally, enhance the edge width through morphological dilation (structure element =3x3 rectangle), expand the edge pixels from 1 pixel to 2-3 pixels, and the visual clarity is improved by 25%.

[0152] Parameter dynamic tuning: Adjust the threshold according to the flight height (such as when the height <50m, the low threshold is reduced to 30, to capture more window details).

[0153] Second step: perspective distortion correction algorithm:

[0154] Principle: For the perspective distortion caused by the tilt shooting of the UAV, the distortion matrix is calculated through the camera intrinsic parameters (focal length, pixel size) and flight attitude angles (pitch angle, roll angle), and real-time correction is achieved.

[0155] Operation details: Obtain the intrinsic matrix of the current camera (such as focal length f=12mm, pixel size s=1.12μm) and attitude angle (such as pitch angle=30°) from the flight control system, establish a perspective projection model; through inverse projection transformation, the pixel coordinates of the distorted image are converted into ideal coordinates, and the pixel values are filled through bilinear interpolation; the verticality error of the corrected building is reduced from 15° to within 3°, meeting the surveying and mapping accuracy requirements (such as 1:500 topographic map standard).

[0156] Third step: color saturation optimization algorithm:

[0157] Principle: In urban building scenes, vegetation and wall color are easily affected by light changes (such as overcast, backlight) and have low saturation. Through saturation adjustment in HSV color space, the true color is restored.

[0158] Operation details: Convert the image from RGB space to HSV space, extract the saturation component (S channel); calculate the mean and standard deviation of the S channel, if the mean <0.3 (low saturation), perform nonlinear enhancement (enhancement coefficient=1.2+(0.3-mean)×0.5); at the same time, set the upper limit of saturation (S≤0.8) to avoid color overflow (such as red wall over-saturation to pure color block).

[0159] Algorithm coordination and efficiency guarantee in core optimization stage

[0160] 1. Algorithm modularization and parallel computing

[0161] To meet the real-time processing requirements (30fps), all algorithms in the core optimization stage are designed with modularization, encapsulated as independent algorithm modules (such as noise reduction module, brightness enhancement module, edge enhancement module), and implemented through GPU parallel computing framework (such as CUDA) for multi-module parallel processing. For example, in low-light scenes, the layered noise reduction module and the brightness enhancement module can be started simultaneously, processing different areas of the image respectively, and the single-frame processing time is shortened from 50ms in serial computation to within 20ms.

[0162] 2. Cross-module parameter linkage

[0163] The parameters between different algorithm modules are linked to avoid parameter conflicts and reduce the optimization effect. For example, in the urban building scene, the edge threshold of the edge enhancement module is linked to the interpolation accuracy of the perspective distortion correction module. When the corrected image resolution changes (e.g., from 4K to 2K), the edge threshold is automatically adjusted (from 80 to 60), ensuring stable edge detection effect.

[0164] 3. Lightweight quality evaluation and feedback

[0165] After each step of algorithm processing in the core optimization stage, a lightweight quality evaluation model is embedded to quickly verify the effect by calculating key indicators (such as PSNR value after noise reduction, SSIM value after enhancement). If PSNR < 28 dB (poor noise reduction effect), automatically increase the filter strength of the noise reduction module; if SSIM < 0.8 (insufficient detail retention), reduce the filter parameter, forming a closed loop of "processing-evaluation-feedback-adjustment", and the optimization effect compliance rate is improved to more than 98%.

[0166] The technology receives the "scene adaptation strategy" output by the AI-driven scene recognition, and after completing the image optimization, transmits the "optimized image data" and "parameter feedback in the image processing process" (such as noise intensity, blur degree) to the flight control data fusion optimization technology, providing data support for subsequent parameter pre-adjustment.

[0167] Figure 5 is a flowchart of step S40 according to an example embodiment, as shown in Figure 5 , step S40 includes:

[0168] Step S401, extract the GPS data, IMU data and trajectory planning data of the unmanned aerial vehicle from the flight control parameters.

[0169] Step S402, predict the flight state of the unmanned aerial vehicle based on the GPS data, IMU data and trajectory planning data.

[0170] Specifically includes:

[0171] 1) Preprocess the GPS data, IMU data and trajectory planning data respectively;

[0172] 2) Based on the unmanned aerial vehicle motion model, construct state equations and observation equations describing the dynamic changes of the flight state and the observation relationship;

[0173] 3) Use the state equation to predict the future position of the unmanned aerial vehicle, and use the observation residual obtained by the observation equation to correct the future position, to obtain the flight state of the unmanned aerial vehicle at the future time.

[0174] During the flight of the UAV, single sensor data is susceptible to interference (such as GPS positioning drift in high-rise sheltered areas, IMU cumulative error due to temperature drift), resulting in inaccurate flight state judgment, which in turn affects the accuracy of ISP parameter pre-adjustment. By using the Extended Kalman Filter (EKF) algorithm, GPS, IMU and trajectory planning multi-source data can be fused to achieve high-precision estimation and short-time prediction of flight state, providing a reliable basis for ISP parameter pre-adjustment. The core logic is to correct sensor errors in real time through a "prediction-update" closed loop, output the optimal flight state estimation value, and predict the flight state change in the next 100-200 ms based on the estimation model. The specific implementation process is as follows:

[0175] Multi-source data characteristic analysis and preprocessing

[0176] Firstly, the characteristics of the three types of fused data are analyzed, and targeted preprocessing is carried out to ensure data consistency and effectiveness:

[0177] GPS data (position, speed):

[0178] Characteristics: Low sampling frequency (usually 1-10 Hz), but high long-term accuracy (horizontal positioning error ±1-3 m), susceptible to shelter (such as urban canyons, forests) leading to data interruption or drift (drift error up to 5-10 m).

[0179] Preprocessing: "Sliding window mean filtering" is used to remove instantaneous drift, and the window size is set to 5 frames. When the deviation of consecutive 3 frames is >5 m, it is determined that the GPS is invalid, and the state estimation dominated by IMU is automatically switched. At the same time, the WGS84 coordinate system (latitude, longitude, altitude) output by GPS is converted to the UAV body coordinate system (x-axis: flight direction, y-axis: horizontal perpendicular to flight direction, z-axis: vertical to ground), which is convenient for alignment with IMU data.

[0180] IMU data (acceleration, angular velocity):

[0181] Characteristics: High sampling frequency (usually 100-1000 Hz), short-term response fast (can capture millisecond-level attitude changes), but has temperature drift (zero offset error changes with temperature, such as angular velocity zero offset ±0.1-1° / h) and cumulative error (position error up to 10-20 m / min after long-time integration).

[0182] Preprocessing: Zero offset error is corrected by "temperature compensation model" (based on IMU built-in temperature sensor data, query pre-calibrated temperature-zero offset mapping table); "high-pass filter" (cutoff frequency 0.1 Hz) is used to remove the gravity component interference of the accelerometer, retaining the motion acceleration; "low-pass filter" (cutoff frequency 10 Hz) is applied to the gyroscope data to suppress high-frequency noise (such as noise caused by body vibration).

[0183] Trajectory planning data (e.g., for an upcoming low-altitude crossing scenario):

[0184] Features: Outputted by the path planning module of the UAV flight control system, it includes the expected flight path for the next 5-10 seconds (such as altitude changes, turning angles, and speed planning). The data has high determinism, but it needs to be matched with the real-time flight status.

[0185] Preprocessing: Extract key state nodes from trajectory planning (such as the aircraft will drop from 100m to 80m in the next 100ms, or turn 30° in the next 200ms), and convert them into expected position, velocity, and acceleration target values ​​in the fuselage coordinate system. These values ​​serve as the "prior expectation" input for Kalman filtering, thereby improving prediction accuracy.

[0186] Based on the UAV kinematic model, a Kalman filter-based state equation and observation equation are constructed to describe the dynamic changes in flight state and their relationship with observation.

[0187] State vector definition:

[0188] Select a 6-dimensional state vector X=[x,vx,ax,y,vy,ay,z,vz,az]^T in the fuselage coordinate system of the UAV, where: x / y / z: position (unit: m), corresponding to the flight direction, horizontal and vertical flight direction, and vertical ground direction, respectively;

[0189] vx / vy / vz: velocity (unit: m / s), corresponding to the instantaneous velocity in the three directions;

[0190] ax / ay / az: Acceleration (unit: m / s²), corresponding to the acceleration in three directions (gravitational component removed).

[0191] This state vector covers three key dimensions: position, velocity, and acceleration, and can comprehensively reflect the flight state that affects ISP parameters (such as altitude changes affecting light intensity and velocity changes affecting image blur).

[0192] State equations: Based on Newton's kinematics formulas, the state transition equations in discrete time are established:

[0193] X(k|k-1)=F(k)×X(k-1|k-1)+B(k)×u(k)+w(k)

[0194] Where: X(k|k-1): the state prediction value at time k based on time k-1;

[0195] F(k): State transition matrix (9x9), describing the dynamic relationship between states, such as position x(k) = x(k-1) + vx(k-1) x At + 0.5 x ax(k-1) x At2 (At is the filter period, set to 10 ms, consistent with the IMU sampling period);

[0196] B(k): Control input matrix (9x3), where u(k) is the acceleration measurement output by the IMU, used to update the acceleration component in the state vector;

[0197] w(k): Process noise vector, following a Gaussian distribution N(0, Q(k)), Q(k) is the process noise covariance matrix, dynamically adjusted according to sensor accuracy (e.g., when GPS fails, increase the noise variance of the position component from 0.1² to 1.0²).

[0198] Observation equation: Establish the mapping relationship between observation and state vector, take multi-source data as observation input:

[0199] Z(k) = H(k) x X(k|k-1) + v(k)

[0200] Where: Z(k): observation vector at time k, including GPS position (x_gps, y_gps, z_gps), velocity (vx_gps, vy_gps, vz_gps), and expected position (x_plan, y_plan, z_plan) of trajectory planning, a total of 9 dimensions;

[0201] H(k): Observation matrix (9x9), realizing the mapping from state vector to observation value (e.g., GPS position observation value directly corresponds to x / y / z component in state vector, H matrix corresponding to position row is set to [1, 0, 0, 0, 0, 0, 0, 0, 0], the rest is 0);

[0202] v(k): Observation noise vector, following a Gaussian distribution N(0, R(k)), R(k) is the observation noise covariance matrix, dynamically adjusted according to data reliability (e.g., when GPS signal is good, position observation noise variance is set to 0.5²; when signal is weak, increase to 5.0²; noise variance of trajectory planning data is fixed at 0.1², because of its high certainty).

[0203] (1) Prediction stage: predict the current state based on historical state and motion model

[0204] Step 1: State prediction:

[0205] Based on the state equation, the state value X(k|k-1) at time k is predicted using the optimal state estimate X(k-1|k-1) at time k-1. For example, if the x-direction position x(k-1) = 100m, velocity vx(k-1) = 10m / s, acceleration ax(k-1) = 0.5m / s², and Δt = 0.01s at time k, then the predicted position x(k|k-1) at time k is 100 + 10 × 0.01 + 0.5 × 0.5 × (0.01)² ≈ 100.100025m, the predicted velocity x(k|k-1) is 10 + 0.5 × 0.01 = 10.005m / s, and the predicted acceleration ax(k|k-1) is 0.5m / s² (assuming no change in control input).

[0206] Step 2: Predicting the process noise covariance:

[0207] The process noise covariance matrix at time k is calculated as P(k|k-1) = F(k) × P(k-1|k-1) × F(k)^T + Q(k), where P(k-1|k-1) is the state estimation covariance matrix at time k-1, reflecting the uncertainty of the state estimation. The smaller the variance, the lower the uncertainty.

[0208] (2) Update phase: Correct the predicted state by combining multi-source observation data.

[0209] Step 1: Calculate the Kalman gain:

[0210] The Kalman gain K(k) = P(k|k-1)×H(k)^T×[H(k)×P(k|k-1)×H(k)^T+R(k)]^(-1) serves to balance the uncertainty of the predicted state (P matrix) with the uncertainty of the observed data (R matrix), determining the weight of the observed data in the state correction. For example, when the GPS signal is good (small variance of the position component of the R matrix), the gain of the K matrix at the corresponding position increases, and the contribution of the observed data to the position correction is higher; when GPS fails (extremely large variance of the position component of the R matrix), the gain of the K matrix at the corresponding position approaches 0, and correction mainly relies on IMU and trajectory planning data.

[0211] Step 2: State estimation update:

[0212] The predicted state is corrected according to the observation residual (the difference between the observation value and the predicted observation value) to obtain the optimal state estimation value X(k|k) at time k = X(k|k-1) + K(k) x [Z(k) - H(k) x X(k|k-1)]. For example, if the GPS observation x-direction position Z_x(k) = 100.1 m, the predicted observation value H_x x X(k|k-1) = 100.100025 m, and the observation residual is -0.000025 m, if K_x = 0.8 (GPS signal is good), then the corrected x position X_x(k|k) = 100.100025 + 0.8 x (-0.000025) = 100.100005 m, which is closer to the GPS observation value, while retaining the dynamic response advantage of the IMU.

[0213] Step 3: State estimation covariance update:

[0214] The state estimation covariance matrix P(k|k) at time k is updated as P(k|k) = [I - K(k) x H(k)] x P(k|k-1), where I is the identity matrix, to ensure that the uncertainty of the P matrix is reduced (variance is reduced) after each update, and the state estimation accuracy is gradually improved.

[0215] Step S403, if the prediction result exceeds the flight state change threshold, the imaging pre-adjustment parameter is used to control the shooting of the unmanned aerial vehicle.

[0216] After obtaining the optimal state estimation value X(k|k) at the current time (k), based on the unmanned aerial vehicle kinematics model and the trajectory planning data, the flight state at future 100 ms (k+10 time) and 200 ms (k+20 time) is predicted to provide a time window for ISP parameter pre-adjustment:

[0217] Prediction model selection: a "constant acceleration model" is used for short-term prediction (100-200 ms belongs to short time scale, and acceleration change can be ignored), and the prediction formula is:

[0218] X_pred(k+t|k) = X(k|k) + V(k|k) x t + 0.5 x A(k|k) x t²

[0219] Where t is the prediction time (t = 0.1 s or 0.2 s), X_pred is the predicted position, V is the current speed, and A is the current acceleration (all from the optimal state estimation at time k).

[0220] If the trajectory planning data contains future acceleration changes (such as deceleration in the future 100 ms, and the acceleration changes from 0.5 m / s² to -0.3 m / s²), a "piecewise acceleration model" is used, and the expected acceleration of the trajectory planning is introduced during prediction to improve the prediction accuracy.

[0221] Prediction accuracy verification and error control: Verify the prediction accuracy through historical data backtesting. Under a 100ms prediction length:

[0222] Position prediction error: ≤0.1m in horizontal direction (x / y axis) and ≤0.05m in vertical direction (z axis, height), meeting the accuracy requirement of ISP parameter pre-adjustment for height changes (e.g. when height changes >10m, the light parameter needs to be adjusted, and 0.05m error is negligible);

[0223] Speed prediction error: ≤0.1m / s, which can accurately determine whether the UAV is in high-speed flight (speed >50km / h ≈13.89m / s), providing a basis for pre-adjustment of the dynamic blur suppression algorithm;

[0224] When the prediction error > threshold (e.g. position error >0.3m), automatically shorten the prediction length (from 200ms to 100ms) and increase the observation weight of Kalman filter (e.g. increase the sampling frequency of IMU) to ensure the effectiveness of prediction.

[0225] After predicting the flight state in the next 100-200ms (e.g. height will drop from 100m to 80m, speed will increase from 10m / s to 15m / s), pre-adjust the ISP parameters through the following logic linkage:

[0226] Height change linkage: If the predicted height decreases by 20m in the next 100ms (from high altitude to low altitude), according to the mapping relationship between light intensity and height (low altitude light intensity is usually 20%-30% higher than high altitude), pre-adjust the automatic aperture from F2.8 to F4.0 and the ISO gain from 400 to 200 to avoid overexposure in low altitude;

[0227] Speed change linkage: If the predicted speed increases by 5m / s in the next 200ms (entering high-speed flight state), pre-adjust the dynamic blur suppression algorithm parameters of the intelligent image processing module, adjusting the blur kernel size from 5x5 pixels to 8x8 pixels and expanding the search window of optical flow estimation from 24x24 pixels to 32x32 pixels to prepare for blur suppression in advance;

[0228] Scene switching linkage: If the trajectory planning data shows that the UAV will enter a low-altitude crossing scene (e.g. crossing a forest) in the next 200ms, combined with the predicted position and speed, pre-adjust the threshold of the edge enhancement algorithm to enhance the recognition of tree outlines, and adjust the automatic aperture to F5.6 to balance the light intake and depth of field.

[0229] This step receives the "parameter feedback" output by the intelligent image processing and the "real-time flight data" from the flight control system, generates "ISP parameter pre-adjustment instructions" through the flight state prediction algorithm, and transmits them to the automatic aperture innovation control technology and intelligent image processing technology to realize pre-adjustment of parameters and eliminate adjustment delay.

[0230] To sum up, the embodiment of the application provides an unmanned aerial vehicle imaging control method based on AI and flight control parameter fusion, which improves the adaptability of the camera in different scenes by constructing an AI scene adaptation strategy according to image features, and solves the defect of weak light adaptation capability in the prior art by constructing a light intensity prediction model in combination with flight control parameters, solving the light intensity prediction model, and obtaining the optimal imaging parameter set of the corresponding camera.

[0231] In a second aspect, the embodiment of the application provides an unmanned aerial vehicle imaging control device 600 based on AI and flight control parameter fusion. Figure 6 is a block diagram of an unmanned aerial vehicle imaging control device 600 based on AI and flight control parameter fusion. As shown in Figure 6 , it includes:

[0232] The adaptation strategy construction module 610 is configured to obtain historical imaging data of the camera, perform scene feature extraction on the historical imaging data based on an attention mechanism to obtain image features in multiple scales, and construct an AI scene adaptation strategy according to the image features.

[0233] The imaging parameter set acquisition module 620 is configured to obtain real-time imaging data of the camera, construct a light intensity prediction model in combination with flight control parameters, solve the light intensity prediction model, and obtain the optimal imaging parameter set of the corresponding camera.

[0234] The optimization parameter extraction module 630 is configured to obtain original image data photographed by the camera under the optimal imaging parameter set, optimize the original image data based on the AI scene adaptation strategy, and extract optimization parameters in the optimization process.

[0235] The imaging control module 640 is configured to predict the flight state of the unmanned aerial vehicle based on the flight control parameters, generate imaging pre-adjustment parameters for the unmanned aerial vehicle in combination with the optimization parameters, and control the photographing of the unmanned aerial vehicle by using the imaging pre-adjustment parameters.

[0236] To sum up, the embodiment of the application provides an unmanned aerial vehicle imaging control device based on AI and flight control parameter fusion, which solves the problem of poor scene adaptability in the prior art by using AI to construct an AI scene adaptation strategy in combination with historical imaging data, and solves the defect of weak light adaptation capability in the prior art by constructing a light intensity prediction model in combination with flight control parameters, solving the light intensity prediction model, and obtaining the optimal imaging parameter set of the corresponding camera.

[0237] It should be noted that the embodiment provided in the present embodiment is used for realizing the above-mentioned embodiment, and the description of which has been made. As used above, the terms "module", "unit", "sub-unit" and the like can be a combination of software and / or hardware that realizes a predetermined function. Although the above embodiment describes the device preferably realized in software, the realization of hardware, or a combination of software and hardware is also possible and conceived.

[0238] In a third aspect, the embodiments of the present application provide an electronic device, Figure 7 is a block diagram of an electronic device according to an exemplary embodiment. As shown in the figure, Figure 7 The electronic device can include a processor 71 and a memory 72 storing computer program instructions.

[0239] Specifically, the processor 71 described above can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured to implement one or more integrated circuits of the embodiments of the present application.

[0240] The memory 72 can include a mass storage for data or instructions. By way of example and not limitation, the memory 72 can include a Hard Disk Drive (HDD), floppy disk drive, a Solid State Drive (SSD), flash memory, a Compact Disc Read Only Memory (CDROM), a Digital Versatile Disk (DVD), a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. The memory 72 can be removable and / or non-removable (or fixed) as appropriate. The memory 72 can be internal or external as appropriate. In certain embodiments, the memory 72 is a non-volatile memory. In certain embodiments, the memory 72 includes a Read-Only Memory (ROM) and a Random Access Memory (RAM). The ROM can be a mask-programmed ROM, a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), an Electrically Alterable ROM (EAROM), or a FLASH, or a combination of two or more of these, as appropriate. The RAM can be a Static Random-Access Memory (SRAM) or a Dynamic Random Access Memory (DRAM), which can be a Fast Page Mode Dynamic Random Access Memory (FPMDRAM), an Extended Data Output Dynamic Random Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), or the like, as appropriate.

[0241] The memory 72 can be used to store or buffer various data files required for processing and / or communication, and possible computer program instructions executed by the processor 71.

[0242] The processor 71 reads and executes the computer program instructions stored in the memory 72 to implement any one of the above-mentioned embodiments of the unmanned aerial vehicle imaging control method based on AI fusion with flight control parameters.

[0243] In an embodiment, the electronic device can further include a communication interface 73 and a bus 70. As shown, the processor 71, the memory 72, and the communication interface 73 are connected through the bus 70 and complete communication with each other. Figure 7

[0244] The communication interface 73 is used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application. The communication interface 73 can also realize data communication with other components, such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations, etc.

[0245] ​Bus 70 includes hardware, software, or both, to couple components of the electronic device to each other and to couple components of the electronic device to other components, including those that are internal or external to the electronic device. Bus 70 includes, but is not limited to, at least one of a data bus, an address bus, a control bus, an expansion bus, a local bus, and the like. By way of example and not limitation, bus 70 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 70 can include one or more buses. Although the present application is described and illustrated with a particular bus, it is contemplated that any suitable bus or interconnect can be used.

[0246] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, having a program stored thereon, where the program, when executed by a processor, implements the unmanned aerial vehicle imaging control method based on AI and flight control parameter fusion provided in the first aspect.

[0247] More specifically, the readable storage medium can include, but is not limited to, a portable disc, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0248] In possible implementation manners, the present application can also be implemented in the form of a program product, which comprises program codes for causing terminal equipment to execute steps of the unmanned aerial vehicle imaging control method based on AI and flight control parameter fusion provided by the first aspect when the program product is run on the terminal equipment.

[0249] The program codes for executing the present application can be written in any combination of one or more programming languages, and can be executed entirely on the user equipment, partly on the user equipment, as a stand-alone software package, partly on the user equipment and partly on a remote device, or entirely on a remote device.

[0250] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, but it should be considered that any combination of the technical features is within the scope of the present disclosure, as long as the combination does not result in contradictions.

[0251] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An unmanned aerial vehicle imaging control method based on AI and flight control parameter fusion, characterized in that, The application comprises the following steps: acquiring historical imaging data of a camera, extracting scene features from the historical imaging data based on an attention mechanism to obtain image features at multiple scales, and constructing an AI scene adaptation strategy according to the image features; acquiring real-time imaging data of the camera, constructing a light intensity prediction model in combination with the flight control parameters, solving the light intensity prediction model, and obtaining an optimal imaging parameter set corresponding to the camera; acquiring original image data captured by the camera under the optimal imaging parameter set, optimizing the original image data based on the AI scene adaptation strategy, and extracting optimization parameters in the optimization process; predicting the flight state of the unmanned aerial vehicle based on the flight control parameters, generating imaging pre-adjustment parameters for the unmanned aerial vehicle in combination with the optimization parameters, and controlling the shooting of the unmanned aerial vehicle using the imaging pre-adjustment parameters; wherein the acquisition of real-time imaging data of the camera, the construction of a light intensity prediction model in combination with the flight control parameters, the solving of the light intensity prediction model, and the obtaining of an optimal imaging parameter set corresponding to the camera comprise the following steps: acquiring the real-time imaging data from the camera, and extracting real-time image brightness mean and ambient light sensor data from the real-time imaging data; extracting the flight height and current flight time of the unmanned aerial vehicle from the flight control parameters, and constructing the light intensity prediction model in combination with the real-time image brightness mean and the ambient light sensor data; constructing a target function from three dimensions of picture signal-to-noise ratio, dynamic range, and color deviation, solving the light intensity prediction model, and obtaining an optimal imaging parameter set corresponding to optimal implementation imaging data; the prediction of the flight state of the unmanned aerial vehicle based on the flight control parameters, the generation of imaging pre-adjustment parameters for the unmanned aerial vehicle in combination with the optimization parameters, and the control of the shooting of the unmanned aerial vehicle using the imaging pre-adjustment parameters comprise the following steps: extracting GPS data, IMU data, and trajectory planning data of the unmanned aerial vehicle from the flight control parameters; predicting the flight state of the unmanned aerial vehicle based on the GPS data, the IMU data, and the trajectory planning data; if the prediction result exceeds a flight state change threshold, controlling the shooting of the unmanned aerial vehicle using the imaging pre-adjustment parameters.

2. The unmanned aerial vehicle imaging control method based on AI and flight control parameter fusion according to claim 1, characterized in that, the acquisition of historical imaging data of a camera, the extraction of scene features from the historical imaging data based on an attention mechanism to obtain image features at multiple scales, and the construction of an AI scene adaptation strategy according to the image features comprise the following steps: acquiring the historical imaging data, extracting scene features from the historical imaging data using a convolution block attention module, and fusing the scene features at multiple scales to obtain the image features; adjusting image processing parameters in a pre-constructed basic strategy library based on the image features using AI to obtain the AI scene adaptation strategy corresponding to different scenes.

3. The unmanned aerial vehicle imaging control method based on AI and flight control parameter fusion according to claim 1, characterized in that, the acquisition of original image data captured by the camera under the optimal imaging parameter set, the optimization of the original image data based on the AI scene adaptation strategy, and the extraction of optimization parameters in the optimization process comprise the following steps: The original image data captured by the camera under the optimal imaging parameter set is acquired, image features in the original image data are extracted, and an original scene corresponding to the original image data is determined according to the image features; An optimization algorithm corresponding to the original scene is selected based on an AI scene adaptation strategy, and algorithm parameters adapted to the current scene are generated through the optimization algorithm; The optimization effect of the algorithm parameters is verified through a lightweight quality evaluation model, and if the expected result is not achieved, the algorithm parameters are reselected; and the algorithm parameters that meet the scene requirements in terms of optimization effect of each frame of image are selected as the optimization parameters.

4. The unmanned aerial vehicle imaging control method based on AI and flight control parameter fusion according to claim 3, characterized in that, The optimization algorithm corresponding to the original scene is selected based on the AI scene adaptation strategy, and the algorithm parameters adapted to the current scene are generated through the optimization algorithm, including: An optimization target is determined according to the original scene; The optimization target is analyzed based on the AI scene adaptation strategy, the image optimization steps corresponding to the optimization target are determined, the optimization algorithm corresponding to the original scene is selected according to the image optimization steps, and the algorithm parameters adapted to the current scene are generated through the optimization algorithm. The flight state of the unmanned aerial vehicle is predicted based on the GPS data, the IMU data and the trajectory planning data, including:

5. The unmanned aerial vehicle imaging control method based on AI and flight control parameter fusion according to claim 1, characterized in that, The GPS data, the IMU data and the trajectory planning data are preprocessed respectively; State equations and observation equations describing the dynamic changes of the flight state and the observation relationship are constructed based on an unmanned aerial vehicle motion model; The future position of the unmanned aerial vehicle is predicted using the state equations, and the future position is corrected using the observation residuals obtained by the observation equations to obtain the flight state of the unmanned aerial vehicle at a future time. It includes:

6. An unmanned aerial vehicle imaging control device based on AI and flight control parameter fusion, characterized in that, An adaptation strategy construction module is configured to acquire historical imaging data of a camera, extract scene features from the historical imaging data based on an attention mechanism to obtain image features at multiple scales, and construct an AI scene adaptation strategy according to the image features; An imaging parameter set acquisition module is configured to acquire real-time imaging data of the camera, construct a light intensity prediction model in combination with the flight control parameters, solve the light intensity prediction model, and obtain an optimal imaging parameter set corresponding to the camera; An optimization parameter extraction module is configured to acquire original image data captured by the camera under the optimal imaging parameter set, optimize the original image data based on the AI scene adaptation strategy, and extract optimization parameters in the optimization process; An imaging control module is configured to predict the flight state of the unmanned aerial vehicle based on the flight control parameters, generate imaging pre-adjustment parameters for the unmanned aerial vehicle in combination with the optimization parameters, and control the shooting of the unmanned aerial vehicle using the imaging pre-adjustment parameters. The imaging parameter set acquisition module is specifically configured to: Acquire the real-time imaging data from the camera, and extract real-time image brightness mean and ambient light sensor data from the real-time imaging data; Extract the flight height and current flight time of the unmanned aerial vehicle from the flight control parameters, and construct the light intensity prediction model in combination with the real-time image brightness mean and the ambient light sensor data. ​ A target function is constructed from three dimensions of picture signal-to-noise ratio, dynamic range and color deviation, the illumination intensity prediction model is solved to obtain an optimal imaging parameter set corresponding to optimal implementation imaging data; The imaging control module is specifically configured to: extracting GPS data, IMU data and trajectory planning data of the UAV from the flight control parameters; predicting the flight state of the UAV based on the GPS data, the IMU data and the trajectory planning data; if the prediction result exceeds the flight state change threshold, using the imaging pre-adjustment parameter to control the shooting of the UAV.

7. An electronic device, comprising: comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the computer program to implement the UAV imaging control method based on AI and flight control parameter fusion according to any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the UAV imaging control method based on AI and flight control parameter fusion according to any one of claims 1 to 5.

Images