Night image and video processing system and method suitable for unmanned aerial vehicle

Abstract

The invention relates to the technical field of computer vision, and discloses a night image and video processing system and method suitable for an unmanned aerial vehicle. The method comprises the following steps: establishing an unmanned aerial vehicle night scene image collection module, an unmanned aerial vehicle night scene image preprocessing module, an unmanned aerial vehicle night scene data calculation module, an unmanned aerial vehicle night scene data analysis module and an unmanned aerial vehicle night image and video optimization module, the unmanned aerial vehicle night scene image preprocessing module performs different formatting preprocessing on acquired night scene imaging pictures and videos, and adopts a bilateral filtering image enhancement algorithm formula to suppress noise, and the unmanned aerial vehicle night scene data calculation module is used for calculating data; the unmanned aerial vehicle night scene data analysis module compensates a shadow image of the unmanned aerial vehicle in night flight according to a calculation result, and the unmanned aerial vehicle night image and video optimization module optimizes parameters of an image enhancement algorithm according to an analysis result.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and particularly to a system and method for processing unmanned aerial vehicle (UAV) night images and videos. Background Art

[0002] With the continuous development of the UAV technology field, its application scope is becoming increasingly wide, gradually expanding from military reconnaissance and geographical mapping to multiple fields such as logistics distribution, agricultural monitoring, search and rescue, and security monitoring. However, there are still some deficiencies in the current UAV technology in image and video processing, especially when operating at night. Most existing UAVs rely on visible light imaging. In the low-light environment at night, due to insufficient light, it is difficult to capture clear and high-quality images and videos. The performance of sensors is limited under low-light conditions, resulting in increased noise in the image, reduced contrast, poor color restoration, and a large amount of detail information loss, which cannot meet the requirements of tasks such as feature recognition of suspicious targets in night security monitoring, precise positioning of missing persons or objects during search and rescue, and accurate assessment of the growth status of night crops in agricultural monitoring. Traditional image enhancement algorithms often over-amplify noise while increasing the image brightness, further affecting the image quality, and it is difficult to adapt to complex and changing night scenes, such as interference from artificial light sources of different intensities and dynamic shadow changes. Although deep learning algorithms have shown great potential in the field of image processing, in the application of UAV night image and video processing, they still face problems such as difficulty in obtaining training data and model deployment being limited by the hardware computing resources of UAVs, resulting in their real-time performance, accuracy, and generalization ability being difficult to reach an ideal state. These technical bottlenecks severely limit the effective application of UAVs in night scenes, and there is an urgent need to develop a system and method specifically applicable to UAV night image and video processing to break through the limitations of existing technologies and improve the efficiency and reliability of UAV operations at night. Summary of the Invention

[0003] Aiming at the deficiencies of the existing technology, the present invention provides a system and method for processing UAV night images and videos, which have the advantage of being able to capture clear and high-quality images and videos, and solves the problem that it is difficult for existing UAVs to capture clear and high-quality images and videos.

[0004] To achieve the above object, the present invention provides the following technical solution: A system for processing UAV night images and videos includes a UAV night scene image collection module, a UAV night scene image preprocessing module, a UAV night scene data calculation module, a UAV night scene data analysis module, and a UAV night image and video optimization module;

[0005] The drone night scene image collection module includes a night scene visible light imaging collection unit, a night scene infrared imaging collection unit and a night scene lidar detection collection unit, and the drone night scene image collection module is connected to the drone night scene image preprocessing module through a network;

[0006] The drone night scene image preprocessing module performs different formatting preprocessing on the night scene imaging pictures and videos collected in the night scene visible light imaging acquisition unit, the night scene infrared imaging acquisition unit and the night scene lidar detection acquisition unit, and obtains the night scene visible light imaging data, the night scene infrared imaging data and the night scene lidar detection data respectively. The drone night scene image preprocessing module includes an adaptive filtering noise reduction unit, and the adaptive filtering noise reduction unit is used to collect noise data in the night scene imaging video during the preprocessing process. The drone night scene image preprocessing module is connected to the drone night scene data calculation module through a network;

[0007] The drone night scene data calculation module calculates the static scene shadow illumination compensation coefficient Fv and the dynamic scene perception compensation coefficient Cv according to the data in the drone night scene image preprocessing module.

[0008] Preferably, the adaptive filtering noise reduction unit is used to collect noise data in the night scene imaging video during the preprocessing process, and use a bilateral filtering image enhancement algorithm to suppress noise. The calculation formula of the bilateral filtering image enhancement algorithm is:

[0009]

[0010] In the formula, L(X) represents the intensity value of the drone night scene output video at pixel X, and I(X i ) represents the input video of the drone night scene in pixels X i The intensity value at the pixel, Ω represents the neighborhood centered on pixel X, G σs and G σr denote the spatial Gaussian kernel and the intensity Gaussian kernel respectively, σs and σr are the standard deviations of space and intensity respectively.

[0011] Preferably, the nighttime scene visible light imaging data includes an intensity value and a weight of the visible light imaging data at pixel i.

[0012] Preferably, the nighttime scene infrared imaging data includes intensity values ​​and weights of the infrared imaging data at pixels i and j.

[0013] Preferably, the night scene lidar detection data includes the distance value and weight of the lidar detection data at the qth detection point.

[0014] Preferably, the UAV night scene data calculation module includes a static scene shadow illumination compensation unit and a dynamic scene perception compensation unit.

[0015] Preferably, the static scene shadow illumination compensation unit calculates the static scene shadow illumination compensation coefficient Fv according to the visible light imaging data and the infrared imaging data of the night scene. The calculation formula is as follows:

[0016]

[0017] In the formula, Fv represents the static scene shadow illumination compensation coefficient, K(i) represents the intensity value of the visible light imaging data at pixel i, a(i) represents the weight of the visible light imaging data at pixel i, H(i) represents the intensity value of the infrared imaging data at pixel i, b(i) represents the weight of the infrared imaging data at pixel i, and n represents the total number of pixels in the image.

[0018] Preferably, the dynamic scene perception compensation unit calculates the dynamic scene perception compensation coefficient Cv according to the visible light imaging data, the infrared imaging data and the lidar detection data of the night scene. The calculation formula is as follows:

[0019]

[0020] In the formula, Cv represents the dynamic scene perception compensation coefficient, K(i) represents the intensity value of the visible light imaging data at pixel i, a(i) represents the weight of the visible light imaging data at pixel i, H(j) represents the intensity value of the infrared imaging data at pixel j, b(j) represents the weight of the infrared imaging data at pixel j, F(q) represents the distance value of the lidar detection data at the q-th detection point, c(q) represents the weight of the lidar detection data at the q-th detection point, N represents the total number of pixels in the visible light image, M represents the total number of pixels in the infrared image, L represents the total number of lidar detection points, and α, β, γ are respectively used to adjust the relative importance of the visible light imaging data, the infrared imaging data and the lidar detection data when calculating the dynamic scene perception compensation coefficient, and α + β + γ = 1.

[0021] Preferably, the UAV night scene data analysis module compensates the shadow images collected by the UAV during night flight according to the static scene shadow illumination compensation coefficient Fv, and compensates the shadow images of the dynamic scene collected by the UAV during night flight according to the dynamic scene perception compensation coefficient Cv.

[0022] A method for processing UAV night images and videos includes the following steps:

[0023] Step 1: Establish a UAV night scene image collection module, a UAV night scene image preprocessing module, a UAV night scene data calculation module, a UAV night scene data analysis module, and a UAV night image and video optimization module;

[0024] Step 2: The UAV night scene image collection module collects UAV flight night scene images and videos through a visible light imager, an infrared imager, and a lidar detector;

[0025] Step 3: The UAV night scene image preprocessing module performs different formatting preprocessing on the collected night scene imaging pictures and videos, and uses the bilateral filtering image enhancement algorithm formula to suppress noise;

[0026] Step 4: The UAV night scene data calculation module calculates the static scene shadow illumination compensation coefficient Fv and the dynamic scene perception compensation coefficient Cv according to the data in the UAV night scene image preprocessing module;

[0027] Step 5: The UAV night scene data analysis module compensates the shadow images of the UAV during night flight according to the static scene shadow illumination compensation coefficient Fv and the dynamic scene perception compensation coefficient Cv;

[0028] Step 6: The UAV night image and video optimization module optimizes the parameters of the image enhancement algorithm according to the analysis results.

[0029] Compared with the prior art, the present invention provides a UAV night image and video processing system and method, which have the following beneficial effects:

[0030] 1. In the system of the present invention, the night scene visible light imaging acquisition unit, the night scene infrared imaging acquisition unit, and the night scene lidar detection acquisition unit simultaneously collect data through a visible light imager, an infrared imager, and a lidar detector respectively, and can obtain night scene information from different dimensions. The visible light imager captures the general outline and texture of the object, the infrared imager uses the thermal radiation characteristics to image in low light, and the lidar detector obtains the distance and spatial position information of the object. Multiple data sources complement each other, comprehensively covering various characteristics of the night scene, and providing a rich data basis for subsequent processing.

[0031] 2. By calculating the static scene shadow illumination compensation coefficient Fv and the dynamic scene perception compensation coefficient Cv, the present invention can compensate the shadow images of the UAV during night flight. According to different compensation coefficients, the shadow images in static and dynamic scenes are compensated respectively. For the static scene, the shadow influence is reduced by adjusting the illumination intensity and contrast; for the dynamic scene, not only the illumination conditions are improved, but also the image dynamic range is enhanced to improve the target recognition ability. This targeted compensation method can better adapt to different scene requirements and improve the usability of the image. Description of the Drawings

[0032] Figure 1 This is the system flowchart of the present invention. Detailed Embodiments

[0033] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] Please refer to Figure 1 , a system suitable for processing drone night images and videos, including a drone night scene image collection module, a drone night scene image preprocessing module, a drone night scene data calculation module, a drone night scene data analysis module, and a drone night image and video optimization module;

[0035] The drone night scene image collection module includes a night scene visible light imaging acquisition unit, a night scene infrared imaging acquisition unit, and a night scene lidar detection acquisition unit. The drone night scene image collection module is connected to the drone night scene image preprocessing module through a network;

[0036] The drone night scene image preprocessing module performs different formatting preprocessing on the night scene imaging pictures and videos collected by the night scene visible light imaging acquisition unit, the night scene infrared imaging acquisition unit, and the night scene lidar detection acquisition unit, and respectively obtains night scene visible light imaging data, night scene infrared imaging data, and night scene lidar detection data. The drone night scene image preprocessing module includes an adaptive filtering and noise reduction unit, and the adaptive filtering and noise reduction unit is used to collect noise data in the night scene imaging video during the preprocessing process. The drone night scene image preprocessing module is connected to the drone night scene data calculation module through a network;

[0037] The drone night scene data calculation module calculates the static scene shadow illumination compensation coefficient Fv and the dynamic scene perception compensation coefficient Cv according to the data in the drone night scene image preprocessing module.

[0038] The adaptive filtering and noise reduction unit is used to collect noise data in the night scene imaging video during the preprocessing process, and adopts a bilateral filtering image enhancement algorithm to suppress noise. The calculation formula of the bilateral filtering image enhancement algorithm is:

[0039]

[0040] In the formula, L(X) represents the intensity value of the output video of the UAV at night scene at pixel X, and I(X i ) represents the intensity value of the input video of the UAV at night scene at pixel X i . Ω represents the neighborhood centered on pixel X, and G σs and G σr represent the spatial Gaussian kernel and the intensity Gaussian kernel respectively. σs and σr are the standard deviations of space and intensity respectively.

[0041] The visible light imaging data of the night scene includes the intensity value and weight of the visible light imaging data at pixel i.

[0042] The infrared imaging data of the night scene includes the intensity value and weight of the infrared imaging data at pixels i and j.

[0043] The lidar detection data of the night scene includes the distance value and weight of the lidar detection data at the q-th detection point.

[0044] The UAV night scene data calculation module includes a static scene shadow illumination compensation unit and a dynamic scene perception compensation unit.

[0045] The static scene shadow illumination compensation unit calculates the static scene shadow illumination compensation coefficient Fv according to the visible light imaging data and infrared imaging data of the night scene. Its calculation formula is:

[0046]

[0047] In the formula, Fv represents the static scene shadow illumination compensation coefficient, K(i) represents the intensity value of the visible light imaging data at pixel i, a(i) represents the weight of the visible light imaging data at pixel i, H(i) represents the intensity value of the infrared imaging data at pixel i, b(i) represents the weight of the infrared imaging data at pixel i, and n represents the total number of pixels in the image.

[0048] The advantage is that the calculation formula of the static scene shadow illumination compensation coefficient Fv comprehensively considers the visible light imaging data and infrared imaging data, utilizes the advantages of the two data sources in different aspects. The visible light imaging data has advantages in presenting object contours and texture details, and the infrared imaging data can capture object information through thermal radiation characteristics in low-light environments. The combination of the two comprehensively covers the characteristics of the static scene, making the calculated compensation coefficient more able to reflect the actual scene illumination situation.

[0049] The dynamic scene perception compensation unit calculates the dynamic scene perception compensation coefficient Cv according to the visible light imaging data, infrared imaging data and lidar detection data of the night scene. Its calculation formula is:

[0050]

[0051] In the formula, Cv represents the dynamic scene perception compensation coefficient, K(i) represents the intensity value of visible light imaging data at pixel i, a(i) represents the weight of visible light imaging data at pixel i, H(j) represents the intensity value of infrared imaging data at pixel j, b(j) represents the weight of infrared imaging data at pixel j, F(q) represents the distance value of lidar detection data at the q-th detection point, c(q) represents the weight of lidar detection data at the q-th detection point, N represents the total number of pixels in the visible light image, M represents the total number of pixels in the infrared image, L represents the total number of lidar detection points, α, β, and γ are respectively used to adjust the relative importance of visible light imaging data, infrared imaging data, and lidar detection data when calculating the dynamic scene perception compensation coefficient, and α + β + γ = 1. By adjusting these three factors, the weights of the three types of data can be flexibly allocated according to different application scenarios and requirements, so that the calculated compensation coefficient is more in line with the actual situation.

[0052] The advantages are as follows: Through the calculation formula of the dynamic scene perception compensation coefficient Cv, it is possible to calculate different data separately according to pixels (visible light and infrared) and detection points (lidar) for the fast-changing characteristics of the dynamic scene. Since factors such as the movement of objects and the instantaneous change of light in the dynamic scene will have different effects on each pixel point and detection point, their weights are different. Such a design enables the calculation result to capture the dynamic changes of the scene in real time, providing a reliable basis for image compensation in the dynamic scene, thereby effectively improving the quality of images in the dynamic scene.

[0053] The UAV night scene data analysis module compensates the shadow images collected by the UAV during night flight according to the static scene shadow illumination compensation coefficient Fv. This process reduces the impact of shadows on the image quality by adjusting the illumination intensity and contrast of the image, making the image clearer and easier to identify. According to the dynamic scene perception compensation coefficient Cv, it compensates the shadow images of the dynamic scene collected by the UAV during night flight. This compensation process can not only improve the illumination conditions of the image but also enhance the target recognition ability by increasing the dynamic range of the image. The increase in the dynamic range enables the image to maintain good detail performance under different illumination conditions, thus better supporting the requirements of night tasks such as target detection, tracking, and recognition.

[0054] The UAV night image and video optimization module optimizes the parameters of the image enhancement algorithm (such as the long and short exposure fusion weight, perceptual loss weight) and adjusts the model training strategy (such as the adaptive curriculum learning weight) according to the analysis results to further improve the image quality.

[0055] A method for processing UAV night images and videos, comprising the following steps:

[0056] Step 1: Establish a UAV night scene image collection module, a UAV night scene image preprocessing module, a UAV night scene data calculation module, a UAV night scene data analysis module, and a UAV night image and video optimization module;

[0057] Step 2: The UAV night scene image collection module collects UAV flight night scene images and videos through a visible light imager, an infrared imager, and a lidar detector. The night scene visible light imaging acquisition unit, the night scene infrared imaging acquisition unit, and the night scene lidar detection acquisition unit collect data simultaneously through the visible light imager, the infrared imager, and the lidar detector, respectively, and can obtain night scene information from different dimensions. The visible light imager captures the general outline and texture of objects, the infrared imager uses the thermal radiation characteristics to image in low light, and the lidar detector obtains the distance and spatial position information of objects. Multiple data sources complement each other, comprehensively covering various features of the night scene, and providing a rich data basis for subsequent processing;

[0058] Step 3: The UAV night scene image preprocessing module performs different formatting preprocessing on the collected night scene imaging pictures and videos, and uses the bilateral filtering image enhancement algorithm formula to suppress noise. By performing formatting preprocessing on the collected data, it meets the requirements of subsequent processing and improves processing efficiency. At the same time, the bilateral filtering image enhancement algorithm is used to suppress noise. While enhancing the image brightness, this algorithm effectively reduces the impact of noise on the image quality by considering the spatial distance and pixel value differences, and retains image details, providing high-quality data for subsequent data analysis and calculation;

[0059] Step 4: The UAV night scene data calculation module calculates the static scene shadow illumination compensation coefficient Fv and the dynamic scene perception compensation coefficient Cv according to the data in the UAV night scene image preprocessing module. The calculation of the static scene shadow illumination compensation coefficient is through combining the intensity values and weights of visible light and infrared imaging data, and the calculation of the dynamic scene perception compensation coefficient is through fusing visible light, infrared, and lidar detection data, and flexibly adjusting the importance of each data source through a weight adjustment factor, making the calculation result more accurately reflect the scene characteristics and providing a scientific basis for image compensation;

[0060] Step 5: The UAV night scene data analysis module compensates the shadow images of the UAV during night flight according to the static scene shadow illumination compensation coefficient Fv and the dynamic scene perception compensation coefficient Cv. The shadow images in static and dynamic scenes are compensated respectively according to different compensation coefficients. For the static scene, the shadow influence is reduced by adjusting the illumination intensity and contrast. For the dynamic scene, not only the illumination condition is improved, but also the dynamic range of the image is enhanced to improve the target recognition ability. This targeted compensation method can better meet the requirements of different scenes and improve the usability of the image.

[0061] Step 6: The UAV night image and video optimization module optimizes the parameters of the image enhancement algorithm according to the analysis results. With the continuous accumulation and analysis of data, the algorithm parameters can be continuously adjusted to further improve the image quality to adapt to the complex and changeable night environment and meet the higher requirements for UAV night image and video processing in different application scenarios.

[0062] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An image and video processing system applicable to unmanned aerial vehicle (UAV) night vision, characterized in that, It includes a UAV night scene image collection module, a UAV night scene image preprocessing module, a UAV night scene data calculation module, a UAV night scene data analysis module, and a UAV night image and video optimization module; The UAV night scene image collection module includes a night scene visible light imaging acquisition unit, a night scene infrared imaging acquisition unit, and a night scene lidar detection acquisition unit. The UAV night scene image collection module is connected to the UAV night scene image preprocessing module through a network; The UAV night scene image preprocessing module performs different formatting preprocessing on the night scene imaging pictures and videos collected by the night scene visible light imaging acquisition unit, the night scene infrared imaging acquisition unit, and the night scene lidar detection acquisition unit, and respectively obtains night scene visible light imaging data, night scene infrared imaging data, and night scene lidar detection data. The UAV night scene image preprocessing module contains an adaptive filtering and noise reduction unit, which is used to collect the noise data in the night scene imaging video during the preprocessing process. The UAV night scene image preprocessing module is connected to the UAV night scene data calculation module through a network; The UAV night scene data calculation module calculates the static scene shadow illumination compensation coefficient Fv and the dynamic scene perception compensation coefficient Cv according to the data in the UAV night scene image preprocessing module.

2. The night image and video processing system for unmanned aerial vehicles according to claim 1, wherein: The adaptive filtering and noise reduction unit is used to collect the noise data in the night scene imaging video during the preprocessing process and uses the bilateral filtering image enhancement algorithm to suppress noise. The calculation formula of the bilateral filtering image enhancement algorithm is: In the formula, L(X) represents the intensity value of the output video of the drone's night scene at pixel X, and I(X i ) represents the intensity value of the input video of the drone's night scene at pixel X i . Ω represents the neighborhood centered on pixel X, and G σs and G σr represent the spatial Gaussian kernel and the intensity Gaussian kernel respectively, and σs and σr are the standard deviations of space and intensity respectively.

3. The night image and video processing system for drones according to claim 1, characterized in that: The night scene visible light imaging data includes the intensity value and weight of the visible light imaging data at pixel i.

4. The night image and video processing system for drones according to claim 1, wherein: The night scene infrared imaging data includes the intensity value and weight of the infrared imaging data at pixels i and j.

5. The image and video processing system for drones applicable at night according to claim 1, wherein: The night scene lidar detection data includes the distance value and weight of the lidar detection data at the qth detection point.

6. The night image and video processing system for drones according to claim 1, wherein: The UAV night scene data calculation module includes a static scene shadow illumination compensation unit and a dynamic scene perception compensation unit.

7. The night image and video processing system for drones according to claim 6, characterized in that: The static scene shadow illumination compensation unit calculates the static scene shadow illumination compensation coefficient Fv according to the night scene visible light imaging data and the night scene infrared imaging data. Its calculation formula is: In the formula, Fv represents the static scene shadow illumination compensation coefficient, K(i) represents the intensity value of the visible light imaging data at pixel i, a(i) represents the weight of the visible light imaging data at pixel i, H(i) represents the intensity value of the infrared imaging data at pixel i, b(i) represents the weight of the infrared imaging data at pixel i, and n represents the total number of pixels in the image.

8. An image and video processing system for drones applicable at night according to claim 6, characterized in that: The dynamic scene perception compensation unit calculates the dynamic scene perception compensation coefficient Cv according to the night scene visible light imaging data, the night scene infrared imaging data, and the night scene lidar detection data. Its calculation formula is: In the formula, Cv represents the dynamic scene perception compensation coefficient, K(i) represents the intensity value of the visible light imaging data at pixel i, a(i) represents the weight of the visible light imaging data at pixel i, H(j) represents the intensity value of the infrared imaging data at pixel j, b(j) represents the weight of the infrared imaging data at pixel j, F(q) represents the distance value of the lidar detection data at the q-th detection point, c(q) represents the weight of the lidar detection data at the q-th detection point, N represents the total number of pixels in the visible light image, M represents the total number of pixels in the infrared image, L represents the total number of lidar detection points, and α, β, and γ are respectively used to adjust the relative importance of the visible light imaging data, infrared imaging data, and lidar detection data when calculating the dynamic scene perception compensation coefficient, and α + β + γ = 1.

9. The night image and video processing system for drones according to claim 8, wherein: The UAV night scene data analysis module compensates the shadow images collected by the UAV during night flight according to the static scene shadow illumination compensation coefficient Fv, and compensates the shadow images of the dynamic scene collected by the UAV during night flight according to the dynamic scene perception compensation coefficient Cv; The UAV night image and video optimization module optimizes the parameters of the image enhancement algorithm according to the analysis results.

10. A method for processing night-time images and videos of drones, characterized in that, It includes the following steps: Step 1: Establish a UAV night scene image collection module, a UAV night scene image preprocessing module, a UAV night scene data calculation module, a UAV night scene data analysis module, and a UAV night image and video optimization module; Step 2: The UAV night scene image collection module collects UAV flight night scene images and videos through a visible light imager, an infrared imager, and a lidar detector; Step 3: The UAV night scene image preprocessing module performs different formatting preprocessing on the collected night scene imaging pictures and videos, and uses the image enhancement algorithm formula of bilateral filtering to suppress noise; Step 4: The UAV night scene data calculation module calculates the static scene shadow illumination compensation coefficient Fv and the dynamic scene perception compensation coefficient Cv according to the data in the UAV night scene image preprocessing module; Step 5: The UAV night scene data analysis module compensates the shadow images of the UAV during night flight according to the static scene shadow illumination compensation coefficient Fv and the dynamic scene perception compensation coefficient Cv; Step 6: The UAV night image and video optimization module optimizes the parameters of the image enhancement algorithm according to the analysis results.

Images