Target-level fusion method and system for visible light-infrared images of unmanned aerial vehicles
The target-level fusion method is used to register, semantic segmentation and temperature information fusion of visible light and infrared images collected by the drone, which solves the problems of strong image alignment dependence and information loss, and achieves high-precision and high-rootability images Fusion, suitable for real-time processing of images with large resolution differences.
Patent Information
- Application Number
- CN202411259447.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-09-10
Smart Images

Figure CN118780997B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for target-level fusion of visible light and infrared images of unmanned aerial vehicles. Background Art
[0002] In recent years, multi-payload imaging systems for UAVs have attracted great interest. Compared with traditional single-payload imaging systems, multi-payload imaging systems can obtain more image information. Multi-payload imaging systems integrate image sensors of multiple modes, so that image information of different modes of the same target can be obtained at the same time and location. With the development of image sensor technology, the hardware cost of multi-payload imaging systems is getting lower and lower, and their application on UAV platforms is becoming more and more extensive. Among them, the most common multi-payload imaging system is the visible light infrared imaging system.
[0003] Visible light images can provide texture details with high spatial resolution and clarity in a manner consistent with the human visual system. In contrast, infrared images can distinguish targets from backgrounds based on radiation differences and are applicable to different climates and lighting conditions. Therefore, fusing the two images can combine the detailed texture information in visible light images and the thermal radiation information in infrared images to provide a more accurate description of the scene than any single-source image.
[0004] With the rapid development of image processing technology, more and more visible light infrared fusion methods have been proposed. The common point of these fusion methods is that first, the input visible light infrared images are required to be accurately aligned, and secondly, the entire image is fused. However, in the case of large differences in the resolution of visible light infrared images obtained by actual UAV platforms, it is difficult to accurately align visible light infrared images, and direct fusion is prone to information loss. At the same time, when the size of the original visible light infrared image obtained by the UAV platform is large, the computational cost of registering and fusing the entire image is high, and a lot of useless background information is fused. Summary of the invention
[0005] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a method and system for target-level fusion of visible light and infrared images of unmanned aerial vehicles are provided. The present invention aims to solve the problem that visible light infrared images in the existing fusion technology are highly dependent on precise alignment, and to solve the problem that fused image information is lost and visual effects are poor due to large differences in resolution of visible light infrared images.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0007] A method for target-level fusion of visible light and infrared images of unmanned aerial vehicles comprises the following steps:
[0008] S101, obtaining a pair of visible light original images and infrared original images collected by the drone;
[0009] S102, extracting a visible light target slice and an infrared target slice from the visible light original image and the infrared original image;
[0010] S103, registering the visible light target slice and the infrared target slice;
[0011] S104, performing semantic segmentation on the visible light target slice using an unsupervised method to obtain a visible light region segmentation map;
[0012] S105, extracting temperature information from the infrared target slice and injecting it into the visible light region segmentation map to obtain a preliminary fusion result;
[0013] S106, performing guided filtering on the preliminary fusion result to obtain the final fusion result.
[0014] Optionally, when obtaining the visible light original image and the infrared original image collected by the drone in step S101, it includes capturing the corresponding number of frames of the video stream according to the frame number relationship between the visible light video stream and the infrared video stream to obtain a pair of visible light original images and infrared original images, and the function expression of the frame number relationship between the visible light video stream and the infrared video stream is:
[0015] ,
[0016] In the above formula, is the number of frames captured from the visible light video stream, is the number of frames captured from the infrared video stream, is the frame rate of the visible light video stream, is the frame rate of the infrared video stream. Optionally, step S102 includes: performing target detection on the visible light original image and the infrared original image respectively, cutting the visible light original image or the infrared original image according to the position of the detection result box obtained by the target detection, and finally obtaining a visible light target slice and an infrared target slice, and performing target detection includes:
[0017] S201, the input image The prediction output is obtained by processing through the convolutional neural network CNN ;
[0018] S202, output according to prediction The center coordinates of each prediction box are calculated according to the following formula , Width and Height and confidence ;
[0019] ,
[0020] ,
[0021] ,
[0022] In the above formula, is the sigmoid activation function, is the offset of the center coordinate of the prediction box, is the auxiliary item of the center coordinate offset, is the offset of the predicted box width and height, It is an auxiliary item for width and height offset. is the confidence level, is the confidence prediction value;
[0023] S203, using a non-maximum suppression algorithm to remove prediction boxes with a high degree of overlap in the prediction boxes of each category, and only retain the prediction box with the highest confidence as the final detection result box.
[0024] Optionally, step S103 includes:
[0025] S301, calculating the correlation coefficient between the visible light target slice and the infrared target slice according to the following formula:
[0026] ,
[0027] In the above formula, represents the mutual correlation coefficient between the visible light target slice and the infrared target slice, , is the translation parameter in the xy direction, Slice for visible light targets Coordinates Pixels at Slice for infrared targets Coordinates The pixel at; select the initial xy translation parameter as the xy translation parameter before iteration, and initialize the number of iterations to 1;
[0028] S302, calculating the translation parameter in the xy direction after iteration according to the following formula:
[0029] ,
[0030] ,
[0031] In the above formula, and is the translation parameter in the xy direction after iteration, and is the translation parameter in the xy direction before iteration, is the learning rate, is the mutual correlation coefficient between the visible light target slice and the infrared target slice;
[0032] S303, add 1 to the number of iterations to determine whether the number of iterations is equal to a preset threshold. If not, jump to step S302; otherwise, use the final translation parameters in the iterative xy direction to align the visible light target slice and the infrared target slice, and then remove the non-aligned parts of the visible light target slice and the infrared target slice to obtain the registered visible light target slice and the infrared target slice.
[0033] Optionally, step S104 includes:
[0034] S401, randomly selecting the centroids of K clusters for the visible light target slice;
[0035] S402, calculating the Euclidean distances from all pixel points in the visible light target slice to each centroid;
[0036] S403, based on the obtained Euclidean distance, all pixels in the visible light target slice are classified into the cluster where the centroid with the smallest distance is located. After all pixels are classified, the average values of the pixels in each cluster on the red R, green G and blue B channels are calculated to obtain a new centroid. , is the mean on the red R, is the mean value on green G, is the mean value on blue B;
[0037] S404, determine whether the end condition is met, where the end condition refers to the number of iterations being equal to a preset threshold or the centroid no longer changing. If the end condition is not met, jump to step S402; otherwise, the visible light target slice is segmented into K regions through the K clusters obtained after classification to obtain a visible light region segmentation map.
[0038] Optionally, in step S105, the function expression for extracting temperature information from the infrared target slice and injecting it into the visible light region segmentation map is:
[0039] ,
[0040] In the above formula, Represents the first Regions, For the The pixel points corresponding to the area are is the sum of the pixels in the nth region, is the pixel coordinate, Slice for infrared targets Coordinates The pixel value at .
[0041] Optionally, in step S106, the function expression for obtaining the final fusion result by performing guided filtering on the preliminary fusion result is:
[0042] ,
[0043] In the above formula, The coordinates of the final fusion result The pixel value at Pixel The filter window is centered at The size is , is the filter radius, For this filter window The pixel coordinates in For coordinates The weight of Slice for visible light targets In coordinates The pixel value at , and the weighted calculation function expression is:
[0044] ,
[0045] In the above formula, Slice for visible light targets The local variance of the pixel, is the strength parameter.
[0046] In addition, the present invention also provides a UAV visible light-infrared image target level fusion system, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the UAV visible light-infrared image target level fusion method.
[0047] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program / instruction, and the computer program / instruction is programmed or configured to execute the UAV visible light-infrared image target-level fusion method through a processor.
[0048] In addition, the present invention also provides a computer program product, including a computer program / instruction, which is programmed or configured to execute the UAV visible light-infrared image target level fusion method through a processor.
[0049] Compared with the prior art, the present invention mainly has the following advantages:
[0050] 1. High image fusion accuracy. The present invention includes obtaining pairs of visible light original images and infrared original images collected by a drone; extracting visible light target slices and infrared target slices from the visible light original images and the infrared original images; registering the visible light target slices and the infrared target slices; performing semantic segmentation on the visible light target slices using an unsupervised method to obtain a visible light region segmentation map; extracting temperature information from the infrared target slices and injecting it into the visible light region segmentation map to obtain a preliminary fusion result; and performing guided filtering on the preliminary fusion result to obtain the final fusion result. The present invention can effectively solve the problem of the high dependence of visible light infrared images on precise alignment in the existing fusion technology, as well as the problem of information loss and poor visual effect of the fused image caused by large differences in the resolution of visible light infrared images. It can effectively and real-time fuse the complementary information of the target in the visible light infrared image when there are large differences in the resolution of the visible light infrared image, thereby improving the image fusion accuracy.
[0051] 2. High robustness. Visible light infrared images taken by drones have different resolutions and are difficult to align accurately. The present invention effectively extracts and fuses the complementary information of visible light infrared slices, so that the fusion results of visible light infrared images are still highly robust even when the alignment accuracy is not high.
[0052] 3. High timeliness. Compared with the traditional image processing fusion method, the present invention does not need to register and fuse the entire image, but only processes the target slice, and the processing flow calculation cost is simple, so it has obvious advantages in processing speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.
[0054] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.
[0055] Figure 2 Schematic diagram of the principle of target-level visible light infrared fusion in an embodiment of the present invention.
[0056] Figure 3 The figure is a comparison chart of the fusion effect of the method according to the embodiment of the present invention and the existing method on the first target.
[0057] Figure 4 The figure is a comparison chart of the fusion effect of the embodiment method of the present invention and the existing method on the second goal.
[0058] Figure 5 The figure is a comparison chart of the fusion effect of the method according to the embodiment of the present invention and the existing method on the third goal. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be explained and described in detail in combination with the flow chart and the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0060] like Figure 1 As shown, the target-level fusion method of the drone visible light-infrared image in this embodiment includes the following steps:
[0061] S101, obtaining a pair of visible light original images and infrared original images collected by the drone;
[0062] S102, extracting a visible light target slice and an infrared target slice from the visible light original image and the infrared original image;
[0063] S103, registering the visible light target slice and the infrared target slice;
[0064] S104, performing semantic segmentation on the visible light target slice using an unsupervised method to obtain a visible light region segmentation map;
[0065] S105, extracting temperature information from the infrared target slice and injecting it into the visible light region segmentation map to obtain a preliminary fusion result;
[0066] S106, performing guided filtering on the preliminary fusion result to obtain the final fusion result.
[0067] During the real-time shooting process of the drone's onboard camera, the frame rates of the visible light video stream and the infrared video stream are usually different. Therefore, in order to make the visible light infrared images processed each time to be paired images of the same target, the same time and the same position, when obtaining the visible light original image and the infrared original image collected by the drone in step S101 of this embodiment, it includes capturing the corresponding frames of the video stream according to the frame number relationship between the visible light video stream and the infrared video stream to obtain a pair of visible light original images and infrared original images. The functional expression of the frame number relationship between the visible light video stream and the infrared video stream is:
[0068] ,
[0069] In the above formula, is the number of frames captured from the visible light video stream, is the number of frames captured from the infrared video stream, is the frame rate of the visible light video stream, is the frame rate of the infrared video stream. By capturing the corresponding frames of the video stream according to the frame relationship between the visible light and infrared video streams, we can obtain pairs of visible light original images and infrared original images of the same target, at the same time and at the same position.
[0070] Since the field of view of visible light infrared images is biased, even paired images of the same target, at the same time and at the same position are not aligned. In order to effectively fuse the common target areas in the visible light and infrared images and avoid interference from image details in irrelevant areas, a target detection algorithm is used to acquire the same target areas in the visible light and infrared images. By constructing a data set of visible light and infrared modalities of the target to be detected, target detection models of the visible light and infrared modalities are trained respectively, and target detection is performed on the visible light and infrared image pairs parsed in real time respectively. The visible light and infrared images are cut according to the position of the detection result box, and finally target slices of the two modalities are obtained. In this embodiment, step S102 includes: performing target detection on the visible light original image and the infrared original image respectively, cutting the visible light original image or the infrared original image according to the position of the detection result box obtained by the target detection, and finally obtaining visible light target slices and infrared target slices, and performing target detection includes:
[0071] S201, the input image The prediction output is obtained by processing through the convolutional neural network CNN , which can be expressed as:
[0072] ,
[0073] In the above formula, Represents convolutional neural network CNN; predict output Including the offset of the center coordinates of the prediction box , auxiliary items for center coordinate offset , the offset of the predicted box width and height , auxiliary items for width and height offset And the confidence prediction value ;
[0074] S202, output according to prediction The center coordinates of each prediction box are calculated according to the following formula , Width and Height and confidence ;
[0075] ,
[0076] ,
[0077] ,
[0078] In the above formula, is the sigmoid activation function, is the offset of the center coordinate of the prediction box, is the auxiliary item of the center coordinate offset, is the offset of the predicted box width and height, It is an auxiliary item for width and height offset. is the confidence level, is the confidence prediction value;
[0079] S203, using a non-maximum suppression algorithm to remove prediction boxes with a high degree of overlap in the prediction boxes of each category, and only retain the prediction box with the highest confidence as the final detection result box.
[0080] The target slices of the two modes obtained through target detection contain basically the same image target area. However, since the position and size of the prediction boxes finally output by the two modal target detection algorithms may have certain deviations, this example performs a rough registration on the target slices of the two modes obtained through target detection so that the image contents can be further aligned. First, the infrared target slices are made consistent in width with the visible light target slices while maintaining the original aspect ratio. Secondly, the infrared target slices are translated based on the visible light target slices so that the infrared target slices and the visible light target slices are basically aligned. In this embodiment, step S103 includes:
[0081] S301, calculating the correlation coefficient between the visible light target slice and the infrared target slice according to the following formula:
[0082] ,
[0083] In the above formula, represents the mutual correlation coefficient between the visible light target slice and the infrared target slice, , is the translation parameter in the xy direction, Slice for visible light targets Coordinates Pixels at Slice for infrared targets Coordinates The pixel at; select the initial xy translation parameter as the xy translation parameter before iteration, and initialize the number of iterations to 1;
[0084] S302, calculating the translation parameter in the xy direction after iteration according to the following formula:
[0085] ,
[0086] ,
[0087] In the above formula, and is the translation parameter in the xy direction after iteration, and is the translation parameter in the xy direction before iteration, is the learning rate, is the mutual correlation coefficient between the visible light target slice and the infrared target slice;
[0088] S303, add 1 to the number of iterations to determine whether the number of iterations is equal to a preset threshold. If not, jump to step S302; otherwise, use the final translation parameters in the iterative xy direction to align the visible light target slice and the infrared target slice, and then remove the non-aligned parts of the visible light target slice and the infrared target slice to obtain the registered visible light target slice and the infrared target slice.
[0089] When the final iterative xy direction translation parameters are used to align the visible light target slice and the infrared target slice, the translation scaling matrix between the visible light target slice and the infrared target slice can be obtained. :
[0090] ,
[0091] Multiply the visible light target slice or one of the infrared target slices by the translation and scaling matrix , the visible light target slice and the infrared target slice can be aligned. In addition, the pyramid multi-scale strategy can be further adopted to perform step-by-step registration to improve the accuracy of registration. Finally, the non-aligned part of the registered visible light infrared target slice is cut off to obtain the registered visible light target slice and infrared target slice.
[0092] Since visible light target slices have rich texture information, such as Figure 2 As shown, in this embodiment, the K-means clustering method is used to perform unsupervised semantic segmentation on the visible light target slice, which can effectively retain the important texture information in the target slice. Step S104 in this embodiment includes:
[0093] S401, randomly selecting the centroids of K clusters for the visible light target slice;
[0094] S402, calculate the Euclidean distance from all pixels in the visible light target slice to each centroid; assuming that the centroid pixel is , the pixel points to be calculated are: , the Euclidean distance between two pixels is , then the calculation function expression of Euclidean distance is:
[0095] ,
[0096] S403, based on the obtained Euclidean distance, all pixels in the visible light target slice are classified into the cluster where the centroid with the smallest distance is located. After all pixels are classified, the average values of the pixels in each cluster on the red R, green G and blue B channels are calculated to obtain a new centroid. , is the mean on the red R, is the mean value on green G, is the mean value on blue B;
[0097] S404, determine whether the end condition is met, the end condition means that the number of iterations is equal to the preset threshold or the centroid no longer changes. If the end condition is not met, jump to step S402; otherwise, the visible light target slice is divided into K regions by the K clusters obtained after classification to obtain a visible light region segmentation map. Through the above process, the visible light target slice can be divided into K regions, effectively retaining the rich texture information in the image, and finally generating a visible light region segmentation map.
[0098] The infrared slice has rich temperature information. The overall temperature of each area of the infrared target slice corresponding to the visible light segmentation map is averaged, and the visible light area segmentation map is filled, thereby effectively fusing the texture information of the visible light target slice and the temperature information of the infrared target slice to obtain an initial fusion map. Specifically, the function expression of extracting temperature information from the infrared target slice and injecting it into the visible light area segmentation map in step S105 of this embodiment is:
[0099] ,
[0100] In the above formula, Represents the first Regions, For the The pixel points corresponding to the area are is the sum of the pixels in the nth region, is the pixel coordinate, Slice for infrared targets Coordinates The pixel value at .
[0101] In this embodiment, step S106 is used to use visible light target slices to perform guided filtering on the initial fusion image to remove image redundant information. In the process of unsupervised clustering segmentation of visible light target slices, while retaining the rich texture information of the target area, redundant information such as noise is also retained. The initial fusion image is guided and filtered using the registered visible light target slices as reference images to further enhance the texture details and edges in the initial fusion image, while filtering out redundant information such as noise in the initial fusion image to obtain the final target-level visible light infrared fusion image. In step S106 of this embodiment, the function expression for guiding the preliminary fusion result to obtain the final fusion result is:
[0102] ,
[0103] In the above formula, The coordinates of the final fusion result The pixel value at Pixel The filter window is centered at The size is , is the filter radius, The filter window The pixel coordinates in For coordinates The weight of Slice for visible light targets In coordinates The pixel value at , and the weighted calculation function expression is:
[0104] ,
[0105] In the above formula, Slice for visible light targets The local variance of the pixel, is the strength parameter. Assume the filter radius is , the strength parameter is , and for visible light images Apply mean filtering to obtain the local mean of each pixel ; Calculate each pixel Weight , the local variance of the pixel is Representation, the weight can be calculated.
[0106] In order to verify the real-time target-level visible light infrared image fusion system based on the drone platform in this example, the image data obtained by the visible light infrared imaging system on the DJI Mavic II drone was used for verification experiments. The resolution of the obtained visible light image is: 4000 3000, infrared resolution: 640 512, the image contents include three categories: armored vehicle model, tank model and truck model.
[0107] In order to accurately evaluate the performance of the proposed system and more intuitively compare the advantages and disadvantages of different fusion methods, two fusion evaluation indicators are mainly used to measure the fusion effect, namely local contrast and running time. The local contrast can reflect the prominence and clarity of the texture details in the fused image, and the running time can reflect the timeliness of the algorithm execution process. In the experiment, the method of this embodiment is compared with three existing methods, including Joint-BU (Kopf J, Cohen MF, Lischinski D, et al. Joint bilateral upsampling[J]. ACM Transactions on Graphics (ToG), 2007, 26(3): 96-es.), Pix (Lutio R, D'aronco S,Wegner JD, et al. Guided super-resolution as pixel-to-pixel transformation[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.2019: 8829-8837.), and UGSR (Gupta H, Mitra K. Toward unaligned guided thermal super-resolution[J]. IEEE Transactions on Image Processing, 2021, 31: 433-445.). Figure 3 to Figure 5 The fusion effect comparison between the method in this embodiment and three existing methods on a self-made data set is shown in Figure 2. Figure 3 This is a comparison chart of the fusion effect of the method in this embodiment and three existing methods on the first target. Figure 4 This is a comparison chart of the fusion effect of the method in this embodiment and three existing methods on the second goal. Figure 5 This is a comparison chart of the fusion effect of the method in this embodiment and three existing methods on the third goal. Figure 3 (a) Figure 4 (a) and Figure 5 (a) in the figure shows a visible light target slice, which has high clarity and a lot of texture details of the target. Figure 3 (b) Figure 4 (b) and Figure 5 (b) in the figure shows an infrared target slice, which is blurry and contains the temperature distribution information of the target. Figure 3 (c) Figure 4 (c) and Figure 5 (c) shows that the Joint-BU method smoothes the noise of the infrared target slice and fuses some texture information of the visible light image, but the image clarity is still low and the target cannot be clearly distinguished; Figure 3 (d) Figure 4 (d) and Figure 5 (d) shows that the Pix method improves the overall brightness of the image, enhances the edge information of the target, and has a certain degree of discrimination; Figure 3 (e) Figure 4 (e) and Figure 5 (e) shows that the UGSR method highlights the target, but introduces a lot of noise, making the target recognition in the image still low; Figure 3 (f) Figure 4 (f) and Figure 5 (f) in the figure shows that the method of this embodiment effectively fuses the texture information of the visible light target slice, retains the temperature information of the infrared target slice, and avoids the interference of useless background information, thus achieving good visual effects. Table 1 shows the local contrast of the fusion results obtained by the method of this embodiment and three existing methods on the self-made dataset. Table 2 shows the running time of the method of this embodiment and three existing methods on the self-made dataset.
[0108] Table 1: Local contrast of the method in this embodiment and three existing methods on the self-made dataset
[0109]
[0110] As can be seen from Table 1, the local contrast of the fusion result of the Joint-BU method is slightly reduced. This is because it blurs the edge of the target while removing noise, resulting in the loss of texture information. The local contrast of the fusion results obtained by the Pix and UGSR methods is slightly improved, but the fused visible light texture information is still insufficient. The method of this embodiment has a significant improvement in local contrast, fully integrating the texture information of visible light, making the local information more obvious.
[0111] Table 2: Running time of the method in this example and three existing methods on a self-made dataset
[0112]
[0113] According to Table 2, the method of this embodiment and Joint-BU have short running times and can basically achieve real-time fusion; the UGSR method needs to use a deep learning reasoning framework during the fusion process, and the fusion time is more than seconds; the Pix method fuses pixel by pixel, takes a long time to run, and is not real-time.
[0114] In summary, the method of this embodiment includes parsing the video stream taken by the drone to obtain paired visible light infrared images; performing target detection on the paired visible light infrared images respectively to obtain target slices of the two visible light infrared modes; performing coarse alignment on the target slices of the two modes so that the target slices of the two modes are basically aligned; using an unsupervised clustering method to divide the visible light target slices into regions and extract the texture information of the visible light target slices; taking the average of the temperature information of the corresponding region of the infrared target slice, and filling it into the visible light target slice after the region division to obtain an initial fused slice; using the visible light target slice to perform guided filtering on the initial fused slice, removing redundant information in the image, and obtaining the final target-level visible light infrared fused image. The method of this embodiment can realize target-level visible light infrared image fusion, has high stability, good real-time performance, and is suitable for the fusion of visible light infrared images with large resolution differences on drone platforms. The method of this embodiment can effectively solve the problem of high dependence of visible light infrared images on precise alignment in existing fusion technology, as well as the problem of loss of fused image information and poor visual effect caused by large differences in resolution of visible light infrared images. It can effectively fuse the complementary information of targets in visible light infrared images in real time when the resolution of visible light infrared images is greatly different, and improve the image fusion accuracy. The visible light infrared images taken by drones have differences in resolution and are difficult to align accurately. The method of this embodiment effectively extracts and fuses the complementary information of visible light infrared slices, so that the fusion results of visible light infrared images are still highly robust when the alignment accuracy is not high. Compared with the fusion method of traditional image processing, the method of this embodiment does not need to register and fuse the entire image, but only processes the target slice, and the calculation cost of the processing flow is simple, so it has obvious advantages in processing speed. Moreover, the method of this embodiment is implemented in the VScode integrated development environment, combined with the open source OpenCV machine vision image processing library, and the algorithm code of the program is written in C++. This software can run stably in WINDOWS7 and above operating systems, including embedded system environments, and has the advantage of good versatility.
[0115] In addition, this embodiment also provides a UAV visible light-infrared image target level fusion system, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the UAV visible light-infrared image target level fusion method.
[0116] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program / instruction, and the computer program / instruction is programmed or configured to execute the UAV visible light-infrared image target level fusion method through a processor.
[0117] In addition, this embodiment also provides a computer program product, including a computer program / instruction, which is programmed or configured to execute the UAV visible light-infrared image target level fusion method through a processor.
[0118] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0119] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A method for target-level fusion of visible light and infrared images of unmanned aerial vehicles, characterized in that: The steps include: S101, obtaining a pair of visible light original images and infrared original images collected by the UAV; S102, extracting a visible light target slice and an infrared target slice from the visible light original image and the infrared original image; S103, registering the visible light target slice and the infrared target slice; S104, performing semantic segmentation on the visible light target slice using an unsupervised method to obtain a visible light region segmentation map; S105, extracting temperature information from the infrared target slice and injecting it into the visible light region segmentation map to obtain a preliminary fusion result; S106, performing guided filtering on the preliminary fusion result to obtain the final fusion result; Step S102 includes: performing target detection on the visible light original image and the infrared original image respectively, cutting the visible light original image or the infrared original image according to the position of the detection result frame obtained by the target detection, and finally obtaining visible light target slices and infrared target slices. The target detection includes: S201, the input image The prediction output is obtained by processing through the convolutional neural network CNN ; S202, output according to prediction The center coordinates of each prediction box are calculated according to the following formula , width and height and confidence ; , , , In the above formula, is the sigmoid activation function, is the offset of the center coordinate of the prediction box, is the auxiliary item for the center coordinate offset, is the offset of the predicted box width and height, It is an auxiliary item for width and height offset. is the confidence level, is the confidence prediction value; S203, using a non-maximum suppression algorithm to remove prediction boxes with high overlap in each category, and only retaining the prediction box with the highest confidence as the final detection result box; Step S103 includes: S301, calculating the correlation coefficient between the visible light target slice and the infrared target slice according to the following formula: , In the above formula, represents the mutual correlation coefficient between the visible light target slice and the infrared target slice, 、 are the translation parameters in the xy directions, Slice for visible light targets Coordinates Pixels at Slice for infrared targets Coordinates The pixel at ; select the initial xy direction translation parameter as the xy direction translation parameter before iteration, and initialize the number of iterations to 1; S302, calculate the translation parameter in the xy direction after iteration according to the following formula: , , In the above formula, and is the translation parameter in the xy direction after iteration, and is the translation parameter in the xy direction before iteration, is the learning rate, is the cross-correlation coefficient between the visible light target slice and the infrared target slice; In step S303, the number of iterations is incremented by 1 to determine whether it is equal to a preset threshold. If not, the process proceeds to step S302. Otherwise, the visible light target slice and the infrared target slice are aligned using the final translation parameters in the iterative xy direction. The non-aligned portions of the visible light target slice and the infrared target slice are then removed to obtain the registered visible light target slice and the infrared target slice. The function expression for obtaining the final fusion result by performing guided filtering on the preliminary fusion result in step S106 is: , In the above formula, The coordinates of the final fusion result The pixel value at Pixels The filter window is centered on The size is , is the filter radius, For the filter window The pixel coordinates in For coordinates The weight of Slice for visible light targets In coordinates The pixel value at , and the weighted calculation function expression is: , In the above formula, Slice for visible light targets The local variance of the pixel, is the intensity parameter.
2. The method for target-level fusion of UAV visible light and infrared images according to claim 1, characterized in that: Acquiring the visible light original image and infrared original image captured by the drone in step S101 includes capturing corresponding frames of the video stream according to the frame number relationship between the visible light video stream and the infrared video stream to obtain a pair of visible light original images and infrared original images. The function expression of the frame number relationship between the visible light video stream and the infrared video stream is: , In the above formula, is the number of frames captured from the visible light video stream, is the number of frames captured from the infrared video stream, is the frame rate of the visible light video stream, is the frame rate of the infrared video stream.
3. The method for target-level fusion of UAV visible light and infrared images according to claim 1, characterized in that: Step S104 includes: S401, randomly selecting the centroids of K clusters for the visible light target slice; S402, calculating the Euclidean distances from all pixels in the visible light target slice to each centroid; S403: Based on the obtained Euclidean distance, all pixels in the visible light target slice are classified into the cluster with the smallest distance to the centroid. After all pixels are classified, the average value of the red R, green G and blue B channels of each pixel in each cluster is calculated to obtain the new centroid. , is the mean on the red R, is the mean value on green G, is the mean value on blue B; S404, determine whether the end condition is met, where the end condition refers to the number of iterations being equal to a preset threshold or the centroid no longer changing. If the end condition is not met, jump to step S402; otherwise, segment the visible light target slice into K regions using the K clusters obtained after classification to obtain a visible light region segmentation map.
4. The method for target-level fusion of UAV visible light and infrared images according to claim 1, characterized in that: The function expression for extracting temperature information from the infrared target slice and injecting it into the visible light region segmentation map in step S105 is: , In the above formula, Represents the first regions, For the The pixel points corresponding to the area, is the sum of the pixels in the nth region, is the pixel coordinate, Slice the infrared target Coordinates The pixel value at .
5. A visible light-infrared image target-level fusion system for unmanned aerial vehicles, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the target-level fusion method for visible light and infrared images of unmanned aerial vehicles described in any one of claims 1 to 4.
6. A computer-readable storage medium having a computer program / instruction stored therein, characterized in that: The computer program / instruction is programmed or configured to execute the target-level fusion method of the UAV visible light-infrared image according to any one of claims 1 to 4 through a processor.
7. A computer program product comprising a computer program / instructions, characterized in that The computer program / instruction is programmed or configured to execute the target-level fusion method of the UAV visible light-infrared image according to any one of claims 1 to 4 through a processor.
Citation Information
Patent Citations
Embedded platform-oriented dual-mode target detection method and system
CN113255521A
Crop disease and pest detection method and system based on multi-modal image, equipment and storage medium
CN113569786A
Building change detection method based on visible light and infrared satellite image fusion
CN118587537A
Object-level infrared-and-visible-light image fusion method based on fully convolutional neural network
WO2024174488A1
Cited By
Distributed multi-source image interactive fusion method based on multistage space-time calibration
CN119540074A
A distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration
CN119540074B
