Picture target object detection method based on fusion network

Abstract

The invention provides a picture target object detection method based on a fusion network. The method comprises the following steps: acquiring an infrared image of a to-be-detected target object picture and a visible light image of the to-be-detected target object picture; respectively inputting the infrared image and the visible light image into a fusion network, carrying out fusion processing and outputting, and further obtaining an output fusion image; wherein the fusion network comprises a first feature extraction module, a second feature extraction module and a decoding module which are connected in sequence; and performing detection processing operation on the output fusion image to determine a target detection object. Therefore, the precision and reliability of target detection are greatly improved in a scene that multiple unmanned aerial vehicles cooperatively execute tasks.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to the field of target detection. Background Art

[0002] Drones need to shuttle through different complex terrains and be deployed in areas where humans cannot survive. They can perform tasks in extremely harsh weather conditions and extremely dangerous areas, accurately detect and identify targets, and form a unique asymmetric advantage, thereby greatly improving detection efficiency. As the environmental situation becomes increasingly complex and diverse, various emergencies will occur in the videos shot by drones, such as occlusion, scale changes, in and out of view, lighting changes, and changes in movement speed.

[0003] The accuracy and reliability of target detection directly determine whether the drone can accurately locate and identify the mission target, which in turn affects the quality and efficiency of mission completion. With the increasing complexity of the environment and tasks, it is impossible to perceive complete target information through a single sensor or a single drone. Therefore, some tasks require multiple drones to work together to complete the task. When multiple drones work together to complete the task, the complex environment background will disrupt the effective visual information during the long-distance perception process, resulting in low light or exposure causing the image to lose the target, thus affecting the quality and efficiency of the drone mission. Therefore, in the process of multiple drones working together to perform tasks, how to improve the accuracy and reliability of target detection is undoubtedly a question of great research value. Summary of the invention

[0004] The present invention provides a method for detecting target objects in images based on a fusion network, which is used to solve the problem of low accuracy and reliability of target detection when multiple drones perform tasks in a collaborative manner.

[0005] To achieve the above object, the present invention provides a method for detecting target objects in images based on a fusion network, comprising: acquiring an infrared image of a target object image to be detected and a visible light image of the target object image to be detected;

[0006] The infrared image and the visible light image are respectively input into a fusion network, fused and output, and then a fused image is obtained; wherein the fusion network includes a first feature extraction module, a second feature extraction module and a decoding module which are connected in sequence.

[0007] A detection operation is performed on the output fused image to determine a target detection object.

[0008] Preferably, the infrared image and the visible light image are respectively input into a fusion network, fused and output, and the step of obtaining the output fused image specifically includes:

[0009] Inputting the infrared image and the visible light image into the first feature extraction module respectively, performing shallow feature extraction processing, and obtaining first infrared shallow feature information and first visible light shallow feature information;

[0010] Input the first infrared shallow feature information and the first visible light shallow feature information into the second feature extraction module respectively, perform deep feature extraction processing, and obtain thermal information of the target in the infrared image and texture information of the target in the visible light image;

[0011] The thermal information of the target in the infrared image and the texture information of the visible light image are fused to obtain a fused image, and the fused image is input into the decoding module for decoding processing to obtain the output fused image.

[0012] Preferably, the first feature extraction module includes a first convolutional layer, and the activation function adopted is a LeakyReLU function.

[0013] Preferably, the second feature extraction module comprises a first gradient residual dense block and a second gradient residual dense block which are connected in sequence.

[0014] Preferably, the decoding module includes a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and an output layer which are connected in sequence.

[0015] Preferably, the first gradient residual dense block comprises a dense flow branch and a residual flow branch.

[0016] Preferably, the specific steps of performing a detection processing operation on the output fused image to determine the target detection object are:

[0017] Based on the output fused image, determining network parameters of the fusion network;

[0018] Determine a quality-enhanced fusion image based on the network parameters of the fusion network; wherein the quality-enhanced fusion image is a fusion image with effectively improved visual quality;

[0019] After the quality-enhanced fused image is input into a visual perception enhancement module, an enhanced fused image is output;

[0020] The enhanced fused image is input into a target detection network for detection to determine the target detection object.

[0021] Preferably, the visual perception enhancement module includes the following two processing methods for the output fused image: illumination information compensation and Laplace sharpening.

[0022] The present invention has the following beneficial effects:

[0023] This application uses infrared images and visible light images of the target object to be detected to carry out fusion processing operations in the fusion network. Through such a processing method, the detail feature retention of the target object can be significantly improved. Thereafter, the fused image obtained by the fusion processing operation is preprocessed to determine the target detection object. Then, in the scenario where multiple drones perform tasks collaboratively, the accuracy and reliability of target detection are greatly improved. In this way, not only can the target object be identified more accurately, but also the stability and adaptability of the entire detection system in complex environments and mission conditions can be enhanced, providing more powerful technical support for drone collaborative operations, ensuring that various tasks can be completed efficiently and accurately.

[0024] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 It is a flowchart of a target object detection method according to a preferred embodiment of the present invention.

[0027] Figure 2 It is a flowchart of the overall structure of image processing according to a preferred embodiment of the present invention.

[0028] Figure 3 It is a schematic diagram of the process of obtaining the output fused image according to the preferred embodiment of the present invention.

[0029] Figure 4 It is a schematic diagram of the first gradient dense module structure of the preferred embodiment of the present invention.

[0030] Figure 5 It is a schematic diagram of the typical target layout of the experimental site of the present invention.

[0031] Figure 6 It is a schematic diagram of the deployment of UAVs and the actual position of the target in the experiment of the present invention.

[0032] Figure 7 Schematic diagram of the UAV platform used for conducting experiments in the present invention.

[0033] Figure 8 It is an experimental effect diagram of the large-field visible light guided infrared target detection process of the present invention.

[0034] Fig. 9It is an experimental effect diagram of the large-field infrared-guided visible light target detection process conducted by the present invention.

[0035] Fig.10 It is an experimental effect diagram of the target fusion effect under different lighting conditions of the experiment conducted by the present invention.

[0036] Fig.11 It is an experimental effect diagram of the single UAV and multi-UAV collaborative target detection result diagram of the experiment conducted by the present invention. DETAILED DESCRIPTION

[0037] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0038] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variants are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] The specific application scenarios and implementation process of this application are introduced as follows.

[0040] Nowadays, drones are active in all aspects of life, such as photogrammetry, logistics and distribution, and are widely used. In the application field of drones, there is a close and critical connection between target detection accuracy and mission completion. High-precision target detection technology can provide drones with accurate target identification information, ensuring that they can accurately perform operations in various tasks such as search and rescue, geographic mapping, etc. On the contrary, if the target detection accuracy is poor, the drone is very likely to deviate during the mission, and even cause the mission to fail to be successfully completed, which will significantly affect its application results in actual scenarios.

[0041] As the complexity of the environment and tasks continues to increase, it is impossible to perceive complete target information through a single sensor or a single drone. Therefore, some tasks require the cooperation of multiple drones to complete. When multiple drones work together to complete tasks, the complex environment background will disrupt effective visual information during the long-distance perception process, resulting in low light or exposure causing the image to lose the target, thus affecting the quality and efficiency of the drone mission. Therefore, in the process of multiple drones working together to perform tasks, how to improve the accuracy and reliability of target detection is a very worthy issue.

[0042] According to the above, in order to solve the problem of low target detection accuracy, this application proposes a method for detecting target objects in images based on a fusion network, such as Figure 1 As shown, this application includes the following specific steps:

[0043] S101: Acquire an infrared image of a target object to be detected and a visible light image of the target object to be detected;

[0044] With the increasing complexity of the environment and tasks, it is impossible to perceive complete target information with only a single sensor or a single drone. Therefore, the application scenario mentioned in this application, collaborative target detection, is mainly completed by the cooperation of multiple drones to detect the target. More specifically, this application is based on multiple drones shooting the target object from different perspectives, and the obtained dual-light images (visible light and infrared images) are fused, so as to improve the field of view and accuracy of target detection based on the processing of the fused image.

[0045] During the flight, the drone will capture and process images of the route. The target object to be detected mentioned here can be understood as the object that the drone needs to complete the specified task. For example, in the distribution field, the drone needs to accurately locate and identify the field of item distribution. Here, the scope of the field of item distribution can be regarded as the target object to be detected in this application. It should be noted that the target object to be detected in this application is determined according to the scenario of the specific task application and is not specifically limited here.

[0046] The picture of the target object to be detected can be understood as the picture obtained after the drone takes a picture of the target object to be detected in the process of completing the specified task. Since the taken picture may include other information, not just the target object to be detected. Therefore, in general, the target object to be detected is located in the picture of the object to be detected. It should be noted that since multiple drones are completing the task. Therefore, the pictures of the object to be detected taken here are taken by different drones based on different angles, with clear picture quality and unique field of view, which helps to quickly identify the object to be detected.

[0047] In this application, the infrared image of the target object to be detected and the visible light image of the target object to be detected are mainly used to achieve image fusion. Figure 2 In the figure, the picture on the left is an infrared image of the target object to be detected, and the picture on the right is a visible light image of the target object to be detected. The dual-light image is used in this application to detect the target object because the dual-light fusion image has obvious advantages. It can achieve information complementarity, and the visible light and infrared light information complement each other, enhance the target recognition ability, and improve the recognition. It can also improve environmental adaptability, overcome lighting changes, enhance penetration ability, and assist in analysis and decision-making, with high application value.

[0048] S102: Input the infrared image and the visible light image into a fusion network respectively, perform fusion processing and output, and then obtain an output fused image; wherein the fusion network includes a first feature extraction module, a second feature extraction module and a decoding module connected in sequence.

[0049] like Figure 2 As shown, the fusion network 210 of the present application mainly includes a first feature extraction module 211, a second feature extraction module 212, and a decoding module 213. The first feature extraction module 211 in the present application generally realizes the extraction of shallow features of the image, the second feature extraction module 212 generally realizes the extraction of deep features of the image, and the decoding module 213 is generally a decoder in the field of image processing, and no specific restrictions are made here. The fusion network 210 has powerful functions and advantages. It can integrate different image features and realize multi-source information fusion. It can enhance the image expression and improve the accuracy of tasks such as target detection and segmentation. It effectively overcomes the limitations of a single data source and provides a more comprehensive and accurate solution for image processing.

[0050] According to the structural diagram of the above fusion network 210, combined with Figure 3 Here, the steps of inputting the infrared image and the visible light image into the fusion network 210 respectively, performing fusion processing and outputting them, and then obtaining the output fusion image are specifically described:

[0051] Step S301: inputting the infrared image and the visible light image into the first feature extraction module 210 respectively, performing shallow feature extraction processing, and obtaining first infrared shallow feature information and first visible light shallow feature information;

[0052] like Figure 2As shown, the first feature extraction module 211 includes a first convolution layer, and the activation function used is the LeakyReLU function. The first convolution layer is a common convolution layer, which performs a convolution operation (3*3conv) with a convolution kernel size of 3×3 and a batch normalization operation (BN) on the input image, and uses the LeakyReLU activation function (ReLU) to extract shallow features.

[0053] Therefore, the first infrared shallow feature information in this application can be understood as the shallow features extracted by the public convolution layer after performing a convolution operation (3*3conv) with a convolution kernel size of 3×3, a batch normalization operation (BN), and a LeakyReLU activation function (ReLU) on the infrared image of the target object to be detected. It should be noted that the shallow features here are 16 channels.

[0054] In this application, the first infrared shallow feature information can be understood as the shallow features extracted by the public convolution layer after performing a convolution operation (3*3conv) with a convolution kernel size of 3×3, a batch normalization operation (BN), and a LeakyReLU activation function (ReLU) on the visible light image of the target object to be detected. It should be noted that the shallow features here are 16 channels.

[0055] It should be noted that in the present application, when executing step 301, when extracting shallow features, the Sobel gradient operator is used for edge detection to find the turning points where the brightness changes significantly in the image, distinguish the high-frequency bands of edges and noise, calculate the gradient information and direction, and finally detect the boundary based on the gradient information.

[0056] Step S302: inputting the first infrared shallow feature information and the first visible light shallow feature information into the second feature extraction module respectively, performing deep feature extraction processing, and obtaining thermal information of the target in the infrared image and texture information of the target in the visible light image;

[0057] The first infrared shallow feature information and the first visible light shallow feature information obtained in step S301 are respectively input into the second feature extraction module 220 for deep feature extraction processing to obtain the thermal information of the target in the infrared image and the texture information of the target in the visible light image.

[0058] like Figure 2 As shown, the second feature extraction module in the present application is mainly composed of two layers, including a first gradient residual dense block 221 and a second gradient residual dense block 222 connected in sequence. The first gradient residual dense block 221 inputs 16-channel features, outputs 32-channel features, which are input into the second gradient residual dense block 222, and outputs 48-channel features.

[0059] The first gradient residual dense block 2121 in the present application is mainly an RDG module. The present application adopts the RDG module because the RDG module can significantly enhance the detail features of the image. When processing low-resolution images, the image details can be refined and enhanced through its advanced algorithm. The specific implementation process is described in detail as follows.

[0060] like Figure 4 As shown, the first gradient residual dense block includes a dense stream branch 410 and a residual stream branch 420 .

[0061] The dense flow branch 410 uses two densely connected convolutional layers with a kernel size of 3×3 and a common convolutional layer with a kernel size of 1×1. The dense connection means that the input features of the gradient residual dense block are used as the inputs of two convolutional layers (ConvBNReLU) and the common convolutional layer (Conv), and the outputs of the two convolutional layers are used as the inputs of the common convolutional layer. Jump dense connections are introduced to make full use of the features extracted by various convolutional layers.

[0062] The residual stream branch 420 includes a Sobel operator and a standard convolution layer with a kernel size of 1×1. The input features of the residual stream branch are calculated by the operator to obtain the gradient amplitude, and then the size difference of the channel is eliminated by the standard convolution layer, and then the output features of the gradient residual dense block are obtained by summing with the output of the dense stream branch.

[0063] In the present application, the depth features and fine-grained detail features of the image of the target object to be detected are integrated by element-wise stacking of the outputs of the dense stream branch 410 and the residual stream branch 420 .

[0064] The quality of the fused image depends largely on the loss function. The fusion network 210 in the present application designs a loss function based on content loss. Assuming that a pair of registered visible light and infrared images are I k ,I h , the fused image is I f , using the feature extraction module E F , deep features with rich fine-grained detail information are extracted from infrared and visible light images and can be expressed as:

[0065] {F ki ,F hi}={E F (I ki ),E F (I hi )} (1)

[0066] The feature representation of the gradient residual dense module (it should be noted that the gradient residual dense module mentioned here can be the first gradient residual dense module or the second gradient residual dense module) is (2), where Fi is the input feature.

[0067]

[0068] In order to better fuse the salient targets in the infrared image and the texture details in the visible light image, the content loss is designed to ensure the visual fidelity of the fused image. Let the content loss be L con , including the overall intensity loss L of the fused image int and texture detail loss L text , the intensity loss measures the difference between the fused image and the original image at the pixel level, the texture loss represents the difference in fine-grained features, and the content loss function is:

[0069] L con =L int +αL text (3)

[0070] The intensity loss of the content loss component is defined as:

[0071]

[0072] Where H and W are the height and width of the image, ||||1 represents the f1-norm, and max() represents the maximum selection at the element level.

[0073] The texture loss is defined as follows, which calculates the difference in gradient values ​​to compare the texture changes between the fused image and the original image.

[0074]

[0075] The α parameter plays a role in adjusting the balance between the two losses, limiting the intensity loss and texture loss within a certain range. The fusion network based on the gradient residual dense block can achieve a better intensity distribution and maintain rich detail information under the guidance of the content loss. The content loss can effectively improve the visual quality and statistical evaluation indicators of the fused image.

[0076] It should be noted that since there is almost no difference in structure and function between the second gradient residual dense block 2122 and the first gradient residual dense block 2121, this will not be described in detail here, and the structure and implemented functions of the first gradient residual module can be referred to.

[0077] Step S303: After fusing the thermal information of the target in the infrared image and the texture information of the target in the visible light image, a fused image is obtained, and the fused image is input into the decoding module for decoding processing to obtain the output fused image.

[0078] like Figure 2As shown, the second gradient residual dense block 2122 outputs the thermal information of the target in the infrared image and the texture information of the target in the visible light image respectively, and obtains a fused image after fusion. The fused image is input into the decoding module 213 for decoding processing, so as to obtain the output fused image.

[0079] like Figure 2 As shown, the structure of the decoding module 213 includes a second convolution layer 2131, a third convolution layer 2132, a fourth convolution layer 2133, and an output layer 2134 connected in sequence. The decoding module 213 is used to perform a convolution operation with a convolution kernel size of 3×3, a batch normalization operation, and an activation using a LeakyReLU activation function on the features of the input fused image for decoding and restoration.

[0080] It should be noted that the second convolution layer 2131, the third convolution layer 2132, and the fourth convolution layer 2133 perform approximately the same processing on the fusion features of the input image, so they will not be described one by one.

[0081] The only difference is that Figure 2 As shown, after the fusion features of the input image are operated in the second convolution layer 2131, the output image features are 48 channels, which are then input into the third convolution layer 2132. The output image features are 32 channels, and the 32-channel image features are input into the fourth convolution layer 3133, and the output image features are 16 channels, which are then transmitted to the output layer 2134 for corresponding processing.

[0082] The output layer 2134 is used to perform a convolution operation with a convolution kernel size of 1×1 and a batch normalization operation on the input image features, and activate them using a Tanh activation function to obtain an output fused image.

[0083] S103: Performing a detection processing operation on the output fused image to determine a target detection object.

[0084] like Figure 2 As shown, the specific steps of performing the detection processing operation on the output fusion image to determine the target detection object are:

[0085] Step S501: based on the output fused image, determining the network parameters of the fusion network; based on the network parameters of the fusion network, determining the quality-enhanced fused image; wherein the quality-enhanced fused image is a fused image with effectively improved visual quality; outputting the quality-enhanced fused image to obtain an enhanced fused image;

[0086] In the process of obtaining the output fused image, the fusion network 210 is used, and there is a one-to-one correspondence between the output fused image and the network parameters of the fusion network 210. Therefore, when the output fused image is obtained, the network parameters of the fusion network are also obtained. After obtaining the output fused image and determining the network parameters of the fusion network, the quality-upgraded fused image is determined based on the network parameters of the fusion network, that is, the loss function based on the content loss is adjusted to balance the intensity loss and the texture loss, so as to obtain a fused image with effectively improved visual quality. After the quality-upgraded fused image is input into the visual perception enhancement 220 module, it is enhanced and the enhanced fused image is output. The visual perception enhancement module 220 includes the following two processing methods for the output fused image: illumination information compensation and Laplace sharpening.

[0087] For small targets in the fused image (here mainly refers to small targets that are small in size and difficult to be found in the image), the illumination information is first compensated. When the light is too dark, the night image needs illumination information to eliminate the influence of some darkness and "supplement light" to the image. That is, the visual perception enhancement module 220 restores the attribute color of the fused image, greatly eliminating the impact of poor lighting conditions on the image quality, and makes small targets that are difficult to detect appear. At the same time, Laplace sharpening is performed to enhance the edge details of small targets in the image, so that the subsequent target detection network 230 can extract richer shallow features.

[0088] In the visual perception enhancement module, the more detailed steps of fusion image processing are as follows:

[0089] The visual perception enhancement module first starts with the minimum image obtained by downsampling the image pyramid, compares each pixel with the adjacent pixels in the other 8 directions, and uses the Gaussian center function to estimate the image reflection component R; proceeds downward in sequence, and after the calculation of the previous layer is completed, the estimated image reflection component is interpolated so that the interpolated reflectivity estimation image is the same size as the next layer image, and then the same comparison operation is performed on the next layer; finally, the final result is obtained after the adjacent pixels of the original image are compared, and the enhanced image without illumination is obtained. In this calculation process, it is assumed that S1, S2, ..., S m are points on the path, arranged from far to near, where r c Represents the final reflectance estimate at the center location.

[0090] The calculation steps are as follows:

[0091] 1. Transform the original image to the logarithmic domain S(x,y) to obtain the number of pyramid layers;

[0092] 2. Initialize the constant image matrix R0(x,y) as the initial value of the image for iterative calculation;

[0093] 3. Perform eight-neighborhood pixel comparison operations from the top layer to the last layer, and calculate the reflectance R of the pixels on the path according to the Retinex theory

[22] according to formulas (6) and (7). After the operation of the nth layer is completed, the operation result R of the nth layer is interpolated to twice the original size, which is the same as the size of the n+1th layer. When the bottom layer is calculated, the obtained R is the final de-illuminated enhanced image.

[0094] S m =r m +l m (6)

[0095]

[0096] Among them, r m is the logarithmic value of the original reflection component of the image, l m Represents the logarithmic value of the incident illumination component of the image.

[0097] For the de-illuminated fused image, sharpening is performed to compensate for the image contour, enhance the image edge and grayscale jump part, and improve the image clarity and contrast. By comparing the grayscale of the central pixel with the average grayscale of other pixels in the neighborhood, when the central pixel grayscale is lower than the average grayscale of other pixels, the template convolution result of the enhanced fused image is added to the original pixel to further reduce the central pixel grayscale; when the central pixel grayscale is higher than the average grayscale of other pixels, the template convolution result of the enhanced fused image is added to the original pixel to further increase the central pixel grayscale.

[0098] Step S502: Input the enhanced fused image into the target detection network 230 for detection processing to determine the target detection object.

[0099] The target detection network 230 is used in images or videos to extract features, predict target categories and positions, and obtain the final target detection results after processing. Therefore, in this application, after the fused image is enhanced in the visual perception enhancement module 220, an enhanced fused image is obtained, and the image is input into the target detection network 230 for detection processing, thereby determining the target detection object.

[0100] In order to demonstrate the performance of the method proposed in this application for typical UAV ground-view target detection, the algorithm was trained and tested using different targets in self-constructed ground scenes to verify that the proposed method has good detection performance.

[0101] Experimental environment

[0102] The experimental site selected in this paper is located in an experimental field, which has grassland, jungle, cement road, buildings, lakes and other environmental facilities. Multiple targets (including tank models, HIMARS models, sniper models, mortar models, etc., which are almost impossible to observe in the horizontal scene) are randomly deployed in the grass. Figure 5 shown.

[0103] Multiple drones detect targets in the experimental site from three perspectives at pre-set positions, as shown in the schematic diagram. Figure 6 shown.

[0104] Experimental Platform

[0105] This application built the following experiment Figure 7 The drone platform shown is used for the collection of typical ground target data and the deployment verification of the algorithm (the algorithm provided by the method of this application). The drone platform is equipped with an optoelectronic pod (with visible light and infrared sensors) for real-time acquisition of ground optoelectronic images to achieve perception of the ground environment. In addition, the drone is also equipped with an inertial navigation module and GPS to obtain the drone's own attitude and position information, and then performs online target detection on the aircraft through the edge computing platform and uses the communication radio module to establish communication with the ground end to achieve data transmission and information interaction.

[0106] Multi-UAV collaborative detection experiment

[0107] 1) Large field of view visible light guides infrared fine identification

[0108] Under good lighting conditions, the algorithm in this paper adaptively selects visible light to guide infrared for active perception. Figure 8 The process shown is mainly used to test the multi-mode collaborative detection capability of drones in a wide area. UAV1 uses a visible light sensor with a large field of view to detect regional targets. After finding the target, it guides UAV2 and UAV3 to detect the target using an infrared sensor with a small field of view through azimuth. The detailed steps are as follows:

[0109] ① After the mission begins, three drones patrol the target area, and UAV1 uses a visible light sensor to search the target area with a large field of view. At this time, due to the high altitude of the host and the small zoom factor, the target size is usually small and the features are not obvious;

[0110] ② When UAV1 finds the target, it executes the single-machine detection algorithm, firstly performs a rough detection of the target, and the recognition result is as follows: Figure 8As shown in the second column, the blue rectangle in the figure indicates that the target has been detected and the detection confidence is higher than 30%. Subsequently, UAV1 actively selects low-confidence targets: target 1 (confidence is 35%), target 2 (confidence is lower than 30%), and sends the spatial position to UAV2 and UAV3, guiding the infrared sensor of the slave to perform fine recognition;

[0111] ③UAV2 and UAV3 receive the target position information, rotate the pods to make the low-confidence target in the center of the picture, and obtain the target fine recognition results. Target 1 is identified as a tank with a confidence of 83%, and target 2 is identified as a HIMARS with a confidence of 67%.

[0112] By comparing the target categories and positions with those actually deployed, the orientation information transmitted by UAV1 to UAV2 and UAV3 is accurate, and the three UAV nodes can provide detection results for all set targets within the site through collaborative active perception.

[0113] 2) Large field of view infrared guides visible light for fine identification

[0114] In bad weather or low light conditions, infrared images can still maintain a certain degree of clarity and contrast, showing the radiation difference between the target and the background, but cannot provide detailed appearance information of the target. Visible light images can provide certain contour information, effectively making up for the low resolution of infrared images. Therefore, in low light conditions, Fig. 9 For this paper, the method of adaptively selecting infrared to guide visible light for active perception process is used. UAV1 uses the infrared sensor with a large field of view to detect regional targets. After finding the target, it guides UAV2 and UAV3 to use the visible light sensor with a small field of view to detect the target through azimuth. The detailed steps are as follows:

[0115] ① After the mission begins, three drones patrol the target area, and UAV1 uses an infrared sensor to conduct a wide field of view search in the target area. At this time, due to the high flight altitude of the host and the small zoom factor of the infrared sensor, the target size is usually small and the features are not obvious;

[0116] ② When UAV1 finds the target, it executes the single-machine detection algorithm and first performs a rough recognition of the target. The recognition result is as follows: Fig. 9 As shown in the second column, the blue rectangle in the figure indicates that the target has been detected and the detection confidence is higher than 30%. Subsequently, UAV1 actively selects low-confidence targets: target 1 (confidence is 39%), target 2 (confidence is lower than 30%), calculates their spatial positions, and sends them to UAV2 and UAV3, guiding the visible light sensor of the slave to perform fine recognition;

[0117] ③UAV2 and UAV3 receive the target position information, rotate the pods to make the low-confidence target in the center of the picture, and obtain the target fine detection results. Target 1 is identified as HIMARS, and target 2 is identified as a tank, with confidence levels above 80%.

[0118] Table 1 records 4 sets of test data at different times and for different target types, and statistically analyzes the changes in target detection effects and detection confidence before and after the active perception algorithm is applied. In the column "Whether to use the active perception algorithm", "√" indicates that the active perception algorithm is used, and "×" indicates that the active perception algorithm is not used. In the column "Target detection effect", "√" indicates that the target is detected, and "×" indicates that the target is not detected. It can be seen from the table that after using the multi-UAV collaborative active perception algorithm, the target detection effect and detection confidence have been significantly improved.

[0119] Table 1

[0120]

[0121] Fig.10 The fusion effect of the algorithm in this paper under different lighting conditions, and the weight of the visible light mode during fusion is printed below the image, from top to bottom, there are three different scenes: morning, noon, and evening. As can be seen from the figure, the lighting conditions at noon are the least, and the thermal radiation information contained in the image is the least. Therefore, the overall background color of the image is closest to the background of the visible light image. Compared with noon, the lighting conditions in the morning are slightly weaker, so the infrared weight accounts for a slightly larger proportion, and the target has certain infrared thermal radiation information. The lighting conditions in the evening are the weakest, so the infrared weight accounts for the largest proportion in the fused image, the overall image is closer to the grayscale image, and the target and the background are also clearly distinguished. After calculation, in the morning, the proportion of infrared weight is about 30%-70%. At noon, the proportion of infrared weight is about 0%-30%, and in the evening, the proportion of infrared weight is about 70%-100%. This shows that the algorithm has good adaptive ability under different lighting conditions.

[0122] 3) UAV multi-mode target detection

[0123] Prepare three UAV nodes, each equipped with an edge computing platform based on ARM+FPGA acceleration, a task scheduling board module, a multi-mode optoelectronic pod, a self-organizing network radio, and a UAV attitude and positioning device. Arrange a ground radio on the ground side and connect it to a display and control computer to display UAV status data and fusion perception processing results, and test whether the control and communication link systems between the airborne and ground sides are normal.

[0124] like Fig.11The following are the results of collaborative target detection of a single UAV (first column) and multiple UAVs (last three columns) in the field experiment. It can be seen that the single UAV detection results have missed detections (the ellipse position in the figure), while the missed targets can be detected through active perception of multiple UAVs, with an average accuracy of 75.8%, proving the effectiveness of this method.

[0125] In summary, this application studies a task-driven UAV target detection method based on a convolutional fusion network and visual perception enhancement using visible light and infrared fusion to improve the retention of target detail features. This application designs a multi-UAV collaborative active perception process based on a distributed bionic cluster, making full use of multiple UAV detection units to achieve collaborative detection of unknown targets in unknown environments. Experimental results show that the average accuracy of this multi-UAV ground target detection algorithm reaches 75.8%, with a latency of 56.55ms and an inference speed of 45.84FPS on an intelligent edge computing platform, and has certain robustness and real-time performance.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting target objects in images based on a fusion network, characterized in that: include: Acquire an infrared image of a target object to be detected and a visible light image of the target object to be detected; The infrared image and the visible light image are respectively input into a fusion network, fused and output, and then a fused image is obtained; wherein the fusion network comprises a first feature extraction module, a second feature extraction module and a decoding module connected in sequence; A detection processing operation is performed on the output fused image to determine the target detection object.

2. The method according to claim 1, characterized in that The steps of inputting the infrared image and the visible light image into the fusion network respectively, performing fusion processing and outputting them, and then obtaining the output fused image specifically include: Inputting the infrared image and the visible light image into the first feature extraction module respectively, performing shallow feature extraction processing, and obtaining infrared shallow feature information and visible light shallow feature information; Input the first infrared shallow feature information and the first visible light shallow feature information into the second feature extraction module respectively, perform deep feature extraction processing, and obtain thermal information of the target in the infrared image and texture information of the target in the visible light image; The thermal information of the target in the infrared image and the texture information of the visible light image are fused to obtain a fused image, and the fused image is input into the decoding module for decoding processing to obtain the output fused image.

3. The detection method according to claim 2, characterized in that: The first feature extraction module includes a first convolutional layer, and the activation function adopted is a LeakyReLU function.

4. The detection method according to claim 2, characterized in that: The second feature extraction module includes a first gradient residual dense block and a second gradient residual dense block which are connected in sequence.

5. The detection method according to claim 2, characterized in that: The decoding module includes a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and an output layer which are connected in sequence.

6. The detection method according to claim 4, characterized in that: The first gradient residual dense block includes a dense flow branch and a residual flow branch.

7. The detection method according to claim 1, characterized in that: The specific steps of performing the detection processing operation on the output fusion image to determine the target detection object are: Based on the output fused image, determining network parameters of the fusion network; Determine a quality-enhanced fusion image based on the network parameters of the fusion network; wherein the quality-enhanced fusion image is a fusion image with effectively improved visual quality; After the quality-enhanced fused image is input into a visual perception enhancement module, an enhanced fused image is output; The enhanced fused image is input into a target detection network to determine the target detection object.

8. The detection method according to claim 7, characterized in that: The visual perception enhancement module includes the following two processing methods for the output fused image: illumination information compensation and Laplace sharpening.

Images