A target detection method based on fusion of two modal images

Abstract

The application discloses a target detection method based on two kinds of modal image fusion, which comprises the following steps: based on the AirSim platform, a related scene is built to carry out simulation and simulation, and image data sets of different modes are collected; a dual-modal image fusion network is constructed, and the fusion network is trained according to the back propagation principle; the Siamese neural network is used to extract the features of the images of the two kinds of modes at the same time, and the features are reconstructed into a fusion image through a deconvolution operation; the quality of the reconstructed fusion image is tested and evaluated through a quantitative evaluation index, and the fusion network is modified according to the evaluation result; the fusion features output by the trained fusion network are directly input into a single-stage YOLOv7 target detection network, the network parameters of the trained fusion network are frozen, the target detection network is trained, and an optimized fusion and detection network is obtained. The application is convenient for realizing efficient and accurate dual-modal image fusion and detection, and is helpful to improve the target detection capability of the unmanned aerial vehicle in emergency search and rescue scenes.

Description

Technical Field

[0001] This invention relates to the field of image fusion and target detection technology, and more specifically to a target detection method based on the fusion of two modal images. Background Technology

[0002] With the decrease in the cost of drones and the increase in computing resources, drones equipped with deep learning-based target detection methods and optical sensors can ignore terrain and perform photoelectric detection on the ground to accurately and timely detect targets.

[0003] However, a single sensor is insufficient to effectively reflect all the information of a scene, especially in drone emergency rescue scenarios. Visible light images alone are difficult to reflect all the environmental information under adverse weather conditions. Infrared images, while ignoring light intensity, lack texture information of objects. Synthetic Aperture Radar (SAR) has the characteristics of all-weather operation and cloud penetration, but due to a large amount of speckle noise, it is difficult to interpret relevant information. Depth cameras use photoelectric elements to convert the light signals of the target scene into electrical signals. They are passive and produce images with rich detail information. They have the advantages of long range, high resolution, and good concealment. The disadvantage is that they are relatively sensitive to light, and the data accuracy is affected in strong or low light environments. Fusion of images from different modalities can complement each other's advantages and improve the detection capabilities of drones. At the same time, the computing resources of drone onboard platforms are limited, and current deep learning-based target detection methods consume a lot of computing resources, resulting in poor detection capabilities for drones.

[0004] Therefore, how to improve the accuracy and efficiency of target detection in fused images, so as to enhance the target detection capabilities of UAVs in emergency search and rescue scenarios, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a target detection method based on two-modal image fusion that at least solves some of the above-mentioned technical problems, which facilitates efficient and accurate dual-modal image fusion and detection, and helps to improve the target detection capability of UAVs in emergency search and rescue and other scenarios.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] This invention provides a target detection method based on two-modal image fusion, the method comprising:

[0008] Build relevant scenarios for simulation and collect image datasets of different modalities;

[0009] A dual-modal image fusion network is constructed and trained according to the backpropagation principle. The Siamese neural network is used to extract features from the images of the two modalities simultaneously, and the features are reconstructed into a fused image through deconvolution operation.

[0010] The quality of the reconstructed fused image will be tested and evaluated using quantitative evaluation metrics, and the fusion network will be modified based on the evaluation results.

[0011] The fusion features output by the fusion network are directly input into a single-stage YOLOv7 object detection network before the fusion image is reconstructed. The network parameters of the fusion network are frozen, and the object detection network is trained to obtain an optimized fusion and detection network.

[0012] Target detection is performed using an optimized fusion and detection network.

[0013] Preferably, relevant urban and outdoor scenes are built on the AirSim platform, including rain, snow, fog, fire, and flood scenes; the environmental season and time are changed to simulate changes in different light intensities; the sensors of visible light camera, infrared camera and depth camera are turned on, and the drone is controlled to switch different positions and perspectives to collect image datasets with different modalities; the image dataset is divided into training set, validation set and test set, and offline image enhancement operations are performed.

[0014] Preferably, a recursive gated convolutional structure is introduced into the fusion network to extract information within the deep feature domain and information between the superimposed fusion feature domains through the recursive gated convolutional mining.

[0015] Preferably, the fusion network includes a feature extraction module, a feature fusion module, and an image reconstruction module, wherein:

[0016] The feature extraction module includes early convolution extraction and intra-domain and inter-domain information mining based on recursive loop gate convolution.

[0017] The feature fusion module is used to fuse the extracted features;

[0018] The image reconstruction module includes convolution and activation functions, which are used to reconstruct the fused image from the fused features.

[0019] Preferably, after extracting features from two image modalities simultaneously using a Siamese neural network, the extracted features are fused to obtain fused feature I. F =Concat(I1+I2), and based on the requirements for image fusion, the loss function is defined as:

[0020] L = L SSIM +L m

[0021] LSSIM =w1×(1-ssim(I f ,I1))+w2×(1-ssim(I f ,I2))

[0022] L m =|H(I1)+H(I2)-H(I) f )|

[0023] Among them, L SSIM L represents the structural similarity loss. m I represents the information entropy loss, I1 represents the features extracted from the first modality of the image, I2 represents the features extracted from the second modality of the image, and I... F The fusion represents the features obtained; w1 represents the weight coefficient of the first modality image, w2 represents the weight coefficient of the second modality image, H(·) represents the calculation of the information entropy of the image, and ssim(·) represents the calculation of the structural similarity of the image.

[0024] Preferably, the evaluation metrics include: feature mutual information, structural similarity, and peak signal-to-noise ratio. The calculation formula for the evaluation metrics is as follows:

[0025]

[0026]

[0027]

[0028] Where FMI represents feature mutual information, SSIM represents structural similarity, PSNR represents peak signal-to-noise ratio, and I F1 I represents the amount of feature information of the first modality image contained in the fused image. F2 Let μ1 represent the amount of feature information from the second modality image contained in the fused image, H represent the information entropy of the different modalities, μ1 represent the average pixel value of the first modality image, and μ2 represent the average pixel value of the second modality image. This represents the pixel variance of the first modality image. σ represents the pixel variance of the second modality image. 12 Let I(i,j) and K(i,j) represent the pixel covariance of the two modal images, respectively, where i and j represent the corresponding image coordinates, m represents the width of the image, n represents the height of the image, and C1 and C2 are constants.

[0029] Preferably, gradient descent is used to train the object detection network.

[0030] Preferably, a normalized Gaussian distance function is used to replace the target loss of YOLOv7, and the calculation formula is as follows:

[0031] L loc =1-NWD(N a N b )

[0032]

[0033]

[0034]

[0035]

[0036] Among them, L loc N represents the localization loss. a N b Let W2 represent the Gaussian distributions of the predicted anchor frame and the ground truth anchor frame, respectively. NWD represents the calculated normalized Gaussian distance between the two distributions. 2 (·) denotes the calculation of the second-order Gaussian distance, C represents the average absolute size of the targets in the dataset, f(x|μ,Σ) represents the probability density function of the second-order Gaussian distribution, and x, μ, and Σ represent the coordinates, mean vector, and covariance matrix of the Gaussian distribution, respectively. x C y Let ω and h represent the center coordinates of the anchor frame, respectively, and let h and h represent the width and height of the anchor frame, respectively.

[0037] Preferably, a parameterless attention module is embedded in the object detection network to calculate attention weights and enhance the feature map.

[0038] Compared with the prior art, the present invention has at least the following beneficial effects:

[0039] This invention provides a target detection method based on two-modal image fusion. Firstly, considering the difficulty in collecting relevant datasets for target application scenarios, the method simulates the relevant scenarios using the AirSim simulation platform to collect datasets. Then, it leverages the advantages of different modal images for fusion and complementarity, improving the quality of the fused image. Furthermore, considering the limited onboard computing resources of UAVs, the fusion and detection networks are optimized. This improves detection accuracy and reduces computational energy consumption, facilitating efficient and accurate dual-modal image fusion and detection. Ultimately, this enhances the UAV's target detection capabilities in various scenarios, helping to maximize the advantages of UAVs in emergency rescue scenarios.

[0040] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0041] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0044] Figure 1 This is a schematic diagram of a target detection method based on the fusion of two modal images provided in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of the fusion network structure provided in an embodiment of the present invention.

[0046] Figure 3 This is a schematic diagram of the detection network structure provided in an embodiment of the present invention.

[0047] Figure 4 This is a schematic diagram of the visible light image before fusion, provided for an embodiment of the present invention.

[0048] Figure 5 A schematic diagram of the infrared image before fusion provided in an embodiment of the present invention.

[0049] Figure 6 This is a schematic diagram of the reconstructed fused image provided in an embodiment of the present invention.

[0050] Figure 7 This is a schematic diagram of the target detection result of the fused image provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. In addition, various numbers, etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0052] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0053] In the description of this invention, it should be noted that some processes described in this application specification and drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may be performed in any order or in parallel.

[0054] See Figure 1 As shown, this embodiment of the invention provides a target detection method based on the fusion of two modal images. The method mainly includes:

[0055] 1) Real-time scene simulation construction: The simulation includes special weather conditions such as rain, snow, fog, fire, and flood. The simulation uses the perspective of a drone to collect the data set by simulating real drone flight. The labeled image dataset is divided into training set, validation set, and test set, and offline image enhancement is performed.

[0056] 2) Modify the YOLOv7 backbone network by introducing a recursive gated convolutional structure. Use a Siamese network to simultaneously extract features from both modalities of the image. After fusion, deconvolution is used to restore the fused image. Based on the backpropagation principle, train the network, and after multiple iterations, allow the loss function to stabilize, obtaining a fused image that contains more information and has a structure similar to the original image. Simultaneously, considering the generally different resolutions of images from different modalities, deconvolution is used to adjust the image to a feature map of the same size.

[0057] 3) Quantitative indicators are used to examine the quality of the fused image. The feature fusion network is modified (the main modification method is to change the number of layers in the inter-domain information mining module) and the network depth is adjusted to improve the quality of the fused image.

[0058] 4) Connect the trained fusion network to the subsequent feature extraction network and the downstream detection head, and use the weights of the YOLOv7 pre-trained model to replace the weights of the same part of the network to train the object detection network, thus obtaining the trained fusion and detection network.

[0059] To better illustrate the steps of the present invention, more specific implementation steps are as follows:

[0060] In step 1) above, on the installed AirSim platform, relevant urban and outdoor scenes are built. Different weather conditions such as wind, sand, rain, and snow are rendered for different scenes, and the environmental season and time are changed to simulate changes in different light intensities. Visible light cameras, infrared cameras, and depth cameras are activated, and the drone is controlled to switch between different positions and perspectives to collect diverse image datasets. Finally, the collected datasets are enhanced by translation, flipping, noise perturbation, etc.

[0061] In step 2) above, after simultaneously extracting features I1 and I2 from the two modalities using a Siamese neural network, the extracted features are fused to obtain the fused feature I. F =Concat(I1+I2), based on the requirements of image fusion, the loss function is defined as: L=L SSIM +L m ,in:

[0062] L SSIM =w1×(1-ssim(I f ,I1))+w2×(1-ssim(I f ,I2))

[0063] L m =|H(I1)+H(I2)-H(I) f )|

[0064] In this method, H(·) is used to calculate the information entropy of the image, and ssim(·) is used to calculate the structural similarity of the image. The fusion network is trained to ensure that the fused features contain more information, reduce artifacts, and have a similar structure to the original image. This invention generates different fused images based on different weather conditions by adjusting the image weight coefficients w1 and w2. For example, when more infrared image information is needed at night, its weight is increased by changing the values ​​of the weight coefficients. Then, as... Figure 2 The image is reconstructed from the features using a deconvolution operation.

[0065] In step 3) above, the reconstructed fused image is evaluated using objective metrics such as Feature Mutual Information (FMI), Structural Similarity (SSIM), and Peak Signal-to-Noise Ratio (PSNR). In a specific embodiment, the formula for the evaluation metrics is as follows:

[0066]

[0067]

[0068]

[0069] Among them, I F1with I F2 These represent the amount of feature information from the two modalities contained in the fused image, respectively; H represents the information entropy of the different modalities; and μ1 and μ2 represent the average pixel values ​​of the images. With σ 12 Let I(i,j) and K(i,j) represent the pixel variance of the first image and the pixel covariance of the second image, respectively. I(i,j) and K(i,j) represent the pixels at corresponding positions in the first image and the corresponding pixels in the second image. i and j represent the image coordinates with the top-left corner of the image as the origin. m represents the width of the image, n represents the height of the image, and C1 and C2 are constants to avoid instability caused by the denominator being close to 0. In this embodiment, after generating the fused image, the quality of the generated fused image is evaluated using the above metrics, compared with other optimal fusion algorithms, and the number of network layers is modified to improve the quality of the generated image.

[0070] In step 4) above, images of different modalities are used as input to the fusion network to obtain fused features, which are then fed into the feature extraction and detection network. With the parameters of the already trained fusion network frozen, gradient descent is used to train the detection network, thereby improving target detection accuracy and optimizing the UAV target detection capability. Specific optimization measures include improving the target detection loss function and incorporating the SimAM attention mechanism for small target detection.

[0071] The relevant technical means used in the method of this invention will be described in detail below:

[0072] Recursive Loop-Gated Convolution: Transformer is a neural network model based on self-attention mechanism, which can better establish long-distance dependencies and enable the model to better understand global semantics. However, self-attention mechanism is computationally complex and not conducive to real-time detection on UAVs. Recursive Loop-Gated Convolution can explore the execution of high-order spatial interactions through gated loops and recursion. It is constructed by standard convolution, linear projection and element-wise multiplication, without introducing significant additional computation, and simplifies the complex computational operations of Transformer. Recursive Loop-Gated Convolution is used to encourage the model to capture global information of the image while reducing computational resource consumption. The specific calculation steps of recursive loop-gated convolution are shown in formulas (4)-(7):

[0073]

[0074] p k+1 =f k (q k )⊙g k (p k ) / α,k=0,1,...,n-1 (5)

[0075]

[0076] y = Φ out (q n (7)

[0077] Where x represents the input image features, and the Φ(·) function represents a set of features obtained by projecting the features, denoted as p0 and... Then, as shown in formula (5), the gate convolution operation is performed recursively, g k The function (·) shown in formula (6) is used to match feature dimensions in different orders. When k is 0, the feature dimensions remain unchanged. When k is not 0, the feature dimensions are matched using a projection operation. k (·) represents a set of deep convolutional layers, ⊙ represents element-wise multiplication, and α is a scaling parameter used to scale the input and stabilize training. Finally, the feature q obtained from the last recursive step is... n Projection is performed to obtain the output image features.

[0078] Loss function: The loss function of YOLOv7 consists of three parts: classification loss, target loss, and localization loss, L = L cls +L obj +L loc .in:

[0079] Classification loss is the error between the predicted object category and the actual category; target loss is the error of whether the predicted bounding box contains an object; and localization loss is the error between the predicted bounding box position and the actual position. The main application scenario of this invention is small object detection. Since the intersection-union ratio (IU) metric is highly sensitive to the positional deviation of small objects, this embodiment uses a normalized Gaussian distance function as the loss function to replace the target loss, thereby enhancing the model's ability to detect small objects in specific application scenarios. Its working principle is to design a feature that better measures the similarity of small objects to mitigate the sensitivity of the IU to the positional deviation of small objects. First, the bounding boxes are modeled as two-dimensional Gaussian distributions, and Gaussian distance is used to measure the distance between the Gaussian distributions of two bounding boxes.

[0080] L loc =1-NWD(N a N b (8)

[0081]

[0082]

[0083]

[0084]

[0085] Among them, Lloc N represents the localization loss. a With N b It is the Gaussian distribution of the predicted anchor frame and the ground truth anchor frame. The anchor frame is modeled as a second-order Gaussian distribution. The probability density function is calculated as shown in formula (10). NWD represents the result of the normalized Gaussian distance between the two distributions. W2 2 (·) is used to calculate the second-order Gaussian distance, where C is the average absolute size of the targets in the dataset, and x, μ, and Σ represent the coordinates, mean vector, and covariance matrix of the Gaussian distribution, respectively. For the four parameters of the anchor frame to be transformed, the center coordinates of the anchor frame and the width and height (c...) of the anchor frame are... x ,c y ,w,h), using formulas (11) and (12) to prepare for the subsequent calculation of the Gaussian distribution.

[0086] Attention Mechanism: To enhance the detector's ability to detect small targets while maintaining detection efficiency, this embodiment employs a parameter-free attention module, SimAM (A Simple, Parameter-Free Attention Module), to calculate attention weights. This module is embedded into the target detection network to enhance features, thereby improving the detector's ability to detect small targets. The specific location of SimAM is as follows: Figure 3 The image is shown in the head of the detection network. The specific calculation formula is as follows: an energy function designed with reference to the human brain's attention mechanism is used to calculate the attention weights, and then the feature map is enhanced according to the importance of each neuron.

[0087]

[0088]

[0089] Where t represents the target neuron of the input features. and Let represent the mean and variance of all neurons in the channel except t, and λ be the regularization coefficient. X is the original value of the input feature map. It represents the energy of each neuron, i.e., the importance of each neuron, and ⊙ represents element-wise multiplication. The enhanced feature map is represented by the result of element-wise multiplication of the input feature map with the power of the sigmoid function.

[0090] In this embodiment, the training steps for the fusion network and the detection network are as follows: Images from different modalities are input into the fusion network, and the loss function is calculated simultaneously. Based on backpropagation, the weights are updated in the direction of minimum loss function using gradient descent, iterating continuously until the model results stabilize and converge. After the fusion network is trained, its parameters are fixed, and the detection network is connected before the image reconstruction module of the fusion network. The detection network is then trained using the same method. Finally, the desired target detection network based on the fusion of two modalities is obtained.

[0091] The principle and technical effects of the method of the present invention will be described in detail below with reference to the accompanying drawings:

[0092] like Figure 1 As shown, firstly, a relevant scene was built based on the AirSim platform, and image datasets of different modalities were collected. Secondly, the bimodal images were input into a Siamese neural network to fuse image features. The fusion network was designed considering the different image resolutions of the different modalities, feature extraction efficiency, and the quality of the fused image. Then, the fusion quality of the bimodal images was evaluated, and based on the objective evaluation results, the fusion network was modified to improve the quality of the fused image as much as possible. Finally, the fused features were fed into a detection network for feature extraction, using a more efficient single-stage target detection algorithm, and further improvements were made to meet the real-time and detection accuracy requirements of UAV target detection.

[0093] like Figure 2 As shown, image information from different modalities is first deconvolutionally processed to obtain feature maps of consistent size. Then, early feature extraction is performed on the features. Next, intra-domain information mining is performed on different modalities. Finally, inter-domain information is extracted and fused to obtain the features of the fused bimodal image. The fused features are reconstructed into an image. The quality of image fusion is judged based on indicators. The depth of the feature extraction network is increased to improve the network and enhance the quality of the fused image.

[0094] like Figure 3 As shown, the fused features obtained by the dual-modal image fusion network are directly input into the single-stage target detection network YOLOv7 (omitting the process of deconvolution reconstruction of features). The network parameters of the fusion network are frozen, and the target detection network is trained. At the same time, the target detection network is improved to enhance the UAV's ability to detect small ground targets. The SimAM attention mechanism is added during the feature extraction process to derive 3D attention weights for the feature map and explore the importance of neurons, thereby improving the detection performance of the target detection network. Meanwhile, a normalized Gaussian distance function that is sensitive to the positional deviation of small targets is used to replace the target loss of YOLOv7 to improve detection accuracy without affecting detection efficiency.

[0095] Figure 4 and Figure 5 These are images of different modalities before fusion. Figure 6 The fused image is reconstructed from the fused features, and it can be seen that the fusion significantly improves the image quality. Figure 7 The image shows the fusion detection result obtained by inputting images from the VEDAI dataset into a trained fusion detection network. The result is an image of the fused infrared and visible light images. The target detection results are accurate, and the results display different vehicle categories and their confidence scores. It can be seen that there are no missed or false detections, and the confidence scores are high. It should be noted that the embodiments and accompanying drawings of this invention use the fusion of infrared and visible light images as an example. In addition, to simultaneously obtain image information from other different modalities, any two or more different modalities can be selected for fusion. The fusion detection implementation and training method are the same as in the example presented in this paper; only the parameters of the trained neural network differ.

[0096] As described in the above embodiments, those skilled in the art will understand that the present invention provides a target detection method based on the fusion of two modal images, which can improve the efficiency and accuracy of the target detection algorithm that fuses dual modal images. Using the present invention makes it easier to better leverage the aerial advantages of UAVs in certain missions.

[0097] In addition, embodiments of the present invention also provide a storage medium storing one or more programs readable by a computing device, the one or more programs including instructions that, when executed by the computing device, cause the computing device to perform a target detection method based on two-modal image fusion as described above.

[0098] Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, DVD-ROM, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or high-speed digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device described above configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer such that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, and servers. Furthermore, in one example, the computer program, along with any associated data, data files, and data structures, is distributed across a networked computer system, such that the computer program, along with any associated data, data files, and data structures, is stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0099] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products, etc. Therefore, the present invention can take the form of a completely software embodiment, a completely hardware embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0102] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target detection method based on two modalities of image fusion, characterized in that, The method includes: Build relevant scenarios for simulation and collect image datasets of different modalities; A dual-modal image fusion network is constructed, in which a recursive gated convolutional structure is introduced. This recursive gated convolutional structure is used to extract information within deep feature domains and to mine information between the superimposed fused feature domains. The fusion network includes a feature extraction module, a feature fusion module, and an image reconstruction module, wherein: The feature extraction module is used for early convolution extraction and intra-domain and inter-domain information mining based on recursive loop gate convolution. The feature fusion module is used to fuse the extracted features; The image reconstruction module includes convolution and activation functions, which are used to reconstruct the fused image from the fused features; The fusion network is trained based on the backpropagation principle, and the Siamese neural network is used to extract features from the images of the two modalities at the same time. The features are then reconstructed into a fused image through deconvolution. The quality of the reconstructed fused image will be tested and evaluated using quantitative evaluation metrics, and the fusion network will be modified based on the evaluation results. The fusion features output by the fusion network are directly input into a single-stage YOLOv7 object detection network before the fusion image is reconstructed. The network parameters of the fusion network are frozen, and the object detection network is trained to obtain an optimized fusion and detection network. In this process, a normalized Gaussian distance function is used to replace the target loss of YOLOv7, and a parameterless attention module is embedded in the object detection network to calculate attention weights and enhance the feature map. Target detection is performed using an optimized fusion and detection network.

2. The target detection method based on two-modal image fusion according to claim 1, characterized in that, Build relevant urban and wilderness scenes on the AirSim platform, including rain, snow, fog, fire, and flood scenarios; change the environmental season and time to simulate changes in light intensity; activate the sensors of visible light camera, infrared camera, and depth camera, and control the drone to switch different positions and perspectives to collect image datasets with different modalities; divide the image dataset into training set, validation set, and test set, and perform offline image enhancement operations.

3. The target detection method based on two-modal image fusion according to claim 1, characterized in that, After simultaneously extracting features from two image modalities using a Siamese neural network, the extracted features are fused to obtain fused features. Based on the requirements for image fusion, the loss function is defined as follows: ； in, Represents structural similarity loss. This represents the information entropy loss. This represents the features extracted from the first modality image. This represents the features extracted from the second modality image. This indicates that the features obtained through fusion are... This represents the weighting coefficients of the first modality image. The weighting coefficients of the second modality image, This represents the calculation of the information entropy of an image. This indicates the calculation of structural similarity between images.

4. The target detection method based on two-modal image fusion according to claim 3, characterized in that, The evaluation metrics include: feature mutual information, structural similarity, and peak signal-to-noise ratio. The formulas for calculating the evaluation metrics are as follows: ； in, Representing feature mutual information, Indicates structural similarity, Indicates peak signal-to-noise ratio. This represents the amount of feature information from the first modality image contained in the fused image. This represents the amount of feature information from the second modality image contained in the fused image. The information entropy of images representing different modalities This represents the average pixel value of the first modality image. This represents the average pixel value of the second modality image. This represents the pixel variance of the first modality image. This represents the pixel variance of the second modality image. This represents the pixel covariance of the two modalities of the image. , These represent the pixels at corresponding positions in the two modalities of the image. and These represent the corresponding image coordinate points. Represents the width of the image. Represents the height of the image. It is a constant.

5. The target detection method based on two-modal image fusion according to claim 1, characterized in that, The object detection network was trained using gradient descent.

6. The target detection method based on two-modal image fusion according to claim 1, characterized in that, The target loss of YOLOv7 is replaced with the normalized Gaussian distance function, and the calculation formula is as follows: ； in, Indicates location loss. These represent the Gaussian distributions of the predicted anchor frame and the ground truth anchor frame, respectively. This represents the result of calculating the normalized Gaussian distance between the two distributions. This indicates the calculation of the second-order Gaussian distance. This represents the average absolute size of the target in the dataset. This represents the probability density function of a second-order Gaussian distribution. , and Let represent the coordinates, mean vector, and covariance matrix of the Gaussian distribution, respectively. These represent the center coordinates of the anchor frame, These represent the width and height of the anchor frame, respectively.

Images