Target detection method based on infrared multiband image fusion

Abstract

A target detection method based on infrared multiband image fusion belongs to the technical field of target detection, and comprises the following steps: analyzing and constructing a trunk structure fusion network model based on Transform; carrying out block processing and coding on the input graph of the network; all embedded information is input into a Transform encoder based on medium wave / long wave multi-head self-attention; a target detection mark output by the encoder is sent to the multi-layer sensor, and other block marks are used for fused image reconstruction; and training the constructed network by using the training image data set, and outputting a final target detection result and an infrared fusion image. According to the method, good target detection performance can be obtained, infrared image fusion can be completed, and a result can be output. Therefore, the method is an effective target detection method based on infrared multiband image fusion.

Description

Technical Field

[0001] The invention relates to a target detection method based on infrared multi-band image fusion, and belongs to the technical field of image processing. Background Art

[0002] Image fusion is a research hotspot in the field of information fusion. Infrared multi-band image fusion is an image information fusion technology that can effectively fuse infrared medium-wave images and long-wave images. It uses correlation and complementarity to reconstruct infrared medium-wave images and long-wave image data obtained by infrared sensors for the same scene into a new infrared image. The new infrared image takes into account the rich information of medium-wave and long-wave infrared images. Infrared multi-band image fusion technology has important application value for infrared images in remote sensing mapping, military target detection, agricultural meteorological analysis and other fields.

[0003] Infrared fusion images can be used for both manual image judgment and high-level visual tasks such as target detection. The existing fusion framework faces significant problems: the fusion oriented to the underlying visual tasks is out of touch with the high-level visual tasks, and cannot fully utilize the high-level visual task features of multi-band images, resulting in limited intelligent analysis applications. Summary of the invention

[0004] The technical problem solved by the present invention is: to overcome the deficiencies of the prior art and provide a target detection method based on infrared multi-band image fusion, so as to achieve the completion of low-level visual tasks and high-level visual tasks within a unified network framework.

[0005] The technical solution of the present invention is: a target detection method based on infrared multi-band image fusion, comprising:

[0006] Step 1: construct a Transformer-based backbone structure fusion network model, wherein the backbone structure fusion network model includes medium-wave and long-wave multi-head self-attention Transformer encoders; the Transformer encoder includes a multi-layer perceptron;

[0007] Step 2: According to the backbone structure fusion network model, the original input image is processed and encoded in blocks, the input blocks are cropped and linearly projected into a block embedding matrix, a learnable object detection embedding information is added, and a learnable position embedding information is added at the same time to construct a complete embedding information;

[0008] Step 3: All embedded information is input into the medium-wavelength and long-wavelength multi-head self-attention Transformer encoder;

[0009] Step 4: The target detection tag output by the Transformer encoder is sent to the multi-layer perceptron, and other patch tags are used for fusion image reconstruction;

[0010] Step 5: Use the training image data set to train the backbone structure fusion network model and output the final target detection results and infrared fusion images.

[0011] Furthermore, during the block processing and encoding, the initially input medium wave or long wave images are stacked as a whole input, and p×p clipping and block division are performed.

[0012] Furthermore, when the input original image is processed and encoded in blocks, a learnable object detection embedding information is added to the block embedding matrix for learning labels for the object detection task.

[0013] Furthermore, when the input original image is processed and encoded in blocks, a learnable position embedding information is added to the block embedding matrix to maintain the correct order of the image block sequence when reconstructing the image.

[0014] Furthermore, the output tag of the Transformer encoder includes target detection information, spatial information and spectral information.

[0015] Furthermore, the target detection information output tag of the Transformer encoder is used by a multi-layer perceptron to complete the target detection task.

[0016] Furthermore, the spatial information and spectral information output marks of the Transformer encoder are used in the convolutional reconstruction layer to complete the fused image reconstruction task.

[0017] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a target detection method based on infrared multi-band image fusion are implemented.

[0018] A target detection device based on infrared multi-band image fusion includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the target detection method based on infrared multi-band image fusion are implemented.

[0019] The advantages of the present invention compared with the prior art are:

[0020] 1. The present invention discloses a target detection method based on infrared multi-band image fusion, which solves the problem of using Transformer encoder for infrared medium-wave / long-wave image fusion.

[0021] 2. The present invention discloses a target detection method based on infrared multi-band image fusion, which solves the problem of separation of image fusion task and target detection task, completes unified network joint training to directly output fused image and target detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0023] Figure 1 This is a flowchart of the steps of a target detection method based on infrared multi-band image fusion in an embodiment of the present invention;

[0024] Figure 2 It is a principle block diagram of a target detection method based on infrared multi-band image fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to better understand the above technical scheme, the technical scheme of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical scheme of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0026] The following is a further detailed description of a target detection method based on infrared multi-band image fusion provided by an embodiment of the present invention in conjunction with the accompanying drawings. Figure 1 , the specific implementation methods may include:

[0027] (1) Analyze and construct a Transformer-based backbone structure fusion network model;

[0028] (2) According to the model of step (1), the network input image is divided into blocks and encoded, the input blocks are cropped and linearly projected into a block embedding matrix, and then a learnable object detection embedding information is added, and a learnable position embedding information is added at the same time, so as to construct a complete embedding information;

[0029] (3) All embedded information is input into the medium / long wavelength multi-head self-attention Transformer encoder;

[0030] (4) The target detection tags output by the encoder are sent to a multi-layer perceptron, and other tags are used to fused image reconstruction;

[0031] (5) Use the training image dataset to train the constructed network and output the final target detection results and infrared fusion images.

[0032] In the solution provided in the embodiment of the present invention, Figure 1In this embodiment, the target detection method based on infrared multi-band image fusion is shown in the following steps:

[0033] Analyze and construct a Transformer-based backbone structure fusion network model 1001; according to the model of step one, the input graph of the network is divided into blocks and encoded 1002; all embedded information is input into the medium-wave / long-wave multi-head self-attention Transformer encoder 1003; the target detection mark output by the encoder is sent to the multi-layer perceptron, and other block marks are used for fusion image reconstruction 1004; use the training image data set to train the constructed network and output the final target detection result and infrared fusion image 1005.

[0034] Figure 2 The principle block diagram of the target detection method of infrared multi-band image fusion is shown, and the block diagram can correspond to each step of the invention:

[0035] (1) Analyze and construct the Transformer-based backbone structure fusion network model 1001;

[0036] (2) According to the model of step 1, the input graph of the network is divided into blocks and encoded 1002;

[0037] (3) All embedded information is input into the medium / long wavelength multi-head self-attention Transformer encoder 1003;

[0038] (4) The target detection label output by the encoder is sent to the multi-layer perceptron, and other patch labels are used for fusion image reconstruction 1004;

[0039] (5) Using the training image data set, the constructed network is trained to output the final target detection result and infrared fusion image 1005.

[0040] Specifically, in step (1), the constructed backbone structure fusion network model includes an input image processing module, a Transformer encoder and an output module.

[0041] In the step (2), the initially input medium wave / long wave image is taken as the whole input:

[0042] I=Concat(I 中波 I 长波 )

[0043] I is then cropped into non-overlapping input tiles of size p, and these tiles are then flattened into a vector, where N represents the number of tiles:

[0044] N=HW / p 2 ,

[0045] These flattened patches are then linearly projected into a block embedding matrix of length D. In order to complete the object detection task, the additional learnable object detection embedding X class is added to the block embedding. In addition, learnable position information is added to ensure the reconstruction order of the blocks during image reconstruction. The input of the encoder can then be described as:

[0046]

[0047] Among them, E I represents the input embedding matrix, and the position embedding matrix is ​​E pos .

[0048] In the steps (3) and (4), the Transformer encoder based on medium / long wave multi-head self-attention is composed of a medium / long wave multi-head self-attention module, a multi-layer perceptron and layer normalization. The output of each layer in the encoder is used as the input of the next layer, which can be described as:

[0049] Z′ I =f SSMSA (f LN (Z I )),

[0050] Z O =f MLP (f LN (Z′ I )),

[0051] Among them, f LN (·) is the normalization operation in the middle layer of the encoder. SSMSA (·) and f MLP (·) are medium / long wavelength multi-head self-attention operation and multi-layer perceptron operation respectively; Z O is the output tag containing all target detection information and medium wave / long wave information. Then, Z O The first tag in is used as a marker for the target detection task, and other markers are used to reconstruct the fused image. is sent to the detection center and obtains the final target detection result Y label :

[0052]

[0053] Finally, after reconstruction by the convolution group, the final infrared fusion image I is obtained. fused :

[0054] In step (5), end-to-end model training is completed. Since the network needs to complete the target detection task and the image fusion task at the same time, the loss function needs to consider the target detection task loss and the image fusion quality loss, and the total loss function is:

[0055] L total =L detection +ηL fusion ,

[0056] Among them, L detection is the target detection task loss, L fusion is the image fusion quality loss, and η is the weight parameter. fusion The mean square error is used as the loss metric. detection The commonly used cross entropy loss function is used.

[0057] For the trained model, medium-wave infrared images and long-wave infrared images are used as network input, and infrared fusion images and target detection results are used as network output.

[0058] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes Figure 1 The method described.

[0059] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0060] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0061] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0062] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0063] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0064] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A target detection method based on infrared multi-band image fusion, characterized in that: include: Step 1: construct a Transformer-based backbone structure fusion network model, wherein the backbone structure fusion network model includes medium-wave and long-wave multi-head self-attention Transformer encoders; the Transformer encoder includes a multi-layer perceptron; Step 2: According to the backbone structure fusion network model, the original input image is processed and encoded in blocks, the input blocks are cropped and linearly projected into a block embedding matrix, a learnable object detection embedding information is added, and a learnable position embedding information is added at the same time to construct a complete embedding information; Step 3: All embedded information is input into the medium-wavelength and long-wavelength multi-head self-attention Transformer encoder; Step 4: The target detection tag output by the Transformer encoder is sent to the multi-layer perceptron, and other patch tags are used for fusion image reconstruction; Step 5: Use the training image data set to train the backbone structure fusion network model and output the final target detection results and infrared fusion images.

2. The target detection method based on infrared multi-band image fusion according to claim 1 is characterized in that: During the block processing and encoding, the initially input medium wave or long wave images are stacked as a whole input, and p×p clipping and block division are performed.

3. The target detection method based on infrared multi-band image fusion according to claim 1 is characterized in that: When the input original image is processed and encoded in blocks, a learnable object detection embedding information is added to the block embedding matrix for learning labels for the object detection task.

4. The target detection method based on infrared multi-band image fusion according to claim 1 is characterized in that: When the input original image is processed and encoded in blocks, a learnable position embedding information is added to the block embedding matrix to maintain the correct order of the image block sequence when reconstructing the image.

5. The target detection method based on infrared multi-band image fusion according to claim 1 is characterized in that: The output token of the Transformer encoder contains object detection information, spatial information and spectral information.

6. The target detection method based on infrared multi-band image fusion according to claim 5 is characterized in that: The target detection information output tag of the Transformer encoder is used by the multi-layer perceptron to complete the target detection task.

7. The target detection method based on infrared multi-band image fusion according to claim 5 is characterized in that: The spatial information and spectral information output tags of the Transformer encoder are used in the convolutional reconstruction layer to complete the fusion image reconstruction task.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A target detection device based on infrared multi-band image fusion, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images