A multi-band image fusion method, device, storage medium and equipment

Abstract

In a multi-band image fusion method, device, storage medium and equipment provided in this specification, an infrared image and a visible light image are obtained, an initial fused image is determined, and based on the temperature distribution information of the initial fused image, a foreground sub-image and a background sub-image are segmented, and then the frequency domain features of the foreground sub-image are input into a trained foreground generator to obtain a foreground frequency domain feature, and the frequency domain features of the background sub-image are input into a trained background generator to obtain a background frequency domain feature, and a fusion feature determined according to the foreground frequency domain feature and the background frequency domain feature is obtained to obtain a target fused image. Through the foreground generator and the background generator, a foreground frequency domain feature fitting the foreground information in the infrared image and a background frequency domain feature fitting the background information in the visible light image are obtained, and then a fusion feature determined according to the foreground frequency domain feature and the background frequency domain feature is obtained, and a target fused image with a prominent foreground and rich background information can be obtained, and the fusion effect is good.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a multi-band image fusion method, device, storage medium and equipment. Background Art

[0002] Multi-band image fusion technology is a method that combines image data from different sensors or different bands to generate a fused image that contains important information from all input images. This technology has a wide range of applications in remote sensing, military reconnaissance, medical imaging and other fields.

[0003] Taking the fusion of infrared and visible light images as an example, infrared images can reveal the thermal radiation characteristics of objects, and visible light images have rich texture details, that is, the two images provide image information of different dimensions. At present, the fusion image of infrared images and visible light images is often obtained by simple superposition fusion, but this simple superposition fusion method may lead to the loss of texture details, and the fusion effect of the obtained fused image is poor.

[0004] Based on this, this specification provides a multi-band image fusion method, device, storage medium and equipment. Summary of the invention

[0005] This specification provides a multi-band image fusion method and device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a multi-band image fusion method, the method comprising:

[0008] Acquire an infrared image of a target area and a visible light image of the target area;

[0009] Determine an initial fused image of the infrared image and the visible light image, and segment the initial fused image based on temperature distribution information of the initial fused image to obtain a foreground sub-image and a background sub-image;

[0010] Determine frequency domain features corresponding to the foreground sub-image and the background sub-image respectively;

[0011] Inputting the frequency domain features of the foreground sub-image into a trained foreground generator to obtain foreground frequency domain features that fit the foreground information in the infrared image; and inputting the frequency domain features of the background sub-image into a trained background generator to obtain background frequency domain features that fit the background information in the visible light image;

[0012] A fusion feature is determined according to the foreground frequency domain feature and the background frequency domain feature, and a target fused image is obtained according to the fusion feature.

[0013] Optionally, determining an initial fused image of the infrared image and the visible light image specifically includes:

[0014] Determine the infrared image and the frequency domain features respectively corresponding to the infrared image, wherein the frequency domain features include amplitude features and phase features;

[0015] Determine a multi-source amplitude feature according to the amplitude feature of the infrared image and the amplitude feature of the visible light image; and determine a multi-source phase feature according to the phase feature of the infrared image and the phase feature of the visible light image;

[0016] An initial fused image is determined according to the multi-source amplitude features and the multi-source phase features.

[0017] Optionally, the background generator and the foreground generator are trained using the following method, wherein:

[0018] Acquire infrared images of multiple areas and visible light images of the multiple areas;

[0019] For the infrared image and the visible light image of each region, the above method is used to obtain the target fused image corresponding to the region as the first fused image;

[0020] Determining a first frequency domain discrimination result through a first frequency domain discriminator according to the frequency domain features of the foreground in the first fused image and the frequency domain features of the foreground in the infrared image; and determining a second frequency domain discrimination result through a second frequency domain discriminator according to the frequency domain features of the background in the first fused image and the frequency domain features of the background in the visible light image;

[0021] Determine a first joint loss according to the first frequency domain discrimination result, and train the first frequency domain discriminator according to the first joint loss; and determine a second joint loss according to the second frequency domain discrimination result, and train the second frequency domain discriminator according to the second joint loss;

[0022] Determine a foreground difference between a foreground in the first fused image and a foreground in the infrared image, and train the foreground generator based on the foreground difference; and determine a background difference between a background in the first fused image and a background in the visible light image, and train the background generator based on the background difference.

[0023] Optionally, the first frequency domain discriminator includes a first amplitude discriminator and a first phase discriminator; the frequency domain feature includes an amplitude feature and a phase feature;

[0024] Determining a first frequency domain discrimination result by a first frequency domain discriminator according to the frequency domain features of the foreground in the first fused image and the frequency domain features of the foreground in the infrared image specifically includes:

[0025] According to the amplitude characteristics of the foreground in the first fused image and the amplitude characteristics of the foreground in the infrared image, an amplitude discrimination result of the foreground in the infrared image and an amplitude discrimination result of the foreground in the first fused image are obtained by a first amplitude discriminator; and

[0026] According to the phase characteristics of the foreground in the first fused image and the phase characteristics of the foreground in the infrared image, a phase discrimination result of the foreground in the infrared image and a phase discrimination result of the foreground in the first fused image are obtained by a first phase discriminator;

[0027] taking the amplitude discrimination result of the foreground in the infrared image, the amplitude discrimination result of the foreground in the first fused image, the phase discrimination result of the foreground in the infrared image, and the phase discrimination result of the foreground in the first fused image as the first frequency domain discrimination result;

[0028] Determining a first joint loss according to the first frequency domain discrimination result specifically includes:

[0029] A first joint loss is determined according to an amplitude discrimination result of the foreground in the infrared image, an amplitude discrimination result of the foreground in the first fused image, a phase discrimination result of the foreground in the infrared image, and a phase discrimination result of the foreground in the first fused image.

[0030] Optionally, determining a first joint loss according to an amplitude discrimination result of the foreground in the infrared image, an amplitude discrimination result of the foreground in the first fused image, a phase discrimination result of the foreground in the infrared image, and a phase discrimination result of the foreground in the first fused image specifically includes:

[0031] determining a first amplitude loss according to an amplitude discrimination result of the foreground in the infrared image and an amplitude discrimination result of the foreground in the first fused image;

[0032] determining a first phase loss according to a phase discrimination result of the foreground in the infrared image and a phase discrimination result of the foreground in the first fused image;

[0033] A first combined loss is determined according to the first amplitude loss and the first phase loss.

[0034] Optionally, the first frequency domain discrimination result at least includes a frequency domain discrimination result of a foreground in the infrared image;

[0035] Determining a foreground difference between a foreground in the first fused image and a foreground in the infrared image specifically includes:

[0036] Determining the frequency domain adversarial loss of the foreground according to the frequency domain discrimination result of the foreground in the infrared image;

[0037] determining a frequency domain generation loss of the foreground according to the frequency domain characteristics of the foreground in the first fused image and the frequency domain characteristics of the foreground in the infrared image;

[0038] A foreground difference is determined according to a frequency domain adversarial loss of the foreground and a frequency domain generative loss of the foreground.

[0039] Optionally, the frequency domain discrimination result includes an amplitude discrimination result and a phase discrimination result; the frequency domain feature includes an amplitude feature and a phase feature;

[0040] Identify differences in outlook, including:

[0041] Determining the amplitude countermeasure loss of the foreground according to the amplitude discrimination result of the foreground in the infrared image;

[0042] Determining a phase confrontation loss of the foreground according to a phase discrimination result of the foreground in the infrared image;

[0043] determining an amplitude generation loss of the foreground according to an amplitude feature of the foreground in the first fused image and an amplitude feature of the foreground in the infrared image;

[0044] Determining a phase generation loss of the foreground according to a phase feature of the foreground in the first fused image and a phase feature of the foreground in the infrared image;

[0045] Determine a third amplitude loss according to the amplitude confrontation loss of the foreground and the amplitude generation loss of the foreground; and determine a third phase loss according to the phase confrontation loss of the foreground and the phase generation loss of the foreground;

[0046] A foreground difference is obtained according to the third amplitude loss and the third phase loss.

[0047] Optionally, the second frequency domain discriminator includes a second amplitude discriminator and a second phase discriminator; the frequency domain feature includes an amplitude feature and a phase feature;

[0048] Determining a second frequency domain discrimination result by a second frequency domain discriminator according to the frequency domain features of the background in the first fused image and the frequency domain features of the background in the visible light image specifically includes:

[0049] According to the amplitude characteristics of the background in the first fused image and the amplitude characteristics of the background in the visible light image, an amplitude discrimination result of the background in the visible light image and an amplitude discrimination result of the background in the first fused image are obtained by a second amplitude discriminator; and

[0050] According to the phase characteristics of the background in the first fused image and the phase characteristics of the background in the visible light image, a phase discrimination result of the background in the visible light image and a phase discrimination result of the background in the first fused image are obtained by a second phase discriminator;

[0051] taking the amplitude discrimination result of the background in the visible light image, the amplitude discrimination result of the background in the first fused image, the phase discrimination result of the background in the visible light image, and the phase discrimination result of the background in the first fused image as the second frequency domain discrimination result;

[0052] Determining a second joint loss according to the second frequency domain discrimination result specifically includes:

[0053] A second joint loss is determined according to an amplitude discrimination result of the background in the visible light image, an amplitude discrimination result of the background in the first fused image, a phase discrimination result of the background in the visible light image, and a phase discrimination result of the background in the first fused image.

[0054] Optionally, determining a second joint loss according to an amplitude discrimination result of the background in the visible light image, an amplitude discrimination result of the background in the first fused image, a phase discrimination result of the background in the visible light image, and a phase discrimination result of the background in the first fused image specifically includes:

[0055] determining a second amplitude loss according to an amplitude discrimination result of the background in the visible light image and an amplitude discrimination result of the background in the first fused image;

[0056] determining a second phase loss according to a phase discrimination result of the background in the visible light image and a phase discrimination result of the background in the first fused image;

[0057] A second combined loss is determined according to the second amplitude loss and the second phase loss.

[0058] Optionally, the second frequency domain discrimination result at least includes a frequency domain discrimination result of a background in the visible light image;

[0059] Determining a background difference between a background in the first fused image and a background in the visible light image specifically includes:

[0060] Determining the frequency domain adversarial loss of the background according to the frequency domain discrimination result of the background in the visible light image;

[0061] determining a frequency domain generation loss of the background according to the frequency domain characteristics of the background in the first fused image and the frequency domain characteristics of the background in the visible light image;

[0062] A background difference is determined according to a frequency domain adversarial loss of the background and a frequency domain generative loss of the background.

[0063] Optionally, the frequency domain discrimination result includes an amplitude discrimination result and a phase discrimination result; the frequency domain feature includes an amplitude feature and a phase feature;

[0064] Identify background differences, including:

[0065] Determining the amplitude countermeasure loss of the foreground according to the amplitude discrimination result of the background in the visible light image;

[0066] Determining a phase adversarial loss of a foreground according to a phase discrimination result of the background in the visible light image;

[0067] Determining an amplitude generation loss of the background according to an amplitude feature of the background in the first fused image and an amplitude feature of the background in the visible light image;

[0068] Determining a phase generation loss of the background according to a phase feature of the background in the first fused image and a phase feature of the background in the visible light image;

[0069] Determine a fourth amplitude loss according to the amplitude confrontation loss of the background and the amplitude generation loss of the background; and determine a fourth phase loss according to the phase confrontation loss of the background and the phase generation loss of the background;

[0070] A background difference is obtained according to the fourth amplitude loss and the fourth phase loss.

[0071] This specification provides a multi-band image fusion device, including:

[0072] An acquisition module, used to acquire an infrared image of a target area and a visible light image of the target area;

[0073] A segmentation module, used to determine an initial fused image of the infrared image and the visible light image, and segment the initial fused image based on temperature distribution information of the initial fused image to obtain a foreground sub-image and a background sub-image;

[0074] A frequency domain feature module, used to determine the frequency domain features corresponding to the foreground sub-image and the background sub-image respectively;

[0075] A generator module, configured to input the frequency domain features of the foreground sub-image into a trained foreground generator to obtain foreground frequency domain features that fit the foreground information in the infrared image; and input the frequency domain features of the background sub-image into a trained background generator to obtain background frequency domain features that fit the background information in the visible light image;

[0076] The fusion module is used to determine fusion features according to the foreground frequency domain features and the background frequency domain features, and obtain a target fused image according to the fusion features.

[0077] This specification provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, a multi-band image fusion method is implemented.

[0078] The present specification provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a multi-band image fusion method when executing the program.

[0079] At least one of the above-mentioned technical solutions adopted in the present specification can achieve the following beneficial effects: in a multi-band image fusion method provided in the present specification, an infrared image and a visible light image of a target area are obtained, an initial fused image of the infrared image and the visible light image is determined, and based on the temperature distribution information of the initial fused image, the initial fused image is segmented to obtain a foreground sub-image and a background sub-image, and then, the frequency domain features of the foreground sub-image and the frequency domain features of the background sub-image are determined, the frequency domain features of the foreground sub-image are input into a trained foreground generator to obtain foreground frequency domain features that fit the foreground information in the infrared image, and the frequency domain features of the background sub-image are input into a trained background generator to obtain background frequency domain features that fit the background information in the visible light image, and according to the foreground frequency domain features and the background frequency domain features, fusion features are determined, and according to the fusion features, a target fused image is obtained.

[0080] It can be seen from the above method that based on the foreground sub-image and the background sub-image obtained by segmentation of the initial fused image, the foreground frequency domain features that fit the foreground information in the infrared image and the background frequency domain features that fit the background information in the visible light image are obtained through the foreground generator and the background generator. Then, according to the fusion features determined by the foreground frequency domain features and the background frequency domain features, a target fused image with prominent foreground and rich background information can be obtained, and the fusion effect is good. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0082] Figure 1 A flowchart of an infrared and visible light image fusion method provided in this specification;

[0083] Figure 2 A schematic diagram of determining an initial fused image provided in this specification;

[0084] Figure 3 A schematic diagram of the structure of the joint training of the four discriminators, the foreground generator and the background generator provided in this specification;

[0085] Figure 4 A schematic diagram of the structure of the generator provided for this specification;

[0086] Figure 5 A schematic diagram of the structure of the discriminator provided in this specification;

[0087] Figure 6 A schematic diagram of an infrared and visible light image fusion device provided in this specification;

[0088] Figure 7 A schematic diagram of the structure of an electronic device corresponding to a method for implementing infrared and visible light image fusion provided in this specification. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0090] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0091] Figure 1 The following is a flow chart of a method for fusion of infrared and visible light images provided in this specification, comprising the following steps:

[0092] S100: Acquire an infrared image of a target area and a visible light image of the target area.

[0093] In one or more embodiments of this specification, it is not limited to which device is used to implement the process of a method for fusion of infrared and visible light images, such as a personal computer, a mobile terminal, and a server. However, since the subsequent steps involve operations such as data processing, and such operations with high requirements for computing resources are generally performed by a server, this specification will also use the server to implement a method for fusion of infrared and visible light images as an example for description. Among them, the server can be a single device, or composed of multiple devices, such as a distributed server, and this specification does not limit this.

[0094] In one or more embodiments of the present specification, in order to determine an initial fused image of the infrared image and the visible light image in subsequent steps, in this step, the server needs to first obtain an infrared image and a visible light image of the target area (the same target).

[0095] Specifically, the server may obtain an infrared image of the target area and a visible light image of the target area. The spatial consistency of the infrared image and the visible light image is ensured. If there is a spatial difference between the acquired infrared image and the visible light image, it is necessary to perform preprocessing operations such as image registration to ensure that the infrared image and the visible light image are images of the same target area. Of course, the infrared image and the visible light image may be images taken at the same time, thereby reducing the influence of changes in factors such as object movement, person movement, and lighting conditions on the consistency of the acquired infrared image and the visible light image. This specification does not limit the method of acquiring infrared images and visible light images, such as infrared cameras and visible light cameras.

[0096] S102: Determine an initial fused image of the infrared image and the visible light image, and segment the initial fused image based on temperature distribution information of the initial fused image to obtain a foreground sub-image and a background sub-image.

[0097] In one or more embodiments of the present specification, in order to determine the foreground frequency domain characteristics through the foreground generator in the subsequent steps, and to determine the background frequency domain characteristics through the background generator, in this step, the server needs to determine the initial fused image corresponding to the infrared image and the visible light image obtained in step S100, and segment the initial fused image to obtain a foreground sub-image and a background sub-image.

[0098] Specifically, the server can determine the initial fusion image of the infrared image and the visible light image, and segment the initial fusion image based on the temperature distribution information of the initial fusion image to obtain a foreground sub-image and a background sub-image. The specific method of segmenting the initial fusion image based on the temperature distribution information to obtain the foreground sub-image and the background sub-image is not limited in this specification. For example, based on the mapping relationship between the pixel value / frequency and the temperature, a threshold of the pixel value / frequency is pre-set, and the foreground and background in the initial fusion image are marked respectively according to the threshold, and then the foreground sub-image and the background sub-image are segmented. Alternatively, the initial fusion image is segmented by a saliency detection method, that is, a pre-trained saliency detection network is used to generate a foreground saliency map based on the infrared image, and the foreground in the initial fusion image is identified based on the pixel dot multiplication of the foreground saliency map and the initial fusion image, so that the foreground sub-image and the background sub-image can be segmented. Of course, a background saliency map can also be generated based on the visible light image through the saliency detection network, and the background in the initial fused image can be identified based on the pixel multiplication of the background saliency map and the initial fused image, so as to segment the foreground sub-image and the background sub-image.

[0099] It should be noted that the specific method of determining the initial fused image is not limited in this specification, such as weighting and fusing the corresponding pixels of the infrared image and the visible light image in the spatial domain to obtain the initial fused image. In one or more embodiments of this specification, the server may determine the initial fused image after the frequency domain features are fused based on the frequency domain features of the infrared image and the frequency domain features of the visible light image, wherein the frequency domain features include phase features and amplitude features. Specifically:

[0100] In one or more embodiments of the present specification, first, the server can determine the multi-source amplitude feature according to the amplitude feature of the infrared image and the amplitude feature of the visible light image, and obtain the multi-source phase feature according to the phase feature of the infrared image and the phase feature of the visible light image. Secondly, the server determines the initial fused image according to the obtained multi-source amplitude feature and multi-source phase feature.

[0101] The specific method of determining the amplitude feature and the phase feature is not limited in this specification. For example, the frequency domain feature is obtained by two-dimensional discrete Fourier transform, and the amplitude feature and the phase feature are extracted according to the frequency domain feature, as follows:

[0102]

[0103]

[0104] in, It is the conversion result from spatial domain to frequency domain. u can be used to represent the row index of frequency domain, and v can be used to represent the column index of frequency domain. Represents the original image, such as an infrared image or a visible light image. h can be used to represent the row index of the spatial domain, w can be used to represent the column index of the spatial domain, H can be used to represent the image width, and W can be used to represent the image height; Indicates amplitude characteristics; Indicates phase characteristics; Frequency domain operations that convert complex numbers into amplitude-phase expressions; represents the real part in the frequency domain; Represents the imaginary part in the frequency domain.

[0105] According to the amplitude characteristics of the infrared image and the amplitude characteristics of the visible light image, a multi-source amplitude characteristic is obtained by fusion, and according to the phase characteristics of the infrared image and the phase characteristics of the visible light image, a multi-source phase characteristic is obtained by fusion, so as to determine the calculation process of the initial fused image according to the multi-source amplitude characteristic and the multi-source phase characteristic, as follows:

[0106]

[0107]

[0108]

[0109]

[0110] in, and They respectively represent the multi-source amplitude characteristics and multi-source phase characteristics after frequency domain fusion; represents feature fusion operation; as well as Respectively represent the amplitude characteristics and phase characteristics of the infrared image; as well as Respectively represent the amplitude characteristics and phase characteristics of the visible light image; as well as Respectively represent the real and imaginary parts after frequency domain fusion, Inverse frequency domain operation to convert the amplitude-phase expression into complex form; Represents the initial fused image after frequency domain feature fusion; Represents the conversion result from frequency domain to spatial domain.

[0111] like Figure 2 As shown, it is a schematic diagram of determining the initial fused image provided in this specification. The multi-source amplitude characteristics are determined according to the amplitude characteristics of the infrared image and the amplitude characteristics of the visible light image. At the same time, the multi-source phase characteristics are determined according to the phase characteristics of the infrared image and the phase characteristics of the visible light image. Then, the initial fused image is determined according to the multi-source amplitude characteristics and the multi-source phase characteristics.

[0112] S104: Determine frequency domain features corresponding to the foreground sub-image and the background sub-image respectively.

[0113] S106: Input the frequency domain features of the foreground sub-image into the trained foreground generator to obtain the foreground frequency domain features that fit the foreground information in the infrared image; and input the frequency domain features of the background sub-image into the trained background generator to obtain the background frequency domain features that fit the background information in the visible light image.

[0114] In one or more embodiments of the present specification, in order to obtain a target fused image based on foreground frequency domain features and background frequency domain features in subsequent steps, in this step, the server may input the frequency domain features of the foreground sub-image obtained in step S102 into a trained foreground generator to obtain foreground frequency domain features, and input the frequency domain features of the background sub-image obtained in step S102 into a trained background generator to obtain background frequency domain features, and then obtain the target fused image based on the foreground frequency domain features and the background frequency domain features.

[0115] Specifically, the server can determine the frequency domain features corresponding to the foreground sub-image and the background sub-image, respectively, and then input the frequency domain features of the foreground sub-image into the trained foreground generator to obtain the output foreground frequency domain features that fit the foreground information in the infrared image, and input the frequency domain features of the background sub-image into the trained background generator to obtain the output background frequency domain features that fit the background information in the visible light image.

[0116] S108: Determine fusion features according to the foreground frequency domain features and the background frequency domain features, and obtain a target fusion image according to the fusion features.

[0117] In one or more embodiments of the present specification, in this step, the server may obtain a target fused image according to the foreground frequency domain features and the background frequency domain features obtained in step S104.

[0118] Specifically, the server determines the fusion feature according to the foreground frequency domain feature and the background frequency domain feature, and obtains the target fused image according to the fusion feature. The specific method of determining the fusion feature is not limited in this specification, such as the method of frequency domain feature fusion in step S102.

[0119] In the above method, the server obtains the infrared image and the visible light image corresponding to the target area, determines the initial fused image of the infrared image and the visible light image, and then, based on the temperature distribution information of the initial fused image, segmentates to obtain a foreground sub-image and a background sub-image, and then determines the frequency domain features of the foreground sub-image and the frequency domain features of the background sub-image, and obtains the foreground frequency domain features that fit the foreground information in the infrared image and the background frequency domain features that fit the background information in the visible light image through the foreground generator and the background generator, so that the fusion features can be determined based on the feature fusion of the foreground frequency domain features and the background frequency domain features, and according to the fusion features, a target fused image with prominent foreground and rich background information can be obtained.

[0120] In addition, Figure 3 As shown, a schematic diagram of the joint training of four discriminators, a foreground generator, and a background generator is provided for this specification. The joint training process is as follows:

[0121] First, the server can obtain infrared images of multiple areas and the visible light images corresponding to each area, and determine the initial fused image corresponding to each area for the infrared image and visible light image of the area, and segment the initial fused image to obtain a foreground sub-image and a background sub-image. The frequency domain features of the foreground sub-image are input into the foreground generator to be trained to obtain the foreground frequency domain features, and the frequency domain features of the background sub-image are input into the background generator to be trained to obtain the background frequency domain features, and then the target fused image is obtained according to the foreground frequency domain features and the background frequency domain features as the first fused image. The specific process of determining the target fused image will not be repeated here, and the contents of steps S100~S108 can be referred to.

[0122] Secondly, the server can determine the first frequency domain discrimination result through the first frequency domain discriminator according to the frequency domain characteristics of the foreground in the first fused image and the frequency domain characteristics of the foreground in the infrared image. At the same time, the server can determine the second frequency domain discrimination result through the second frequency domain discriminator according to the frequency domain characteristics of the background in the first fused image and the frequency domain characteristics of the background in the visible light image. Then, the server can determine the first joint loss according to the first frequency domain discrimination result, and train the first frequency domain discriminator according to the first joint loss; and determine the second joint loss according to the second frequency domain discrimination result, and train the second frequency domain discriminator according to the second joint loss.

[0123] Optionally, the frequency domain features include phase features and amplitude features, so a first amplitude discriminator and a first phase discriminator may be set to discriminate the phase features and amplitude features of the foreground of the "infrared image / first fused image" respectively. Similarly, a second amplitude discriminator and a second phase discriminator may be set to discriminate the phase features and amplitude features of the "visible light image / first fused image" respectively.

[0124] In one or more embodiments of the present specification, the server may obtain an amplitude discrimination result of the foreground in the infrared image and an amplitude discrimination result of the foreground in the first fused image through a first amplitude discriminator based on the amplitude characteristics of the foreground in the first fused image and the amplitude characteristics of the foreground in the infrared image; and the server may obtain a phase discrimination result of the foreground in the infrared image and a phase discrimination result of the foreground in the first fused image through a first phase discriminator based on the phase characteristics of the foreground in the first fused image and the phase characteristics of the foreground in the infrared image, and use the amplitude discrimination result of the foreground in the infrared image, the amplitude discrimination result of the foreground in the first fused image, the phase discrimination result of the foreground in the infrared image, and the phase discrimination result of the foreground in the first fused image as the first frequency domain discrimination result.

[0125] In one or more embodiments of the present specification, the server may obtain an amplitude discrimination result of the background in the visible light image and an amplitude discrimination result of the background in the first fused image through a second amplitude discriminator based on the amplitude characteristics of the background in the first fused image and the amplitude characteristics of the background in the visible light image; and the server may obtain a phase discrimination result of the background in the visible light image and a phase discrimination result of the background in the first fused image through a second phase discriminator based on the phase characteristics of the background in the first fused image and the phase characteristics of the background in the visible light image; and use the amplitude discrimination result of the background in the visible light image, the amplitude discrimination result of the background in the first fused image, the phase discrimination result of the background in the visible light image, and the phase discrimination result of the background in the first fused image as the second frequency domain discrimination result.

[0126] It should be noted that this specification does not limit the method of determining the first joint loss based on the first frequency domain discrimination result and the method of determining the second joint loss based on the second frequency domain discrimination result. For this reason, this specification provides a specific process for determining the first joint loss and the second joint loss based on the mean square error (MSE) loss function.

[0127] The first frequency domain discrimination result includes the amplitude discrimination result of the foreground in the infrared image, the amplitude discrimination result of the foreground in the first fused image, the phase discrimination result of the foreground in the infrared image, and the phase discrimination result of the foreground in the first fused image. Then the server can determine the first amplitude loss according to the amplitude discrimination result of the foreground in the infrared image and the amplitude discrimination result of the foreground in the first fused image. At the same time, the server can determine the first phase loss according to the phase discrimination result of the foreground in the infrared image and the phase discrimination result of the foreground in the first fused image. Then, the process of determining the first joint loss according to the first amplitude loss and the first phase loss is as follows:

[0128]

[0129]

[0130]

[0131] in, is the first amplitude loss, is the amplitude discrimination result of the foreground in the infrared image, is the amplitude discrimination result of the foreground in the first fused image; is the first phase loss, is the phase discrimination result of the foreground in the infrared image, is the phase discrimination result of the foreground in the first fused image; is the first joint loss, , , , , , They are all balance parameters, and N represents the number of images. The bigger The smaller it is, the stronger the ability of the first amplitude discriminator is, that is, it can distinguish the amplitude characteristics of the foreground in the infrared image and the amplitude characteristics of the foreground in the first fused image. The bigger The smaller it is, the stronger the first phase discriminator is, that is, it can distinguish the phase characteristics of the foreground in the infrared image and the phase characteristics of the foreground in the first fused image.

[0132] The second frequency domain discrimination result includes the amplitude discrimination result of the background in the visible light image, the amplitude discrimination result of the background in the first fused image, the phase discrimination result of the background in the visible light image, and the phase discrimination result of the background in the first fused image. Then the server can determine the second amplitude loss according to the amplitude discrimination result of the background in the visible light image and the amplitude discrimination result of the background in the first fused image. At the same time, the server can determine the second phase loss according to the phase discrimination result of the background in the visible light image and the phase discrimination result of the background in the first fused image. Then, the process of determining the second joint loss according to the second amplitude loss and the second phase loss is as follows:

[0133]

[0134]

[0135]

[0136] in, is the second amplitude loss, is the amplitude discrimination result of the background in the visible light image, is the amplitude discrimination result of the background in the first fused image; is the second phase loss, is the phase discrimination result of the background in the visible light image, is the phase discrimination result of the background in the first fused image; is the second joint loss, , , , , , These are equilibrium parameters. The bigger The smaller it is, the stronger the ability of the second amplitude discriminator is, that is, it can distinguish the amplitude characteristics of the background in the visible light image and the amplitude characteristics of the background in the first fused image. The bigger The smaller it is, the stronger the second phase discriminator is, that is, it can distinguish the phase characteristics of the background in the visible light image and the phase characteristics of the background in the first fused image.

[0137] Finally, the server may determine the foreground difference between the foreground in the first fused image and the foreground in the infrared image, train the foreground generator, determine the background difference between the background in the first fused image and the background in the visible light image, and train the background generator based on the background difference.

[0138] It should be noted that the purpose of "foreground difference or background difference" is to constrain the generator output result to maintain the basic structural characteristics in order to fit the foreground information in the infrared image and the background information in the visible light image.

[0139] The process of determining the prospect difference is as follows:

[0140] In one or more embodiments of the present specification, the server may determine the frequency domain generation loss of the foreground based on the frequency domain features of the foreground in the first fused image and the frequency domain features of the foreground in the infrared image. Optionally, the frequency domain features include phase features and amplitude features. Therefore, in one or more embodiments of the present specification, first, the server may determine the amplitude generation loss of the foreground based on the amplitude features of the foreground in the first fused image and the amplitude features of the foreground in the infrared image. Secondly, the server may determine the phase generation loss of the foreground based on the phase features of the foreground in the first fused image and the phase features of the foreground in the infrared image. Finally, the server may determine the foreground difference based on the amplitude generation loss of the foreground and the phase generation loss of the foreground. The calculation process is as follows:

[0141]

[0142]

[0143]

[0144] in, For the difference in prospects, Generates losses for the magnitude of the outlook, Generate loss for the foreground phase, is the amplitude feature of the foreground in the infrared image, is the amplitude feature of the foreground in the first fused image, is the phase feature of the foreground in the infrared image, is the phase feature of the foreground in the first fused image; and is a balance parameter.

[0145] In addition, the first frequency domain discriminator includes a first amplitude discriminator and a first phase discriminator, that is, the adversarial loss of the first amplitude discriminator and the first phase discriminator can be introduced in the foreground difference to train the generation ability of the foreground generator. That is, first, the server can determine the amplitude adversarial loss of the foreground according to the amplitude discrimination result of the foreground in the infrared image by the first amplitude discriminator, and determine the phase adversarial loss of the foreground according to the phase discrimination result of the foreground in the infrared image by the first phase discriminator. Secondly, the server can determine the third amplitude loss according to the amplitude adversarial loss of the foreground and the amplitude generation loss of the foreground, and determine the third phase loss according to the phase adversarial loss of the foreground and the phase generation loss of the foreground. Finally, the server can determine the foreground difference according to the third amplitude loss and the third phase loss. The calculation process is as follows:

[0146]

[0147]

[0148]

[0149]

[0150]

[0151] in, For the difference in prospects, is the third amplitude loss, is the fourth amplitude loss, To counter the loss for the magnitude of the prospect, Phase counter loss for the foreground; is the amplitude discrimination result of the first amplitude discriminator on the foreground in the infrared image, is the phase discrimination result of the first phase discriminator on the foreground in the infrared image, It is a soft label. Generates losses for the magnitude of the outlook, The process of determining the foreground phase generation loss, the foreground amplitude generation loss and the foreground phase generation loss will not be described in detail here.

[0152] The process of determining background differences is as follows:

[0153] In one or more embodiments of the present specification, the server may determine the frequency domain generation loss of the foreground based on the frequency domain features of the background in the visible light image and the frequency domain features of the foreground in the infrared image. Optionally, the frequency domain features include phase features and amplitude features. Therefore, in one or more embodiments of the present specification, first, the server may determine the amplitude generation loss of the background based on the amplitude features of the background in the first fused image and the amplitude features of the background in the visible light image. Secondly, the server determines the phase generation loss of the background based on the phase features of the background in the first fused image and the phase features of the background in the visible light image. Finally, the server may determine the background difference based on the amplitude generation loss of the background and the phase generation loss of the background. The calculation process is as follows:

[0154]

[0155]

[0156]

[0157] in, For background differences, Generates loss for the amplitude of the background, The phase loss for the background is generated, is the amplitude feature of the background in the visible light image, is the amplitude feature of the background in the first fused image; is the phase feature of the background in the visible light image, is the phase feature of the background in the first fused image; and is a balance parameter.

[0158] In addition, the second frequency domain discriminator includes a second amplitude discriminator and a second phase discriminator, that is, the adversarial loss of the second amplitude discriminator and the second phase discriminator can be introduced in the background difference to train the generation ability of the background generator. That is, first, the server can determine the amplitude adversarial loss of the background according to the amplitude discrimination result of the background in the visible light image by the second amplitude discriminator, and determine the phase adversarial loss of the background according to the phase discrimination result of the background in the visible light image by the second phase discriminator. Secondly, the server can determine the fourth amplitude loss according to the amplitude adversarial loss of the background and the amplitude generation loss of the background, and determine the fourth phase loss according to the phase adversarial loss of the background and the phase generation loss of the background. Finally, the server can determine the background difference according to the fourth amplitude loss and the fourth phase loss. The calculation process is as follows:

[0159]

[0160]

[0161]

[0162]

[0163]

[0164] in, For background differences, is the fourth amplitude loss, is the fourth phase loss; is the background amplitude adversarial loss, Phase adversarial loss for background; is the amplitude discrimination result of the second amplitude discriminator on the background in the visible light image, is the phase discrimination result of the second phase discriminator on the background in the visible light image, It is a soft label. Generates loss for the amplitude of the background, The process of determining the background phase generation loss, the background amplitude generation loss and the background phase generation loss will not be described in detail here.

[0165] It should be noted that the clarity of the foreground often depends on the size of the pixel distribution intensity, and the richness of the background texture details depends on the amount of potential gradient information in the image. The frequency is an indicator of the intensity of the grayscale value change and is the gradient of the grayscale in the plane space. Therefore, four discriminators are used to discriminate image information of different dimensions, that is, the first amplitude discriminator / first phase discriminator performs foreground-oriented discrimination in the frequency domain of the image, and the second amplitude discriminator / first phase discriminator performs background-oriented discrimination in the frequency domain of the image, so as to realize the joint training of the four discriminators, the foreground generator and the background generator, so that the method of using the trained foreground generator / background generator in steps S100 to S108 can retain the foreground information in the infrared image / background information in the visible light image, and obtain a target fused image with a better fusion effect.

[0166] In addition, the residual network can be introduced into the foreground generator and the background generator, and more residual information can be retained and the features of the previous layer can be transferred and retained through residual block series and jump connections. Figure 4 As shown, it is a schematic diagram of the structure of a generator provided in this specification, which is composed of a residual block, a skip connection, a convolution layer, a batch normalization layer, and an activation function, wherein the convolution layer, the batch normalization layer, and the activation function constitute a convolution block, or the convolution layer and the activation function constitute a convolution block. Of course, in this specification Figure 4 The structure shown in is only an example and can be adjusted according to actual needs. Figure 5 As shown in FIG. 1 , a schematic diagram of the structure of a discriminator provided in this specification is composed of a convolutional layer, a batch normalization layer, an activation function, and a fully connected layer, wherein the convolutional layer, the batch normalization layer, and the activation function constitute a convolutional block. Of course, in this specification Figure 5 The structure shown in is only an example and can be adjusted according to actual needs.

[0167] The above is an infrared and visible light image fusion method provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding infrared and visible light image fusion device, such as Figure 6 shown.

[0168] An acquisition module 600 is used to acquire an infrared image of a target area and a visible light image of the target area;

[0169] A segmentation module 601 determines an initial fused image of the infrared image and the visible light image, and segments the initial fused image based on temperature distribution information of the initial fused image to obtain a foreground sub-image and a background sub-image;

[0170] A frequency domain feature module 602 determines frequency domain features corresponding to the foreground sub-image and the background sub-image respectively;

[0171] The generator module 603 inputs the frequency domain features of the foreground sub-image into the trained foreground generator to obtain the foreground frequency domain features that fit the foreground information in the infrared image; and inputs the frequency domain features of the background sub-image into the trained background generator to obtain the background frequency domain features that fit the background information in the visible light image;

[0172] The fusion module 604 determines a fusion feature according to the foreground frequency domain feature and the background frequency domain feature, and obtains a target fused image according to the fusion feature.

[0173] Optionally, the segmentation module 601 is specifically used to determine multi-source amplitude features based on the amplitude features of the infrared image and the amplitude features of the visible light image, and to determine multi-source phase features based on the phase features of the infrared image and the phase features of the visible light image; and to determine an initial fused image based on the multi-source amplitude features and the multi-source phase features.

[0174] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A method for fusing infrared and visible light images is provided.

[0175] This manual also provides Figure 7 The schematic structure diagram of the electronic device shown in FIG. Figure 7 As mentioned above, at the hardware level, the device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The infrared and visible light image fusion method. Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0176] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0177] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0178] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0179] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0180] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0181] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0184] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0185] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0186] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0187] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0188] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0189] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0190] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A multi-band image fusion method, characterized in that: include: Acquire an infrared image of a target area and a visible light image of the target area; Determine an initial fused image of the infrared image and the visible light image, and segment the initial fused image based on temperature distribution information of the initial fused image to obtain a foreground sub-image and a background sub-image; Determine frequency domain features corresponding to the foreground sub-image and the background sub-image respectively; Inputting the frequency domain features of the foreground sub-image into the trained foreground generator to obtain the foreground frequency domain features fitting the foreground information in the infrared image; And, inputting the frequency domain features of the background sub-image into a trained background generator to obtain background frequency domain features that fit the background information in the visible light image; A fusion feature is determined according to the foreground frequency domain feature and the background frequency domain feature, and a target fused image is obtained according to the fusion feature.

2. The method according to claim 1, characterized in that: Determining an initial fused image of the infrared image and the visible light image specifically includes: Determine the infrared image and the frequency domain features respectively corresponding to the infrared image, wherein the frequency domain features include amplitude features and phase features; Determine a multi-source amplitude feature according to the amplitude feature of the infrared image and the amplitude feature of the visible light image; and determine a multi-source phase feature according to the phase feature of the infrared image and the phase feature of the visible light image; An initial fused image is determined according to the multi-source amplitude features and the multi-source phase features.

3. The method according to claim 1, characterized in that: The background generator and the foreground generator are trained using the following method, wherein: Acquire infrared images of multiple areas and visible light images of the multiple areas; For the infrared image and the visible light image of each region, an initial fused image of the infrared image and the visible light image is determined, and based on the temperature distribution information of the initial fused image, the initial fused image is segmented to obtain a foreground sub-image and a background sub-image; the frequency domain features corresponding to the foreground sub-image and the background sub-image are determined; the frequency domain features of the foreground sub-image are input into a foreground generator to obtain a foreground frequency domain feature that fits the foreground information in the infrared image; and the frequency domain features of the background sub-image are input into a background generator to obtain a background frequency domain feature that fits the background information in the visible light image; a fusion feature is determined based on the foreground frequency domain feature and the background frequency domain feature, and a first fused image is obtained based on the fusion feature; Determining a first frequency domain discrimination result through a first frequency domain discriminator according to the frequency domain features of the foreground in the first fused image and the frequency domain features of the foreground in the infrared image; and determining a second frequency domain discrimination result through a second frequency domain discriminator according to the frequency domain features of the background in the first fused image and the frequency domain features of the background in the visible light image; Determine a first joint loss according to the first frequency domain discrimination result, and train the first frequency domain discriminator according to the first joint loss; and determine a second joint loss according to the second frequency domain discrimination result, and train the second frequency domain discriminator according to the second joint loss; Determine a foreground difference between a foreground in the first fused image and a foreground in the infrared image, and train the foreground generator based on the foreground difference; and determine a background difference between a background in the first fused image and a background in the visible light image, and train the background generator based on the background difference.

4. The method according to claim 3, characterized in that: The first frequency domain discriminator includes a first amplitude discriminator and a first phase discriminator; the frequency domain feature includes an amplitude feature and a phase feature; Determining a first frequency domain discrimination result by a first frequency domain discriminator according to the frequency domain features of the foreground in the first fused image and the frequency domain features of the foreground in the infrared image specifically includes: According to the amplitude characteristics of the foreground in the first fused image and the amplitude characteristics of the foreground in the infrared image, an amplitude discrimination result of the foreground in the infrared image and an amplitude discrimination result of the foreground in the first fused image are obtained by a first amplitude discriminator; and According to the phase characteristics of the foreground in the first fused image and the phase characteristics of the foreground in the infrared image, a phase discrimination result of the foreground in the infrared image and a phase discrimination result of the foreground in the first fused image are obtained by a first phase discriminator; taking the amplitude discrimination result of the foreground in the infrared image, the amplitude discrimination result of the foreground in the first fused image, the phase discrimination result of the foreground in the infrared image, and the phase discrimination result of the foreground in the first fused image as the first frequency domain discrimination result; Determining a first joint loss according to the first frequency domain discrimination result specifically includes: A first joint loss is determined according to an amplitude discrimination result of the foreground in the infrared image, an amplitude discrimination result of the foreground in the first fused image, a phase discrimination result of the foreground in the infrared image, and a phase discrimination result of the foreground in the first fused image.

5. The method according to claim 4, characterized in that: Determining a first joint loss according to an amplitude discrimination result of the foreground in the infrared image, an amplitude discrimination result of the foreground in the first fused image, a phase discrimination result of the foreground in the infrared image, and a phase discrimination result of the foreground in the first fused image specifically includes: determining a first amplitude loss according to an amplitude discrimination result of the foreground in the infrared image and an amplitude discrimination result of the foreground in the first fused image; determining a first phase loss according to a phase discrimination result of the foreground in the infrared image and a phase discrimination result of the foreground in the first fused image; A first combined loss is determined according to the first amplitude loss and the first phase loss.

6. The method according to claim 3, characterized in that: The first frequency domain discrimination result at least includes a frequency domain discrimination result of a foreground in the infrared image; Determining a foreground difference between a foreground in the first fused image and a foreground in the infrared image specifically includes: Determining the frequency domain adversarial loss of the foreground according to the frequency domain discrimination result of the foreground in the infrared image; determining a frequency domain generation loss of the foreground according to the frequency domain characteristics of the foreground in the first fused image and the frequency domain characteristics of the foreground in the infrared image; A foreground difference is determined according to a frequency domain adversarial loss of the foreground and a frequency domain generative loss of the foreground.

7. The method according to claim 6, characterized in that: The frequency domain discrimination result includes an amplitude discrimination result and a phase discrimination result; the frequency domain feature includes an amplitude feature and a phase feature; Identify differences in outlook, including: Determining the amplitude countermeasure loss of the foreground according to the amplitude discrimination result of the foreground in the infrared image; Determining a phase confrontation loss of the foreground according to a phase discrimination result of the foreground in the infrared image; determining an amplitude generation loss of the foreground according to an amplitude feature of the foreground in the first fused image and an amplitude feature of the foreground in the infrared image; Determining a phase generation loss of the foreground according to a phase feature of the foreground in the first fused image and a phase feature of the foreground in the infrared image; Determine a third amplitude loss according to the amplitude confrontation loss of the foreground and the amplitude generation loss of the foreground; and determine a third phase loss according to the phase confrontation loss of the foreground and the phase generation loss of the foreground; A foreground difference is obtained according to the third amplitude loss and the third phase loss.

8. The method according to claim 3, characterized in that: The second frequency domain discriminator includes a second amplitude discriminator and a second phase discriminator; the frequency domain features include amplitude features and phase features; Determining a second frequency domain discrimination result by a second frequency domain discriminator according to the frequency domain features of the background in the first fused image and the frequency domain features of the background in the visible light image specifically includes: According to the amplitude characteristics of the background in the first fused image and the amplitude characteristics of the background in the visible light image, an amplitude discrimination result of the background in the visible light image and an amplitude discrimination result of the background in the first fused image are obtained by a second amplitude discriminator; and According to the phase characteristics of the background in the first fused image and the phase characteristics of the background in the visible light image, a phase discrimination result of the background in the visible light image and a phase discrimination result of the background in the first fused image are obtained by a second phase discriminator; taking the amplitude discrimination result of the background in the visible light image, the amplitude discrimination result of the background in the first fused image, the phase discrimination result of the background in the visible light image, and the phase discrimination result of the background in the first fused image as the second frequency domain discrimination result; Determining a second joint loss according to the second frequency domain discrimination result specifically includes: A second joint loss is determined according to an amplitude discrimination result of the background in the visible light image, an amplitude discrimination result of the background in the first fused image, a phase discrimination result of the background in the visible light image, and a phase discrimination result of the background in the first fused image.

9. The method according to claim 8, characterized in that Determining a second joint loss according to an amplitude discrimination result of the background in the visible light image, an amplitude discrimination result of the background in the first fused image, a phase discrimination result of the background in the visible light image, and a phase discrimination result of the background in the first fused image specifically includes: determining a second amplitude loss according to an amplitude discrimination result of the background in the visible light image and an amplitude discrimination result of the background in the first fused image; determining a second phase loss according to a phase discrimination result of the background in the visible light image and a phase discrimination result of the background in the first fused image; A second combined loss is determined according to the second amplitude loss and the second phase loss.

10. The method according to claim 3, characterized in that: The second frequency domain discrimination result at least includes a frequency domain discrimination result of the background in the visible light image; Determining a background difference between a background in the first fused image and a background in the visible light image specifically includes: Determining a frequency domain adversarial loss of the background according to a frequency domain discrimination result of the background in the visible light image; determining a frequency domain generation loss of the background according to the frequency domain characteristics of the background in the first fused image and the frequency domain characteristics of the background in the visible light image; A background difference is determined according to a frequency domain adversarial loss of the background and a frequency domain generative loss of the background.

11. The method according to claim 10, characterized in that: The frequency domain discrimination result includes an amplitude discrimination result and a phase discrimination result; the frequency domain feature includes an amplitude feature and a phase feature; Identify background differences, including: Determining the amplitude countermeasure loss of the foreground according to the amplitude discrimination result of the background in the visible light image; Determining a phase adversarial loss of a foreground according to a phase discrimination result of the background in the visible light image; Determining an amplitude generation loss of the background according to an amplitude feature of the background in the first fused image and an amplitude feature of the background in the visible light image; Determining a phase generation loss of the background according to a phase feature of the background in the first fused image and a phase feature of the background in the visible light image; Determine a fourth amplitude loss according to the amplitude confrontation loss of the background and the amplitude generation loss of the background; and determine a fourth phase loss according to the phase confrontation loss of the background and the phase generation loss of the background; A background difference is obtained according to the fourth amplitude loss and the fourth phase loss.

12. A multi-band image fusion device, characterized in that: include: An acquisition module, used to acquire an infrared image of a target area and a visible light image of the target area; A segmentation module, used to determine an initial fused image of the infrared image and the visible light image, and segment the initial fused image based on temperature distribution information of the initial fused image to obtain a foreground sub-image and a background sub-image; A frequency domain feature module, used to determine the frequency domain features corresponding to the foreground sub-image and the background sub-image respectively; A generator module, used for inputting the frequency domain features of the foreground sub-image into the trained foreground generator to obtain the foreground frequency domain features fitting the foreground information in the infrared image; And, inputting the frequency domain features of the background sub-image into a trained background generator to obtain background frequency domain features that fit the background information in the visible light image; The fusion module is used to determine fusion features according to the foreground frequency domain features and the background frequency domain features, and obtain a target fused image according to the fusion features.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 11 is implemented.

14. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 11 when executing the program.

Images