Image fusion method, model training method, device, equipment and storage medium

Abstract

The disclosed embodiments relate to an image fusion method, a model training method, an apparatus, a device and a storage medium. The image fusion model training method includes: obtaining visible light image samples and infrared image samples; roughly aligning the infrared image samples with the visible light image samples to generate infrared registered image samples; inputting the visible light image samples and the infrared registered image samples into the initial fusion model for overall iterative training to generate a target fusion model; the initial fusion model includes an initial registration unit, an initial extraction unit, an initial segmentation unit, an initial fusion unit and an initial reconstruction unit. Through the above technical solution, an overall optimal target fusion model including a dual-modal image registration function is trained, thereby improving the accuracy of the dual-modal fusion model, and then improving the registration accuracy of visible light images and infrared images and the fusion effect and image quality of their corresponding fused images.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image fusion method, a model training method, a device, a equipment and a storage medium. Background Art

[0002] Image fusion is the process of fusing images from different modalities of the same scene to generate an image with richer information. Visible light images have higher resolution and richer image details, while infrared images can better resist interference from extreme environments. Fusion of these two modal images can better utilize the complementarity between the two images to obtain an image with rich details and adaptable to severe weather.

[0003] In the current algorithms for fusing visible light images and infrared images, on the one hand, it is necessary to obtain strictly registered infrared and visible light bimodal data under the same scene and the same viewing angle before fusion, which is mainly achieved through affine transformation. However, because the accuracy of this image registration method is not high, the registration accuracy between the two images is not high, which reduces the quality of the final fused image. On the other hand, the final fused image is obtained by connecting multiple functional modules in the algorithm, which cannot achieve the overall optimal fusion effect, and will also reduce the quality of the final fused image. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image fusion method, a model training method, an apparatus, a device and a storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides an image fusion model training method, the method comprising:

[0006] Acquire visible light image samples and infrared image samples;

[0007] Roughly registering the infrared image sample with the visible light image sample to generate an infrared registered image sample;

[0008] The visible light image samples and the infrared registered image samples are input into an initial fusion model for overall iterative training to generate a target fusion model; wherein the initial fusion model includes an initial registration unit, an initial extraction unit, an initial segmentation unit, an initial fusion unit and an initial reconstruction unit.

[0009] In a second aspect, the present disclosure also provides an image fusion method, the method comprising:

[0010] Acquire a target infrared image and a target visible light image, and determine a target fusion model; wherein the target fusion model is pre-trained and obtained by the image fusion model training method in any embodiment of the present disclosure; the target fusion model includes a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit;

[0011] Inputting the target infrared image and the target visible light image into the target registration unit to generate a target registration result;

[0012] Inputting the target registration result into the target infrared extraction unit and the target visible light extraction unit respectively to generate target infrared features and target visible light features;

[0013] Inputting the target infrared feature into the target infrared segmentation unit to generate a target infrared mask;

[0014] The target infrared feature, the target visible light feature and the target infrared mask are sequentially input into the target fusion unit and the target reconstruction unit to generate a target fusion image corresponding to the target infrared image and the target visible light image.

[0015] In a third aspect, the present disclosure also provides an image fusion model training device, the device comprising:

[0016] A sample acquisition module, used for acquiring visible light image samples and infrared image samples;

[0017] A coarse registration module, used for coarsely registering the infrared image sample with the visible light image sample to generate an infrared registered image sample;

[0018] The model training module is used to input the visible light image samples and the infrared registration image samples into the initial fusion model for overall iterative training to generate a target fusion model; wherein the initial fusion model includes an initial registration unit, an initial extraction unit, an initial segmentation unit, an initial fusion unit and an initial reconstruction unit.

[0019] In a fourth aspect, the present disclosure also provides an image fusion device, the device comprising:

[0020] An image acquisition module, used to acquire a target infrared image and a target visible light image, and determine a target fusion model; wherein the target fusion model is pre-trained and obtained by the image fusion model training method in any embodiment of the present disclosure; the target fusion model includes a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit;

[0021] A target registration result generating module, used for inputting the target infrared image and the target visible light image into the target registration unit to generate a target registration result;

[0022] A feature generation module, used for inputting the target registration result into the target infrared extraction unit and the target visible light extraction unit respectively, to generate a target infrared feature and a target visible light feature;

[0023] A target infrared mask generation module, used for inputting the target infrared feature into the target infrared segmentation unit to generate a target infrared mask;

[0024] The target fusion image generation module is used to input the target infrared feature, the target visible light feature and the target infrared mask into the target fusion unit and the target reconstruction unit in sequence to generate a target fusion image corresponding to the target infrared image and the target visible light image.

[0025] In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0026] Processor and memory;

[0027] The processor is used to execute the image fusion model training method described in any embodiment of the present disclosure or execute the image fusion method described in any embodiment of the present disclosure by calling the program or instruction stored in the memory.

[0028] In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instructions, wherein the program or the instructions enable a computer to execute the image fusion model training method described in any embodiment of the present disclosure or execute the image fusion method described in any embodiment of the present disclosure.

[0029] The technical solution for image fusion model training provided by the embodiment of the present disclosure can obtain visible light image samples and infrared image samples; roughly align the infrared image samples with the visible light image samples to generate infrared registered image samples; input the visible light image samples and the infrared registered image samples into the initial fusion model for overall iterative training to generate a target fusion model; the initial fusion model includes an initial registration unit, an initial extraction unit, an initial segmentation unit, an initial fusion unit and an initial reconstruction unit; it realizes the use of roughly aligned visible light images and infrared images as training samples to perform overall training on the initial fusion model including the initial registration unit containing the image registration function and the initial segmentation unit of the object corresponding to the focus fusion task, which can enhance the image quality. Model training for the registration function is added to improve the image registration accuracy, thereby improving the fusion accuracy of the trained target fusion model, and further improving the quality of the fused image obtained by fusing the infrared image and the visible light image; on the other hand, the segmentation unit can be used to focus on the fusion processing of the object of concern in the fusion task, thereby improving the adaptability of the target fusion model to the image fusion task and further improving the accuracy of the fused image; on the other hand, the functional units included in the fusion model can be jointly trained as a whole to avoid the local optimal problem caused by the separate training and optimization of each functional unit, so that the target fusion model can achieve the overall optimal training effect, thereby improving the image fusion effect of the target fusion model and further improving the quality of the fused image obtained.

[0030] The technical solution for image fusion provided by the embodiment of the present disclosure can obtain a target infrared image and a target visible light image, and determine a target fusion model; the target fusion model includes a pre-trained target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit; the target infrared image and the target visible light image are input into the target registration unit to generate a target registration result; the target registration result is respectively input into the target infrared extraction unit and the target visible light extraction unit to generate a target infrared feature and a target visible light feature; the target infrared feature is input into the target infrared segmentation unit to generate a target infrared mask; the target infrared feature, the target visible light feature and the target infrared mask are input into the target fusion unit and the target reconstruction unit in sequence to generate a target fusion image corresponding to the target infrared image and the target visible light image; the target fusion model with higher registration accuracy, more suitable fusion area for fusion task and overall optimization training is used to fuse the visible light image and the infrared image, thereby improving the adaptability of the obtained target fusion image to the fusion task, thereby improving the quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0033] Figure 1 A flowchart of an image fusion model training method provided in an embodiment of the present disclosure;

[0034] Figure 2 A schematic diagram of the structure of an initial fusion model provided in an embodiment of the present disclosure;

[0035] Figure 3 A schematic diagram of the structure of an initial extraction unit and an initial segmentation unit of a multiplexed feature extraction model structure provided by an embodiment of the present disclosure;

[0036] Figure 4 It is a flowchart of the overall iterative training process of a model in an image fusion model training method provided by an embodiment of the present disclosure;

[0037] Figure 5 A schematic diagram of the structure of an initial fusion model in a training process provided by an embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram of the structure of an initial fusion model in another training process provided by an embodiment of the present disclosure;

[0039] Figure 7 A schematic diagram of the structure of an initial fusion model in another training process provided by an embodiment of the present disclosure;

[0040] Figure 8 is a flowchart of an overall iterative training process of a model in another image fusion model training method provided by an embodiment of the present disclosure;

[0041] Fig. 9 A flowchart of an image fusion method provided by an embodiment of the present disclosure;

[0042] Fig.10 A schematic diagram of the structure of a target fusion model provided in an embodiment of the present disclosure;

[0043] Fig.11 A schematic diagram of the structure of an image fusion model training device provided in an embodiment of the present disclosure;

[0044] Fig.12 A schematic diagram of the structure of an image fusion device provided in an embodiment of the present disclosure;

[0045] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described in detail below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0048] Figure 1 This is a flow chart of an image fusion model training method provided by an embodiment of the present disclosure. The image fusion model training method is mainly applicable to the training scenario of a model for fusing visible light images and infrared images. The image fusion model training method provided by an embodiment of the present disclosure can be executed by an image fusion model training device, which can be implemented by software and / or hardware, and the device can be integrated in an electronic device with certain computing capabilities, such as a laptop computer, a desktop computer or a server.

[0049] See also Figure 1 , the image fusion model training method specifically includes:

[0050] S110: Obtain visible light image samples and infrared image samples.

[0051] The visible light image samples are pre-collected images of visible light modality for fusion model training, and the infrared image samples are pre-collected images of infrared light modality for fusion model training.

[0052] Specifically, the electronic device can obtain visible light image samples and infrared image samples from a sensor storage space, a local storage space or a network terminal. The visible light image samples and infrared image samples are obtained by collecting data for the same scene at the same viewing angle.

[0053] S120 , roughly register the infrared image sample with the visible light image sample to generate an infrared registered image sample.

[0054] Among them, the infrared registered image samples are images obtained after registering the infrared image samples, which are used as samples for subsequent model training.

[0055] Specifically, there are differences in installation position, field of view, and spatial resolution between visible light sensors and infrared sensors, so there is a deviation between the infrared image samples and visible light image samples collected for the same scene. Therefore, before the fusion process, the images must be registered.

[0056] In order to improve the model training efficiency to a certain extent, in the embodiment of the present disclosure, the above infrared image samples and visible light image samples can be roughly registered by using the registration method in related technologies such as affine transformation. In view of the high spatial resolution and rich detail information of visible light image samples, the infrared image samples can be registered with the visible light image samples as a reference to obtain infrared registered image samples.

[0057] S130, inputting visible light image samples and infrared registration image samples into an initial fusion model for overall iterative training to generate a target fusion model; the initial fusion model includes an initial registration unit, an initial extraction unit, an initial segmentation unit, an initial fusion unit and an initial reconstruction unit.

[0058] Among them, the initial fusion model is an untrained overall model with image fusion function, which includes image registration function, feature extraction function, mask extraction function, feature fusion function and image reconstruction function. The local models corresponding to these local model functions are initial registration unit, initial extraction unit, initial segmentation unit, initial fusion unit and initial reconstruction unit. The model structure of these functional units can be implemented as a machine learning model or a deep learning model. The initial extraction unit here can be divided into an initial infrared extraction unit and an initial visible light extraction unit for infrared images and visible light images, and the model structures of the two extraction units can be the same or different. Similarly, the initial segmentation unit can be divided into an initial infrared segmentation unit and an initial visible light segmentation unit for infrared images and visible light images, and the model structures of the two segmentation units can be the same or different.

[0059] Specifically, based on the above description, the visible light image samples and the infrared registration image samples are still not accurately registered. Therefore, an initial registration unit is added to the initial fusion model to achieve a more accurate registration effect.

[0060] Image fusion in related technologies is to superimpose or replace the whole or part of an image based on the global information of the image, which easily causes the loss of image details and the inability to better focus on the object of the fusion task for fusion. Therefore, an initial segmentation unit based on the idea of ​​instance segmentation is added to the initial fusion model, so that the fusion model can perform accurate fusion processing at least in the area where the object corresponding to the fusion task is located.

[0061] The fused image in the related art is obtained directly after the fusion operation of the two modal images is performed. It does not undergo special image reconstruction processing, which makes the quality of the fused image unstable. Therefore, an initial reconstruction unit for image reconstruction based on fused features is added to the initial fusion model to improve the image quality by performing image quality improvement processing such as denoising and smoothing during the reconstruction process, thereby improving the quality of the fused image.

[0062] In addition, in the fusion algorithm in the related art, the local models corresponding to the feature extraction function, feature fusion function and image reconstruction function are all trained separately, and the training of these local models is optimized separately, but such a training method is prone to fall into the dilemma of local optimality, so that the combined fusion algorithm cannot reach the overall optimality, resulting in the fusion effect of the fusion image obtained by the fusion algorithm is not good and the image quality is not high. Based on the above description, an initial fusion model including the above-mentioned functional units is constructed in the embodiment of the present disclosure, and the initial fusion model is iteratively trained as a whole, that is, the functional units are combined in the model training process for overall iterative training, so that the target fusion model obtained by the final training can achieve the overall optimal fusion effect.

[0063] See also Figure 2 , the electronic device uses the rough registration 210 to register the infrared image sample to the visible light image sample to obtain the infrared registered image sample. Then, the visible light image sample and the infrared registered image sample are sequentially input into the initial registration unit 220, the initial extraction unit 230, the initial segmentation unit 240, the initial fusion unit 250 and the initial reconstruction unit 260 to obtain the final fused image. The fused image output by the model and the standard / reference fused image are used to calculate the loss function to obtain the loss value, and the loss value is used for feedback training to adjust the model parameters in the initial registration unit 220, the initial extraction unit 230, the initial segmentation unit 240, the initial fusion unit 250 and the initial reconstruction unit 260 to complete the overall model training. If the model convergence condition is not met, the above model training process is cyclically performed using the visible light image sample and the infrared registered image sample until the model convergence condition is met.

[0064] The standard / reference fusion image in the above process is the "true value" of the fusion image generated in advance based on the infrared image samples and the visible light image samples. The above model convergence condition can be a preset upper limit of the number of model training times, or a preset lower limit of the loss value.

[0065] It can be understood that the trained target fusion model includes a target registration unit, a target extraction unit, a target segmentation unit, a target fusion unit and a target reconstruction unit. Similarly, the target extraction unit may include a target infrared extraction unit and a target visible light extraction unit. In view of the fact that the two modal images processed by the target registration unit are precisely aligned, the masks of the objects of interest of the fusion task extracted from the two images are also aligned, so one can be used. Therefore, the target segmentation unit may include a target infrared segmentation unit or a target visible light segmentation unit. In order to retain more infrared image features, the target infrared segmentation unit may be retained in the embodiment of the present disclosure, that is, the target segmentation unit in the target fusion model is a target infrared segmentation unit.

[0066] In some embodiments, during the training process of the initial fusion model, a loss function corresponding to the initial fusion unit is constructed based on the information density of the mask area.

[0067] Specifically, in this embodiment, a loss function corresponding to the initial fusion unit is also constructed. The loss function is constructed by the information density of the mask area output by the segmentation unit. The information density can be represented by the gradient density, for example. In this way, the construction process of the loss function can be:

[0068] First, the gradient loss is calculated, and the Sobel operator is used to calculate the gradients of the original image (including infrared image samples and visible light image samples) and the image output by the fusion unit. Then, the difference between these gradients is calculated as part of the loss. This helps ensure that the image output by the fusion unit is consistent with the original image in terms of edges and details. Then the gradient loss of the gradient (second-order gradient loss) is calculated, that is, the gradient of the gradient (that is, the second-order derivative) is calculated, and the second-order gradient difference between the original image and the image output by the fusion unit is compared. This can further emphasize the subtle structure and high-frequency information in the image. Then the loss of the structural similarity index (Structure Similarity Index Measure, SSIM) is calculated, that is, the SSIM metric is used to compare the structural similarity between the original image and the image output by the fusion unit. SSIM takes into account brightness, contrast, and structural information, which helps to retain the structural information in the image. Finally, the above three losses are weighted and combined to obtain a total loss function.

[0069] Based on this embodiment, in the process of overall training of the initial fusion model, in addition to the loss calculation of the final fusion image and feedback adjustment of the model parameters, the electronic device also uses the image output by the initial fusion unit to calculate the loss value of the loss function constructed based on the information density of the mask area, and uses the loss value feedback to adjust the model parameters. This can further improve the accuracy of model training.

[0070] In some embodiments, the initial extraction unit and the initial segmentation unit reuse the same feature extraction network. That is, the feature extraction branch and the segmentation branch of the object detection and semantic segmentation (You Only Look Once version 8 with Semantic Segmentation, YOLOv8-seg) model based on YOLOv8 are split, and the model structure of the initial extraction unit is designed as the structure of the feature extraction branch, and the model structure of the initial segmentation unit is designed as the structure of the segmentation branch. In this way, the model parameters can be reduced by reusing the feature extraction network, thereby improving the training efficiency of the initial fusion model; and the features of the extraction unit can be constrained by the association between the feature extraction branch and the segmentation branch, further improving the accuracy of the fusion model.

[0071] See also Figure 3 , the feature extraction branch of the YOLOv8-seg model includes a combination layer 310, a first downsampling layer 320, a second downsampling layer 330, a third downsampling layer 340 and a fourth downsampling layer 350. During the training process of the initial fusion model, the initial infrared extraction unit and the initial visible light extraction unit both adopt the model structure of the feature extraction branch. Among them, the combination layer 310, the first downsampling layer 320 and the second downsampling layer 330 are used to extract detail features in the image, and their extraction results are input into the initial fusion unit 3100 for fusion processing. At the same time, the extraction results of these layers are also input into the segmentation branch for instance segmentation. The third downsampling layer 340 and the fourth downsampling layer 350 are used to extract position features and morphological features in the image, and their output features are input into the segmentation branch for instance segmentation.

[0072] Continue to see Figure 3 , the segmentation branch includes a first upsampling layer 360, a second upsampling layer 370, a third upsampling layer 380 and a fourth upsampling layer 390. In the training process of the initial fusion model, the initial infrared segmentation unit and the initial visible light segmentation unit both use the segmentation branch, and the mask of the object concerned by the fusion task outputted by the segmentation branch is input into the initial fusion unit 3100 for fusion processing.

[0073] It should be noted that, during the training process of the initial fusion model, the initial infrared segmentation unit and the initial visible light segmentation unit are required to extract the corresponding mask to constrain the model parameters of the corresponding feature extraction unit. Therefore, the model structures of the initial infrared extraction unit and the initial visible light extraction unit are the same, both of which include a combination layer 310, a first downsampling layer 320, a second downsampling layer 330, a third downsampling layer 340 and a fourth downsampling layer 350. However, during the application of the target fusion model, in view of the fact that the masks output by the two modalities are aligned, in order to reduce the amount of model calculation, only the target infrared segmentation unit can be retained. In this way, the position features and morphological features output by the target visible light extraction unit become redundant features. Then, in order to further reduce the amount of model calculation, it is not necessary to execute the third downsampling layer 340 and the fourth downsampling layer 350, that is, the target visible light extraction unit only includes the combination layer 310, the first downsampling layer 320 and the second downsampling layer 330.

[0074] In some embodiments, assuming that the instance segmentation accuracy of the segmentation branch is high, the segmentation unit can be used to constrain the model training of the extraction unit. Therefore, during the training of the initial fusion model, the model parameters of the third downsampling layer, the fourth downsampling layer and the segmentation branch can be frozen. In this way, the instance segmentation capability of the segmentation branch can be inherited, and it can be used to constrain the training process of the model parameters of the combination layer, the first downsampling layer and the second downsampling layer, thereby improving the accuracy of feature extraction of the initial extraction unit and the training efficiency of the initial fusion model.

[0075] The image fusion model training method provided by the above-mentioned embodiments of the present disclosure can obtain visible light image samples and infrared image samples; roughly align the infrared image samples with the visible light image samples to generate infrared registered image samples; input the visible light image samples and the infrared registered image samples into the initial fusion model for overall iterative training to generate a target fusion model; the initial fusion model includes an initial registration unit, an initial extraction unit, an initial segmentation unit, an initial fusion unit and an initial reconstruction unit; it realizes the use of roughly aligned visible light images and infrared images as training samples to perform overall training on the initial fusion model including the initial registration unit containing the image registration function and the initial segmentation unit of the object corresponding to the focus fusion task, which can enhance the image quality. Model training for the registration function is added to improve the image registration accuracy, thereby improving the fusion accuracy of the trained target fusion model, and further improving the quality of the fused image obtained by fusing the infrared image and the visible light image; on the other hand, the segmentation unit can be used to focus on the fusion processing of the object of concern in the fusion task, thereby improving the adaptability of the target fusion model to the image fusion task and further improving the accuracy of the fused image; on the other hand, the functional units included in the fusion model can be jointly trained as a whole to avoid the local optimal problem caused by the separate training and optimization of each functional unit, so that the target fusion model can achieve the overall optimal training effect, thereby improving the image fusion effect of the target fusion model and further improving the quality of the fused image obtained.

[0076] Figure 4 1 is a flowchart of the overall iterative training process of a model in an image fusion model training method provided by an embodiment of the present disclosure. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0077] See also Figure 4 The overall iterative training process of the model in the image fusion model training method includes:

[0078] S410: Based on the visible light image samples and the infrared registration image samples, iteratively train the initial registration unit to generate a first registration unit.

[0079] Specifically, see Figure 5The electronic device uses visible light image samples and roughly registered infrared registered image samples as training data for the initial registration unit 510 for input, and outputs the registration result after calculation by the initial registration unit 510. The registration result can be a registered visible light image sample obtained by registering the visible light image sample with the infrared registered image sample, or it can be a re-registered infrared registered image sample obtained by registering the infrared registered image sample with the visible light image sample. In the embodiment of the present disclosure, the visible light image sample is used as the registration reference, and the registration result is the re-registered infrared registered image sample. At the same time, a standard / reference infrared registered image is pre-designed. The loss value of the loss function is calculated by the image difference between the standard / reference infrared registered image and the re-registered infrared registered image sample, and the loss value is used to feedback and adjust the model parameters of the initial registration unit 510. The training process is executed cyclically until the model convergence condition is reached, and the first registration unit is obtained.

[0080] It should be noted that Figure 5 The initial infrared extraction unit 520, the initial infrared segmentation unit 530, the initial visible light extraction unit 540, the initial visible light segmentation unit 550, the initial fusion unit 560 and the initial reconstruction unit 570 are exemplified by dotted lines, indicating that in this training step, these units in the initial fusion model do not participate in the iterative training.

[0081] S420, input the visible light image sample and the infrared registration image sample into the first registration unit to generate a first registration result, and input the first registration result into the initial infrared extraction unit and the initial infrared segmentation unit in sequence to generate a first infrared mask, and input the first registration result into the initial visible light extraction unit and the initial visible light segmentation unit in sequence to generate a visible light mask.

[0082] Among them, the first registration result is a registered image output by the first registration unit during the joint training process of the three types of units, namely, the registration unit, the extraction unit and the segmentation unit. The first registration result can be a registered visible light image sample obtained by taking the infrared registration image sample as a benchmark, or it can be a re-registered infrared registration image sample obtained by taking the visible light image sample as a benchmark. The re-registered infrared registration image sample in the disclosed embodiment is used as an example for explanation. The first infrared mask is the area where the object of interest of the fusion task is extracted from the infrared registration image during the joint training process of the above three types of units. The visible light mask is the area where the object of interest of the fusion task is extracted from the visible light image sample.

[0083] Specifically, see Figure 6, using visible light image samples and roughly registered infrared registered image samples as training data for joint training of the above three types of units. The electronic device inputs the visible light image samples and the roughly registered infrared registered image samples into the first registration unit 610 for registration processing, and outputs a first registration result. Then, the first registration result is respectively input into the initial infrared extraction unit 620 and the initial visible light extraction unit 640 to extract infrared features and visible light features. The features extracted here include detail features, position features and morphological features. Then, the infrared features are input into the initial infrared segmentation unit 630 to calculate the first infrared mask mask. And the visible light features are input into the initial visible light segmentation unit 650 to calculate the visible light mask mask.

[0084] S430. Based on the first infrared mask and the visible light mask, iteratively train the first registration unit, the initial infrared extraction unit, the initial infrared segmentation unit, the initial visible light extraction unit and the initial visible light segmentation unit to generate a second registration unit, an intermediate infrared extraction unit, an intermediate infrared segmentation unit and an intermediate visible light extraction unit.

[0085] Specifically, if the visible light image sample and the infrared registration image sample are strictly aligned, then the masks obtained by the two are also strictly aligned. Based on this, the electronic device can use the difference in the mask area between the first infrared mask and the visible light mask to calculate the loss value of the joint training of the above three types of units. Then, the mask-based loss value can be used to feedback and adjust the model parameters of the above three types of units. After that, continue to refer to Figure 6 The electronic device can cyclically execute the joint training process of the first registration unit 610, the initial infrared extraction unit 620, the initial infrared segmentation unit 630, the initial visible light extraction unit 640 and the initial visible light segmentation unit 650 until the model convergence condition is reached, and obtain the second registration unit, the intermediate infrared extraction unit, the intermediate infrared segmentation unit and the intermediate visible light extraction unit.

[0086] It should be noted that since the visible light segmentation unit does not participate in the subsequent model training process, after the training is completed, there is no need to pay attention to the trained visible light segmentation unit. In addition, based on the model parameters of the frozen segmentation unit, the initial infrared segmentation unit and the intermediate infrared segmentation unit are the same.

[0087] Through the settings of S420 and S430, the precise edges of the objects of interest in the fusion task extracted from the two modal images can be used to constrain the parameter training of the registration unit and the extraction unit, thereby further improving the accuracy of the registration and the extracted features.

[0088] S440, input the visible light image sample and the infrared registration image sample into the second registration unit to generate a second registration result, and input the second registration result into the intermediate infrared extraction unit and the intermediate visible light extraction unit respectively to generate infrared features and visible light features, and input the infrared features into the intermediate infrared segmentation unit to generate a second infrared mask.

[0089] The second registration result is a registered image output by the second registration unit during the overall joint training of the units included in the fusion model. Similarly, in the disclosed embodiment, the second registration result is an example of a re-registered infrared registered image sample. The second infrared mask is the area where the object of interest of the fusion task is located extracted from the infrared registered image during the above-mentioned overall joint training process.

[0090] Specifically, see Figure 7 , using visible light image samples and roughly registered infrared registered image samples as training data for overall joint training. The electronic device inputs the visible light image samples and the roughly registered infrared registered image samples into the second registration unit 710 for registration processing, and outputs a second registration result. Then, the second registration result is respectively input into the intermediate infrared extraction unit 720 and the second visible light extraction unit 740 to extract infrared features and visible light features. Infrared features include detail features, position features and morphological features. Since the visible light modality does not perform mask segmentation, the visible light features can only retain detail features. Then, the infrared features are input into the intermediate infrared segmentation unit 730 to calculate the second infrared mask mask′.

[0091] S450, input the infrared features, visible light features and the second infrared mask into the initial fusion unit and the initial reconstruction unit in sequence to generate fused features and a reconstructed image, and iteratively train the second registration unit, the intermediate infrared extraction unit, the intermediate infrared segmentation unit, the intermediate visible light extraction unit, the initial fusion unit and the initial reconstruction unit based on the fused features and the reconstructed image to generate a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit to form a target fusion model.

[0092] Specifically, the electronic device inputs the infrared feature, the visible light feature and the second infrared mask into the initial fusion unit 760, and outputs the fused feature / image. Then, the fused feature / image is input into the initial reconstruction unit 770, and the final fused image is output. Then, the electronic device can use the fused feature / image, the visible light image sample and the infrared registration image sample to calculate the loss value corresponding to the fusion unit through the loss function constructed based on the information density. In addition, the loss value corresponding to the reconstruction unit can be calculated using the pre-constructed standard / reference fusion image. Then, the electronic device can use the two loss values ​​to feedback and adjust the model parameters of each unit in the initial fusion model (the frozen model parameters remain unchanged). After that, the electronic device can cyclically execute the joint training process of the above-mentioned second registration unit 710, the intermediate infrared extraction unit 720, the intermediate infrared segmentation unit 730, the intermediate visible light extraction unit 740, the initial fusion unit 760 and the initial reconstruction unit 770 until the model convergence condition is reached, and the target registration unit, the target infrared extraction unit, the target infrared segmentation unit, the target visible light extraction unit, the target fusion unit and the target reconstruction unit are obtained to form a target fusion model.

[0093] It should be noted that the initial visible light segmentation model 750 marked by the dotted line does not participate in this part of the model training process.

[0094] The method for overall iterative training of the initial fusion model provided in the above-mentioned embodiment of the present disclosure realizes the separate training of the registration unit, the joint training of the registration unit, the extraction unit and the segmentation unit, and the joint training of the registration unit, the extraction unit, the segmentation unit, the fusion unit and the reconstruction unit. Through three iterative training processes of different degrees, the training of the model parameters of the fusion model is constrained by multi-dimensional indicators such as the mask of the object of interest of the fusion task, the information density of the fused image and the quality of the reconstructed image, so as to further improve the registration accuracy of the target registration unit and further improve the model accuracy of the target fusion model, thereby further improving the fusion effect and image quality of the object of interest of the fusion task in the output reconstructed image.

[0095] Figure 8 1 is a flowchart of the overall iterative training process of the model in another image fusion model training method provided by an embodiment of the present disclosure. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 8 The overall iterative training process of the model in the image fusion model training method includes:

[0096] S810: Determine the initial registration unit as the current registration unit, and determine the initial fusion model as the current fusion model.

[0097] Specifically, the overall iterative training method in this embodiment is to first independently train the registration unit, then train the fusion model as a whole; then independently train the registration unit, then train the fusion model as a whole... until the model convergence condition is reached. Therefore, in this cyclic training process, the initial registration unit is first determined as the current registration unit of the cyclic amount. At the same time, the initial fusion model is determined as the current fusion model of the cyclic amount.

[0098] S820: Iteratively train the current registration unit based on the visible light image samples and the infrared registration image samples to generate a trained current registration unit, and use the trained current registration unit to update the current registration unit.

[0099] Specifically, the electronic device uses visible light image samples and infrared registration image samples as training data to iteratively train the current registration unit to obtain a trained current registration unit, and then uses the trained current registration unit as a new current registration unit to continue to participate in subsequent model training.

[0100] S830: Use the current registration unit to replace the registration unit in the current fusion model, and update the current fusion model.

[0101] Specifically, the electronic device replaces the registration unit in the current fusion model with the new current registration unit obtained above to obtain a new current fusion model.

[0102] S840, based on the visible light image samples and the infrared registration image samples, perform overall iterative training on the current fusion model to generate a trained current fusion model, and use the trained current fusion model to update the current fusion model.

[0103] Specifically, the electronic device uses visible light image samples and infrared registration image samples as training data to perform model training on the new current fusion model from beginning to end to obtain a trained current fusion model, and then uses the trained current fusion model as the new current fusion model.

[0104] S850. If the model convergence condition is not met, the current registration unit is updated using the registration unit in the current fusion model, and the process returns to the step of iteratively training the current registration unit based on visible light image samples and infrared registration image samples to generate a trained current registration unit, and updating the current registration unit using the trained current registration unit until the model convergence condition is met to generate the target fusion model.

[0105] Among them, the target fusion model includes a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit.

[0106] Specifically, the electronic device first determines whether the current model training degree reaches the model convergence condition. If not, the model training process needs to continue. At this time, the alignment unit in the above-mentioned new current fusion model is used as the new current alignment unit, and the training process of S820 to S850 is returned to execute. If it is reached, the model training process is stopped, and the current fusion model obtained by the last training is determined as the final target fusion model.

[0107] Fig. 9 : is a flow chart of an image fusion method provided by an embodiment of the present disclosure. The image fusion method is mainly applicable to the scene of obtaining a fused image by using a visible light image and an infrared image. The image fusion method provided by an embodiment of the present disclosure can be performed by an image fusion device, which can be implemented by software and / or hardware, and the device can be integrated in an electronic device with certain computing capabilities, such as a laptop computer, a desktop computer, a server or an edge computing platform.

[0108] See also Fig. 9 , the image fusion method specifically includes:

[0109] S910, acquiring a target infrared image and a target visible light image, and determining a target fusion model; the target fusion model includes a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit.

[0110] The target infrared image and the target visible light image are images to be subjected to dual-modal image fusion processing. The target fusion model is pre-trained by the image fusion model training method described in any of the above embodiments. Thus, the model structure and model parameters of each functional unit in the target fusion model can refer to the detailed description of the above embodiments.

[0111] Specifically, the electronic device can obtain the target infrared image and the target visible light image from a dual-modal sensor, or a local storage space, or a network terminal.

[0112] At the same time, the electronic device may obtain the target fusion model, for example, it may read and load the target fusion model from a local / external storage space.

[0113] S920: Input the target infrared image and the target visible light image into a target registration unit to generate a target registration result.

[0114] Specifically, see Fig.10The specific process of the electronic device realizing image fusion using the target fusion model may be: firstly inputting the target infrared image and the target visible light image into the target registration unit 1010, and generating a registration result (i.e., a target registration result) of accurate alignment of the dual-modal images, such as the target visible light image and the target infrared registration image.

[0115] S930 , input the target registration result into the target infrared extraction unit and the target visible light extraction unit respectively, and generate the target infrared feature and the target visible light feature.

[0116] Specifically, the electronic device can continue to input the target registration results into the target infrared extraction unit 1020 and the target visible light extraction unit 1040 respectively, to generate target infrared features including detail features, position features and morphological features, and to generate target visible light features including only detail features.

[0117] S940, inputting the target infrared feature into the target infrared segmentation unit to generate a target infrared mask.

[0118] Specifically, the electronic device continues to input the target infrared features into the target infrared segmentation unit 1030, and outputs the target infrared mask corresponding to the object of interest of the fusion task.

[0119] S950, inputting the target infrared feature, the target visible light feature and the target infrared mask into the target fusion unit and the target reconstruction unit in sequence, and generating a target fusion image corresponding to the target infrared image and the target visible light image.

[0120] Specifically, the electronic device inputs the target infrared feature, the target visible light feature and the target infrared mask into the target fusion unit 1060, and outputs the fusion feature. Then, the electronic device continues to input the fusion feature into the target reconstruction unit 1070, and outputs the final fusion image of the target infrared image and the target visible light image (i.e., the target fusion image).

[0121] The image fusion method provided by the above-mentioned embodiment of the present disclosure can obtain a target infrared image and a target visible light image, and determine a target fusion model; the target fusion model includes a pre-trained target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit; the target infrared image and the target visible light image are input into the target registration unit to generate a target registration result; the target registration result is respectively input into the target infrared extraction unit and the target visible light extraction unit to generate a target infrared feature and a target visible light feature; the target infrared feature is input into the target infrared segmentation unit to generate a target infrared mask; the target infrared feature, the target visible light feature and the target infrared mask are sequentially input into the target fusion unit and the target reconstruction unit to generate a target fusion image corresponding to the target infrared image and the target visible light image; the target fusion model with higher registration accuracy, more suitable fusion area for fusion task and overall optimization training is used to fuse the visible light image and the infrared image, thereby improving the adaptability of the obtained target fusion image to the fusion task, thereby improving the quality of the fused image.

[0122] Fig.11 The structure diagram of an image fusion model training device provided by the embodiment of the present disclosure is shown in FIG. Fig.11 As shown, the image fusion model training device 1100 includes:

[0123] The sample acquisition module 1110 is used to acquire visible light image samples and infrared image samples;

[0124] A coarse registration module 1120 is used to coarsely register the infrared image sample with the visible light image sample to generate an infrared registered image sample;

[0125] The model training module 1130 is used to input visible light image samples and infrared registration image samples into the initial fusion model for overall iterative training to generate a target fusion model; wherein the initial fusion model includes an initial registration unit, an initial extraction unit, an initial segmentation unit, an initial fusion unit and an initial reconstruction unit.

[0126] In some embodiments, the model training module 1130 is specifically used to:

[0127] Iteratively training the initial registration unit based on the visible light image samples and the infrared registration image samples to generate a first registration unit;

[0128] Inputting the visible light image sample and the infrared registration image sample into a first registration unit to generate a first registration result, and sequentially inputting the first registration result into an initial infrared extraction unit and an initial infrared segmentation unit to generate a first infrared mask, and sequentially inputting the first registration result into an initial visible light extraction unit and an initial visible light segmentation unit to generate a visible light mask;

[0129] Based on the first infrared mask and the visible light mask, iteratively training the first registration unit, the initial infrared extraction unit, the initial infrared segmentation unit, the initial visible light extraction unit and the initial visible light segmentation unit to generate the second registration unit, the intermediate infrared extraction unit, the intermediate infrared segmentation unit and the intermediate visible light extraction unit;

[0130] Input the visible light image sample and the infrared registration image sample into the second registration unit to generate a second registration result, and input the second registration result into the intermediate infrared extraction unit and the intermediate visible light extraction unit respectively to generate infrared features and visible light features, and input the infrared features into the intermediate infrared segmentation unit to generate a second infrared mask;

[0131] The infrared features, visible light features and the second infrared mask are sequentially input into the initial fusion unit and the initial reconstruction unit to generate fused features and a reconstructed image, and the second registration unit, the intermediate infrared extraction unit, the intermediate infrared segmentation unit, the intermediate visible light extraction unit, the initial fusion unit and the initial reconstruction unit are iteratively trained based on the fused features and the reconstructed image to generate a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit to form a target fusion model.

[0132] In some other embodiments, the model training module 1130 is specifically used for:

[0133] Determining the initial registration unit as the current registration unit, and determining the initial fusion model as the current fusion model;

[0134] Iteratively train the current registration unit based on the visible light image samples and the infrared registration image samples to generate the trained current registration unit, and use the trained current registration unit to update the current registration unit;

[0135] Using the current registration unit to replace the registration unit in the current fusion model, and updating the current fusion model;

[0136] Based on the visible light image samples and the infrared registration image samples, the current fusion model is iteratively trained as a whole to generate the trained current fusion model, and the trained current fusion model is used to update the current fusion model;

[0137] If the model convergence condition is not met, the current registration unit is updated with the registration unit in the current fusion model, and the process returns to execute iterative training of the current registration unit based on visible light image samples and infrared registration image samples to generate a trained current registration unit, and the step of updating the current registration unit with the trained current registration unit is performed until the model convergence condition is met to generate a target fusion model; wherein the target fusion model includes a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit.

[0138] In some embodiments, the initial extraction unit includes a feature extraction branch based on a YOLOv8 target detection and semantic segmentation model; the feature extraction branch includes at least a combination layer, a first downsampling layer, a second downsampling layer, a third downsampling layer, and a fourth downsampling layer;

[0139] The initial segmentation unit includes a segmentation branch based on a YOLOv8 target detection and semantic segmentation model; the segmentation branch includes a first upsampling layer, a second upsampling layer, a third upsampling layer, and a fourth upsampling layer.

[0140] Furthermore, during the training process of the initial fusion model, the model parameters of the third downsampling layer, the fourth downsampling layer and the segmentation branch are frozen.

[0141] In some embodiments, during the training process of the initial fusion model, a loss function corresponding to the initial fusion unit is constructed based on the information density of the mask area.

[0142] The image fusion model training device provided in the embodiments of the present disclosure can execute the image fusion model training method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0143] Fig.12 FIG. 1 is a schematic diagram of the structure of an image fusion device provided in an embodiment of the present disclosure. Fig.12 As shown, the image fusion device 1200 includes:

[0144] An image acquisition module 1210 is used to acquire a target infrared image and a target visible light image, and determine a target fusion model; wherein the target fusion model is pre-trained by the image fusion model training method according to any one of claims 1 to 7; the target fusion model includes a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit;

[0145] The target registration result generating module 1220 is used to input the target infrared image and the target visible light image into the target registration unit to generate a target registration result;

[0146] A feature generation module 1230 is used to input the target registration result into the target infrared extraction unit and the target visible light extraction unit respectively to generate a target infrared feature and a target visible light feature;

[0147] The target infrared mask generation module 1240 is used to input the target infrared feature into the target infrared segmentation unit to generate a target infrared mask;

[0148] The target fusion image generation module 1250 is used to input the target infrared features, target visible light features and target infrared mask into the target fusion unit and the target reconstruction unit in sequence to generate a target fusion image corresponding to the target infrared image and the target visible light image.

[0149] The image fusion device provided in the embodiments of the present disclosure can execute the image fusion method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0150] It is worth noting that in the embodiments of the above-mentioned devices, the modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present disclosure.

[0151] Fig.13 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Fig.13 As shown, the electronic device 1300 includes one or more processors 1301 and a memory 1302 .

[0152] The processor 1301 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1300 to perform desired functions.

[0153] The memory 1302 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1301 may run the program instructions to implement the image fusion model training method, image fusion method, and / or other desired functions of the embodiment of the present disclosure described above. Various contents such as visible light images / samples, infrared images / samples, fusion models and their model parameters may also be stored in the computer-readable storage medium.

[0154] In one example, the electronic device 1300 may further include: an input device 1303 and an output device 1304, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 1303 may include, for example, a keyboard, a mouse, etc. The output device 1304 may output various information to the outside, including model parameters obtained through training and model-related descriptive information, target fusion images, etc. The output device 1304 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0155] Of course, to simplify, Fig.13 Only some of the components related to the present disclosure in the electronic device 1300 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 1300 may also include any other appropriate components.

[0156] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the image fusion model training method or image fusion method provided by the embodiments of the present disclosure.

[0157] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0158] In addition, the embodiments of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the image fusion model training method or image fusion method provided by the embodiments of the present disclosure.

[0159] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0160] It should be noted that the terms used in the present disclosure are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present disclosure specification and claims, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The term "and / or" includes any and all combinations of one or more related listed items. Relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method or device including the elements.

[0161] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training an image fusion model, characterized in that: include: Acquire visible light image samples and infrared image samples; Roughly registering the infrared image sample with the visible light image sample to generate an infrared registered image sample; Input the visible light image samples and the infrared registered image samples into the initial fusion model for overall iterative training to generate a target fusion model; wherein the initial fusion model includes an initial registration unit, an initial extraction unit, an initial segmentation unit, an initial fusion unit and an initial reconstruction unit; The step of inputting the visible light image samples and the infrared registration image samples into the initial fusion model for overall iterative training to generate a target fusion model includes: Based on the visible light image samples and the infrared registration image samples, iteratively training the initial registration unit to generate a first registration unit; Inputting the visible light image sample and the infrared registration image sample into the first registration unit to generate a first registration result, and sequentially inputting the first registration result into an initial infrared extraction unit and an initial infrared segmentation unit to generate a first infrared mask, and sequentially inputting the first registration result into an initial visible light extraction unit and an initial visible light segmentation unit to generate a visible light mask; Based on the first infrared mask and the visible light mask, iteratively train the first registration unit, the initial infrared extraction unit, the initial infrared segmentation unit, the initial visible light extraction unit, and the initial visible light segmentation unit to generate a second registration unit, an intermediate infrared extraction unit, an intermediate infrared segmentation unit, and an intermediate visible light extraction unit; Input the visible light image sample and the infrared registration image sample into the second registration unit to generate a second registration result, and input the second registration result into the intermediate infrared extraction unit and the intermediate visible light extraction unit respectively to generate infrared features and visible light features, and input the infrared features into the intermediate infrared segmentation unit to generate a second infrared mask; The infrared feature, the visible light feature and the second infrared mask are sequentially input into the initial fusion unit and the initial reconstruction unit to generate fused features and a reconstructed image, and based on the fused features and the reconstructed image, the second registration unit, the intermediate infrared extraction unit, the intermediate infrared segmentation unit, the intermediate visible light extraction unit, the initial fusion unit and the initial reconstruction unit are iteratively trained to generate a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit to form the target fusion model.

2. The method according to claim 1, characterized in that The initial extraction unit includes a feature extraction branch based on the object detection and semantic segmentation model of YOLOv8; The feature extraction branch at least includes a combination layer, a first downsampling layer, a second downsampling layer, a third downsampling layer and a fourth downsampling layer; The initial segmentation unit includes a segmentation branch of the YOLOv8-based target detection and semantic segmentation model; the segmentation branch includes a first upsampling layer, a second upsampling layer, a third upsampling layer and a fourth upsampling layer.

3. The method according to claim 2, characterized in that During the training process of the initial fusion model, model parameters of the third downsampling layer, the fourth downsampling layer and the segmentation branch are frozen.

4. The method according to claim 1, characterized in that: During the training process of the initial fusion model, a loss function corresponding to the initial fusion unit is constructed based on the information density of the mask area.

5. An image fusion method, characterized in that: include: Acquire a target infrared image and a target visible light image, and determine a target fusion model; wherein the target fusion model is pre-trained by the image fusion model training method according to any one of claims 1 to 4; the target fusion model includes a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit; Inputting the target infrared image and the target visible light image into the target registration unit to generate a target registration result; Inputting the target registration result into the target infrared extraction unit and the target visible light extraction unit respectively to generate target infrared features and target visible light features; Inputting the target infrared feature into the target infrared segmentation unit to generate a target infrared mask; The target infrared feature, the target visible light feature and the target infrared mask are sequentially input into the target fusion unit and the target reconstruction unit to generate a target fusion image corresponding to the target infrared image and the target visible light image.

6. An image fusion model training device, characterized in that: include: A sample acquisition module, used for acquiring visible light image samples and infrared image samples; A coarse registration module, used for coarsely registering the infrared image sample with the visible light image sample to generate an infrared registered image sample; A model training module, used for inputting the visible light image samples and the infrared registration image samples into an initial fusion model for overall iterative training to generate a target fusion model; wherein the initial fusion model includes an initial registration unit, an initial extraction unit, an initial segmentation unit, an initial fusion unit and an initial reconstruction unit; The model training module is specifically used for: Based on the visible light image samples and the infrared registration image samples, iteratively training the initial registration unit to generate a first registration unit; Inputting the visible light image sample and the infrared registration image sample into the first registration unit to generate a first registration result, and sequentially inputting the first registration result into an initial infrared extraction unit and an initial infrared segmentation unit to generate a first infrared mask, and sequentially inputting the first registration result into an initial visible light extraction unit and an initial visible light segmentation unit to generate a visible light mask; Based on the first infrared mask and the visible light mask, iteratively train the first registration unit, the initial infrared extraction unit, the initial infrared segmentation unit, the initial visible light extraction unit, and the initial visible light segmentation unit to generate a second registration unit, an intermediate infrared extraction unit, an intermediate infrared segmentation unit, and an intermediate visible light extraction unit; Input the visible light image sample and the infrared registration image sample into the second registration unit to generate a second registration result, and input the second registration result into the intermediate infrared extraction unit and the intermediate visible light extraction unit respectively to generate infrared features and visible light features, and input the infrared features into the intermediate infrared segmentation unit to generate a second infrared mask; The infrared feature, the visible light feature and the second infrared mask are sequentially input into the initial fusion unit and the initial reconstruction unit to generate fused features and a reconstructed image, and based on the fused features and the reconstructed image, the second registration unit, the intermediate infrared extraction unit, the intermediate infrared segmentation unit, the intermediate visible light extraction unit, the initial fusion unit and the initial reconstruction unit are iteratively trained to generate a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit to form the target fusion model.

7. An image fusion device, characterized in that: include: An image acquisition module, used to acquire a target infrared image and a target visible light image, and determine a target fusion model; wherein the target fusion model is pre-trained by the image fusion model training method according to any one of claims 1 to 4; the target fusion model includes a target registration unit, a target infrared extraction unit, a target infrared segmentation unit, a target visible light extraction unit, a target fusion unit and a target reconstruction unit; A target registration result generating module, used for inputting the target infrared image and the target visible light image into the target registration unit to generate a target registration result; A feature generation module, used for inputting the target registration result into the target infrared extraction unit and the target visible light extraction unit respectively, to generate a target infrared feature and a target visible light feature; A target infrared mask generation module, used for inputting the target infrared feature into the target infrared segmentation unit to generate a target infrared mask; The target fusion image generation module is used to input the target infrared feature, the target visible light feature and the target infrared mask into the target fusion unit and the target reconstruction unit in sequence to generate a target fusion image corresponding to the target infrared image and the target visible light image.

8. An electronic device, characterized in that: The electronic device comprises: Processor and memory; The processor is used to execute the image fusion model training method as described in any one of claims 1 to 4, or to execute the image fusion method as described in claim 5, by calling the program or instruction stored in the memory.

Images