An infrared and visible light image fusion method, system, device and medium

Abstract

This application provides an infrared-visible light image fusion method, system, device, and medium, relating to the field of image processing technology. The method includes: constructing a student model using a pre-trained infrared-visible light fusion model; embedding fine-tuning units to be trained into each layer of the pre-trained infrared-visible light fusion model to obtain a teacher model to be trained; inputting infrared sample images and visible light sample images into the student fusion sample image obtained by the student model into a semantic extraction model to extract a multi-region semantic sample mask image; using this mask image as a guide, fine-tuning the student fusion sample image through the teacher model to obtain a teacher fusion sample image; extracting perceptual features from the two fusion sample images using a perceptual feature extraction model to obtain their respective perceptual features; and updating the model parameters of the student model based on these two perceptual features to obtain a student model for fusing infrared and visible light images in real-world scenarios. The aim is to improve the fusion effect of infrared and visible light images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an infrared and visible light image fusion method, system, device, and medium. Background Technology

[0002] Infrared images are sensitive to thermal radiation, highlighting the thermal features of targets and are robust to smoke, obstructions, and low light. Visible light images, on the other hand, offer high resolution, contrast, rich texture details, and conform to human visual perception. These features are highly complementary. Fusing infrared and visible light images to generate images that combine thermal features and texture details has significant research and application value, and this technology has been widely used in fields such as security monitoring, military reconnaissance, and autonomous driving.

[0003] Current infrared and visible light image fusion technologies are primarily based on the PVM framework (pretrained visual models, PVMs). While pretrained models have shown great potential in multimodal image fusion, the high computational cost of inference for fully pretrained models poses a significant challenge in practical applications. This computational burden severely limits the deployment and application of pretrained model-based fusion methods in resource-constrained scenarios, making them difficult to implement effectively in mobile devices or edge computing environments. Summary of the Invention

[0004] In view of this, this application provides an infrared-visible light image fusion method, which aims to solve or partially solve the problems existing in the background art.

[0005] A first aspect of this application provides an infrared-visible light image fusion method, the method comprising: Using a pre-trained infrared-visible light fusion model, an image fusion student model to be trained is constructed, and prior fine-tuning units to be trained are embedded in each layer of the pre-trained infrared-visible light fusion model to obtain an image fusion teacher model to be trained. Infrared and visible light sample images of the sample scene are input into the image fusion student model to be trained to obtain the student fused sample image; Semantic knowledge is extracted from the student fusion sample image using a pre-trained semantic extraction model to obtain a multi-region semantic sample mask image; the student fusion sample image is then fine-tuned using the multi-region semantic sample mask image as a guide by the image fusion teacher model to be trained to obtain a teacher fusion sample image. For the student fusion sample image and the teacher fusion sample image, perceptual features are extracted using a pre-trained perceptual feature extraction model to obtain the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image, respectively. Based at least on the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image, the model parameters of the image fusion student model to be trained are updated to obtain the trained image fusion student model; The infrared target image and the visible light target image for the target scene are input into the trained image fusion student model to obtain the first fused image.

[0006] A second aspect of this application provides an infrared-visible light image fusion system, the system comprising: The model building module is used to construct a student model of image fusion to be trained using a pre-trained infrared and visible light fusion model, and to embed a prior fine-tuning unit to be trained into each layer of the pre-trained infrared and visible light fusion model to obtain a teacher model of image fusion to be trained. The first fusion module is used to input infrared sample images and visible light sample images for the sample scene into the image fusion student model to be trained, so as to obtain the student fused sample image; The semantic extraction module is used to extract semantic knowledge from the student fused sample image through a pre-trained semantic extraction model to obtain a multi-region semantic sample mask map; The second fusion module is used to fine-tune the student fusion sample image by using the image fusion teacher model to be trained and guided by the multi-region semantic sample mask image to obtain the teacher fusion sample image; The perceptual feature extraction module is used to extract perceptual features from the student fusion sample image and the teacher fusion sample image using a pre-trained perceptual feature extraction model, so as to obtain the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image respectively. The parameter update module is used to update the model parameters of the image fusion student model to be trained based at least on the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image, so as to obtain the trained image fusion student model. The target fusion module is used to input infrared target images and visible light target images for the target scene into the trained image fusion student model to obtain the first fused image.

[0007] A third aspect of this application provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and running on the processor, wherein the computer program, when executed by the processor, implements the steps of an infrared-visible light image fusion method as described in the first aspect of this application.

[0008] The fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of an infrared-visible light image fusion method as described in the first aspect of this application.

[0009] The infrared-visible light image fusion method provided in this application has the following advantages: The infrared and visible light image fusion method provided in this application introduces a pre-trained semantic extraction model (such as the SAM (Segment Anything Model) model) and integrates the semantic prior knowledge extracted by the pre-trained semantic extraction model into the fusion process of multimodal images. This deeply mines the semantic information in the scene, which can make the semantic boundary between the target and the background in the final fused image clearer and the fusion effect better.

[0010] Furthermore, this application effectively transfers the complex representations of semantic prior knowledge contained in the image fusion teacher model to the image fusion student model during the training phase (i.e., the process of updating the model parameters of the image fusion student model to be trained based on the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image is the process of transferring the complex representations of semantic prior knowledge contained in the image fusion teacher model to the image fusion student model). This allows the trained image fusion student model to run independently without relying on computationally intensive semantic extraction models (such as the SAM (Segment Anything Model) model) during practical application inference. This can significantly reduce computational complexity while maintaining the excellent fusion performance of the image fusion teacher model (e.g., making the semantic boundaries between the target and the background in the final fused image clearer and the fusion effect better). This can greatly improve the application value of the trained image fusion student model in real-world scenarios. For example, the computational complexity of the trained image fusion student model in this application is significantly reduced, allowing it to be deployed on mobile devices or edge computing environments. It can also effectively adapt to tasks such as security monitoring and night vision navigation in drone scenarios, achieving significant improvements in both the visual quality of the fused image and the adaptability to downstream tasks. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating an infrared-visible light image fusion method according to one embodiment of this application; Figure 2 This is another flowchart illustrating an infrared-visible light image fusion method according to one embodiment of this application; Figure 3 This is a schematic diagram of the structure of a priori fine-tuning unit in an infrared-visible light image fusion method according to an embodiment of this application; Figure 4 This is a schematic diagram of an infrared-visible light image fusion system according to one embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] Before describing the solution of this application, the application scenarios of the infrared and visible light image fusion method provided in this application will be explained first. The method provided in this application is used to fuse infrared and visible light images of the same scene to obtain a fused image that contains both the thermal target features of the infrared image and the texture detail features of the visible light image. It is particularly suitable for fusing infrared and visible light images of the same scene acquired by UAVs. Here, "same scene" refers to the same three-dimensional physical space observed jointly after the camera coordinate systems of the infrared and visible light images are aligned through pre-calibration. It can also be selectively limited to ensuring temporal consistency between the acquired infrared and visible light images of the same scene. The method provided in this application can also fuse infrared and visible light images of different scenes, but in these images of different scenes, there needs to be a partial overlap in the three-dimensional physical space.

[0015] refer to Figure 1 , Figure 1 This is a flowchart illustrating an infrared-visible light image fusion method according to one embodiment of this application. Figure 1 As shown, the method includes: Step S01: Construct a student model for image fusion to be trained using a pre-trained infrared-visible light fusion model, and embed the prior fine-tuning unit to be trained into each layer of the pre-trained infrared-visible light fusion model to obtain a teacher model for image fusion to be trained.

[0016] In this embodiment, before describing the infrared-visible light image fusion method provided by this application, the image fusion training framework on which the infrared-visible light image fusion method provided by this application is based will be described first. For example... Figure 2As shown, the framework consists of a student model for image fusion to be trained, a teacher model for image fusion to be trained, a pre-trained semantic extraction model, and a pre-trained perceptual feature extraction model. The pre-trained semantic extraction model is the SAM (Segment Anything Model), and the pre-trained perceptual feature extraction model is the DINOv3 model.

[0017] The image fusion student model to be trained is a pre-trained infrared and visible image fusion model (IVIF model). This pre-trained infrared and visible image fusion model is an existing fusion model that can be directly deployed and applied for fusing infrared and visible images in the corresponding scene. However, because this pre-trained infrared and visible image fusion model does not consider semantic priors as in this application, its performance is significantly lower than that of the finally trained image fusion student model in this application.

[0018] A copy of the pre-trained infrared-visible light fusion model constituting the image fusion student model to be trained is used to construct the image fusion teacher model to be trained. Prior tuning units to be trained are embedded into each layer of the copied pre-trained infrared-visible light fusion model to construct the image fusion teacher model to be trained. The prior tuning units are SPT (Simultaneous Prior Tuning Units).

[0019] Step S02: Input the infrared sample image and visible light sample image for the sample scene into the image fusion student model to be trained to obtain the student fused sample image.

[0020] In this embodiment, infrared and visible light sample images are acquired in the model training scenario. These images are then input into the image fusion student model to be trained within the image fusion training framework for preliminary image fusion, resulting in a student fused sample image output by the model. Specifically, the student model first extracts thermal target features from the input infrared sample image and texture detail features from the input visible light sample image using its own feature extraction modules (such as convolutional layers and Transformer layers). Then, the two types of features are preliminarily concatenated and weighted to obtain the corresponding preliminary fusion result, which is the output student fused sample image. For the student model to be trained before being trained using the image fusion training framework of this application, the preliminary fusion result it outputs often suffers from semantic ambiguity (e.g., unclear boundaries between thermal targets and background) or insufficient matching between texture and target.

[0021] The sample scene can be either a nighttime scene or a daytime scene. When the sample scene is a nighttime scene, the infrared image acquired at this time can highlight heat targets such as pedestrians and vehicles, but the image is blurry and lacks detail. In contrast, the visible light image shows the texture of buildings and roads, but targets in dark areas are almost invisible.

[0022] Step S03: Extract semantic knowledge from the student fusion sample image using a pre-trained semantic extraction model to obtain a multi-region semantic sample mask image; use the image fusion teacher model to be trained, guided by the multi-region semantic sample mask image, to fine-tune the student fusion sample image to obtain a teacher fusion sample image.

[0023] In this embodiment, since the training method is the same for each training round, and a new round of training is performed only after the model parameters are updated, the following explanation will take one training round as an example.

[0024] In the current training epoch, after obtaining the student fusion sample image output by the image fusion student model to be trained, this student fusion sample image is input into the pre-trained semantic extraction model in the image fusion training framework. This pre-trained semantic extraction model possesses powerful general semantic segmentation capabilities, capable of extracting multi-region semantic masks from arbitrary images without annotation. For example, it can extract pedestrians, vehicles, building backgrounds, road edges, etc., from images input to this pre-trained semantic extraction model and output corresponding binary semantic masks (regions with a value of 1 in the mask correspond to the identified semantic target, and regions with a value of 0 correspond to the background). These extracted multi-region semantic masks can provide precise semantic guidance to the image fusion teacher model to be trained, clearly informing the model which regions are the core targets that need to be preserved and which regions are backgrounds that need to be supplemented with texture, thus providing a natural semantic prior for the fusion task.

[0025] The pre-trained semantic extraction model will extract semantic knowledge from the input student fusion sample image to obtain a multi-region semantic sample mask map in the input student fusion sample image.

[0026] After obtaining the multi-region semantic sample mask image extracted by the pre-trained semantic extraction model, both the extracted multi-region semantic sample mask image and the obtained student fusion sample image are input into the image fusion teacher model to be trained. Using the input multi-region semantic sample mask image as semantic guidance, the input student fusion sample image is fine-tuned to obtain the corresponding teacher fusion sample image.

[0027] In the feature extraction process, different intermediate layers of the image fusion teacher model to be trained correspond to features at different scales. This application embeds corresponding prior fine-tuning units into these intermediate layers of the image fusion teacher model to be trained. The shallow features extracted by the shallow intermediate layers of the image fusion teacher model to be trained focus on pixel-level texture, while the deep features extracted by the deep intermediate layers of the image fusion teacher model to be trained focus on region-level semantics. For example, in the shallow features, the weight of infrared thermal target features is enhanced for the pedestrian region identified by the pre-trained semantic extraction model, while preserving the visible light clothing texture details; in the deep features, the features of the vehicle region are made more distinguishable from the features of the background region, avoiding the target being submerged by the background. Through this hierarchical semantic fine-tuning, each layer of features in the image fusion teacher model to be trained can fit the actual semantic structure of the image, rather than being optimized only at the final output layer.

[0028] Step S04: For the student fusion sample image and the teacher fusion sample image, perform perceptual feature extraction using a pre-trained perceptual feature extraction model to obtain the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image, respectively.

[0029] In this embodiment, the pre-trained perceptual feature extraction model in this application is a pre-trained DINOv3 model, which is a self-supervised model trained on large-scale unlabeled images. It is trained using an unsupervised teacher-student framework without a labeled dataset, enabling it to capture fine-grained textures and exhibit strong local spatial semantics. This allows it to learn the general perceptual features that high-quality images should possess, such as natural texture distribution, reasonable contrast, and detail levels that conform to human visual habits. This application uses this pre-trained perceptual feature extraction model to extract perceptual features from the obtained student fusion sample image and teacher fusion sample image, respectively, to obtain the perceptual features of the student fusion sample image corresponding to the student fusion sample image and the perceptual features of the teacher fusion sample image corresponding to the teacher fusion sample image.

[0030] Step S05: Based at least on the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image, update the model parameters of the image fusion student model to be trained to obtain the trained image fusion student model.

[0031] In this embodiment, after obtaining the perceptual features of the student fused sample image and the perceptual features of the teacher fused sample image, the difference between them is determined. If the difference between the two does not meet the condition for ending training, the model parameters of the image fusion student model to be trained are updated based on the current difference between them. This process constrains the perceptual features of the student fused sample image and the teacher fused sample image to maintain a match, which is equivalent to making the visual effect of the fused image output by the image fusion student model to be trained closer to the image that humans consider beautiful and clear, solving the problem of clear targets but stiff images and messy textures in traditional fusion methods.

[0032] After updating the model parameters of the image fusion student model to be trained, a new round of model training is performed using the updated parameters. Specifically, returning to step S02, infrared and visible light sample images from the same or different sample scenes in the previous training round are input into the updated parameters of the image fusion student model to be trained for a new round of model training. This continues until the difference between the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image obtained in one training round meets the condition for ending training. At this point, the trained image fusion student model is obtained.

[0033] Because the image fusion teacher model in this application considers semantic prior knowledge during the fusion process, it has better performance in fusing infrared sample images and visible light sample images. For example, it can make the semantic boundary between the target and the background clearer in the final fused image and achieve better fusion results.

[0034] Training the image fusion student model based on the differences between the perceptual features of student-fused sample images and teacher-fused sample images aims to match the semantic perception of the student model with that of the teacher model, enabling the student model to exhibit better semantic perception performance during image fusion, similar to the teacher model. The trained student model, having acquired the complex representation of the semantic prior knowledge included in the teacher model during training, demonstrates superior performance in fusing infrared and visible light images. Like the teacher model, it can achieve clearer semantic boundaries between targets and backgrounds in the fused image, resulting in a better fusion effect.

[0035] In this application, the pre-trained perceptual feature extraction model includes an I layer.

[0036] In this embodiment, the pre-trained perceptual feature extraction model in this application is a multi-layered perceptual feature extraction model, which includes I layers, each of which can extract corresponding perceptual features. Based on this hierarchical structure of the pre-trained perceptual feature extraction model, in order to better train the image fusion student model to be trained, this application calculates the layered loss based on the perceptual features extracted layer by layer by the pre-trained perceptual feature extraction model, and then combines the losses of all layers to obtain the final perceptual distillation loss (e.g., Figure 2 In The perceptual distillation loss is used as the difference between the teacher-fused sample image and the student-fused sample image. The specific calculation of the perceptual distillation loss is as follows: The student fusion sample image is input into a pre-trained perceptual feature extraction model to obtain the perceptual features of the student fusion sample image, including: taking i sequentially from 1 to I, and passing it through the i-th layer of the pre-trained perceptual feature extraction model to obtain the perceptual features of the i-th student fusion sample image; the teacher fusion sample image is input into the pre-trained perceptual feature extraction model to obtain the perceptual features of the teacher fusion sample image, including: taking i sequentially from 1 to I, and passing it through the i-th layer of the pre-trained perceptual feature extraction model to obtain the perceptual features of the i-th teacher fusion sample image.

[0037] In this embodiment, the specific process of inputting the student fusion sample image output by the image fusion student model to be trained into the pre-trained perceptual feature extraction model to obtain the perceptual features of the student fusion sample image is as follows: the student fusion sample image is input into the pre-trained perceptual feature extraction model, and the i-th layer of the pre-trained perceptual feature extraction model will extract the corresponding i-th student fusion sample image perceptual features, i taking values ​​from 1 to I in sequence, that is, each layer of the pre-trained perceptual feature extraction model will extract its corresponding student fusion sample image perceptual features.

[0038] In this embodiment, similarly, the application inputs the teacher fusion sample image output by the image fusion teacher model to be trained into the pre-trained perceptual feature extraction model to obtain the perceptual features of the teacher fusion sample image. The specific process is as follows: the teacher fusion sample image is input into the pre-trained perceptual feature extraction model, and the i-th layer of the pre-trained perceptual feature extraction model will extract the corresponding i-th perceptual feature of the teacher fusion sample image. i takes values ​​from 1 to I in sequence, that is, each layer of the pre-trained perceptual feature extraction model will extract its corresponding perceptual feature of the teacher fusion sample image.

[0039] In this application, the perceptual distillation loss of the image fusion student model to be trained is obtained based on the perceptual features of the teacher fusion sample image and the perceptual features of the student fusion sample image, including: determining the i-th perceptual distillation loss based on the difference between the i-th teacher fusion sample image perceptual features and the i-th student fusion sample image perceptual features; and obtaining the perceptual distillation loss of the image fusion student model to be trained according to the 1st to the 1st perceptual distillation loss.

[0040] In this embodiment, in the current training round, based on the difference between the perceptual features of the i-th teacher fusion sample image and the perceptual features of the i-th student fusion sample image extracted by the i-th layer in the pre-trained perceptual feature extraction model, the i-th perceptual distillation loss is calculated, where i ranges from 1 to 1. That is, each layer in the pre-trained perceptual feature extraction model will have a corresponding perceptual distillation loss calculated. Finally, in the current training round, based on the 1st to 1st perceptual distillation losses, the perceptual distillation loss of the image fusion student model to be trained is calculated. The expression for calculating the perceptual distillation loss of the image fusion student model to be trained is:

[0041] in, This represents the perceptual features of the i-th student fused sample image extracted from the i-th layer of the pre-trained perceptual feature extraction model. This represents the perceptual features of the i-th teacher fusion sample image extracted from the i-th layer of the pre-trained perceptual feature extraction model.

[0042] In conjunction with the above embodiments, in one implementation, this application also provides an infrared-visible light image fusion method. This infrared-visible light image fusion method further includes: obtaining the reconstruction distillation loss of the image fusion student model to be trained based on the difference between the teacher fusion sample image and the student fusion sample image.

[0043] In this embodiment, to enable the image fusion student model to better learn the fusion capabilities of the image fusion teacher model, this application also introduces reconstruction distillation loss (e.g., ...) during the training process of the image fusion student model to be trained. Figure 2 In The reconstruction distillation loss is used to ensure that the output of the image fusion student model matches the output of the image fusion teacher model. Specifically, based on the difference between the teacher fusion sample image output by the image fusion teacher model to be trained in the current training round and the student fusion sample image output by the image fusion student model to be trained in the current training round, the reconstruction distillation loss of the image fusion student model to be trained in the current training round is obtained. The expression for calculating the reconstruction distillation loss is as follows:

[0044] in, This represents the student's fused sample image. This represents the teacher's fused sample image.

[0045] In this application, when the infrared and visible light image fusion method provided in this embodiment further includes reconstruction distillation loss, step S05 will take another implementation method: based on the reconstruction distillation loss and perceptual distillation loss of the image fusion student model to be trained, the model parameters of the image fusion student model to be trained will be updated to obtain the trained image fusion student model.

[0046] In this embodiment, when the reconstruction distillation loss is introduced during the training process of the image fusion student model to be trained, in the current training round, the application will update the model parameters of the image fusion student model to be trained based on the reconstruction distillation loss and perceptual distillation loss of the image fusion student model to be trained.

[0047] Step S06: Input the infrared target image and the visible light target image of the target scene into the trained image fusion student model to obtain the first fused image.

[0048] In this embodiment, after training to obtain a deployable image fusion student model with better fusion performance, the computational complexity can be significantly reduced because this image fusion student model does not rely on a computationally intensive semantic extraction model. Therefore, it can be deployed on mobile devices or edge computing environments for the fusion processing of infrared and visible light images. One possible application scenario is deployment in drone equipment for tasks such as security monitoring and night vision navigation in drone scenarios. The image acquisition device in the deployed equipment acquires infrared and visible light target images of the target scene. The acquired infrared and visible light target images are input into the trained image fusion student model in the device for fusion processing to obtain the corresponding first fused image. This type of fused image can both highlight thermal targets (such as pedestrians and vehicles at night) like an infrared image and present the fine textures of buildings and roads like a visible light image. Simultaneously, the semantic boundaries between the target and the background are clear, and the visual quality of the image is more in line with human visual observation habits. It can be directly applied to scenarios such as security monitoring and night vision navigation that have dual requirements for target recognition and detail richness in images.

[0049] The infrared and visible light image fusion method provided in this application introduces a pre-trained semantic extraction model (such as the SAM (Segment Anything Model) model) and integrates the semantic prior knowledge extracted by the pre-trained semantic extraction model into the fusion process of multimodal images. This deeply mines the semantic information in the scene, which can make the semantic boundary between the target and the background in the final fused image clearer and the fusion effect better.

[0050] Furthermore, this application effectively transfers the complex representations of semantic prior knowledge contained in the image fusion teacher model to the image fusion student model during the training phase (i.e., the process of updating the model parameters of the image fusion student model to be trained based on the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image is the process of transferring the complex representations of semantic prior knowledge contained in the image fusion teacher model to the image fusion student model). This allows the trained image fusion student model to run independently without relying on computationally intensive semantic extraction models (such as the SAM (Segment Anything Model) model) during practical application inference. This can significantly reduce computational complexity while maintaining the excellent fusion performance of the image fusion teacher model (e.g., making the semantic boundaries between the target and the background in the final fused image clearer and the fusion effect better). This can greatly improve the application value of the trained image fusion student model in real-world scenarios. For example, the computational complexity of the trained image fusion student model in this application is significantly reduced, allowing it to be deployed on mobile devices or edge computing environments. It can also effectively adapt to tasks such as security monitoring and night vision navigation in drone scenarios, achieving significant improvements in both the visual quality of the fused image and the adaptability to downstream tasks.

[0051] In conjunction with the above embodiments, in one implementation, this application also provides an infrared-visible light image fusion method. In this infrared-visible light image fusion method, the pre-trained infrared-visible light fusion model in the image fusion teacher model to be trained has N layers, and each layer deploys one infrared-visible light fusion unit.

[0052] Using the image fusion teacher model to be trained, and guided by the multi-region semantic sample mask image, the student fusion sample image is fine-tuned to obtain the teacher fusion sample image, including: Step S03_1: Input the multi-region semantic sample mask map and the student fused sample image into the prior fine-tuning unit to be trained embedded in the first layer to obtain the first fine-tuning intermediate image features.

[0053] In this embodiment, the number of layers in the pre-trained infrared-visible light fusion model of the image fusion teacher model to be trained in this application is N, and the preferred value of N is 6 layers, such as... Figure 2 As shown,Figure 2 The pre-trained infrared-visible light fusion model shown has 6 layers. It should be understood that the number of layers N in the pre-trained infrared-visible light fusion model can also be other values, and no specific limitation is made here. Each layer of the pre-trained infrared-visible light fusion model deploys one infrared-visible light fusion unit.

[0054] In this embodiment, the inputs to the prior fine-tuning units to be trained, embedded in different layers, differ to some extent. For example... Figure 2 As shown, the input to the prior fine-tuning unit embedded in the first layer is: the multi-region semantic sample mask image extracted by the semantic extraction model and the student fused sample image output by the image fusion student model to be trained. The prior fine-tuning unit embedded in the first layer obtains the first fine-tuned intermediate image features, i.e. Figure 2 The fine-tuning results are output from the first SPT. For example... Figure 2 As shown, the prior fine-tuning unit to be trained embedded in the first layer is... Figure 2 The first SPT in the series.

[0055] Step S03_2: Input the student fusion sample image into the infrared-visible light fusion unit of the first layer to obtain the first fusion intermediate image features.

[0056] In this embodiment, the student fused sample image output by the image fusion student model to be trained is simultaneously input into the infrared-visible light fusion unit of the first layer to obtain the first fused intermediate image features, i.e. Figure 2 The result is output by the first infrared-visible light fusion unit.

[0057] Step S03_3: Input the first fine-tuned intermediate image features and the first fused intermediate image features into the prior fine-tuning unit to be trained embedded in the second layer to obtain the second fine-tuned intermediate image features.

[0058] In this embodiment, as Figure 2 As shown, after obtaining the first fine-tuned intermediate image features and the first fused intermediate image features through steps S03_1 to S03_2, both are input into the prior fine-tuning unit to be trained embedded in the second layer to obtain the second fine-tuned intermediate image features, i.e. Figure 2 The fine-tuning results are output from the second SPT. For example... Figure 2 As shown, the prior fine-tuning unit to be trained embedded in the second layer is... Figure 2 The second SPT in the series.

[0059] Step S03_4: Take n sequentially from 2 to N, input the nth fine-tuned intermediate image features into the infrared-visible light fusion unit of the nth layer to obtain the nth fused intermediate image features; input the nth fine-tuned intermediate image features and the nth fused intermediate image features into the prior fine-tuning unit to be trained embedded in the (n+1)th layer to obtain the (n+1)th fine-tuned intermediate image features; obtain the teacher fused sample image based on the Nth fine-tuned intermediate image features output by the infrared-visible light fusion unit of the Nth layer.

[0060] In this embodiment, n is sequentially selected from 2 to N. For any value of n, the nth fine-tuned intermediate image feature is input into the infrared-visible light fusion unit of the nth layer to obtain the nth fused intermediate image feature. Simultaneously, the nth fine-tuned intermediate image feature and the nth fused intermediate image feature are input into the prior fine-tuning unit to be trained embedded in the (n+1)th layer to obtain the (n+1)th fine-tuned intermediate image feature. Finally, based on the Nth fine-tuned intermediate image feature output by the infrared-visible light fusion unit of the Nth layer (the last layer), the teacher fused sample image is obtained. Figure 2 As shown, the prior fine-tuning unit to be trained embedded in the nth layer is... Figure 3 The nth SPT in the process. The nth fine-tuned intermediate image feature is (n takes values ​​from 2 to N): the output of the infrared-visible fusion unit of the (n-1)th layer (i.e., the (n-1)th fused intermediate image feature), and the output of the prior fine-tuning unit to be trained embedded in the (n-1)th layer (i.e., the (n-1)th fine-tuned intermediate image feature), are input into the prior fine-tuning unit to be trained embedded in the nth layer for processing to obtain the output result.

[0061] In conjunction with the above embodiments, in one implementation, this application also provides an infrared-visible light image fusion method. In this infrared-visible light image fusion method, the process of the prior fine-tuning unit embedded in the nth layer outputting the nth fine-tuned intermediate image features includes: Step S001: When n is 1, the multi-region semantic sample mask map and the student fused sample image are spliced ​​by the splicing sub-unit in the prior fine-tuning unit to be trained embedded in the nth layer; and when n is 2 to N, the (n-1)th fine-tuned intermediate image features and the (n-1)th fused intermediate image features are spliced ​​by the splicing sub-unit in the prior fine-tuning unit to be trained embedded in the nth layer.

[0062] In this embodiment, the specific processing procedure for the prior fine-tuning unit to be trained is described, and the specific structure of the prior fine-tuning unit to be trained will be explained accordingly. For example... Figure 3 As shown, the prior fine-tuning unit to be trained consists of spliced ​​sub-units (such as...) Figure 3 In the C), the first subunit, the second subunit, and the linear layer (such as ...). Figure 3It consists of a Bilinear layer. The first and second sub-units have the same structure, both including a first convolutional module, an activation function layer, and a second convolutional module connected in series, as shown below. Figure 2 Conv3×3, ReLU, and Conv3×3 in the context of this.

[0063] In this embodiment, when n is 1, the input multi-region semantic sample mask image and student fused sample image are spliced ​​by the splicing sub-unit in the prior fine-tuning unit to be trained embedded in the first layer to obtain the corresponding output result.

[0064] When n is 2 to N, the corresponding output result is obtained by splicing the (n-1)th fine-tuned intermediate image features and the (n-1)th fused intermediate image features in the splicing sub-unit of the prior fine-tuning unit embedded in the nth layer.

[0065] Step S002: When n is 1 to N, the output of the stitching unit is processed by the first sub-unit in the prior fine-tuning unit embedded in the nth layer to obtain the first intermediate feature; the first intermediate feature is processed by the second sub-unit in the prior fine-tuning unit embedded in the nth layer to obtain the first intermediate feature, and the first intermediate feature is processed by the linear layer in the prior fine-tuning unit embedded in the nth layer to obtain the first intermediate feature; the output of the second sub-unit is fused with the output of the linear layer to obtain the nth fine-tuned intermediate image feature; wherein, the first sub-unit and the second sub-unit have the same structure, both including a first convolutional module, an activation function layer and a second convolutional module connected in series.

[0066] In this embodiment, when n is 1 to N, the first intermediate feature is obtained by processing the output of its own splicing unit through the first sub-unit in the prior fine-tuning unit to be trained embedded in the nth layer.

[0067] The first intermediate feature is then processed by the second sub-unit in the prior fine-tuning unit embedded in the nth layer, and the first intermediate feature is also processed by the linear layer in the prior fine-tuning unit embedded in the nth layer.

[0068] Finally, the output of the second sub-unit in the prior fine-tuning unit embedded in the nth layer is fused with the output of the linear layer to obtain the nth fine-tuned intermediate image features.

[0069] In conjunction with the above embodiments, in one implementation, this application also provides an infrared-visible light image fusion method. This infrared-visible light image fusion method further includes: updating the model parameters of each prior fine-tuning unit in the image fusion teacher model to be trained and the pre-trained infrared-visible light fusion model based on the differences between the teacher fusion sample image and the infrared sample image, and the differences between the teacher fusion sample image and the visible light sample image, to obtain a trained image fusion teacher model; the trained image fusion teacher model, the pre-trained infrared-visible light fusion model, and the pre-trained semantic extraction model form an image fusion system for fusing infrared target images and visible light target images of a target scene; connecting the input of the pre-trained semantic extraction model to the output of the pre-trained infrared-visible light fusion model, and connecting the output of the pre-trained semantic extraction model to the input of the first trained prior fine-tuning unit in the trained image fusion teacher model, to obtain the image fusion system.

[0070] In this embodiment, during the training of the image fusion student model to be trained, the application simultaneously updates the model parameters of each prior fine-tuning unit in the image fusion teacher model to be trained, as well as the model parameters of the pre-trained infrared-visible light fusion model in the image fusion teacher model to be trained. The training of the image fusion student model and the image fusion teacher model are performed synchronously.

[0071] When updating the model parameters of the image fusion teacher model to be trained, this application will simultaneously update the model parameters of each prior fine-tuning unit in the image fusion teacher model to be trained and the pre-trained infrared and visible light fusion model in the image fusion teacher model to be trained, based on the difference between the teacher fusion sample image output by the image fusion teacher model to be trained and the infrared sample image of the input image fusion student model, as well as the difference between the teacher fusion sample image and the visible light sample image of the input image fusion student model.

[0072] When the two differences mentioned above satisfy the pre-set end-of-training conditions, the trained image fusion teacher model is obtained. One optional difference evaluation angle is the image gradient (such as edge, texture, and other features). A good fusion result can preserve significant edges in the source image; therefore, for the teacher fusion sample image output by the image fusion teacher model to be trained, the input infrared sample image participating in the fusion, and the visible light sample image, their respective gradient maps are calculated. Then, based on the gradient similarity between the teacher fusion sample image output by the image fusion teacher model to be trained and the input infrared sample image participating in the fusion, respectively, the end-of-training condition is defined: if the gradient similarity between the obtained teacher fusion sample image and the input infrared sample image participating in the fusion is greater than a first threshold, and the gradient similarity between the obtained teacher fusion sample image and the input visible light sample image participating in the fusion is greater than a second threshold.

[0073] In this embodiment, the finally trained image fusion teacher model, a pre-trained infrared-visible light fusion model, and a pre-trained semantic extraction model constitute an image fusion system. This image fusion system can be used to fuse infrared target images and visible light target images for a target scene. The target scene refers to a real-world application scenario, such as infrared target images and visible light target images collected in actual security monitoring.

[0074] The structure of the image fusion system and Figure 2 The structures shown are similar, but the difference is... Figure 2 The image fusion student model in this system will no longer be a student model requiring training; instead, it will only require an existing pre-trained infrared-visible fusion model at that location that does not require any training. Since this image fusion system no longer has a student model to be trained, therefore... Figure 4 Another difference between the structure shown and the image fusion system is that there is no longer a perceptual feature extraction model for training the student model of image fusion.

[0075] As can be seen, the structure of this image fusion system includes a pre-trained infrared-visible light fusion model for receiving input infrared and visible light images, which performs initial fusion of the two inputs. Simultaneously, the system also includes a pre-trained semantic extraction model that receives the initial fusion result, and a trained image fusion teacher model. The trained teacher model also receives multi-region semantic sample masks extracted by the pre-trained semantic extraction model. Finally, the trained teacher model fine-tunes the initial fusion result based on the multi-region semantic sample masks, resulting in a final fused image with better fusion performance.

[0076] In this embodiment, the specific connection relationships between the various units in the image fusion system provided by this application are as follows: The input of the pre-trained semantic extraction model will be connected to the output of the pre-trained infrared-visible light fusion model. The preliminary fusion result output by the pre-trained infrared-visible light fusion model will serve as the input of the pre-trained semantic extraction model. Simultaneously, the output of the pre-trained semantic extraction model will be connected to the input of the first trained prior fine-tuning unit in the trained image fusion teacher model. The multi-region semantic sample mask image extracted by the pre-trained semantic extraction model will serve as the input of the first trained prior fine-tuning unit in the trained image fusion teacher model.

[0077] In conjunction with the above embodiments, in one implementation, this application also provides an infrared-visible light image fusion method. This infrared-visible light image fusion method further includes: Step S101: Input the infrared target image and the visible light target image of the target scene into the pre-trained infrared and visible light fusion model to obtain a preliminary fused image.

[0078] In this embodiment, the core of the image fusion system is to directly use a trained image fusion teacher model for image fusion. Although this teacher model is larger and more complex than a trained image fusion student model, it also achieves higher accuracy. When lightweight deployment and mobile device (or edge computing environment) requirements are not a priority, this image fusion system can be used to fuse infrared and visible light images in a corresponding scenario to achieve better fusion results.

[0079] The specific fusion process of this image fusion system in application is as follows: First, infrared and visible light target images of the target scene to be fused are acquired. Then, the infrared and visible light target images of the target scene are input into the pre-trained infrared-visible light fusion model of the image fusion system to perform preliminary fusion, resulting in a preliminary fused image.

[0080] Step S102: Input the preliminary fused image into the pre-trained semantic extraction model to obtain a multi-region semantic mask map of the preliminary fused image.

[0081] In this embodiment, the obtained preliminary fusion result is input into the pre-trained semantic extraction model of the image fusion system. The pre-trained semantic extraction model extracts semantic knowledge from the preliminary fusion result to obtain a multi-region semantic sample mask map corresponding to the preliminary fused image. This multi-region semantic sample mask map is used to subsequently fine-tune the preliminary fused image to obtain a better fusion result.

[0082] Step S103: Input the multi-region semantic mask map and the preliminary fused image into the trained image fusion teacher model. Using the trained image fusion teacher model and guided by the multi-region semantic mask map, fine-tune the preliminary fused image to obtain the second fused image.

[0083] In this embodiment, the multi-region semantic mask and the preliminary fused image are simultaneously input into the trained image fusion teacher model in the image fusion system. Using the trained image fusion teacher model, and guided by the multi-region semantic mask, the preliminary fused image is fine-tuned based on the trained prior fine-tuning units embedded in each layer of the model, resulting in a second fused image with better fusion performance.

[0084] The infrared and visible light image fusion method provided in this application first inputs the infrared and visible light images of the corresponding scene into a student model for image fusion to be trained. This student model extracts thermal target features and texture detail features from the two types of images and performs preliminary fusion to obtain a preliminary fusion set. Then, the preliminary fusion result is input into a pre-trained semantic extraction model (such as the SAM model) to extract a multi-region semantic mask map as a semantic prior. Next, through the prior fine-tuning unit in the teacher model for image fusion to be trained, multi-scale intermediate features of the pre-trained infrared and visible light fusion model in the teacher model are hierarchically optimized using the multi-region semantic mask map as a guide. Simultaneously, [the method] introduces... (Reconstruction of distillation loss) and (Perceptual distillation loss) Constructs a two-dimensional constraint, using a distillation framework to transfer representations containing SAM semantic knowledge to the image fusion student model to be trained, enabling independent operation during the inference stage. This two-dimensional constraint ensures the semantic rationality and visual perception quality of the fusion result. The perceptual distillation loss constrains the relationships between semantic regions, making the spatial relationships of semantic regions in the fused image clear and the visual effect better. That is, a pre-trained perceptual feature extraction model is used as a feature extractor to perform semantic perception constraints. The fused image generated by the infrared and visible light image fusion method provided in this application has both the salience of infrared thermal targets and the details of visible light textures. The semantic boundaries are clear and the visual texture conforms to human eye habits. At the same time, it significantly reduces computational complexity and can be effectively adapted to tasks such as security monitoring and night vision navigation in UAV scenarios. It achieves significant improvements in both the visual quality of the fused image and the adaptability to downstream tasks. The method proposed in this application can efficiently utilize the semantic prior of the pre-trained model (i.e. the pre-trained semantic extraction model in this application) to deeply mine scene semantic information, enhance the robustness of the infrared and visible light model to understand different complex scenes and improve processing efficiency, so that the fused image can achieve significant progress in both visual quality and downstream task adaptability.

[0085] Based on the same inventive concept, this application provides an infrared and visible light image fusion system, such as... Figure 1 As shown, the infrared-visible light image fusion system 400 includes: The model building module 401 is used to construct a student model of image fusion to be trained using a pre-trained infrared and visible light fusion model, and to embed a prior fine-tuning unit to be trained into each layer of the pre-trained infrared and visible light fusion model to obtain a teacher model of image fusion to be trained. The first fusion module 402 is used to input infrared sample images and visible light sample images for the sample scene into the image fusion student model to be trained, so as to obtain the student fused sample image; The semantic extraction module 403 is used to extract semantic knowledge from the student fused sample image through a pre-trained semantic extraction model to obtain a multi-region semantic sample mask map. The second fusion module 404 is used to fine-tune the student fusion sample image by using the image fusion teacher model to be trained and guided by the multi-region semantic sample mask image to obtain the teacher fusion sample image; The perceptual feature extraction module 405 is used to extract perceptual features from the student fusion sample image and the teacher fusion sample image using a pre-trained perceptual feature extraction model, and obtain perceptual features of the student fusion sample image and perceptual features of the teacher fusion sample image respectively. The parameter update module 406 is used to update the model parameters of the image fusion student model to be trained based at least on the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image, so as to obtain the trained image fusion student model. The target fusion module 407 is used to input the infrared target image and the visible light target image for the target scene into the trained image fusion student model to obtain the first fused image.

[0086] Based on the same inventive concept, this application provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, it implements the steps of an infrared-visible light image fusion method as described in the first aspect of this application.

[0087] Based on the same inventive concept, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of an infrared-visible light image fusion method as described in the first aspect of this application.

[0088] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of this application.

[0089] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0090] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes ​ The steps of the function specified in one or more boxes.

[0094] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0095] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0096] The above provides a detailed description of an infrared and visible light image fusion method, system, device, and medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An infrared-visible light image fusion method, characterized in that, The method includes: Using a pre-trained infrared-visible light fusion model, an image fusion student model to be trained is constructed, and prior fine-tuning units to be trained are embedded in each layer of the pre-trained infrared-visible light fusion model to obtain an image fusion teacher model to be trained. Infrared and visible light sample images of the sample scene are input into the image fusion student model to be trained to obtain the student fused sample image; Semantic knowledge is extracted from the student fusion sample image using a pre-trained semantic extraction model to obtain a multi-region semantic sample mask image; the student fusion sample image is then fine-tuned using the multi-region semantic sample mask image as a guide by the image fusion teacher model to be trained to obtain a teacher fusion sample image. For the student fusion sample image and the teacher fusion sample image, perceptual features are extracted using a pre-trained perceptual feature extraction model to obtain the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image, respectively. Based at least on the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image, the model parameters of the image fusion student model to be trained are updated to obtain the trained image fusion student model; The infrared target image and the visible light target image for the target scene are input into the trained image fusion student model to obtain the first fused image.

2. The infrared-visible light image fusion method according to claim 1, characterized in that, The pre-trained infrared-visible light fusion model in the image fusion teacher model to be trained has N layers, and each layer is deployed with an infrared-visible light fusion unit. Using the image fusion teacher model to be trained, and guided by the multi-region semantic sample mask image, the student fusion sample image is fine-tuned to obtain the teacher fusion sample image, including: The multi-region semantic sample mask image and the student fused sample image are input into the prior fine-tuning unit to be trained embedded in the first layer to obtain the first fine-tuning intermediate image features. The student fused sample image is input into the infrared-visible light fusion unit of the first layer to obtain the first fused intermediate image features; The first fine-tuned intermediate image features and the first fused intermediate image features are input into the prior fine-tuning unit to be trained embedded in the second layer to obtain the second fine-tuned intermediate image features. By taking n values ​​from 2 to N sequentially, the nth fine-tuned intermediate image features are input into the infrared-visible light fusion unit of the nth layer to obtain the nth fused intermediate image features; the nth fine-tuned intermediate image features and the nth fused intermediate image features are input into the prior fine-tuning unit to be trained embedded in the (n+1)th layer to obtain the (n+1)th fine-tuned intermediate image features; and the teacher fused sample image is obtained based on the Nth fine-tuned intermediate image features output by the infrared-visible light fusion unit of the Nth layer.

3. The infrared-visible light image fusion method according to claim 2, characterized in that, The process of the prior fine-tuning unit embedded in the nth layer outputting the nth fine-tuned intermediate image features includes: When n is 1, the multi-region semantic sample mask map and the student fused sample image are spliced ​​by the splicing sub-unit in the prior fine-tuning unit to be trained embedded in the nth layer; and when n is 2 to N, the (n-1)th fine-tuned intermediate image features and the (n-1)th fused intermediate image features are spliced ​​by the splicing sub-unit in the prior fine-tuning unit to be trained embedded in the nth layer. When n is 1 to N, the output of the stitching unit is processed by the first sub-unit in the prior fine-tuning unit embedded in the nth layer to obtain the first intermediate feature; the first intermediate feature is processed by the second sub-unit in the prior fine-tuning unit embedded in the nth layer to obtain the first intermediate feature, and the first intermediate feature is processed by the linear layer in the prior fine-tuning unit embedded in the nth layer to obtain the first intermediate feature; the output of the second sub-unit is fused with the output of the linear layer to obtain the nth fine-tuned intermediate image feature; The first subunit and the second subunit have the same structure, both including a first convolutional module, an activation function layer and a second convolutional module connected in series.

4. The infrared-visible light image fusion method according to claim 1, characterized in that, The method further includes: Based on the differences between the teacher fusion sample image and the infrared sample image, and the differences between the teacher fusion sample image and the visible light sample image, the model parameters of each prior fine-tuning unit in the image fusion teacher model to be trained and the model parameters of the pre-trained infrared and visible light fusion model are updated to obtain the trained image fusion teacher model. The trained image fusion teacher model, the pre-trained infrared and visible light fusion model, and the pre-trained semantic extraction model constitute an image fusion system for fusing infrared target images and visible light target images for a target scene.

5. The infrared-visible light image fusion method according to claim 4, characterized in that, The method further includes: Connect the input of the pre-trained semantic extraction model to the output of the pre-trained infrared-visible light fusion model, and connect the output of the pre-trained semantic extraction model to the input of the first trained prior fine-tuning unit in the trained image fusion teacher model to obtain the image fusion system. The infrared target image and the visible light target image for the target scene are input into the pre-trained infrared-visible light fusion model to obtain a preliminary fused image; The preliminary fused image is input into the pre-trained semantic extraction model to obtain a multi-region semantic mask map of the preliminary fused image; The multi-region semantic mask and the preliminary fused image are input into the trained image fusion teacher model. Guided by the multi-region semantic mask, the trained image fusion teacher model fine-tunes the preliminary fused image to obtain the second fused image.

6. The infrared-visible light image fusion method according to claim 1, characterized in that, The method further includes: Based on the difference between the teacher fusion sample image and the student fusion sample image, the reconstruction distillation loss of the image fusion student model to be trained is obtained; Based at least on the perceptual features of the student fusion sample images and the perceptual features of the teacher fusion sample images, the model parameters of the image fusion student model to be trained are updated to obtain the trained image fusion student model, including: Based on the reconstruction distillation loss and perceptual distillation loss of the image fusion student model to be trained, the model parameters of the image fusion student model to be trained are updated to obtain the trained image fusion student model.

7. An infrared-visible light image fusion method according to any one of claims 1 to 6, characterized in that, The pre-trained perceptual feature extraction model includes layer I; the student fused sample image is input into the pre-trained perceptual feature extraction model to obtain the perceptual features of the student fused sample image, including: By sequentially selecting i from 1 to I, the i-th layer of the pre-trained perceptual feature extraction model is used to obtain the perceptual features of the fused sample image of the i-th student. The teacher fusion sample image is input into the pre-trained perceptual feature extraction model to obtain the perceptual features of the teacher fusion sample image, including: By sequentially selecting i from 1 to I, the i-th layer of the pre-trained perceptual feature extraction model is used to obtain the perceptual features of the i-th teacher fusion sample image; Based on the perceptual features of the teacher fusion sample image and the perceptual features of the student fusion sample image, the perceptual distillation loss of the image fusion student model to be trained is obtained, including: Based on the difference between the perceptual features of the fused sample image of the i-th teacher and the perceptual features of the fused sample image of the i-th student, determine the perceptual distillation loss of the i-th student; The perceptual distillation loss of the image fusion student model to be trained is obtained based on the first perceptual distillation loss to the I-th perceptual distillation loss.

8. An infrared-visible light image fusion system, characterized in that, The system includes: The model building module is used to construct a student model of image fusion to be trained using a pre-trained infrared and visible light fusion model, and to embed a prior fine-tuning unit to be trained into each layer of the pre-trained infrared and visible light fusion model to obtain a teacher model of image fusion to be trained. The first fusion module is used to input infrared sample images and visible light sample images for the sample scene into the image fusion student model to be trained, so as to obtain the student fused sample image; The semantic extraction module is used to extract semantic knowledge from the student fused sample image through a pre-trained semantic extraction model to obtain a multi-region semantic sample mask map; The second fusion module is used to fine-tune the student fusion sample image by using the image fusion teacher model to be trained and guided by the multi-region semantic sample mask image to obtain the teacher fusion sample image; The perceptual feature extraction module is used to extract perceptual features from the student fusion sample image and the teacher fusion sample image using a pre-trained perceptual feature extraction model, so as to obtain the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image respectively. The parameter update module is used to update the model parameters of the image fusion student model to be trained based at least on the perceptual features of the student fusion sample image and the perceptual features of the teacher fusion sample image, so as to obtain the trained image fusion student model. The target fusion module is used to input infrared target images and visible light target images for the target scene into the trained image fusion student model to obtain the first fused image.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and running on the processor, wherein the computer program, when executed by the processor, implements the steps of an infrared-visible image fusion method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the infrared-visible light image fusion method as described in any one of claims 1 to 7.

Images