Visible light imaging and infrared imaging fusion method and device, electronic equipment and storage medium

By extracting semantic information in visible light imaging and combining infrared imaging, using large language models and transform decoders to generate predicted infrared images, the problem that traditional fusion methods fail to maximize the potential of imaging technology is solved, and higher semantic understanding and images are achieved The blending effect of quality.

CN119027768AInactive Publication Date: 2024-11-26WUXI HONGXIHUI TECHNOLOGY CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411048544.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The traditional infrared imaging and visible light imaging fusion methods are limited to the fusion at the pixel level and have failed to maximize the potential of the two imaging technologies, especially the lack of understanding of the "world" model at the semantic level.

Method used

By acquiring visible light imaging and infrared imaging, semantic information is extracted and input into a large language model with preset task information, generating task prompt words, combining hidden variable input transformation decoding models for visible light and infrared imaging, predicting infrared images, and combining them with visible light images Fusion, using the transformer decoder and multimodal big model for training and editing, to generate semantically rich fusion images.

Benefits of technology

The semantic indicative improvement of infrared images is achieved, so that the predicted infrared images and visible light images can greatly exert the advantages of the two imaging technologies and obtain higher semantic understanding and image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027768A_ABST
    Figure CN119027768A_ABST
Patent Text Reader

Abstract

The invention relates to a visible light imaging and infrared imaging fusion method and device, electronic equipment and a storage medium. The method comprises the steps that visible light imaging and infrared imaging are acquired; semantic information in visible light imaging is extracted, the semantic information and preset task information are input into a first large language model, and task cues are obtained; respectively determining a visible light imaging hidden variable and an infrared imaging hidden variable based on visible light imaging and infrared imaging; a conditional hidden variable synthesized by the task cue word, the visible light imaging hidden variable and the infrared imaging hidden variable is input into a transformation decoding model, a predicted infrared image is obtained, and the transformation decoding model is obtained through training data training based on a converter decoder; and fusing the current frame of visible light image with the predicted infrared image to obtain a target fused image. According to the method, the intelligent model can understand the meaning of the image or the language in the world, the predictive infrared image semantic indication performance is better than that of original infrared imaging, and the advantages of the two imaging technologies are brought into full play.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a visible light imaging and infrared imaging fusion method, device, electronic equipment and storage medium. Background Art

[0002] Infrared imaging is widely used in industrial inspection, military applications and other fields. It has completely different advantages over visible light imaging. In view of the differences between infrared imaging and visible light imaging, each has certain advantages, such as high resolution, natural color, easy to interpret by human eyes, low cost, etc. Infrared imaging can capture images in the dark, detect temperature differences, can be applied to thermal analysis, can be applied to non-contact temperature measurement, etc. The wavelength of the infrared imaging spectrum is relatively large, and the color and edge of the imaging are far inferior to visible light. Therefore, the significance of infrared imaging is actually more reflected at the semantic level rather than the pixel level. For example, in the military field, infrared imaging is used to find hidden targets or identify targets in smoke. By fusing the two imaging methods, the advantages of both technologies can be obtained. However, although image or pixel-level fusion can obtain the advantages of both imaging methods, the traditional fusion method is limited to pixel-level fusion and cannot maximize the potential of the two imaging technologies. Summary of the invention

[0003] Based on this, it is necessary to provide a visible light imaging and infrared imaging fusion method, device, electronic device and storage medium that can maximize the advantages of infrared imaging and visible light imaging technologies in order to address the above technical problems.

[0004] The present invention provides a method for fusing visible light imaging and infrared imaging, the method comprising:

[0005] Obtain visible light imaging and infrared imaging;

[0006] Extract semantic information from visible light imaging, input it into the first language model together with preset task information, and obtain task prompt words;

[0007] Based on visible light imaging and infrared imaging, the visible light imaging latent variables and infrared imaging latent variables are determined respectively;

[0008] Inputting the conditional latent variables synthesized by the task prompt word, the visible light imaging latent variables and the infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image, wherein the transform decoding model is obtained by training the training data based on the transformer decoder;

[0009] The current frame visible light image is fused with the predicted infrared image to obtain a target fused image.

[0010] In one embodiment, extracting semantic information from visible light imaging includes:

[0011] The visible light video is input into the video-to-text conversion model to obtain the semantic information in the video.

[0012] In one embodiment, the step of obtaining the task prompt word further comprises:

[0013] The task prompt words are embedded into words to generate task prompt latent variables.

[0014] In one embodiment, inputting the conditional latent variables synthesized from the task prompt words, the visible light imaging latent variables and the infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image includes:

[0015] Inputting the conditional latent variables into the transform decoding model to obtain an output result;

[0016] The output result is input into an image decoder to obtain the predicted infrared image.

[0017] In one embodiment, the transformer-based decoder is trained to obtain the transform decoding model through training data, including:

[0018] The preset area is calibrated after the visible light imaging and infrared imaging are fused;

[0019] Inputting the preset task information into the second language model to generate image editing prompt words;

[0020] Inputting the calibrated fused image and the image editing prompt words into the multimodal large model to obtain an edited image;

[0021] Training data for training the transformer decoder is obtained based on the edited image and visible light imaging.

[0022] In one embodiment, the fusion of visible light imaging and infrared imaging to calibrate a preset area includes:

[0023] Fusing visible light imaging with infrared imaging to obtain a fused image;

[0024] Based on infrared imaging, key areas with brightness greater than a preset value are calibrated;

[0025] The key area of ​​infrared imaging is calibrated on the fused image to obtain a fused image of the calibrated preset area.

[0026] In one embodiment, the step of obtaining training data for training the transformer encoder based on the edited image and visible light imaging includes:

[0027] Obtaining an infrared layer image based on the difference between the edited image and the visible light imaging;

[0028] Inputting the infrared layer image into a diffusion model to obtain training data;

[0029] The transformer decoder is trained according to the training data to obtain the transformer decoding model.

[0030] The present invention also provides a visible light imaging and infrared imaging fusion device, the device comprising:

[0031] An acquisition module, used for acquiring visible light imaging and infrared imaging;

[0032] An extraction module is used to extract semantic information from visible light imaging, input it into the first language model together with preset task information, and obtain task prompt words;

[0033] A determination module, used to determine the visible light imaging latent variable and the infrared imaging latent variable based on the visible light imaging and the infrared imaging respectively;

[0034] An input module is used to input the conditional latent variables synthesized by the task prompt words, the visible light imaging latent variables and the infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image, wherein the transform decoding model is obtained by training the training data based on the transformer decoder;

[0035] The fusion module is used to fuse the current frame visible light image with the predicted infrared image to obtain a target fused image.

[0036] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the visible light imaging and infrared imaging fusion method as described in any one of the above is implemented.

[0037] The present invention also provides a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the visible light imaging and infrared imaging fusion method as described in any one of the above is implemented.

[0038] The above-mentioned visible light imaging and infrared imaging fusion method, device, electronic device and storage medium, the large language model can realize the understanding of the world, which enables the intelligent model to understand the meaning of images or language in the world. At the same time, the predicted infrared image is enhanced by the preset task information instruction, which is different from the original acquired infrared imaging. The original acquired infrared imaging is only used as a generation condition, that is, the predicted infrared image is completely generative, and its purpose is to generate semantic information on the visible light imaging. Its semantic indicativeness is better than the original infrared imaging. The target fused image obtained by the fusion of the predicted infrared image and the visible light image gives full play to the advantages of the two imaging technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flow chart of a visible light imaging and infrared imaging fusion method according to an embodiment of the present invention;

[0041] Figure 2 A flow chart of a visible light imaging and infrared imaging fusion method according to another embodiment of the present invention;

[0042] Figure 3 A schematic diagram of a visible light imaging and infrared imaging fusion method according to an embodiment of the present invention;

[0043] Figure 4 A flow chart of a method for fusing visible light imaging and infrared imaging according to another embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of a visible light imaging and infrared imaging fusion device according to an embodiment of the present invention;

[0045] Figure 6 The figure is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Infrared imaging is widely used in industrial inspection, military applications and other fields. It has completely different advantages over visible light imaging. In view of the differences between infrared imaging and visible light imaging, each has certain advantages, such as high image resolution, natural color, easy human eye interpretation, low cost, etc. Infrared imaging can capture images in the dark, can detect temperature differences, can be used for thermal analysis, can be used for non-contact temperature measurement, etc. Therefore, in the field of infrared imaging, a new direction is to fuse the two imaging methods to obtain the advantages of both technologies. Some fusion methods have been proposed, but these fusion methods are still fusion at the image level, or pixel level.

[0048] The wavelength of infrared imaging spectrum is relatively long, and the color and edge of the imaging are far inferior to those of visible light. Therefore, the significance of infrared imaging is actually more reflected in the semantic level rather than the pixel level. For example, in the military field, infrared imaging is used to find hidden targets or identify targets in smoke. Although image or pixel-level fusion can obtain the advantages of both imaging technologies, it cannot maximize the potential of the two imaging technologies due to the lack of understanding of the "world" model.

[0049] The development of large language models (LLMs) has brought new breakthroughs to artificial intelligence technology. Through the Transformer architecture, large language models can achieve understanding of the world, which enables intelligent models to understand the meaning of images or languages ​​around the world. Therefore, the present invention provides a visible light imaging and infrared imaging fusion method, device, electronic device and storage medium.

[0050] Combine the following Figure 1-Figure 6 The invention describes the visible light imaging and infrared imaging fusion method, device, electronic device and storage medium.

[0051] like Figure 1 As shown, in one embodiment, a method for fusing visible light imaging and infrared imaging includes the following steps:

[0052] Step S110, obtaining visible light imaging and infrared imaging. A video stream and infrared imaging are obtained by a visible light sensor and an infrared sensor respectively, and the video stream includes multiple frames of visible light imaging.

[0053] Step S120 , extracting semantic information from visible light imaging, and inputting it into a first language model together with preset task information, and obtaining task prompt words.

[0054] For details, see Figure 3, input the visible light video into the video to text conversion model (Video to Text Model) to obtain the semantic information in the video. The video to text conversion model can use an open source model, such as a VideoTransformer model. The model will semantically describe the video of the visible light image to obtain semantic information that can be recognized in the visible light imaging. The semantic information is input into a dedicated first large language model (LLM), and the first large language model receives preset task information and generates a task prompt word (token). The task prompt word is embedded in the word to generate a task prompt latent variable, and the task prompt latent variable is a one-dimensional latent variable. The first large language model can be trained based on dedicated data using the open source LLAMA3 model.

[0055] Step S130 , determining the visible light imaging latent variable and the infrared imaging latent variable based on the visible light imaging and the infrared imaging respectively.

[0056] Infrared imaging and visible light imaging are respectively input into an image encoder, that is, the encoder of the autoencoder decoder. After passing through the encoder, visible light imaging latent variables and infrared imaging latent variables are generated respectively, both of which are one-dimensional latent variables.

[0057] Step S140, inputting the conditional latent variables synthesized by the task prompt word, the visible light imaging latent variables and the infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image, wherein the transform decoding model is obtained by training the training data based on the transformer decoder.

[0058] The visible light imaging latent variables, infrared imaging latent variables and task prompt words are combined to generate conditional latent variables. The conditional latent variables are input into the transform decoding model, and the transform decoding model generates the predicted infrared image. Among them, the transform decoding model is trained by the transformer decoder (TransformerDecoder). The decoder can use the transformer's multi-head attention mechanism to extract and analyze the long-range dependencies of the conditional latent variables.

[0059] Step S150, fusing the current frame visible light image with the predicted infrared image to obtain a target fused image.

[0060] In the visible light imaging and infrared imaging fusion method of this embodiment, the large language model can realize the understanding of the world, which enables the intelligent model to understand the meaning of images or languages ​​in the world. At the same time, the predicted infrared image is enhanced by the preset task information instruction, which is different from the original acquired infrared imaging. The original acquired infrared imaging is only used as a generation condition, that is, the predicted infrared image is completely generative, and its purpose is to generate semantic information on the visible light imaging. Its semantic indicativeness is better than that of the original infrared imaging. The target fused image obtained by the fusion of the predicted infrared image and the visible light image gives full play to the advantages of the two imaging technologies.

[0061] like Figure 2 As shown, in one embodiment, the conditional latent variables synthesized by the task prompt word, the visible light imaging latent variable and the infrared imaging latent variable are input into the transform decoding model to obtain the predicted infrared image, including the following steps:

[0062] Step S142, input the conditional latent variables into the transformation decoding model to obtain the output result.

[0063] Among them, the transform decoding model is trained based on the transformer decoder. The decoder can use the transformer's multi-head attention mechanism to extract and analyze the long-range dependencies of conditional latent variables.

[0064] Step S144, inputting the output result into the image decoder to obtain the predicted infrared image.

[0065] Specifically, the input of Transformer Decoder will be input into an image decoder (auto decoder), and finally a predicted infrared image will be generated.

[0066] The following is a real-life operation scenario. The visible light sensor installed on the autonomous military robot collects video data, and the video to text model generates a description of the environment: "The current environment is a street, and there are multiple pedestrians in front." Further, the task information input by the operator is received, for example: "Search for our own wounded, and it is known that there are 10 of our own personnel in the area." Further, the first language model of the dedicated task receives the environment description information and the task information, and generates at least one task prompt token, for example, "Search for potential personnel in the infrared image, and after identifying the personnel, make a judgment on the enemy and our side, and identify our own personnel, and further, determine whether the person is injured." After word embedding, the token generates a one-dimensional task prompt latent variable.

[0067] On the other side, the infrared sensor on the military robot collects infrared imaging, which is input into the image encoder together with the visible light imaging in the video stream collected by the visible light sensor to obtain one-dimensional visible light imaging latent variables and infrared imaging latent variables, which are then combined with the task prompt token into conditional latent variables.

[0068] See also Figure 3 , after the conditional latent variables are input into the Transformer Decoder, they are sent to the image decoder to generate a predicted infrared image. Note that the predicted infrared image at this time is an image enhanced by instructions, which is different from the original acquired infrared imaging. The original acquired infrared imaging is only used as a generation condition, that is, the predicted infrared image is completely generative, and its purpose is to generate semantic information on visible light imaging, such as generating a human highlight image in a region of visible light imaging. For example, in this example, the generated heat map can be a creative image of a wounded person behind an obstruction. This is because the infrared data itself performs poorly in naked eye recognition. Although the generated image has a certain degree of randomness, its semantic indicativeness is better than the original infrared imaging.

[0069] The transform decoding model can be trained using a diffusion model (transformer decoder). However, certain modifications are required to obtain training data. In the traditional diffusion model training process, the original image data is subjected to multiple steps of diffusion (that is, Gaussian noise is gradually added) to obtain training data. The goal of the training is to train the model end-to-end so that the objective function (generally the noise prediction between two diffusion steps) is reduced. However, in the present invention, the fusion of direct infrared imaging and visible light imaging is not the expected result, because the result of directly superimposing visible light is not optimal from a semantic level.

[0070] like Figure 4 As shown, in one embodiment, based on the transformer decoder, the transform decoding model is obtained by training with training data, comprising the following steps:

[0071] Step S310, calibrating a preset area after fusing the visible light imaging and the infrared imaging.

[0072] Specifically, the visible light imaging is fused with the infrared imaging to obtain a fused image. The collected visible light imaging is directly fused with the infrared imaging to obtain a fused image. Based on the infrared imaging, the key areas with brightness greater than the preset value are calibrated, wherein the key areas of the infrared imaging are obtained, and the area can be extracted by using the preset value judgment method to extract the area where the visible light brightness is greater than the preset value. The key areas of the infrared imaging are calibrated on the fused image to obtain a fused image of the calibrated preset area.

[0073] Step S320: input the preset task information into the second language model to generate image editing prompt words.

[0074] The second largest language model is another language model other than the first largest language model. Image editing prompt words are generated based on preset task information and the second largest language model. Image editing prompt words can also be generated using artificial and task information.

[0075] Step S330, input the calibrated fused image and the image editing prompt words into the multimodal large model to obtain the edited image.

[0076] Optionally, the calibrated image is input into a multimodal large language model, such as GPT-4o, and at the same time, the image editing prompt words are input into the multimodal large language model to obtain an image output. The image has good indicative information as an edited image. Note that the editing prompt words need to limit the editing to only the key area of ​​the calibrated infrared imaging, so the modified image generates a fused image corrected in the important area.

[0077] Step S340, obtaining training data for training a transformer decoder based on the edited image and visible light imaging.

[0078] Specifically, an infrared layer image is obtained based on the difference between the edited image and the visible light imaging, and the infrared layer image is input into a diffusion model to obtain training data. The transformer decoder is trained based on the training data to obtain a transform decoding model. The transform decoding model predicts an infrared imaging layer on top of the visible light imaging, which can generate a new infrared layer with semantic information based on the information about the world contained in the visible light, the task information given by the prompt word, and the collected infrared imaging data.

[0079] The visible light imaging and infrared imaging fusion device provided by the present invention is described below. The visible light imaging and infrared imaging fusion device described below and the visible light imaging and infrared imaging fusion method described above can be referenced to each other.

[0080] like Figure 5 As shown, in one embodiment, a visible light imaging and infrared imaging fusion device includes an acquisition module 410 , an extraction module 420 , a determination module 430 , an input module 440 and a fusion module 450 .

[0081] The acquisition module 410 is used to acquire visible light imaging and infrared imaging.

[0082] The extraction module 420 is used to extract semantic information from visible light imaging, and input it into the first language model together with preset task information to obtain task prompt words.

[0083] The determination module 430 is used to determine the visible light imaging latent variables and the infrared imaging latent variables based on the visible light imaging and the infrared imaging respectively.

[0084] The input module 440 is used to input the conditional latent variables synthesized by the task prompt words, visible light imaging latent variables and infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image, wherein the transform decoding model is obtained by training the training data based on the transformer decoder.

[0085] The fusion module 450 is used to fuse the current frame visible light image with the predicted infrared image to obtain a target fused image.

[0086] Figure 6 An example of a physical structure diagram of an electronic device is shown. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a visible light imaging and infrared imaging fusion method is implemented, and the method includes:

[0087] Obtain visible light imaging and infrared imaging;

[0088] Extract semantic information from visible light imaging, input it into the first language model together with preset task information, and obtain task prompt words;

[0089] Based on visible light imaging and infrared imaging, the visible light imaging latent variables and infrared imaging latent variables are determined respectively;

[0090] Inputting the conditional latent variables synthesized by the task prompt word, the visible light imaging latent variables and the infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image, wherein the transform decoding model is obtained by training the training data based on the transformer decoder;

[0091] The current frame visible light image is fused with the predicted infrared image to obtain a target fused image.

[0092] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the electronic device to which the scheme of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0093] On the other hand, the present invention further provides a computer storage medium storing a computer program, which implements a visible light imaging and infrared imaging fusion method when the computer program is executed by a processor, the method comprising:

[0094] Obtain visible light imaging and infrared imaging;

[0095] Extract semantic information from visible light imaging, input it into the first language model together with preset task information, and obtain task prompt words;

[0096] Based on visible light imaging and infrared imaging, the visible light imaging latent variables and infrared imaging latent variables are determined respectively;

[0097] Inputting the conditional latent variables synthesized by the task prompt word, the visible light imaging latent variables and the infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image, wherein the transform decoding model is obtained by training the training data based on the transformer decoder;

[0098] The current frame visible light image is fused with the predicted infrared image to obtain a target fused image.

[0099] In another aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, a method for fusing visible light imaging and infrared imaging is implemented, the method comprising:

[0100] Obtain visible light imaging and infrared imaging;

[0101] Extract semantic information from visible light imaging, input it into the first language model together with preset task information, and obtain task prompt words;

[0102] Based on visible light imaging and infrared imaging, the visible light imaging latent variables and infrared imaging latent variables are determined respectively;

[0103] Inputting the conditional latent variables synthesized by the task prompt word, the visible light imaging latent variables and the infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image, wherein the transform decoding model is obtained by training the training data based on the transformer decoder;

[0104] The current frame visible light image is fused with the predicted infrared image to obtain a target fused image.

[0105] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0106] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0107] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A method for fusing visible light imaging and infrared imaging, characterized in that: The method comprises: Obtain visible light imaging and infrared imaging; Extract semantic information from visible light imaging, input it into the first language model together with preset task information, and obtain task prompt words; Based on visible light imaging and infrared imaging, the visible light imaging latent variables and infrared imaging latent variables are determined respectively; Inputting the conditional latent variables synthesized by the task prompt word, the visible light imaging latent variables and the infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image, wherein the transform decoding model is obtained by training the training data based on the transformer decoder; The current frame visible light image is fused with the predicted infrared image to obtain a target fused image.

2. The visible light imaging and infrared imaging fusion method according to claim 1, characterized in that: The extracting of semantic information in visible light imaging includes: The visible light video is input into the video-to-text conversion model to obtain the semantic information in the video.

3. The visible light imaging and infrared imaging fusion method according to claim 2, characterized in that: The step of obtaining the task prompt word includes: The task prompt words are embedded into words to generate task prompt latent variables.

4. The visible light imaging and infrared imaging fusion method according to claim 1, characterized in that: The step of inputting the conditional latent variables synthesized from the task prompt words, the visible light imaging latent variables and the infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image comprises: Inputting the conditional latent variables into the transform decoding model to obtain an output result; The output result is input into an image decoder to obtain the predicted infrared image.

5. The visible light imaging and infrared imaging fusion method according to claim 1, characterized in that: The transformer-based decoder, obtaining the transform decoding model through training data training, comprises: The preset area is calibrated after the visible light imaging and infrared imaging are fused; Inputting the preset task information into the second language model to generate image editing prompt words; Inputting the calibrated fused image and the image editing prompt words into the multimodal large model to obtain an edited image; Training data for training the transformer decoder is obtained based on the edited image and visible light imaging.

6. The visible light imaging and infrared imaging fusion method according to claim 5, characterized in that: The method of calibrating a preset area after fusing the visible light imaging and the infrared imaging comprises: Fusing visible light imaging with infrared imaging to obtain a fused image; Based on infrared imaging, key areas with brightness greater than a preset value are calibrated; The key area of ​​infrared imaging is calibrated on the fused image to obtain a fused image of the calibrated preset area.

7. The visible light imaging and infrared imaging fusion method according to claim 6, characterized in that: The step of obtaining training data for training the converter encoder based on the edited image and visible light imaging comprises: Obtaining an infrared layer image based on the difference between the edited image and the visible light imaging; Inputting the infrared layer image into a diffusion model to obtain training data; The transformer decoder is trained according to the training data to obtain the transformer decoding model.

8. A visible light imaging and infrared imaging fusion device, characterized in that: The device comprises: An acquisition module, used for acquiring visible light imaging and infrared imaging; An extraction module is used to extract semantic information from visible light imaging, input it into the first language model together with preset task information, and obtain task prompt words; A determination module, used to determine the visible light imaging latent variable and the infrared imaging latent variable based on the visible light imaging and the infrared imaging respectively; An input module is used to input the conditional latent variables synthesized by the task prompt words, the visible light imaging latent variables and the infrared imaging latent variables into the transform decoding model to obtain the predicted infrared image, wherein the transform decoding model is obtained by training the training data based on the transformer decoder; The fusion module is used to fuse the current frame visible light image with the predicted infrared image to obtain a target fused image.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the visible light imaging and infrared imaging fusion method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the visible light imaging and infrared imaging fusion method described in any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Isovariant consistency image fusion method and system based on task driving

    CN120339092A

  • Task-driven equivariant consistency image fusion method and system

    CN120339092B