A method, diffusion model and system for generating infrared-visible light data in pairs

Abstract

The present invention provides a method, diffusion model and system for generating infrared-visible light data in pairs, the method comprising: obtaining a plurality of pairs of infrared features and visible light features; obtaining a plurality of fusion features; obtaining a plurality of latent space features; obtaining a plurality of pairs of infrared-visible light image data after reconstruction; obtaining features after adding noise; obtaining the loss function of a denoising sub-model of a diffusion model; obtaining a trained diffusion model; performing feature fusion on a visible light image of a target to be measured and a text instruction and inputting the result into a denoising sub-model in a trained diffusion model, performing inference denoising on the denoising sub-model in the trained diffusion model and outputting denoised latent space features, and decoding the denoised latent space features to obtain paired infrared-visible light image data corresponding to the target to be measured. The present invention can simultaneously generate paired infrared-visible light data and maintain a high degree of consistency in information between the two.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal image generation, and in particular to a method, a diffusion model and a system for generating infrared-visible light data in pairs. Background Art

[0002] With the rapid development of artificial intelligence technology, the demand for high-quality data is growing. However, practical applications often face challenges such as high data collection costs and strict privacy protection requirements. Traditional methods are difficult to effectively deal with these problems. In recent years, inspired by the theory of diffusion processes in physics, a new type of diffusion model based on probability statistics has emerged, showing excellent data generation potential and providing innovative solutions to the above problems. The action process of the diffusion model is divided into forward and reverse. The forward process first defines a forward process of gradual noise addition. Through a carefully designed noise addition strategy, the original data is gradually transformed into a distribution state close to Gaussian noise. This process not only simulates the natural law of information diffusion in nature, but also cleverly hides the sensitive information of the original data and enhances the privacy of data processing. The reverse process constructs a powerful reverse denoising network. The network learns to reversely extract and restore the key features of the original data from the Gaussian noise distribution through training, and finally generates samples that are highly consistent with the original data distribution. This process makes full use of the powerful fitting ability of deep neural networks to achieve efficient reconstruction of complex data distribution.

[0003] At present, most of the research content of diffusion models is still focused on the generation of RGB images in the visible light band. However, for computer vision tasks, the model and system's perception of the scene can also come from information of various other modalities, including temperature information, image depth information, and texture perception information. For example, at night or in low-light scenes, RGB images contain less information, while infrared images can detect objects with temperatures above absolute zero by capturing thermal radiation with wavelengths between 0.75-15 microns. Therefore, for a scene, the more modal data that can be obtained, the more robust the model is, and the more stable the effect in actual visual tasks. However, there are currently very few similar paired visible light-infrared data, mainly for the following reasons: First, the equipment used to collect infrared images is usually expensive, and the procurement cost of ordinary research teams is high, and there is a certain research threshold. Secondly, the collection of paired data also involves the problem of consistency, which requires that the corresponding visible light data be obtained at the same time as the infrared data is collected, and after manual proofreading, the paired data of acceptable quality can be obtained. In addition, the performance of existing visible light to infrared models is also relatively limited, and there are certain bottlenecks in the conversion quality. Summary of the invention

[0004] The present invention provides a method, a diffusion model and a system for generating infrared-visible light data in pairs, which can simultaneously generate paired infrared-visible light data and maintain a high degree of consistency of information between the two.

[0005] According to one aspect of the present invention, there is provided a method for generating infrared-visible light data in pairs, the method comprising:

[0006] Convolution feature extraction is performed on a plurality of pairs of infrared-visible light image data to obtain a plurality of pairs of infrared features and visible light features;

[0007] The infrared features and visible light features in each pair are added together to obtain several fused features;

[0008] Inputting several fusion features into the variational autoencoder for encoding respectively, and obtaining several latent space features, wherein each latent space feature includes data information of infrared modality and data information of visible light modality;

[0009] Inputting the data information of the infrared modality in the plurality of latent space features into the infrared decoder respectively, and inputting the data information of the visible light modality in the plurality of latent space features into the visible light decoder respectively, to obtain the reconstructed plurality of pairs of infrared-visible light image data;

[0010] In the case that the reconstructed paired infrared-visible light image data can restore the original paired infrared-visible light image data, any latent space feature is selected, and forward noise is performed on the selected latent space feature to obtain a noise-completed feature;

[0011] Obtaining the loss function of the denoising sub-model of the diffusion model based on the visible light image and text instructions used for training;

[0012] Based on the loss function of the denoising sub-model of the diffusion model and the features after adding noise, the denoising sub-model of the diffusion model is trained to obtain a trained diffusion model;

[0013] The visible light image and text instructions of the target to be measured are feature fused and input into the denoising sub-model in the trained diffusion model. The denoising sub-model in the trained diffusion model performs inference denoising and outputs the denoised latent space features. The denoised latent space features are decoded to obtain the paired infrared-visible light image data corresponding to the target to be measured.

[0014] Preferably, the selected latent space features are forward-noised by the following formula:

[0015]

[0016] In the formula, z trepresents the latent space feature with t levels of noise, t = 1, 2, 3, ..., T, T represents the total number of noise steps, α t represents the noise parameter of the tth step, z0 represents the pure latent space feature, and ε represents the randomly sampled Gaussian noise.

[0017] Preferably, the loss function of the denoising sub-model of the diffusion model is obtained by the following formula:

[0018]

[0019] Where, L θ represents the loss function of the denoising sub-model of the diffusion model, ε θ (z t ,t,c) represents the denoising sub-model of the diffusion model, and c represents the feature fusion result of the visible light image and text instruction of the target to be measured.

[0020] Preferably, the denoising sub-model of the diffusion model is trained for denoising by the following formula:

[0021]

[0022] In the formula, z t-1 Represents the latent space features with t-1 levels of noise.

[0023] Preferably, the method further comprises: using a visible light-infrared coaxial image acquisition device to acquire paired infrared-visible light image data.

[0024] Preferably, the text instruction comprises rotating or translating the visible light image.

[0025] According to another aspect of the present invention, a diffusion model for generating infrared-visible light data in pairs is provided, and the diffusion model is obtained by training using any of the above-mentioned methods.

[0026] According to yet another aspect of the present invention, there is provided a system for generating infrared-visible light data in pairs, the system comprising an image acquisition device, a data processing module and a result output device;

[0027] The image acquisition device is used to acquire a visible light image of the target to be measured;

[0028] The data processing module is deployed with the trained diffusion model, and is used to perform feature fusion on the visible light image and text instructions of the target to be measured; to perform inference denoising based on the feature fusion result of the visible light image and text instructions of the target to be measured, and obtain the denoised latent space features; and to decode the denoised latent space features to obtain paired infrared-visible light image data corresponding to the target to be measured;

[0029] The result output device is used to output the generated paired infrared-visible light image data.

[0030] The technical solution of the present invention combines the powerful capabilities of the image feature encoder ResNet and the CLIP large model to accurately capture the deep information of scene features and text descriptions respectively; using the diffusion model, these two types of features are cleverly integrated in the latent space, and the pure Gaussian noise is gradually converted into information-rich latent space features through the sophisticated operations of noise addition and denoising; in addition, with the help of decoders designed specifically for infrared and visible light, these latent space features are successfully decoded into paired modal data. The present invention can be used to generate infrared and visible light images in pairs, realizing the seamless generation of bimodal data, and can also guide the generation process according to the text content provided by the user, greatly enriching the diversity of the generated results. The present invention can provide more data for military, night driving and other fields, and promote the development of related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A flow chart of a method for generating infrared-visible light data in pairs provided according to an embodiment of the present invention is shown;

[0033] Figure 2 A schematic diagram of a process of reconstructing data provided according to an embodiment of the present invention is shown;

[0034] Figure 3 A schematic diagram of a denoising sub-model reasoning and denoising process of a diffusion model is shown according to an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of the structure of a system for generating infrared-visible light data in pairs provided according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0036] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0039] like Figure 1-Figure 3 As shown, the present invention provides a method for generating infrared-visible light data in pairs, the method comprising:

[0040] Convolution feature extraction is performed on a plurality of pairs of infrared-visible light image data to obtain a plurality of pairs of infrared features and visible light features;

[0041] The infrared features and visible light features in each pair are added together to obtain several fused features;

[0042] Inputting several fusion features into the variational autoencoder for encoding respectively, and obtaining several latent space features, wherein each latent space feature includes data information of infrared modality and data information of visible light modality;

[0043] Inputting the data information of the infrared modality in the plurality of latent space features into the infrared decoder respectively, and inputting the data information of the visible light modality in the plurality of latent space features into the visible light decoder respectively, to obtain the reconstructed plurality of pairs of infrared-visible light image data;

[0044] In the case that the reconstructed paired infrared-visible light image data can restore the original paired infrared-visible light image data, any latent space feature is selected, and forward noise is performed on the selected latent space feature to obtain a noise-completed feature;

[0045] Obtaining the loss function of the denoising sub-model of the diffusion model based on the visible light image and text instructions used for training;

[0046] Based on the loss function of the denoising sub-model of the diffusion model and the features after adding noise, the denoising sub-model of the diffusion model is trained to obtain a trained diffusion model;

[0047] The visible light image and text instructions of the target to be measured are feature fused and input into the denoising sub-model in the trained diffusion model. The denoising sub-model in the trained diffusion model performs inference denoising and outputs the denoised latent space features. The denoised latent space features are decoded to obtain the paired infrared-visible light image data corresponding to the target to be measured.

[0048] The present invention combines the powerful capabilities of the image feature encoder ResNet and the CLIP large model to accurately capture the deep information of scene features and text descriptions respectively; using the diffusion model, these two types of features are cleverly integrated in the latent space, and through the delicate operations of noise addition and denoising, pure Gaussian noise is gradually converted into information-rich latent space features; in addition, with the help of decoders designed specifically for infrared and visible light, these latent space features are successfully decoded into paired modal data. The present invention can be used to generate infrared and visible light images in pairs, realizing the seamless generation of bimodal data, and can also guide the generation process according to the text content provided by the user, greatly enriching the diversity of the generated results. The present invention can provide more data for military, night driving and other fields, and promote the development of related fields.

[0049] The two core parts of the diffusion model are the forward denoising and reverse denoising processes. The diffusion process in this method is performed in the latent space after the variational autoencoder is encoded. Assuming that the pure latent space feature in the latent space is represented as z0, the latent space feature of the tth step can be directly obtained based on the formula of the diffusion model:

[0050]

[0051] In the formula, z trepresents the latent space feature with t levels of noise, t = 1, 2, 3, ..., T, T represents the total number of noise steps, α t represents the noise parameter of the tth step, z0 represents the pure latent space feature, and ε represents the randomly sampled Gaussian noise. According to this formula, when the total number of noise addition steps T is large enough, the final noise-added feature z T will approach a standard Gaussian noise distribution.

[0052] At this point, the loss function of the denoising sub-model of the diffusion model can be expressed as:

[0053]

[0054] Where, L θ represents the loss function of the denoising sub-model of the diffusion model, ε θ (z t ,t,c) represents the denoising sub-model of the diffusion model, c represents the feature fusion result of the visible light image and text instruction of the target to be measured, and θ represents the parameters of the denoising sub-model. Among them, c is used as the input condition of the denoising sub-model of the diffusion model.

[0055] According to an embodiment of the present invention, the denoising sub-model of the diffusion model is trained for denoising by the following formula:

[0056]

[0057] In the formula, z t-1 Represents the latent space features with t-1 levels of noise.

[0058] According to an embodiment of the present invention, the method further comprises: using a visible light-infrared coaxial image acquisition device to acquire paired infrared-visible light image data.

[0059] According to an embodiment of the present invention, the text instruction includes rotating or translating the visible light image.

[0060] According to an embodiment of the present invention, Figure 3 As shown, a schematic diagram of the denoising process of the denoising sub-model inference of the diffusion model is shown.

[0061] In this embodiment, the text instruction is set to turn left so that the visible light image turns left. During the inference process, the visible light image of the target to be measured and the text instruction (turn left) are feature fused and input into the denoising sub-model in the trained diffusion model. The denoising sub-model in the trained diffusion model performs inference denoising and outputs the denoised latent space features. The denoised latent space features are decoded to obtain the paired infrared-visible light image data corresponding to the target to be measured.

[0062] The present invention also provides a diffusion model for generating infrared-visible light data in pairs, and the diffusion model is obtained by training using any of the above methods.

[0063] like Figure 4 As shown, the present invention also provides a system for generating infrared-visible light data in pairs, the system comprising an image acquisition device, a data processing module and a result output device;

[0064] The image acquisition device is used to acquire a visible light image of the target to be measured;

[0065] The data processing module is deployed with the trained diffusion model, and is used to perform feature fusion on the visible light image and text instructions of the target to be measured; to perform inference denoising based on the feature fusion result of the visible light image and text instructions of the target to be measured, and obtain the denoised latent space features; and to decode the denoised latent space features to obtain paired infrared-visible light image data corresponding to the target to be measured;

[0066] The result output device is used to output the generated paired infrared-visible light image data.

[0067] In this embodiment, the image acquisition device may adopt a visible light sensor, the data processing module includes hardware such as a deep learning chip, an FPGA, a graphics card (GPU), and the result output device includes an image display, etc.

[0068] In summary, the present invention provides a method, diffusion model and system for generating infrared-visible light data in pairs. The present invention combines the powerful capabilities of the image feature encoder ResNet and the CLIP large model to accurately capture the deep information of scene features and text descriptions respectively; using the diffusion model, these two types of features are cleverly integrated in the latent space, and the pure Gaussian noise is gradually converted into information-rich latent space features through the fine operations of noise addition and denoising; in addition, with the help of decoders designed specifically for infrared and visible light, these latent space features are successfully decoded into paired modal data. The present invention can be used to generate infrared and visible light images in pairs, realize the seamless generation of dual-modal data, and can also guide the generation process according to the text content provided by the user, greatly enriching the diversity of the generated results. The present invention can provide more data for military, night driving and other fields, and promote the development of related fields.

[0069] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0070] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating infrared-visible light data in pairs, characterized in that: The method comprises: Convolution feature extraction is performed on a plurality of pairs of infrared-visible light image data to obtain a plurality of pairs of infrared features and visible light features; The infrared features and visible light features in each pair are added together to obtain several fused features; Inputting several fusion features into the variational autoencoder for encoding respectively, and obtaining several latent space features, wherein each latent space feature includes data information of infrared modality and data information of visible light modality; Inputting the data information of the infrared modality in the plurality of latent space features into the infrared decoder respectively, and inputting the data information of the visible light modality in the plurality of latent space features into the visible light decoder respectively, to obtain the reconstructed plurality of pairs of infrared-visible light image data; In the case that the reconstructed paired infrared-visible light image data can restore the original paired infrared-visible light image data, any latent space feature is selected, and forward noise is performed on the selected latent space feature to obtain a noise-completed feature; Obtaining the loss function of the denoising sub-model of the diffusion model based on the visible light image and text instructions used for training; Based on the loss function of the denoising sub-model of the diffusion model and the features after adding noise, the denoising sub-model of the diffusion model is trained to obtain a trained diffusion model; The visible light image and text instructions of the target to be measured are feature fused and input into the denoising sub-model in the trained diffusion model. The denoising sub-model in the trained diffusion model performs inference denoising and outputs the denoised latent space features. The denoised latent space features are decoded to obtain the paired infrared-visible light image data corresponding to the target to be measured.

2. The method according to claim 1, characterized in that The selected latent space features are forward-noised using the following formula: In the formula, z t represents the latent space feature with t levels of noise, t = 1, 2, 3, ..., T, T represents the total number of noise steps, α t represents the noise parameter of the tth step, z0 represents the pure latent space feature, and ε represents the randomly sampled Gaussian noise.

3. The method according to claim 1, characterized in that The loss function of the denoising sub-model of the diffusion model is obtained by the following formula: Where, L θ represents the loss function of the denoising sub-model of the diffusion model, ε θ (z t ,t,c) represents the denoising sub-model of the diffusion model, c represents the feature fusion result of the visible light image and text instruction of the target to be measured, z t represents the latent space features with t levels of noise, t = 1, 2, 3, ..., T, T represents the total number of noise adding steps, and ε represents randomly sampled Gaussian noise.

4. The method according to claim 1, characterized in that The denoising sub-model of the diffusion model is trained by the following formula: In the formula, z t-1 represents the latent space feature with t-1 level noise, α t represents the noise parameter at step t, ε θ (z t ,t,c) represents the denoising sub-model of the diffusion model, z t represents the latent space feature with t levels of noise, t = 1, 2, 3, ..., T, T represents the total number of noise addition steps, c represents the feature fusion result of the visible light image and text instruction of the target to be measured, and ε represents randomly sampled Gaussian noise.

5. The method according to claim 1, characterized in that The method further comprises: using a visible light-infrared coaxial image acquisition device to acquire paired infrared-visible light image data.

6. The method according to claim 1, characterized in that The text instruction includes rotating or translating the visible light image.

7. A diffusion model for generating infrared-visible light data in pairs, characterized in that The diffusion model is obtained by training using any one of the methods described in claims 1-6.

8. A system for generating infrared-visible light data in pairs, characterized in that: The system includes an image acquisition device, a data processing module and a result output device; The image acquisition device is used to acquire a visible light image of the target to be measured; The diffusion model of claim 7 is deployed on the data processing module, and the data processing module is used to perform feature fusion on the visible light image and text instructions of the target to be measured; and to perform inference denoising based on the feature fusion result of the visible light image and text instructions of the target to be measured to obtain the denoised latent space features; It is also used to decode the denoised latent space features to obtain paired infrared-visible light image data corresponding to the target to be measured; The result output device is used to output the generated paired infrared-visible light image data.

Images