A method, diffusion model and system for generating infrared images based on target frames

Abstract

The present invention provides a method, a diffusion model and a system for generating an infrared image based on a target frame. The method comprises: obtaining a target frame of an object; obtaining a category coding feature corresponding to a target frame category; obtaining a position coding feature corresponding to a target frame position; obtaining a target frame coding feature; performing forward denoising on latent space features of the infrared image to obtain a feature after denoising; obtaining a loss function of a denoising sub-model of a diffusion model based on the target frame coding feature; performing denoising training on the denoising sub-model of the diffusion model based on the loss function of the denoising sub-model of the diffusion model and the feature after denoising to obtain a trained diffusion model; the trained diffusion model generates a corresponding infrared image based on the target frame coding feature corresponding to the target frame of the object to be tested. The present invention can generate an infrared image close to reality and can accurately generate a corresponding object within a given target frame range.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared image generation, and in particular to a method, a diffusion model and a system for generating an infrared image based on a target frame. Background Art

[0002] In recent years, the diffusion model has attracted much attention as a data generation model based on probability theory. The model was originally derived from the theory of diffusion processes in the field of physics. In the field of machine learning, the diffusion model is widely used in data generation tasks, especially the generation of images, videos, and audio. The diffusion model first drives a process of transformation from the real data distribution to the Gaussian noise distribution by gradually introducing noise, that is, forward diffusion. Subsequently, the model learns the inverse diffusion process through a neural network, that is, gradually denoising from pure noise through reverse steps, and finally generates sample data close to the original data distribution. This method has achieved important results in the fields of image generation and image restoration, and has opened up new research directions for data reconstruction and generation.

[0003] Most of the current diffusion models are used to generate RGB images in the visible light band. However, human perception of the world often comes from multiple perceptual dimensions, such as temperature information. At the same time, in night scenes, human visual ability will be greatly affected, and infrared images can detect objects with temperatures above absolute zero by capturing thermal radiation with wavelengths between 0.75-15 microns. Therefore, infrared images have been proven to be well suited for target detection networks in night driving systems. However, the amount of data currently used for infrared image target detection is still very small, mainly for the following reasons: First, the equipment used to collect infrared images is usually expensive. Second, it takes a lot of time and labor to accurately label objects in infrared images, especially for small target objects in infrared images. Summary of the invention

[0004] The present invention provides a method, a diffusion model and a system for generating an infrared image based on a target frame, which can generate an infrared image close to reality and can accurately generate a corresponding object within a given target frame.

[0005] According to one aspect of the present invention, a method for generating an infrared image based on a target frame is provided, the method comprising:

[0006] Obtain a target frame of the object, wherein the target frame includes a target frame category and a target frame position, and the target frame category corresponds to the target frame position one by one;

[0007] Input the target frame category of the object into the multimodal model to obtain the category coding feature corresponding to the target frame category; perform Fourier transform on the target frame position of the object to obtain the position coding feature corresponding to the target frame position; add the category coding feature and the position coding feature to obtain the comprehensive coding feature; perform linear mapping on the comprehensive coding feature to obtain the target frame coding feature;

[0008] Perform forward noise addition on the latent space features of the infrared image to obtain the noise-completed features;

[0009] The loss function of the denoising sub-model of the diffusion model is obtained based on the target box encoding features;

[0010] Based on the loss function of the denoising sub-model of the diffusion model and the features after adding noise, the denoising sub-model of the diffusion model is trained to obtain a trained diffusion model;

[0011] The trained diffusion model generates the corresponding infrared image based on the target frame encoding features corresponding to the target frame of the object to be tested.

[0012] Preferably, the target frame encoding feature is obtained by the following formula:

[0013] H=MLP(Fourier({b1,...,b n}); CLIP({l1,...,ln}))

[0014] Where H represents the target box encoding feature, MLP represents linear mapping, Fourier represents Fourier transform, CLIP represents multimodal model, b1,...,b n represents the first, ..., nth target box positions, l1, ..., l n The 1st, ..., nth object box categories.

[0015] Preferably, the latent space features of the infrared image are forward denoised by the following formula:

[0016]

[0017] In the formula, z t represents the latent space feature with t levels of noise, t = 1, 2, 3, ..., T, T represents the total number of noise steps, α t represents the noise parameter of the tth step, z0 represents the pure latent space feature, and ε represents the randomly sampled Gaussian noise.

[0018] Preferably, the loss function of the denoising sub-model of the diffusion model is obtained by the following formula:

[0019]

[0020] Where, Lθ represents the loss function of the denoising sub-model of the diffusion model, ε θ (z t ,t,H) represents the denoising sub-model of the diffusion model.

[0021] Preferably, the denoising sub-model of the diffusion model is trained for denoising by the following formula:

[0022]

[0023] In the formula, z t-1 Represents the latent space features with t-1 levels of noise.

[0024] According to another aspect of the present invention, a diffusion model for generating an infrared image based on a target frame is provided, and the diffusion model is trained by using any of the above methods.

[0025] According to another aspect of the present invention, there is provided a system for generating an infrared image based on a target frame, the system comprising an infrared image acquisition device, a data processing module and a result output device;

[0026] The infrared image acquisition device is used to acquire a target frame of an object;

[0027] The data processing module is equipped with the trained diffusion model, and is used to generate a corresponding infrared image based on the target frame coding feature corresponding to the target frame of the object to be tested;

[0028] The result output device is used to output the generated infrared image.

[0029] By applying the technical solution of the present invention, the given target frame position and category are feature encoded by Fourier transform and multimodal model (CLIP model), thereby obtaining the target frame encoding feature; and the feature is denoised and denoised in latent space by diffusion model, thereby obtaining a trained diffusion model; the trained diffusion model generates a corresponding infrared image based on the target frame of the object to be tested. After the diffusion model training is completed, it performs step-by-step denoising reasoning based on the target frame encoding feature, obtains the corresponding latent space feature from pure Gaussian noise, and thus generates the corresponding infrared image. The present invention can generate infrared images that are close to the real one, and can accurately generate corresponding objects within a given target frame range. The infrared images generated by the present invention can be widely used in training data expansion in the fields of severe weather target detection, night driving, etc., to promote the development of related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 A flow chart of a method for generating an infrared image based on a target frame provided according to an embodiment of the present invention is shown;

[0032] Figure 2 A schematic diagram of a target frame feature encoding process provided according to an embodiment of the present invention is shown;

[0033] Figure 3 A schematic diagram showing a process of denoising training / inference of a denoising sub-model of a diffusion model provided in accordance with an embodiment of the present invention is shown;

[0034] Figure 4 A schematic structural diagram of a system for generating an infrared image based on a target frame provided according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0035] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0037] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0038] like Figure 1 As shown, the present invention provides a method for generating an infrared image based on a target frame, the method comprising:

[0039] Obtain a target frame of the object, wherein the target frame includes a target frame category and a target frame position, and the target frame category corresponds to the target frame position one by one;

[0040] The target frame category of the object is input into the multimodal model to obtain the category coding feature corresponding to the target frame category; the target frame position of the object is Fourier transformed to obtain the position coding feature corresponding to the target frame position; the category coding feature and the position coding feature are added to obtain the comprehensive coding feature; the comprehensive coding feature is linearly mapped to obtain the target frame coding feature, such as Figure 2 As shown;

[0041] Perform forward noise addition on the latent space features of the infrared image to obtain the noise-completed features;

[0042] The loss function of the denoising sub-model of the diffusion model is obtained based on the target box encoding features;

[0043] Based on the loss function of the denoising sub-model of the diffusion model and the features after adding noise, the denoising sub-model of the diffusion model is trained to obtain a trained diffusion model; wherein, through the denoising training, the diffusion model can gradually perceive the position and category of the target frame of the target object;

[0044] The trained diffusion model generates the corresponding infrared image based on the target frame encoding features corresponding to the target frame of the object to be tested.

[0045] By applying the technical solution of the present invention, the given target frame position and category are feature encoded by Fourier transform and multimodal model (CLIP model), thereby obtaining the target frame encoding feature; and the feature is denoised and denoised in latent space by diffusion model, thereby obtaining a trained diffusion model; the trained diffusion model generates a corresponding infrared image based on the target frame of the object to be tested. After the diffusion model training is completed, it performs step-by-step denoising reasoning based on the target frame encoding feature, obtains the corresponding latent space feature from pure Gaussian noise, and thus generates the corresponding infrared image. The present invention can generate infrared images that are close to the real one, and can accurately generate corresponding objects within a given target frame range. The infrared images generated by the present invention can be widely used in training data expansion in the fields of severe weather target detection, night driving, etc., to promote the development of related fields.

[0046] According to an embodiment of the present invention, the target frame encoding feature is obtained by the following formula:

[0047] H=MLP(Fourier({b1,...,b n}); CLIP({l1,...,ln}))

[0048] Where H represents the target box encoding feature, MLP represents linear mapping, Fourier represents Fourier transform, CLIP represents multimodal model, b1,...,b n represents the first, ..., nth target box positions, l1, ..., l n The 1st, ..., nth object box categories.

[0049] The core of the diffusion model is the forward denoising and reverse denoising process. In the stable diffusion model, the denoising and denoising process is performed on the latent space features. Assuming that the pure latent space feature is represented as z0, the latent space feature of the infrared image is forward denoised by the following formula to obtain the latent space feature that has been denoised to the tth step:

[0050]

[0051] In the formula, z t represents the latent space feature with t levels of noise, t = 1, 2, 3, ..., T, T represents the total number of noise steps, α t represents the noise parameter of the tth step, z0 represents the pure latent space feature, and ε represents the randomly sampled Gaussian noise. Therefore, when the total number of noise addition steps T is large enough, the final noise-added feature z T will approach a standard Gaussian noise distribution.

[0052] At this point, the loss function of the denoising sub-model of the diffusion model can be expressed as:

[0053]

[0054] Where, L θ represents the loss function of the denoising sub-model of the diffusion model, ε θ (z t ,t,H) represents the denoising sub-model of the diffusion model. Among them, the target box encoding feature H is used as the input condition of the denoising sub-model of the diffusion model.

[0055] According to an embodiment of the present invention, the denoising sub-model of the diffusion model is trained for denoising by the following formula:

[0056]

[0057] In the formula, z t-1 Represents the latent space features with t-1 level noise, that is, the latent space features denoised to the t-1th step.

[0058] According to an embodiment of the present invention, Figure 3 As shown, a schematic diagram of the process of denoising training / inference of the denoising sub-model of the diffusion model is shown.

[0059] In this embodiment, denoising training is taken as an example for detailed description. During the training process, the target frame encoding feature H is used as the input condition of the denoising sub-model of the diffusion model to denoise the feature z T Denoising is performed until the current loss function reaches the minimum and the training of the current step is completed; then the target frame encoding feature H is used as the input condition of the denoising sub-model of the diffusion model to train the feature z T-1 Denoising is performed until the current loss function reaches the minimum, completing the training of the current step; and so on, until the pure latent space feature z0 is obtained, that is, the training of the denoising sub-model of the diffusion model is completed. The reasoning process is similar. Based on the target frame of the object to be tested, the corresponding target frame encoding feature is obtained, which is used as the input condition for reasoning denoising. The denoising sub-model of the trained diffusion model is based on the input condition to add noise to the feature z T Perform inference denoising, which will not be described here.

[0060] The present invention also provides a diffusion model for generating an infrared image based on a target frame, and the diffusion model is obtained by training using any of the above methods.

[0061] like Figure 4 As shown, the present invention also provides a system for generating an infrared image based on a target frame, the system comprising an infrared image acquisition device, a data processing module and a result output device;

[0062] The infrared image acquisition device is used to acquire a target frame of an object;

[0063] The data processing module is equipped with the trained diffusion model, and is used to generate a corresponding infrared image based on the target frame coding feature corresponding to the target frame of the object to be tested;

[0064] The result output device is used to output the generated infrared image.

[0065] In this embodiment, the infrared image acquisition device may be an infrared thermal imager, the data processing module includes hardware such as a deep learning chip, an FPGA, a graphics card (GPU), and the result output device includes an image display, a bounding box display device, and the like.

[0066] In summary, the present invention provides a method, a diffusion model and a system for generating infrared images based on a target frame, wherein the given target frame position and category are feature encoded by Fourier transform and a multimodal model (CLIP model), thereby obtaining target frame encoding features; and the features are denoised and processed in latent space by a diffusion model, thereby obtaining a trained diffusion model; the trained diffusion model generates a corresponding infrared image based on the target frame of the object to be tested. After the diffusion model training is completed, it performs step-by-step denoising reasoning based on the target frame encoding features, obtains the corresponding latent space features from pure Gaussian noise, and generates the corresponding infrared image. The present invention can generate infrared images that are close to reality, and can accurately generate corresponding objects within a given target frame range. The infrared images generated by the present invention can be widely used in training data expansion in the fields of severe weather target detection, night driving, etc., to promote the development of related fields.

[0067] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0068] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating an infrared image based on a target frame, characterized in that: The method comprises: Obtain a target frame of the object, wherein the target frame includes a target frame category and a target frame position, and the target frame category corresponds to the target frame position one by one; Input the target frame category of the object into the multimodal model to obtain the category coding feature corresponding to the target frame category; perform Fourier transform on the target frame position of the object to obtain the position coding feature corresponding to the target frame position; add the category coding feature and the position coding feature to obtain the comprehensive coding feature; perform linear mapping on the comprehensive coding feature to obtain the target frame coding feature; Perform forward noise addition on the latent space features of the infrared image to obtain the noise-completed features; The loss function of the denoising sub-model of the diffusion model is obtained based on the target box encoding features; Based on the loss function of the denoising sub-model of the diffusion model and the features after adding noise, the denoising sub-model of the diffusion model is trained to obtain a trained diffusion model; The trained diffusion model generates the corresponding infrared image based on the target frame encoding features corresponding to the target frame of the object to be tested.

2. The method according to claim 1, characterized in that The target box encoding feature is obtained by the following formula: H=MLP(Fourier({b1,...,b n });CLIP({l1,...,l n })) Where H represents the target box encoding feature, MLP represents linear mapping, Fourier represents Fourier transform, CLIP represents multimodal model, b1,...,b n represents the first, ..., nth target box positions, l1, ..., l n The 1st, ..., nth object box categories.

3. The method according to claim 1, characterized in that The latent space features of the infrared image are forward-noised using the following formula: In the formula, z t represents the latent space feature with t levels of noise, t = 1, 2, 3, ..., T, T represents the total number of noise steps, α t represents the noise parameter of the tth step, z0 represents the pure latent space feature, and ε represents the randomly sampled Gaussian noise.

4. The method according to claim 1, characterized in that: The loss function of the denoising sub-model of the diffusion model is obtained by the following formula: Where, L θ represents the loss function of the denoising sub-model of the diffusion model, ε θ (z t ,t,H) represents the denoising sub-model of the diffusion model, ε represents the randomly sampled Gaussian noise, z t represents the latent space features with t levels of noise, t = 1, 2, 3, ..., T, T represents the total number of noise adding steps, and H represents the target box encoding features.

5. The method according to claim 1, characterized in that: The denoising sub-model of the diffusion model is trained by the following formula: In the formula, z t-1 represents the latent space feature with t-1 level noise, α t represents the noise parameter at step t, z t represents the latent space feature with t levels of noise, t = 1, 2, 3, ..., T, T represents the total number of noise steps, ε θ (z t ,t,H) represents the denoising sub-model of the diffusion model, H represents the target box encoding feature, and ε represents the randomly sampled Gaussian noise.

6. A diffusion model for generating infrared images based on a target frame, characterized in that: The diffusion model is obtained by training using the method described in any one of claims 1-5.

7. A system for generating infrared images based on a target frame, characterized in that: The system includes an infrared image acquisition device, a data processing module and a result output device; The infrared image acquisition device is used to acquire a target frame of an object; The diffusion model according to claim 6 is deployed on the data processing module, and the data processing module is used to generate a corresponding infrared image based on the target frame coding feature corresponding to the target frame of the object to be tested; The result output device is used to output the generated infrared image.

Images