Model training method, image generation method, device, equipment, medium and product

Abstract

The embodiment of the present application provides a model training method, an image generation method, an apparatus, a device, a medium and a product, which relate to the field of computer vision technology. The model training method includes: obtaining multiple sample pairs; each sample pair includes: a sample thermal imaging image containing a first object and corresponding sample input data; the sample input data includes: at least one of a sample visible light image, a sample mask image and a sample description text; using the diffusion model of the initial structure to process each sample pair to obtain a predicted thermal imaging image, and based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise, adjusting the model parameters of the diffusion model of the initial structure until the convergence condition is reached. In this way, the trained diffusion model can be used to generate a thermal imaging image containing the object indicated by the input data, so that a large number of thermal imaging images containing the object indicated by the input data can be obtained conveniently and quickly.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a model training method, image generation method, device, equipment, medium and product. Background Art

[0002] Thermal imaging technology can capture the thermal radiation emitted by target objects (such as animals in a protected area or equipment in a substation) and convert this information into a visual image to obtain a thermal image containing the target object. In some application scenarios, thermal images containing target objects can be detected using a deep learning model. In order to ensure the generalization ability of the model, it is often necessary to use a large number of thermal images containing target objects as sample images to train the model. Therefore, how to obtain a large number of thermal images containing target objects has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a model training method, image generation method, device, equipment, medium and product to conveniently and quickly obtain a large number of thermal imaging images of objects indicated by input data. The specific technical solution is as follows:

[0004] In a first aspect of an embodiment of the present application, a model training method is provided, the method comprising:

[0005] Acquire a plurality of sample pairs; wherein each sample pair comprises: a sample thermal imaging image containing a first object and corresponding sample input data; the sample input data corresponding to any sample thermal imaging image comprises: a sample visible light image consistent with the image content contained in the sample thermal imaging image, a sample mask image indicating the position of the first object in the sample visible light image, and at least one of a sample description text for describing the category and temperature of the first object in the sample visible light image;

[0006] Each sample pair is processed using the diffusion model of the initial structure to obtain a predicted thermal imaging image, and based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until the convergence condition is reached.

[0007] Optionally, the step of processing each sample pair using the diffusion model of the initial structure to obtain a predicted thermal imaging image, and adjusting model parameters of the diffusion model of the initial structure based on a difference between noise added when obtaining the predicted thermal imaging image and the predicted noise until a convergence condition is reached includes:

[0008] The diffusion model of the initial structure is used to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step; the sample image features and the sample text features are processed based on the attention mechanism to obtain a predicted thermal imaging image; based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until the convergence condition is reached;

[0009] Among them, when the sample input data corresponding to any sample thermal imaging image includes an image, the sample image feature is: the fusion result of the noisy image feature and the image feature of the image contained in the sample input data in the sample pair; when the sample input data corresponding to any sample thermal imaging image includes text, the sample text feature is the text feature of the sample description text in the sample pair.

[0010] Optionally, the diffusion model of the initial structure is used to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step; the sample image features and the sample text features are processed based on the attention mechanism to obtain a predicted thermal imaging image; and based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until a convergence condition is reached, including:

[0011] In the case where the sample input data corresponding to any sample thermal imaging image includes a sample visible light image, a sample mask image and a sample description text, the image features of the sample thermal imaging image in each sample pair are denoised according to a preset time step using a diffusion model of an initial structure, and the fusion result of the image features after denoising, the image features of the sample visible light image in the sample pair and the preset text features are processed based on an attention mechanism to obtain a first predicted thermal imaging image;

[0012] Based on the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until a first convergence condition is reached to obtain a preliminarily trained diffusion model;

[0013] Using the initially trained diffusion model, the image features of the sample thermal imaging image in each sample pair are denoised according to the preset time step, and based on the attention mechanism, the fusion result of the noisy image features, the image features of the sample mask image in the sample pair, and the text features of the sample description text in the sample pair are processed to obtain a second predicted thermal imaging image;

[0014] Based on the difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise, the model parameters of the initially trained diffusion model are adjusted until a second convergence condition is met.

[0015] Optionally, the diffusion model is implemented based on Unet, and a cross attention layer is provided between each encoding layer and the corresponding decoding layer of the Unet;

[0016] In the process of obtaining the first predicted thermal imaging image, the input of the Unet is the fusion result of the image features after the noise is added and the image features of the sample visible light image in the sample pair, the Query of the cross attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the preset text feature, and the Value is the fusion result of the Query and the Key;

[0017] In the process of obtaining the second predicted thermal imaging image, the input of the Unet is the fusion result of the noisy image features and the image features of the sample mask image in the sample pair, the Query of the cross-attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the text feature of the sample description text in the sample pair, and the Value is the fusion result of the Query and the Key.

[0018] Optionally, the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise is calculated according to a first formula, wherein the first formula is:

[0019] ;

[0020] represents the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, represents the noise added according to the preset time step when obtaining the first predicted thermal imaging image, represents the noise predicted to be added according to the preset time step when the first predicted thermal imaging image is obtained, represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the visible light image of the sample in the sample pair The image features, Indicates calculation of the square of the 2-norm;

[0021] The difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise is calculated according to a second formula, wherein the second formula is:

[0022] ;

[0023] represents the difference between the noise added and the predicted noise when obtaining the second predicted thermal imaging image, represents the noise added according to the preset time step when obtaining the second predicted thermal imaging image, represents the noise predicted to be added according to the preset time step when obtaining the second predicted thermal imaging image, represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the sample mask image in the sample pair The image features, Indicates the sample description text in this sample pair The text features of It means calculating the square of 2 norm.

[0024] Optionally, the image features of the image in each sample pair are extracted using a pre-trained VAE (Variational Autoencoder).

[0025] Optionally, the VAE is the part of a pre-trained comparative text-image pre-training model used to extract image features, and the text features of the text in each sample pair are extracted using the text encoder in the comparative text-image pre-training model.

[0026] According to a second aspect of the embodiments of the present application, there is also provided an image generation method, the method comprising:

[0027] Acquire a pre-trained diffusion model and input data to be processed; wherein the diffusion model is trained based on any of the above-mentioned model training methods; the type of sample input data used when training the diffusion model includes the type of the input data to be processed; the input data to be processed includes: a visible light image to be processed, a mask image to be processed indicating the position of a second object in the visible light image to be processed, and at least one of a description text to be processed for describing the category and temperature of the second object in the visible light image to be processed;

[0028] The input data to be processed is processed using the diffusion model to obtain a thermal imaging image.

[0029] Optionally, using the diffusion model to process the input data to be processed to obtain a thermal imaging image includes:

[0030] Based on the attention mechanism, the image features to be processed and the text features to be processed are processed to obtain a thermal imaging image; wherein, when the input data to be processed includes an image, the image features to be processed are: a fusion result of preset noise and the image features of the image contained in the input data to be processed; when the input data to be processed includes text, the text features to be processed are text features of the text contained in the input data to be processed.

[0031] Optionally, in the case where the diffusion model is obtained by training according to any of the above-mentioned model training methods involving obtaining a preliminarily trained diffusion model, the input data to be processed includes: the mask image to be processed and the description text to be processed;

[0032] The method of processing the image features to be processed and the text features to be processed based on the attention mechanism to obtain a thermal imaging image includes:

[0033] Based on the attention mechanism, the fusion result of the preset noise and the image features of the mask image to be processed and the text features of the description text to be processed are processed to obtain a thermal imaging image.

[0034] Optionally, the diffusion model is implemented based on Unet, and a cross-attention layer is set between each encoding layer and the corresponding decoding layer of the Unet; the input of the Unet is the fusion result of the preset noise and the image features of the mask image to be processed, the Query of the cross-attention layer corresponding to each encoding layer is the feature output by the encoding layer, the Key is the text feature of the description text to be processed, and the Value is the fusion result of the Query and the Key.

[0035] Optionally, the image features of the image contained in the input data to be processed are extracted using a pre-trained VAE.

[0036] Optionally, the VAE is the part of a pre-trained contrastive text-image pre-training model used to extract image features, and the text features of the text contained in the input data to be processed are extracted using the text encoder in the contrastive text-image pre-training model.

[0037] According to a third aspect of the embodiments of the present application, a model training device is provided, the device comprising:

[0038] A sample pair acquisition module, used to acquire a plurality of sample pairs; wherein each sample pair includes: a sample thermal imaging image containing a first object and corresponding sample input data; the sample input data corresponding to any sample thermal imaging image includes: a sample visible light image consistent with the image content contained in the sample thermal imaging image, a sample mask image indicating the position of the first object in the sample visible light image, and at least one of a sample description text for describing the category and temperature of the first object in the sample visible light image;

[0039] The model parameter adjustment module is used to process each sample pair using the diffusion model of the initial structure to obtain a predicted thermal imaging image, and adjust the model parameters of the diffusion model of the initial structure based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise until the convergence condition is reached.

[0040] Optionally, the model parameter adjustment module includes:

[0041] A model adjustment submodule is used to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step using the diffusion model of the initial structure; process the sample image features and the sample text features based on the attention mechanism to obtain a predicted thermal imaging image; and adjust the model parameters of the diffusion model of the initial structure based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise until the convergence condition is reached;

[0042] Wherein, when the sample input data corresponding to any sample thermal imaging image includes an image, the sample image feature is: a fusion result of the image feature after noise addition and the image feature of the image included in the sample input data in the sample pair; when the sample input data corresponding to any sample thermal imaging image includes text, the sample text feature is a text feature of the sample description text in the sample pair;

[0043] Optionally, the model adjustment submodule is specifically used to, when the sample input data corresponding to any sample thermal imaging image includes a sample visible light image, a sample mask image and a sample description text, use the diffusion model of the initial structure to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step, and process the fusion result of the noisy image features with the image features of the sample visible light image in the sample pair and the preset text features based on the attention mechanism to obtain a first predicted thermal imaging image;

[0044] Based on the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until a first convergence condition is reached to obtain a preliminarily trained diffusion model;

[0045] Using the initially trained diffusion model, the image features of the sample thermal imaging image in each sample pair are denoised according to the preset time step, and based on the attention mechanism, the fusion result of the noisy image features, the image features of the sample mask image in the sample pair, and the text features of the sample description text in the sample pair are processed to obtain a second predicted thermal imaging image;

[0046] Based on the difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise, adjusting the model parameters of the initially trained diffusion model until a second convergence condition is reached;

[0047] Optionally, the diffusion model is implemented based on Unet, and a cross attention layer is provided between each encoding layer and the corresponding decoding layer of the Unet;

[0048] In the process of obtaining the first predicted thermal imaging image, the input of the Unet is the fusion result of the image features after noise addition and the image features of the sample visible light image in the sample pair, the Query of the cross attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the preset text feature, and the Value is the fusion result of the Query and the Key;

[0049] In the process of obtaining the second predicted thermal imaging image, the input of the Unet is the fusion result of the image features after noise addition and the image features of the sample mask image in the sample pair, the Query of the cross attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the text feature of the sample description text in the sample pair, and the Value is the fusion result of the Query and the Key;

[0050] Optionally, the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise is calculated according to a first formula, wherein the first formula is:

[0051] ;

[0052] represents the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, represents the noise added according to the preset time step when obtaining the first predicted thermal imaging image, represents the noise predicted to be added according to the preset time step when the first predicted thermal imaging image is obtained, represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the visible light image of the sample in the sample pair The image features, Indicates calculation of the square of the 2-norm;

[0053] The difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise is calculated according to a second formula, wherein the second formula is:

[0054] ;

[0055] represents the difference between the noise added and the predicted noise when obtaining the second predicted thermal imaging image, represents the noise added according to the preset time step when obtaining the second predicted thermal imaging image, represents the noise predicted to be added according to the preset time step when obtaining the second predicted thermal imaging image, represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the sample mask image in the sample pair The image features, Indicates the sample description text in this sample pair The text features of Indicates calculation of the square of the 2-norm;

[0056] Optionally, the image features of the image in each sample pair are extracted using a pre-trained VAE;

[0057] Optionally, the VAE is the part of a pre-trained comparative text-image pre-training model used to extract image features, and the text features of the text in each sample pair are extracted using the text encoder in the comparative text-image pre-training model.

[0058] In another aspect of the embodiments of the present application, there is further provided an image generating device, the device comprising:

[0059] A data acquisition module, used to acquire a pre-trained diffusion model and input data to be processed; wherein the diffusion model is obtained by training based on any of the above-mentioned model training methods; the type of sample input data used when training the diffusion model includes the type of the input data to be processed; the input data to be processed includes: a visible light image to be processed, a mask image to be processed indicating the position of a second object in the visible light image to be processed, and at least one of a description text to be processed for describing the category and temperature of the second object in the visible light image to be processed;

[0060] The thermal imaging image generation module is used to process the input data to be processed using the diffusion model to obtain a thermal imaging image.

[0061] Optionally, the thermal imaging image generation module includes:

[0062] A feature processing submodule, used for processing the image features to be processed and the text features to be processed based on the attention mechanism to obtain a thermal imaging image; wherein, when the input data to be processed includes an image, the image features to be processed are: a fusion result of preset noise and the image features of the image included in the input data to be processed; when the input data to be processed includes text, the text features to be processed are text features of the text included in the input data to be processed;

[0063] Optionally, in the case where the diffusion model is obtained by training according to any of the above-mentioned model training methods involving obtaining a preliminarily trained diffusion model, the input data to be processed includes: the mask image to be processed and the description text to be processed;

[0064] The feature processing submodule is specifically used to process the fusion result of the preset noise and the image features of the mask image to be processed and the text features of the description text to be processed based on the attention mechanism to obtain a thermal imaging image;

[0065] Optionally, the diffusion model is implemented based on Unet, and a cross attention layer is provided between each encoding layer and the corresponding decoding layer of the Unet; the input of the Unet is the fusion result of the preset noise and the image features of the mask image to be processed, the Query of the cross attention layer corresponding to each encoding layer is the feature output by the encoding layer, the Key is the text feature of the description text to be processed, and the Value is the fusion result of the Query and the Key;

[0066] Optionally, the image features of the image contained in the input data to be processed are extracted using a pre-trained VAE;

[0067] Optionally, the VAE is the part of a pre-trained contrastive text-image pre-training model used to extract image features, and the text features of the text contained in the input data to be processed are extracted using the text encoder in the contrastive text-image pre-training model.

[0068] In another aspect of the embodiments of the present application, an electronic device is provided, including:

[0069] Memory, used to store computer programs;

[0070] The processor is used to implement any of the above-mentioned model training methods or image generation methods when executing the program stored in the memory.

[0071] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any of the above-mentioned model training methods or image generation methods.

[0072] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned model training methods or image generation methods.

[0073] Beneficial effects of the embodiments of the present application:

[0074] A model training method provided in an embodiment of the present application can obtain multiple sample pairs; wherein each sample pair includes: a sample thermal imaging image containing a first object and corresponding sample input data; the sample input data corresponding to any sample thermal imaging image includes: a sample visible light image consistent with the image content contained in the sample thermal imaging image, a sample mask image representing the position of the first object in the sample visible light image, and at least one of the sample description texts used to describe the category and temperature of the first object in the sample visible light image; each sample pair is processed using a diffusion model of an initial structure to obtain a predicted thermal imaging image, and based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until the convergence condition is reached.

[0075] Based on the above processing, the diffusion model can be trained using the sample thermal imaging images containing the first object in each sample pair and the corresponding sample input data. During the training process, the diffusion model can learn the characteristics of the sample input data, as well as the association between the sample input data and the sample thermal imaging images containing the object indicated by the sample input data (i.e., the first object), that is, the diffusion model can learn the ability to generate a thermal imaging image containing the object indicated by the input data based on the input data. In this way, the trained diffusion model can be used to generate a thermal imaging image containing the object indicated by the input data, so that a large number of thermal imaging images containing the object indicated by the input data can be obtained conveniently and quickly.

[0076] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0078] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0079] Figure 2 A schematic diagram of a process for obtaining a sample mask image in a model training method provided in an embodiment of the present application;

[0080] Figure 3 A schematic diagram of the structure of a comparative text-image pre-training model provided in an embodiment of the present application;

[0081] Figure 4 A schematic diagram of the first stage of training in a model training method provided in an embodiment of the present application;

[0082] Figure 5 A schematic diagram of the second stage training process in a model training method provided in an embodiment of the present application;

[0083] Figure 6 A schematic diagram of a flow chart of non-stage training in a model training method provided in an embodiment of the present application;

[0084] Figure 7 A schematic diagram of a first flow chart of an image generation method provided in an embodiment of the present application;

[0085] Figure 8 A second flow chart of the image generation method provided in the embodiment of the present application;

[0086] Fig. 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0087] Fig.10 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0088] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0090] Thermal imaging technology can capture the thermal radiation emitted by target objects (such as animals in a protected area or equipment in a substation) and convert this information into a visual image to obtain a thermal image containing the target object. In some application scenarios, thermal images containing target objects can be detected through deep learning models. For example, deep learning models can be used to achieve target detection, target classification, or target recognition tasks. In order to ensure the generalization ability of the model, it is often necessary to use a large number of thermal images containing target objects as sample images to train the model.

[0091] In order to quickly and easily obtain a large number of thermal imaging images containing objects indicated by input data, the present application embodiment provides a model training method, see Figure 1 , Figure 1 A flow chart of a model training method provided in an embodiment of the present application. The model training method includes:

[0092] Step S101: Acquire multiple sample pairs.

[0093] Wherein, each sample pair includes: a sample thermal imaging image containing a first object and corresponding sample input data; the sample input data corresponding to any sample thermal imaging image includes: a sample visible light image consistent with the image content contained in the sample thermal imaging image, a sample mask image representing the position of the first object in the sample visible light image, and at least one of sample description texts for describing the category and temperature of the first object in the sample visible light image.

[0094] Step S102: Process each sample pair using the diffusion model of the initial structure to obtain a predicted thermal imaging image, and adjust the model parameters of the diffusion model of the initial structure based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise until the convergence condition is reached.

[0095] Based on the above processing, the diffusion model can be trained using the sample thermal imaging images containing the first object in each sample pair and the corresponding sample input data. During the training process, the diffusion model can learn the characteristics of the sample input data, as well as the association between the sample input data and the sample thermal imaging images containing the object indicated by the sample input data (i.e., the first object), that is, the diffusion model can learn the ability to generate a thermal imaging image containing the object indicated by the input data based on the input data. In this way, the trained diffusion model can be used to generate a thermal imaging image containing the object indicated by the input data, so that a large number of thermal imaging images containing the object indicated by the input data can be obtained conveniently and quickly.

[0096] For step S101, multiple sample thermal imaging images containing the first object and sample input data corresponding to each sample thermal imaging image can be obtained. A sample thermal imaging image may contain one or more first objects, and the first object can be set as needed without specific limitation. For example, the first object may be an object that needs to be included in the thermal imaging image that needs to be generated currently, such as an animal in a protected area or equipment in a substation. A sample thermal imaging image and the sample input data corresponding to the sample thermal imaging image may be referred to as a sample pair.

[0097] The sample input data corresponding to any sample thermal imaging image may include at least one of: a sample visible light image, a sample mask image, and a sample description text. That is, the sample input data corresponding to any sample thermal imaging image may be any one, any two, or three of the sample visible light image, the sample mask image, and the sample description text. The type of sample input data can be set as needed. The more types of sample input data there are, the more features the diffusion model can learn during the training process, which can improve the accuracy of the trained diffusion model in generating a thermal imaging image containing the object indicated by the input data based on the input data to a certain extent.

[0098] In the sample input data corresponding to any sample thermal imaging image, the sample visible light image and the sample thermal imaging image contain the same image content. That is, the scenes and objects contained in the two images are consistent. For example, the sample visible light image and the sample thermal imaging image may be collected at the same time for the first object in the same scene. In the sample thermal imaging image, different temperature intervals can be represented by different colors, and each temperature interval and the color corresponding to each temperature interval can be set according to needs without specific limitation.

[0099] The sample mask image may represent the position of the first object in the sample visible light image. For example, the pixel position (which may be referred to as the designated pixel position) belonging to the image area occupied by the first object in the sample mask image and the pixel values ​​of other pixel positions may be different values, and the pixel values ​​of each designated pixel position are the same, and the pixel values ​​of other pixel positions are the same. For example, the pixel value of the designated pixel position may be 1, and the pixel values ​​of other pixel positions may be 0. The sample mask image may be obtained by performing image segmentation on the sample visible light image using a preset image segmentation algorithm. For example, the preset image segmentation algorithm may be a Segment anything model (SAM, an image segmentation model) algorithm, a threshold-based segmentation algorithm, an edge detection segmentation algorithm, or a clustering segmentation algorithm.

[0100] like Figure 2 As shown, Figure 2A schematic diagram of a flow chart for obtaining a sample mask image in a model training method provided in an embodiment of the present application. The process of obtaining a sample mask image can also be referred to as a label generation process, and a visible light image (i.e., a sample visible light image) can be input into a pre-trained segmentation model to obtain a segmented image (i.e., a sample mask image). The obtained segmented image can be used as a pseudo-label, i.e., as a control condition for subsequent training of a diffusion model.

[0101] The sample description text can be used to describe the category and temperature of the first object in the sample visible light image. The sample visible light image and the sample thermal imaging image contain the same image content, that is, the sample description text can also describe the category and temperature of the first object in the sample thermal imaging image. The category of a first object is the type to which the first object belongs. For example, the category of a first object can be a cat or a dog. The temperature of a first object can be described by the temperature difference between the first object and the environment in which it is located. For example, the temperature range to which the temperature of the first object and the environment in which it is located can be determined based on the color of the image area occupied by the first object in the sample thermal imaging image and the color of the image area representing the environment in which the first object is located. Correspondingly, the temperature difference between the first object and the environment in which it is located can be determined based on the temperature range to which the temperature of the first object and the environment in which it is located. In one implementation, the temperature difference interval and the corresponding temperature difference degree may be preset. The temperature difference interval may indicate the temperature range in which the temperature difference is located, and the temperature difference degree may indicate the size of the temperature difference. For example, the temperature difference interval is 0-5 degrees Celsius, and the corresponding temperature difference degree is small; the temperature difference interval is 5-10 degrees Celsius, and the corresponding temperature difference degree is medium; the temperature difference interval is above 10 degrees Celsius, and the corresponding temperature difference degree is large. For example, the sample description text may be: There is a cat in the image, and the temperature difference between the cat and the environment is large.

[0102] For step S102, each sample pair may be processed using the diffusion model of the initial structure to obtain a predicted thermal imaging image. The process of processing the data by the diffusion model may include: a forward diffusion process and a reverse diffusion process. In the forward diffusion process, noise may be gradually added to the data to obtain noise data. In the reverse diffusion process, the noise added in each step may be predicted, and the noise in the noise data may be gradually removed according to the predicted noise, so as to restore the original high-quality data from the noise data to obtain a predicted thermal imaging image.

[0103] Based on the difference between the noise added when the predicted thermal imaging image is obtained and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until the convergence condition is reached. For example, the convergence condition may be that the number of times the model parameters of the diffusion model are adjusted reaches a specified number of times, or the convergence condition may also be that the difference between the noise added when the predicted thermal imaging image is obtained and the predicted noise is less than a preset threshold. Accordingly, a trained diffusion model can be obtained.

[0104] During the training process, the diffusion model can learn the characteristics of the sample input data and the association between the sample input data and the sample thermal imaging image containing the first object, that is, the diffusion model can learn the ability to generate a thermal imaging image containing the object indicated by the input data based on the input data. In this way, the trained diffusion model can be used to generate a thermal imaging image containing the object indicated by the input data, and a large number of thermal imaging images containing the object indicated by the input data can be obtained conveniently and quickly. The specific process of training the diffusion model of the initial structure can refer to the relevant description in the subsequent embodiments.

[0105] In addition, in one method, GAN (Generative Adversarial Networks) can be used to generate thermal imaging images, but the authenticity and diversity of thermal imaging images generated based on this algorithm are lower than the solution of generating thermal imaging images using a diffusion model. And the present application can perform multimodal (image and text) control on the diffusion model, that is, set multimodal control conditions, so that the image quality of thermal imaging images generated using the diffusion model is greatly improved. The specific method of implementing multimodal control can refer to the description in the subsequent embodiments.

[0106] In one embodiment, step S102 includes:

[0107] Step 1: Use the diffusion model of the initial structure to add noise to the image features of the sample thermal imaging image in each sample pair according to the preset time step; process the sample image features and the sample text features based on the attention mechanism to obtain the predicted thermal imaging image; based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise, adjust the model parameters of the diffusion model of the initial structure until the convergence condition is reached.

[0108] Among them, when the sample input data corresponding to any sample thermal imaging image includes an image, the sample image feature is: the fusion result of the noisy image feature and the image feature of the image contained in the sample input data in the sample pair; when the sample input data corresponding to any sample thermal imaging image includes text, the sample text feature is the text feature of the sample description text in the sample pair.

[0109] In an embodiment of the present application, the features of the data in each sample pair can be obtained, and accordingly, the acquired features can be processed using the diffusion model of the initial structure. The image features of the sample thermal imaging image in each sample pair are denoised according to a preset time step using the diffusion model of the initial structure, that is, the forward diffusion process of the diffusion model. The sample image features and the sample text features can be processed based on the attention mechanism to obtain a predicted thermal imaging image, that is, the reverse diffusion process of the diffusion model. The sample image features and the sample text features can be called the control conditions of the diffusion model.

[0110] In the case where the sample input data corresponding to any sample thermal imaging image includes an image (i.e., a sample visible light image and / or a sample mask image), the sample image feature is: a fusion result of the image feature after noise addition and the image feature of the image included in the sample input data in the sample pair. For example, the image feature after noise addition and the image feature of the image included in the sample input data in the sample pair may be spliced, or the weighted sum of the image feature after noise addition and the image feature of the image included in the sample input data in the sample pair may be calculated to obtain a fusion result. In the case where the sample input data corresponding to any sample thermal imaging image does not include an image, the sample image feature is the image feature after noise addition.

[0111] For example, the image in each sample pair may be feature extracted by a preset image encoder to obtain the image features of the image in the sample pair. For example, the image encoder may be a CNN (Convolutional Neural Networks), a transformer (a network structure), or a VAE in subsequent embodiments.

[0112] In the case where the sample input data corresponding to any sample thermal imaging image includes text (i.e., sample description text), the sample text feature is the text feature of the sample description text in the sample pair. For example, the sample description text in each sample pair can be feature extracted by a pre-set text encoder to obtain the text feature of the sample description text. For example, the text encoder can be a text encoder in BERT (a language representation model), LSTM (Long Short Term Memory) or CLIP (Contrastive Language Image Pre training, i.e., the contrastive text-image pre-training model in subsequent embodiments). In the case where the sample input data corresponding to any sample thermal imaging image does not include text, the sample text feature is a preset text feature, for example, the preset text feature can be a random value.

[0113] After obtaining the predicted thermal imaging image, the model parameters of the diffusion model of the initial structure can be adjusted based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise until the convergence condition is reached to obtain a trained diffusion model.

[0114] Based on the above processing, the diffusion model of the initial structure can be used to process the image features of the image and / or the text features of the text in each sample pair based on the attention mechanism to realize the training of the diffusion model. It is ensured that the diffusion model can learn the features of the data contained in each sample pair and the association relationship between the sample input data in each sample pair and the sample thermal imaging image containing the first object during the training process. That is, the diffusion model can learn the ability to generate a thermal imaging image containing the object indicated by the input data according to the input data.

[0115] In this way, the trained diffusion model can be used to generate thermal imaging images containing the object indicated by the input data, and a large number of thermal imaging images containing the object indicated by the input data can be obtained conveniently and quickly. And by using the diffusion model to process the features of the data contained in each sample pair, the complexity of the diffusion model processing each sample pair can be reduced to a certain extent, and the calculation efficiency of the diffusion model can be improved, thereby improving the training efficiency of the diffusion model.

[0116] In one embodiment, the image features of the images in each sample pair are extracted using a pre-trained VAE.

[0117] In an embodiment of the present application, the image features of the images in each sample pair can be extracted by using a pre-trained VAE to obtain the image features of the images in the sample pair. VAE can be used to map the image space to a latent space with a high-dimensional information volume and a low-dimensional data volume. That is, the image features extracted by VAE contain a large amount of information and a low amount of data. Accordingly, the complexity of processing the image features of the images contained in each sample pair using the diffusion model of the initial structure can be reduced, the calculation scale of the diffusion model can be reduced, the calculation efficiency of the diffusion model can be improved, and then, the training efficiency of the diffusion model can be improved.

[0118] In one embodiment, VAE is a part of a pre-trained contrastive text-image pre-trained model used to extract image features, and the text features of the text in each sample pair are extracted using a text encoder in the contrastive text-image pre-trained model.

[0119] In an embodiment of the present application, a plurality of image-text pairs may be acquired in advance, each image-text pair comprising a text and an image, and the text is used to describe the image. The image-text pairs are used to train the initial structure of the contrast text-image pre-training model to obtain a trained contrast text-image pre-training model. The contrast text-image pre-training model may include a part for extracting image features (which may be referred to as an image encoder) and a part for extracting text features (which may be referred to as a text encoder). During the training process, the contrast text-image pre-training model may learn the alignment relationship between the image and the text, so that the text features extracted by the text encoder in the contrast text-image pre-training model can reflect the image features of the image described by the text. For example, the contrast text-image pre-training model may be CLIP.

[0120] like Figure 3 As shown, Figure 3 A schematic diagram of the structure of a contrast text-image pre-training model provided for an embodiment of the present application. In the process of training the contrast text-image pre-training model, for a picture-text pair, the image in the picture-text pair can be input into the image encoder in the contrast text-image pre-training model of the initial structure to obtain the image features of the image. And the text in the picture-text pair can be input into the text encoder in the contrast text-image pre-training model of the initial structure to obtain the text features of the text used to describe the image. Furthermore, the loss value can be calculated based on the difference between the image features of the image and the text features of the text used to describe the image to adjust the model parameters of the contrast text-image pre-training model of the initial structure until the model convergence condition is reached to obtain a trained contrast text-image pre-training model.

[0121] Based on the above processing, it can be ensured that the sample text features extracted by the text encoder can effectively reflect the image features of the image described by the sample description text, and that the image features extracted by the image encoder can be aligned with the text features. Furthermore, it is ensured that the image generation model can learn the features of the image and / or text in the sample input data contained in each sample pair, as well as the association between the sample input data and the sample thermal imaging image containing the first object, that is, the diffusion model can learn the ability to generate a thermal imaging image containing the object indicated by the input data based on the input data. Subsequently, the trained diffusion model can be used to generate a thermal imaging image containing the object indicated by the input data based on the input data.

[0122] In the present application, the diffusion model of the initial structure can be used to process all the features obtained through each sample pair to obtain a predicted thermal imaging image. Based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until the convergence condition is reached. However, when there are multiple types of sample input data corresponding to any sample thermal imaging image, the amount of calculation for the diffusion model to process all the features obtained through each sample pair is often high, resulting in a long training time and low efficiency. In order to further improve the training efficiency of the diffusion model, the diffusion model can also be trained in multiple stages. For each stage, only the diffusion model obtained by the previous stage training of the stage is used to process part of the features obtained through each sample pair to achieve the training of the stage. The diffusion model used in the first stage training is the diffusion model of the initial structure, and the features used in each stage training are different. All features obtained through each sample pair in the multi-stage training are used. The features used in each stage can be set as needed without specific limitation.

[0123] Taking the case where the sample input data corresponding to any sample thermal imaging image includes a sample visible light image, a sample mask image and a sample description text as an example, the following method 1 and method 2 can be used to train the diffusion model by utilizing the image features of the sample visible light image, the image features of the sample mask image, the image features of the sample thermal imaging image and the text features of the sample description text in each sample pair.

[0124] In the first approach, the diffusion model can be trained in two stages. Step 1 includes:

[0125] Step 11: When the sample input data corresponding to any sample thermal imaging image includes a sample visible light image, a sample mask image and a sample description text, the image features of the sample thermal imaging image in each sample pair are denoised according to a preset time step using the diffusion model of the initial structure, and the fusion result of the noisy image features and the image features of the sample visible light image in the sample pair and the preset text features are processed based on the attention mechanism to obtain a first predicted thermal imaging image.

[0126] Step 12: Based on the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until the first convergence condition is reached to obtain a preliminarily trained diffusion model.

[0127] Step 13: using the initially trained diffusion model to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step, and based on the attention mechanism, processing the fusion result of the noisy image features with the image features of the sample mask image in the sample pair and the text features of the sample description text in the sample pair to obtain a second predicted thermal imaging image;

[0128] Step 14: Based on the difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise, the model parameters of the initially trained diffusion model are adjusted until the second convergence condition is reached.

[0129] In the embodiment of the present application, step 11 and step 12 may be referred to as the first stage, and step 13 and step 14 may be referred to as the second stage. During the training process of the first stage, the sample image feature is the fusion result of the image feature after adding noise and the image feature of the sample visible light image in the sample pair. Since the sample text description is not used in the training process of the first stage, the sample text feature may be a preset text feature, for example, the preset text feature may be a random value. After the pre-training of the first stage, a preliminary trained diffusion model may be obtained. For example, the first convergence condition may be that the number of times the model parameters of the diffusion model of the initial structure are adjusted reaches a first specified number, or the first convergence condition may also be that the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise is less than a first preset threshold.

[0130] See also Figure 4 , Figure 4 A flowchart of the first stage of training in a model training method provided in an embodiment of the present application. The visible light image is a sample visible light image, the image coding model is the image encoder in the above embodiment, and the thermal imaging image is a sample thermal imaging image. The image coding model can be used to extract features of the visible light image and the thermal imaging image to obtain their respective image features. The extracted image features can be processed using a diffusion model to achieve the first stage of training. That is, step 11 and step 12 are executed.

[0131] During the second stage of training, the sample image feature is the fusion result of the image feature after adding noise and the image feature of the sample mask image in the sample pair, and the sample text feature is the text feature of the sample description text in the sample pair. After the second stage of training, a trained diffusion model can be obtained. For example, the second convergence condition can be that the number of times the model parameters of the initially trained diffusion model are adjusted reaches a second specified number, or the second convergence condition can also be that the difference between the added noise and the predicted noise when obtaining the second predicted thermal imaging image is less than a second preset threshold.

[0132] See also Figure 5 , Figure 5 A flow chart of the second stage training in a model training method provided in an embodiment of the present application. The segmented image is a sample mask image, the image coding model is the image encoder in the above embodiment, the text coding model is the text encoder in the above embodiment, the text description information is the sample description text, and the thermal imaging image is the sample thermal imaging image. The image coding model can be used to extract features of the segmented image and the thermal imaging image to obtain respective image features, and the text coding model can be used to extract features of the text description information to obtain text features. The extracted image features and text features can be processed using a diffusion model to implement the second stage training. That is, execute steps 13 and 14.

[0133] Based on the above processing, during the training process of the first stage, the diffusion model can learn the features of the sample visible light image and the sample thermal imaging image, for example, the scene information in the sample visible light image can be learned, and the diffusion model can learn the correlation between the features of the sample visible light image and the sample thermal imaging image. The adaptability of the diffusion model to the scene where the first object is located can be improved, that is, the domain adaptation can be accelerated. Furthermore, the difficulty of the training in the second stage can be reduced, and the training efficiency of the second stage can be improved. That is, the memory consumption when generating the predicted thermal imaging image (also referred to as the data generation stage) during the training process of the diffusion model can be reduced, and the efficiency of the reasoning generation of the predicted thermal imaging image can be accelerated. In this way, the training efficiency of the diffusion model can be further improved while ensuring that the trained diffusion model can generate a thermal imaging image containing the object indicated by the input data. It can reduce memory consumption and accelerate reasoning in the data generation stage.

[0134] In one embodiment, the diffusion model is implemented based on Unet, and a cross attention layer is set between each encoding layer and the corresponding decoding layer of Unet;

[0135] In the process of obtaining the first predicted thermal imaging image, the input of Unet is the fusion result of the image features after noise addition and the image features of the sample visible light image in the sample pair, the Query of the cross attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the preset text feature, and the Value is the fusion result of the Query and the Key;

[0136] In the process of obtaining the second predicted thermal imaging image, the input of Unet is the fusion result of the noisy image features and the image features of the sample mask image in the sample pair, the Query of the cross-attention layer corresponding to each encoding layer is the feature output by the encoding layer, the Key is the text feature of the sample description text in the sample pair, and the Value is the fusion result of the Query and the Key.

[0137] In the embodiment of the present application, the diffusion model is implemented based on Unet, and a cross-attention layer is provided between each encoding layer and the corresponding decoding layer of Unet to perform cross-attention learning and realize the attention mechanism. The input of Unet is the sample image feature, the Query of the cross-attention layer corresponding to each encoding layer is the feature output by the encoding layer, the Key is the sample text feature, and the Value is the fusion result of the Query and the Key. For example, the weighted sum or product of the Query and the Key can be calculated to obtain the Value.

[0138] In the first stage of training, the image features after adding noise can be fused with the first sample image features obtained from the sample visible light in the sample pair to obtain the fusion result (i.e., the sample image features) and use it as the input of Unet. The Query of the cross attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the preset text feature (i.e., the sample text feature), and the Value is the fusion result of the Query and the Key.

[0139] In the second stage of training, the image features after adding noise can be fused with the image features of the sample mask image in the sample pair to obtain the fusion result (i.e., the sample image features) and use it as the input of Unet. The Query of the cross-attention layer corresponding to each encoding layer is the feature output by the encoding layer, the Key is the text feature of the sample description text in the sample pair (i.e., the sample text feature), and the Value is the fusion result of the Query and the Key.

[0140] Based on the above processing, it is ensured that the first stage and the second stage of training can be achieved, so that the trained diffusion model can generate a thermal imaging image containing the object indicated by the input data, while further improving the training efficiency of the diffusion model.

[0141] In one embodiment, the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise is calculated according to a first formula, wherein the first formula is:

[0142] ;

[0143] represents the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, represents the noise added according to the preset time step when obtaining the first predicted thermal imaging image, represents the noise added according to the preset time step when the first predicted thermal imaging image is obtained, Represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the visible light image of the sample in the sample pair The image features, Indicates calculation of the square of the 2-norm;

[0144] The difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise is calculated according to the second formula, where the second formula is:

[0145] ;

[0146] represents the difference between the noise added and the predicted noise when obtaining the second predicted thermal imaging image, represents the noise added according to the preset time step when obtaining the second predicted thermal imaging image, represents the noise added according to the preset time step when obtaining the second predicted thermal imaging image, Represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the sample mask image in the sample pair The image features, Indicates the sample description text in this sample pair The text features of It means calculating the square of 2 norm.

[0147] In the embodiment of the present application, the added noise It can conform to a standard Gaussian distribution. A time step range can be preset, the preset time step belongs to the time step range, and the preset time step can represent the number of times noise is added. During the first stage of training, the difference between the noise added when the first predicted thermal imaging image is obtained and the predicted noise can be calculated by the first formula to adjust the model parameters of the diffusion model of the initial structure. The difference between the last added noise and the predicted last added noise according to each preset time step can be calculated by the first formula to calculate the difference between the noise added when the first predicted thermal imaging image is obtained and the predicted noise.

[0148] In the second stage of training, the difference between the noise added when the first predicted thermal imaging image is obtained and the predicted noise can be calculated by the second formula to adjust the model parameters of the initially trained diffusion model. The specific method of calculating the difference between the noise added when the second predicted thermal imaging image is obtained and the predicted noise can refer to the relevant description of the first formula.

[0149] Based on the above processing, noise can be gradually added to the third sample image features according to the preset time step. The third sample image features after adding noise and other input features can be processed by the diffusion model, and the noise added in each step is predicted to obtain the predicted noise, and denoising is performed according to the predicted noise to obtain the predicted thermal imaging image. The diffusion model can learn the ability to gradually denoise and generate thermal imaging images from the noise image, so as to ensure that the required thermal imaging image can be generated by the diffusion model later.

[0150] In the second approach, all features obtained through each sample pair are processed using the diffusion model of the initial structure to obtain a predicted thermal imaging image, that is, no stage-by-stage training is performed.

[0151] In one embodiment, step 1 includes:

[0152] Step 1: Use the diffusion model of the initial structure to add noise to the image features of the sample thermal imaging image in each sample pair according to the preset time step, and process the sample image features and the text features of the sample description text in the sample pair based on the attention mechanism to obtain the third predicted thermal imaging image.

[0153] The sample image features are: the fusion result of the image features after noise addition, the image features of the sample thermal imaging image in the sample pair, and the image features of the sample mask image in the sample pair.

[0154] Step 2: Based on the difference between the noise added when obtaining the third predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until the third convergence condition is met.

[0155] In an embodiment of the present application, the image features after adding noise, the image features of the sample thermal imaging image in the sample pair, and the image features of the sample mask image in the sample pair can be fused to obtain a fusion result (i.e., the sample image features). The sample image features and the text features of the sample description text in the sample pair are processed using the diffusion model of the initial structure to obtain a third predicted thermal imaging image. Furthermore, based on the difference between the noise added when obtaining the third predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure can be adjusted until the third convergence condition is reached. For example, the third convergence condition can be that the number of times the model parameters of the diffusion model of the initial structure are adjusted reaches a third specified number, or the third convergence condition can also be that the difference between the noise added when obtaining the third predicted thermal imaging image and the predicted noise is less than a third preset threshold.

[0156] See also Figure 6 , Figure 6A flowchart of non-stage training in a model training method provided in an embodiment of the present application. The visible light image is a sample visible light image, the segmented image is a sample mask image, the image coding model is the image encoder in the above embodiment, the text coding model is the text encoder in the above embodiment, the text description information is the sample description text, and the thermal imaging image is the sample thermal imaging image. The image coding model can be used to extract features of the visible light image, the segmented image, and the thermal imaging image to obtain respective image features, and the text coding model can be used to extract features of the text description information to obtain text features. The extracted features can be processed using a diffusion model to implement diffusion model training. That is, the above steps one and two are executed.

[0157] Based on the above processing, the diffusion model can be trained using the sample thermal imaging images containing the first object in each sample pair and the corresponding sample input data. During the training process, the diffusion model can learn the characteristics of the sample input data, as well as the association between the sample input data and the sample thermal imaging images containing the object indicated by the sample input data (i.e., the first object), that is, the diffusion model can learn the ability to generate a thermal imaging image containing the object indicated by the input data based on the input data. In this way, the trained diffusion model can be used to generate a thermal imaging image containing the object indicated by the input data, so that a large number of thermal imaging images containing the object indicated by the input data can be obtained conveniently and quickly.

[0158] In one embodiment, the diffusion model is implemented based on Unet, and a cross-attention layer is set between each encoding layer and the corresponding decoding layer of Unet; the input of Unet is the sample image feature; the Query of the cross-attention layer corresponding to each encoding layer is the feature output by the encoding layer, the Key is the text feature of the sample description text in the sample pair, and the Value is the fusion result of the Query and the Key.

[0159] In an embodiment of the present application, a diffusion model can be implemented by Unet, and cross-attention learning can be implemented based on a cross-attention layer. In this way, it can be ensured that the diffusion model can learn the features of the sample visible light image, the sample mask image, the sample description text, and the sample thermal imaging image, and the diffusion model can learn the association between the features of the sample thermal imaging image and the corresponding sample input data. That is, the diffusion model can learn the ability to generate a thermal imaging image containing the object indicated by the input data based on the input data. In this way, the trained diffusion model can be used to generate a thermal imaging image containing the object indicated by the input data, so that a large number of thermal imaging images containing the object indicated by the input data can be obtained quickly and easily.

[0160] In one embodiment, the difference between the noise added when obtaining the third predicted thermal imaging image and the predicted noise is calculated according to a third formula, where the third formula is:

[0161] ;

[0162] represents the difference between the noise added and the predicted noise when obtaining the third predicted thermal imaging image, represents the noise added according to the preset time step when obtaining the third predicted thermal imaging image, It indicates the noise added according to the preset time step when the third predicted thermal imaging image is obtained. Represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the visible light image of the sample in the sample pair The image features, Represents the sample mask image in the sample pair The image features, Indicates the sample description text in this sample pair The text features of It means calculating the square of 2 norm.

[0163] In the embodiment of the present application, the added noise It can conform to the standard Gaussian distribution. During the training process of the diffusion model, the difference between the noise added when the third predicted thermal imaging image is obtained can be calculated by the third formula to adjust the model parameters of the diffusion model of the initial structure.

[0164] Based on the above processing, noise can be gradually added to the third sample image features according to the preset time step. The third sample image features after adding noise and other input features can be processed by the diffusion model, and the noise added in each step is predicted to obtain the predicted noise, and denoising is performed according to the predicted noise to obtain the predicted thermal imaging image. The diffusion model can learn the ability to gradually denoise and generate thermal imaging images from the noise image, so as to ensure that the required thermal imaging image can be generated by the diffusion model later.

[0165] Based on the same inventive concept, the present application embodiment provides an image generation method, see Figure 7 , Figure 7 A first flow chart of an image generation method provided in an embodiment of the present application, the method comprising:

[0166] Step S701: Obtain a pre-trained diffusion model and input data to be processed.

[0167] Among them, the diffusion model is trained based on the model training method described in any of the above embodiments; the type of sample input data used when training the diffusion model includes the type of input data to be processed; the input data to be processed includes: a visible light image to be processed, a mask image to be processed representing the position of the second object in the visible light image to be processed, and at least one of the description texts to be processed for describing the category and temperature of the second object in the visible light image to be processed.

[0168] Step S702: using the diffusion model to process the input data to be processed to obtain a thermal imaging image.

[0169] In an embodiment of the present application, the type of sample input data used when training the diffusion model includes the type of input data to be processed. The type of input data to be processed may be consistent with the type of sample input data used when training the diffusion model, or the type of input data to be processed may also be a partial type of the type of sample input data used when training the diffusion model. And the input data to be processed includes: a visible light image to be processed, a mask image to be processed representing the position of the second object in the visible light image to be processed, and any one, two or three items of the description text to be processed for describing the category and temperature of the second object in the visible light image to be processed. The category of the second object may be the same as the category of the first object, or may be different. The scene where the first object indicated by the sample input data is located is consistent with the scene where the second object indicated by the input data to be processed is located. For example, the scenes where the first object and the second object are located are both protected areas, and the second object may be an animal in the protected area, then the first object may be that kind of animal or other animals in the protected area.

[0170] For example, when the diffusion model is obtained by training according to method one in the above-mentioned model training method, the input data to be processed may include: a mask image to be processed and a description text to be processed; when the diffusion model is obtained by training according to method two in the above-mentioned model training method, the input data to be processed may include: a visible light image to be processed, a mask image to be processed and a description text to be processed.

[0171] Based on the above processing, the diffusion model can learn the ability to generate thermal imaging images containing the objects indicated by the input data according to the input data during the training process. Accordingly, by using the trained diffusion model to process the input data to be processed, it is possible to generate thermal imaging images containing the objects indicated by the input data to be processed, and it is possible to quickly and easily obtain a large number of thermal imaging images containing the objects indicated by the input data to be processed. In this way, it is possible to quickly and easily obtain thermal imaging images containing objects of scarce types, and the generated thermal imaging images can be applied to downstream target detection, target classification, target recognition and other tasks. For example, the generated thermal imaging images can be used as sample images to train deep learning models that implement downstream tasks.

[0172] In one embodiment, step S702 includes: processing the image features to be processed and the text features to be processed based on the attention mechanism to obtain a thermal imaging image.

[0173] Among them, when the input data to be processed includes an image, the image feature to be processed is: the fusion result of the preset noise and the image feature of the image contained in the input data to be processed; when the input data to be processed includes text, the text feature to be processed is the text feature of the text contained in the input data to be processed.

[0174] In one embodiment, when the diffusion model is obtained by training according to the first method in the above-mentioned model training method, the input data to be processed includes: a mask image to be processed and a description text to be processed;

[0175] Based on the attention mechanism, the image features to be processed and the text features to be processed are processed to obtain a thermal imaging image, including:

[0176] Based on the attention mechanism, the fusion results of the preset noise and the image features of the mask image to be processed and the text features of the description text to be processed are processed to obtain a thermal imaging image.

[0177] See also Figure 8 , Figure 8 A second flow chart of the image generation method provided in the embodiment of the present application. The process of generating a thermal imaging image can also be referred to as a process of thermal imaging inference generation. The segmented image is the mask image to be processed, the image encoding model is the image encoder in the above embodiment, the text encoding model is the text encoder in the above embodiment, and the text description information is the description text to be processed. The image encoding model can be used to extract features of the visible light image and the segmented image to obtain respective image features, and the text encoding model can be used to extract features of the text description information to obtain text features. The extracted features can be processed using a trained diffusion model to generate a thermal imaging image.

[0178] In one embodiment, the diffusion model is implemented based on Unet, and a cross-attention layer is set between each encoding layer and the corresponding decoding layer of Unet; the input of Unet is the fusion result of the preset noise and the image features of the mask image to be processed, the Query of the cross-attention layer corresponding to each encoding layer is the feature output by the encoding layer, the Key is the text feature of the description text to be processed, and the Value is the fusion result of the Query and the Key.

[0179] In one embodiment, the image features of the image contained in the input data to be processed are extracted using a pre-trained VAE.

[0180] In one embodiment, VAE is a part of a pre-trained contrastive text-image pre-training model used to extract image features, and the text features of the text contained in the input data to be processed are extracted using a text encoder in the contrastive text-image pre-training model.

[0181] In one embodiment, when the diffusion model is obtained by training according to the second method in the above-mentioned model training method, the input data to be processed includes a visible light image to be processed, a mask image to be processed, and a description text to be processed;

[0182] The features of the image to be processed and the features of the text to be processed are processed based on the attention mechanism to obtain a thermal imaging image, including: the features of the image to be processed and the text features of the description text to be processed are processed based on the attention mechanism to obtain a thermal imaging image.

[0183] The features of the image to be processed are: a fusion result of the preset noise, the image features of the visible light image to be processed, and the image features of the mask image to be processed.

[0184] The present application also provides a model training device, see Fig. 9 , Fig. 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, the device comprising:

[0185] The sample pair acquisition module 901 is used to acquire a plurality of sample pairs; each sample pair includes: a sample thermal imaging image including a first object and corresponding sample input data; the sample input data corresponding to any sample thermal imaging image includes: a sample visible light image consistent with the image content included in the sample thermal imaging image, a sample mask image indicating the position of the first object in the sample visible light image, and at least one of a sample description text for describing the category and temperature of the first object in the sample visible light image;

[0186] The model parameter adjustment module 902 is used to process each sample pair using the diffusion model of the initial structure to obtain a predicted thermal imaging image, and adjust the model parameters of the diffusion model of the initial structure based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise until the convergence condition is reached.

[0187] Based on the model training device provided in the embodiment of the present application, the diffusion model can be trained using the sample thermal imaging image containing the first object in each sample pair and the corresponding sample input data. During the training process, the diffusion model can learn the characteristics of the sample input data, as well as the association between the sample input data and the sample thermal imaging image containing the object indicated by the sample input data (i.e., the first object), that is, the diffusion model can learn the ability to generate a thermal imaging image containing the object indicated by the input data based on the input data. In this way, the trained diffusion model can be used to generate a thermal imaging image containing the object indicated by the input data, so that a large number of thermal imaging images containing the object indicated by the input data can be obtained conveniently and quickly.

[0188] In one embodiment, the model parameter adjustment module 902 includes:

[0189] A model adjustment submodule is used to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step using the diffusion model of the initial structure; process the sample image features and the sample text features based on the attention mechanism to obtain a predicted thermal imaging image; and adjust the model parameters of the diffusion model of the initial structure based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise until the convergence condition is reached;

[0190] Wherein, when the sample input data corresponding to any sample thermal imaging image includes an image, the sample image feature is: a fusion result of the image feature after noise addition and the image feature of the image included in the sample input data in the sample pair; when the sample input data corresponding to any sample thermal imaging image includes text, the sample text feature is a text feature of the sample description text in the sample pair;

[0191] In one embodiment, the model adjustment submodule is specifically used to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step using the diffusion model of the initial structure when the sample input data corresponding to any sample thermal imaging image includes a sample visible light image, a sample mask image and a sample description text, and to process the fusion result of the image features after adding noise with the image features of the sample visible light image in the sample pair and the preset text features based on the attention mechanism to obtain a first predicted thermal imaging image;

[0192] Based on the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until a first convergence condition is reached to obtain a preliminarily trained diffusion model;

[0193] Using the initially trained diffusion model, the image features of the sample thermal imaging image in each sample pair are denoised according to the preset time step, and based on the attention mechanism, the fusion result of the noisy image features, the image features of the sample mask image in the sample pair, and the text features of the sample description text in the sample pair are processed to obtain a second predicted thermal imaging image;

[0194] Based on the difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise, adjusting the model parameters of the initially trained diffusion model until a second convergence condition is reached;

[0195] In one embodiment, the diffusion model is implemented based on Unet, and a cross attention layer is provided between each encoding layer and the corresponding decoding layer of the Unet;

[0196] In the process of obtaining the first predicted thermal imaging image, the input of the Unet is the fusion result of the image features after noise addition and the image features of the sample visible light image in the sample pair, the Query of the cross attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the preset text feature, and the Value is the fusion result of the Query and the Key;

[0197] In the process of obtaining the second predicted thermal imaging image, the input of the Unet is the fusion result of the image features after noise addition and the image features of the sample mask image in the sample pair, the Query of the cross attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the text feature of the sample description text in the sample pair, and the Value is the fusion result of the Query and the Key;

[0198] In one embodiment, the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise is calculated according to a first formula, wherein the first formula is:

[0199] ;

[0200] represents the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, represents the noise added according to the preset time step when obtaining the first predicted thermal imaging image, represents the noise predicted to be added according to the preset time step when the first predicted thermal imaging image is obtained, represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the visible light image of the sample in the sample pair The image features, Indicates calculation of the square of the 2-norm;

[0201] The difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise is calculated according to a second formula, wherein the second formula is:

[0202] ;

[0203] represents the difference between the noise added and the predicted noise when obtaining the second predicted thermal imaging image, represents the noise added according to the preset time step when obtaining the second predicted thermal imaging image, represents the noise predicted to be added according to the preset time step when obtaining the second predicted thermal imaging image, represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the sample mask image in the sample pair The image features, Indicates the sample description text in this sample pair The text features of Indicates calculation of the square of the 2-norm;

[0204] In one embodiment, the image features of the image in each sample pair are extracted using a pre-trained VAE;

[0205] In one embodiment, the VAE is a part of a pre-trained contrastive text-image pre-training model used to extract image features, and the text features of the text in each sample pair are extracted using a text encoder in the contrastive text-image pre-training model.

[0206] The present application also provides an image generation device. Fig.10 , Fig.10 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application, the device comprising:

[0207] The data acquisition module 1001 is used to acquire a pre-trained diffusion model and input data to be processed; wherein the diffusion model is obtained by training based on any of the above-mentioned model training methods; the type of sample input data used when training the diffusion model includes the type of the input data to be processed; the input data to be processed includes: a visible light image to be processed, a mask image to be processed indicating the position of a second object in the visible light image to be processed, and at least one of a description text to be processed for describing the category and temperature of the second object in the visible light image to be processed;

[0208] The thermal imaging image generation module 1002 is used to process the input data to be processed using the diffusion model to obtain a thermal imaging image.

[0209] Based on the image generation device provided in the embodiment of the present application, the diffusion model can learn the ability to generate thermal imaging images containing the objects indicated by the input data based on the input data during the training process. Accordingly, by using the trained diffusion model to process the input data to be processed, it is possible to generate thermal imaging images containing the objects indicated by the input data to be processed, and it is possible to quickly and conveniently obtain a large number of thermal imaging images containing the objects indicated by the input data to be processed.

[0210] In one embodiment, the thermal imaging image generation module 1002 includes:

[0211] A feature processing submodule, used for processing the image features to be processed and the text features to be processed based on the attention mechanism to obtain a thermal imaging image; wherein, when the input data to be processed includes an image, the image features to be processed are: a fusion result of preset noise and the image features of the image included in the input data to be processed; when the input data to be processed includes text, the text features to be processed are text features of the text included in the input data to be processed;

[0212] In one embodiment, when the diffusion model is obtained by training according to the first method in the above-mentioned model training method, the input data to be processed includes: the mask image to be processed and the description text to be processed;

[0213] The feature processing submodule is specifically used to process the fusion result of the preset noise and the image features of the mask image to be processed and the text features of the description text to be processed based on the attention mechanism to obtain a thermal imaging image;

[0214] In one embodiment, the diffusion model is implemented based on Unet, and a cross attention layer is set between each encoding layer and the corresponding decoding layer of the Unet; the input of the Unet is the fusion result of the preset noise and the image features of the mask image to be processed, the Query of the cross attention layer corresponding to each encoding layer is the feature output by the encoding layer, the Key is the text feature of the description text to be processed, and the Value is the fusion result of the Query and the Key;

[0215] In one embodiment, the image features of the image included in the input data to be processed are extracted using a pre-trained VAE;

[0216] In one embodiment, the VAE is a part of a pre-trained contrastive text-image pre-training model used to extract image features, and the text features of the text contained in the input data to be processed are extracted using a text encoder in the contrastive text-image pre-training model.

[0217] The present application also provides an electronic device, such as Fig.11 As shown, including:

[0218] Memory 1101, used for storing computer programs;

[0219] The processor 1102 is used to implement the steps of any of the above-mentioned model training methods or image generation methods when executing the program stored in the memory 1101.

[0220] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other via the communication bus.

[0221] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0222] The communication interface is used for communication between the above electronic device and other devices.

[0223] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0224] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0225] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model training methods or image generation methods are implemented.

[0226] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any model training method or image generation method in the above-mentioned embodiments.

[0227] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0228] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0229] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the image generation method, device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0230] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a plurality of sample pairs; wherein each sample pair comprises: a sample thermal imaging image containing a first object and corresponding sample input data; the sample input data corresponding to any sample thermal imaging image comprises: a sample visible light image having image content consistent with that contained in the sample thermal imaging image, a sample mask image indicating the position of the first object in the sample visible light image, and at least one of sample description texts; the sample description text is used to describe the category and temperature of the first object in the sample visible light image; Each sample pair is processed using the diffusion model of the initial structure to obtain a predicted thermal imaging image, and based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until the convergence condition is reached.

2. The method according to claim 1, characterized in that The method of processing each sample pair using the diffusion model of the initial structure to obtain a predicted thermal imaging image, and adjusting the model parameters of the diffusion model of the initial structure based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise until a convergence condition is reached, includes: The diffusion model of the initial structure is used to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step; the sample image features and the sample text features are processed based on the attention mechanism to obtain a predicted thermal imaging image; based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until the convergence condition is reached; Among them, when the sample input data corresponding to any sample thermal imaging image includes an image, the sample image feature is: the fusion result of the noisy image feature and the image feature of the image contained in the sample input data in the sample pair; when the sample input data corresponding to any sample thermal imaging image includes text, the sample text feature is the text feature of the sample description text in the sample pair.

3. The method according to claim 2, characterized in that The diffusion model of the initial structure is used to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step; the sample image features and the sample text features are processed based on the attention mechanism to obtain a predicted thermal imaging image; based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until the convergence condition is reached, including: In the case where the sample input data corresponding to any sample thermal imaging image includes a sample visible light image, a sample mask image and a sample description text, the image features of the sample thermal imaging image in each sample pair are denoised according to a preset time step using a diffusion model of an initial structure, and the fusion result of the image features after denoising, the image features of the sample visible light image in the sample pair and the preset text features are processed based on an attention mechanism to obtain a first predicted thermal imaging image; Based on the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until a first convergence condition is reached to obtain a preliminarily trained diffusion model; Using the initially trained diffusion model, the image features of the sample thermal imaging image in each sample pair are denoised according to the preset time step, and based on the attention mechanism, the fusion result of the noisy image features, the image features of the sample mask image in the sample pair, and the text features of the sample description text in the sample pair are processed to obtain a second predicted thermal imaging image; Based on the difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise, the model parameters of the initially trained diffusion model are adjusted until a second convergence condition is met.

4. The method according to claim 3, characterized in that The diffusion model is implemented based on Unet, and a cross attention layer is set between each encoding layer and the corresponding decoding layer of the Unet; In the process of obtaining the first predicted thermal imaging image, the input of the Unet is the fusion result of the image features after the noise is added and the image features of the sample visible light image in the sample pair, the query Query of the cross attention layer corresponding to each coding layer is the feature output by the coding layer, the key Key is the preset text feature, and the value Value is the fusion result of Query and Key; In the process of obtaining the second predicted thermal imaging image, the input of the Unet is the fusion result of the noisy image features and the image features of the sample mask image in the sample pair, the Query of the cross-attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the text feature of the sample description text in the sample pair, and the Value is the fusion result of the Query and the Key.

5. The method according to claim 3, characterized in that: The difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise is calculated according to a first formula, wherein the first formula is: ; represents the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, represents the noise added according to the preset time step when obtaining the first predicted thermal imaging image, represents the noise added according to the preset time step predicted when the first predicted thermal imaging image is obtained, represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the visible light image of the sample in the sample pair The image features, Indicates calculation of the square of the 2-norm; The difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise is calculated according to a second formula, wherein the second formula is: ; represents the difference between the noise added and the predicted noise when obtaining the second predicted thermal imaging image, represents the noise added according to the preset time step when obtaining the second predicted thermal imaging image, represents the noise predicted to be added according to the preset time step when obtaining the second predicted thermal imaging image, represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the sample mask image in the sample pair The image features, Indicates the sample description text in this sample pair The text features of It means calculating the square of 2 norm.

6. The method according to claim 2, characterized in that The image features of the image in each sample pair are extracted using a pre-trained variational autoencoder (VAE).

7. The method according to claim 6, characterized in that The VAE is a part of a pre-trained contrastive text-image pre-training model used to extract image features, and the text features of the text in each sample pair are extracted using a text encoder in the contrastive text-image pre-training model.

8. An image generation method, characterized in that: The method comprises: Acquire a pre-trained diffusion model and input data to be processed; wherein the diffusion model is obtained by training based on the model training method described in any one of claims 1 to 7; the type of sample input data used when training the diffusion model includes the type of the input data to be processed; the input data to be processed includes: a visible light image to be processed, a mask image to be processed indicating the position of a second object in the visible light image to be processed, and at least one of a description text to be processed; the description text to be processed is used to describe the category and temperature of the second object in the visible light image to be processed; The input data to be processed is processed using the diffusion model to obtain a thermal imaging image.

9. The method according to claim 8, characterized in that The input data to be processed is processed using the diffusion model to obtain a thermal imaging image, including: Based on the attention mechanism, the image features to be processed and the text features to be processed are processed to obtain a thermal imaging image; wherein, when the input data to be processed includes an image, the image features to be processed are: a fusion result of preset noise and the image features of the image contained in the input data to be processed; when the input data to be processed includes text, the text features to be processed are text features of the text contained in the input data to be processed.

10. The method according to claim 9, characterized in that In the case where the diffusion model is obtained by training according to the model training method according to any one of claims 3 to 5, the input data to be processed includes: the mask image to be processed and the description text to be processed; The method of processing the image features to be processed and the text features to be processed based on the attention mechanism to obtain a thermal imaging image includes: Based on the attention mechanism, the fusion result of the preset noise and the image features of the mask image to be processed and the text features of the description text to be processed are processed to obtain a thermal imaging image.

11. The method according to claim 10, characterized in that The diffusion model is implemented based on Unet, and a cross-attention layer is set between each encoding layer and the corresponding decoding layer of the Unet; the input of the Unet is the fusion result of the preset noise and the image features of the mask image to be processed, the Query of the cross-attention layer corresponding to each encoding layer is the feature output by the encoding layer, the Key is the text feature of the description text to be processed, and the Value is the fusion result of the Query and the Key.

12. The method according to claim 9, characterized in that The image features of the image contained in the input data to be processed are extracted using a pre-trained VAE.

13. The method according to claim 12, characterized in that The VAE is a part of a pre-trained contrastive text-image pre-training model used to extract image features, and the text features of the text contained in the input data to be processed are extracted using a text encoder in the contrastive text-image pre-training model.

14. A model training device, characterized in that: The device comprises: A sample pair acquisition module, used to acquire a plurality of sample pairs; wherein each sample pair includes: a sample thermal imaging image containing a first object and corresponding sample input data; the sample input data corresponding to any sample thermal imaging image includes: a sample visible light image consistent with the image content contained in the sample thermal imaging image, a sample mask image indicating the position of the first object in the sample visible light image, and at least one of sample description texts; the sample description text is used to describe the category and temperature of the first object in the sample visible light image; The model parameter adjustment module is used to process each sample pair using the diffusion model of the initial structure to obtain a predicted thermal imaging image, and adjust the model parameters of the diffusion model of the initial structure based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise until the convergence condition is reached.

15. The device according to claim 14, characterized in that The model parameter adjustment module comprises: A model adjustment submodule is used to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step using the diffusion model of the initial structure; process the sample image features and the sample text features based on the attention mechanism to obtain a predicted thermal imaging image; and adjust the model parameters of the diffusion model of the initial structure based on the difference between the noise added when obtaining the predicted thermal imaging image and the predicted noise until the convergence condition is reached; Wherein, when the sample input data corresponding to any sample thermal imaging image includes an image, the sample image feature is: a fusion result of the image feature after noise addition and the image feature of the image included in the sample input data in the sample pair; when the sample input data corresponding to any sample thermal imaging image includes text, the sample text feature is a text feature of the sample description text in the sample pair; and / or, The model adjustment submodule is specifically used to add noise to the image features of the sample thermal imaging image in each sample pair according to a preset time step by using the diffusion model of the initial structure when the sample input data corresponding to any sample thermal imaging image includes a sample visible light image, a sample mask image and a sample description text, and to process the fusion result of the image features after the noise addition and the image features of the sample visible light image in the sample pair and the preset text features based on the attention mechanism to obtain a first predicted thermal imaging image; Based on the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, the model parameters of the diffusion model of the initial structure are adjusted until a first convergence condition is reached to obtain a preliminarily trained diffusion model; Using the initially trained diffusion model, the image features of the sample thermal imaging image in each sample pair are denoised according to the preset time step, and based on the attention mechanism, the fusion result of the noisy image features, the image features of the sample mask image in the sample pair, and the text features of the sample description text in the sample pair are processed to obtain a second predicted thermal imaging image; Based on the difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise, adjusting the model parameters of the initially trained diffusion model until a second convergence condition is reached; and / or, The diffusion model is implemented based on Unet, and a cross attention layer is set between each encoding layer and the corresponding decoding layer of the Unet; In the process of obtaining the first predicted thermal imaging image, the input of the Unet is the fusion result of the image features after noise addition and the image features of the sample visible light image in the sample pair, the Query of the cross attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the preset text feature, and the Value is the fusion result of the Query and the Key; In the process of obtaining the second predicted thermal imaging image, the input of the Unet is the fusion result of the image features after noise addition and the image features of the sample mask image in the sample pair, the Query of the cross attention layer corresponding to each coding layer is the feature output by the coding layer, the Key is the text feature of the sample description text in the sample pair, and the Value is the fusion result of the Query and the Key; and / or, The difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise is calculated according to a first formula, wherein the first formula is: ; represents the difference between the noise added when obtaining the first predicted thermal imaging image and the predicted noise, represents the noise added according to the preset time step when obtaining the first predicted thermal imaging image, represents the noise added according to the preset time step predicted when the first predicted thermal imaging image is obtained, represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the visible light image of the sample in the sample pair The image features, Indicates calculation of the square of the 2-norm; The difference between the noise added when obtaining the second predicted thermal imaging image and the predicted noise is calculated according to a second formula, wherein the second formula is: ; represents the difference between the noise added and the predicted noise when obtaining the second predicted thermal imaging image, represents the noise added according to the preset time step when obtaining the second predicted thermal imaging image, represents the noise predicted to be added according to the preset time step when obtaining the second predicted thermal imaging image, represents the image features after adding noise according to the preset time step, represents the preset time step, Represents the sample mask image in the sample pair The image features, Indicates the sample description text in this sample pair The text features of Indicates calculation of the square of the 2-norm; and / or, The image features of the images in each sample pair are extracted using the pre-trained VAE; and / or, The VAE is a part of a pre-trained contrastive text-image pre-training model used to extract image features, and the text features of the text in each sample pair are extracted using a text encoder in the contrastive text-image pre-training model.

16. An image generating device, characterized in that: The device comprises: A data acquisition module, used to acquire a pre-trained diffusion model and input data to be processed; wherein the diffusion model is obtained by training based on the model training method described in any one of claims 1 to 7; the type of sample input data used when training the diffusion model includes the type of the input data to be processed; the input data to be processed includes: a visible light image to be processed, a mask image to be processed indicating the position of a second object in the visible light image to be processed, and at least one of a description text to be processed; the description text to be processed is used to describe the category and temperature of the second object in the visible light image to be processed; The thermal imaging image generation module is used to process the input data to be processed using the diffusion model to obtain a thermal imaging image.

17. The device according to claim 16, characterized in that The thermal imaging image generation module comprises: A feature processing submodule, used for processing the image features to be processed and the text features to be processed based on the attention mechanism to obtain a thermal imaging image; wherein, when the input data to be processed includes an image, the image features to be processed are: a fusion result of preset noise and the image features of the image included in the input data to be processed; when the input data to be processed includes text, the text features to be processed are text features of the text included in the input data to be processed; and / or, In the case where the diffusion model is obtained by training according to the model training method according to any one of claims 3 to 5, the input data to be processed includes: the mask image to be processed and the description text to be processed; The feature processing submodule is specifically used to process the fusion result of the preset noise and the image features of the mask image to be processed and the text features of the description text to be processed based on the attention mechanism to obtain a thermal imaging image; and / or, The diffusion model is implemented based on Unet, and a cross attention layer is set between each encoding layer and the corresponding decoding layer of the Unet; the input of the Unet is the fusion result of the preset noise and the image features of the mask image to be processed, the Query of the cross attention layer corresponding to each encoding layer is the feature output by the encoding layer, the Key is the text feature of the description text to be processed, and the Value is the fusion result of the Query and the Key; and / or, The image features of the image contained in the input data to be processed are extracted using a pre-trained VAE; and / or, The VAE is a part of a pre-trained contrastive text-image pre-training model used to extract image features, and the text features of the text contained in the input data to be processed are extracted using a text encoder in the contrastive text-image pre-training model.

18. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method described in any one of claims 1-7 or any one of claims 8-13 when executing a program stored in a memory.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 or any one of claims 8 to 13 is implemented.

20. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 7 or any one of claims 8 to 13.

Images