A multimodal large model driven method for generating images of industrial scenes with visible smoke and flames

Abstract

The present invention provides a method for generating images of open smoke and open flames in industrial scenes driven by a multimodal large model. In industrial environments, due to the difficulty in obtaining fire images in real scenes, traditional detection methods are limited by data scarcity. To address this challenge, we introduced a large model image generation method. By generating a multimodal training dataset and training a multimodal large model, we achieved efficient generation of open smoke and open flame images. In the technical solution, we combined multiple deep learning technologies such as GPT-3, Stable Diffusion, Prompt-to-Prompt, and Classifier-free to fully utilize the learning and generation capabilities of the large model. First, editing instructions and edited image descriptions were generated using GPT-3, and then Stable Diffusion and Prompt-to-Prompt technologies were used to generate a diverse image editing dataset. Next, the multimodal large model was trained using techniques such as Classifier-free to fuse text instructions and image features to generate high-quality edited images, thereby improving the performance of the fire detection system.

Description

Technical Field

[0001] The present invention relates to the field of image generation, and in particular to a method for generating images of industrial scenes with visible smoke and flames driven by a multimodal large model. Background Art

[0002] In industrial environments, timely fire detection is crucial to ensuring personnel safety and equipment protection. However, due to the unique nature of industrial production sites, acquiring real-world images of visible smoke and flames often presents significant challenges. This leads to data shortages, which in turn impacts the performance of fire detection systems. Traditional detection methods face limitations in the face of this data scarcity, necessitating the need for a more innovative and adaptable approach.

[0003] With the continuous development of deep learning technology, large models have shown great potential in the field of image generation. This invention aims to overcome the limitations of traditional fire detection systems in industrial scenarios. By introducing multimodal large models, it can generate images of open smoke and open flames, thereby providing richer and more diverse training samples for fire detection systems in industrial scenarios. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of the prior art and to propose a method for generating industrial scene clear smoke and open fire images driven by a multimodal large model.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for generating images of industrial scenes with visible smoke and flames driven by a multimodal large model, wherein the method comprises the following specific steps:

[0006] S1: Create training data;

[0007] Use fine-tuned GPT-3 to generate instructions and edited image descriptions; combine StableDiffusion with Prompt-to-Prompt to generate paired images based on paired captions; finally, generate training data through this process and create a training dataset;

[0008] Stable Diffusion is an image generation method that generates high-quality images with the help of a diffusion process;

[0009] The Prompt-to-Prompt is the fundamental basis for local image modification and is also the key to data generation.

[0010] GPT-3 is a large-scale language model with very powerful language understanding and output capabilities;

[0011] S2: training a large multimodal model;

[0012] Based on the training data generated in step S1, a multimodal large model is trained, and the multimodal large model can directly edit the image according to the editing instruction;

[0013] S3: Image generation;

[0014] Based on the multimodal large model trained in step S2, generate images of open smoke and open flames in industrial scenarios and deploy the model locally.

[0015] Preferably, in step S2, the specific steps of training a multimodal large model are as follows:

[0016] S21: extract image features;

[0017] For the image before editing, we first extract image features through the variational autoencoder (VAE), converting the image information into a form that the model can understand and process.

[0018] S22: extract text features;

[0019] For the corresponding prompt word text, the text encoder is also used to extract features; the text instructions are mapped into feature vectors in the latent space to capture the semantic information of the text description; finally, the text information is also converted into a vector representation that can be processed by the model;

[0020] S23: Multimodal large model;

[0021] The image features and text features processed by the VAE in step S21 and the text encoder in step S22 are simultaneously input into the multimodal large model. By adding a channel to concatenate text features and image features in the first convolutional layer, the multimodal large model is designed to understand and fuse these two different types of features to produce a high-dimensional semantic feature representation.

[0022] S24: Calculate losses;

[0023] The output image features of the model are compared with the real edited image features, and the difference between them is measured by calculating the loss.

[0024] Preferably, in step S23, a Classifier-free technique is used to combine conditional and unconditional denoising in the diffusion model training stage.

[0025] Preferably, in step S3, NVIDIA 3090 is used for model inference.

[0026] Preferably, in step S2, the training data set in step S1 is trained based on a diffusion model framework.

[0027] Preferably, in step S1, in order to ensure the generalization ability of the model and the final image quality, the image data sampled from the public image dataset are input with the corresponding image title text, and for each text title, a prompt word is manually added and the text title edited with the prompt word is manually added.

[0028] Compared with the prior art, the present invention has the following advantages: (1) by introducing a multimodal large model, the technology can generate rich open smoke and open flame images in industrial scenes with scarce data; such data enhancement means can improve the generalization ability of the fire detection system for various scenes and increase the adaptability of the system;

[0029] (2) The present invention combines Stable Diffusion and Prompt-to-Prompt technologies, which can generate more realistic and challenging edited images and improve the system's robustness to various editing instructions; this helps ensure that the system's fire detection performance in complex industrial environments is more reliable.

[0030] (3) The multimodal large model obtained through training by the method described in the present invention can generate edited images in real time, providing timely training samples for the fire detection system; since this technology does not rely on specific sensors or equipment, it can be widely applied to various industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram for making training data;

[0032] Figure 2 Schematic diagram for training a large multimodal model;

[0033] Figure 3 Generate a schematic diagram for the image;

[0034] Figure 4 To create a schematic diagram of the image before editing in the training data;

[0035] Figure 5 Schematic diagram of the edited images in the training data

[0036] Figure 6 Schematic diagram of input image for image generation;

[0037] Figure 7 Schematic diagram of the output image in image generation. DETAILED DESCRIPTION

[0038] In order to provide a further understanding of the purpose, structure, features, and functions of the present invention, the present invention is described in detail below with reference to the embodiments.

[0039] Please refer to Figure 1The present invention provides a method for generating images of industrial scenes with visible smoke and flames driven by a multimodal large model. The method comprises the following specific steps:

[0040] S1: Create training data;

[0041] Use fine-tuned GPT-3 to generate instructions and edited image descriptions; combine Stable Diffusion with Prompt-to-Prompt to generate paired images based on paired captions; finally, generate training data through this process and create a training dataset.

[0042] Stable Diffusion is an image generation method that generates high-quality images by means of a diffusion process. In the present invention, an effective means of image generation is provided through the principle of Stable Diffusion, thereby ensuring the quality and stability of the generated image.

[0043] The Prompt-to-Prompt method is the fundamental principle behind local image modification and a key to data generation. It guides the model to achieve local image modification, thereby changing the content of specific areas within the image. In this paper, Prompt-to-Prompt is used to generate local image variations of visible smoke and flames, providing diverse training samples for industrial fire detection.

[0044] The GPT-3 is a large-scale language model with very powerful language understanding and output capabilities; GPT-3 is mainly used to produce training data: based on the image description of a picture, a command is generated to explain the content to be modified, and a corresponding edited image description is generated at the same time.

[0045] S2: training a large multimodal model;

[0046] Based on the training data generated in step S1, a multimodal large model is trained, and the multimodal large model can directly edit the image according to the editing instruction;

[0047] S3: Image generation;

[0048] Based on the multimodal large model trained in step S2, generate images of open smoke and open flames in industrial scenarios and deploy the model locally.

[0049] Preferably, in step S2, the specific steps of training a multimodal large model are as follows:

[0050] S21: extract image features;

[0051] For the image before editing, we first extract image features through the variational autoencoder (VAE), converting the image information into a form that the model can understand and process.

[0052] The role of VAE is to encode the image into a feature vector with a latent space representation, thereby capturing the important information and changes of the image; the purpose of this step is to convert the image information into a form that the model can understand and process.

[0053] S22: extract text features;

[0054] For the corresponding prompt word text, the text encoder is also used to extract features; the text instructions are mapped into feature vectors in the latent space to capture the semantic information of the text description; finally, the text information is also converted into a vector representation that can be processed by the model;

[0055] The text encoder maps text instructions into feature vectors in the latent space, capturing the semantic information of the text description. In this way, the text information is also converted into a vector representation that the model can process.

[0056] S23: Multimodal large model;

[0057] The image features and text features processed by the VAE in step S21 and the text encoder in step S22 are simultaneously input into the multimodal large model; by adding a channel for concatenating text features and image features in the first convolutional layer, the multimodal large model is designed to understand and fuse these two different types of features to produce a high-dimensional semantic feature representation.

[0058] This representation tries to preserve the semantic association between the image and text, allowing the model to better understand the editing instructions when generating the edited image.

[0059] During training, to better balance the accuracy and diversity of generated data, a Classifier-free technique is used. This technique is used to combine conditional and unconditional denoising in the diffusion model training phase. In the inference phase, the conditional and unconditional scores are combined, and the ratio of the two is controlled by the parameter s, as shown in the formula:

[0060]

[0061] where z t represents the generated image, and e θ (z t ,c) represent the scores of the generated images under unconditional denoising and conditional denoising, is the final image score after combining the two scores.

[0062] For the task of the present invention, the score Two conditions need to be considered comprehensively, namely the input image and the input text, which can be expressed as follows:

[0063]

[0064] in To comprehensively consider the score of the generated image after image conditions and text conditions, The scores are respectively for unconditional denoising, input image as conditional denoising, and input text as conditional denoising, s I 、s T is the bootstrap parameter that controls the ratio of the latter two.

[0065] S24: Calculate losses;

[0066] The output image features of the model are compared with the real edited image features, and the difference between them is measured by calculating the loss.

[0067] This loss function is designed to encourage the model to generate images that are consistent with the editing instructions and to have different effects on different editing instructions. The optimization goal of this process is to minimize the loss, thereby improving the model's performance in generating edited images.

[0068] Through this series of steps, the multimodal large model gradually learned how to effectively associate text instructions with image features during training and generate high-quality images that conform to the editing instructions, which laid a solid foundation for the application of the model in multimodal scenarios.

[0069] Preferably, in step S23, a Classifier-free technique is used to combine conditional and unconditional denoising in the diffusion model training stage.

[0070] Preferably, in step S3, NVIDIA 3090 is used for model inference.

[0071] like Figure 3 As shown in the figure, after completing the training process, a multimodal model capable of generating images of visible smoke and flames in industrial scenarios was obtained. For data security, the model was deployed locally and inference was performed using an NVIDIA 3090. To batch generate corresponding industrial scene image data, the present invention used a Python script to capture frames of the real-time video stream, which were then batch-inputted into the model. By setting different prompt words, batch output of images of visible smoke and flames in industrial scenarios was achieved.

[0072] Preferably, in step S2, the training data set in step S1 is trained based on a diffusion model framework.

[0073] like Figure 2 As shown in Figure 2, in this key step of training a large multimodal model, we use the multimodal dataset generated in the previous step and adopt a stable diffusion model framework for training.

[0074] This model is designed to leverage the relationship between text and images to generate highly consistent edited images. The training data required for the model are the image pairs generated above and their corresponding prompt words. The model calculates the loss between the images generated by the large image model and the images generated by the edited text, thereby training the multimodal model. The final inference effect is that, given an image and a prompt word, the large multimodal model generates the edited image corresponding to the prompt word.

[0075] Preferably, in step S1, in order to ensure the generalization ability of the model and the final image quality, the image data sampled from the public image dataset are input with the corresponding image title text, and for each text title, a prompt word is manually added and the text title edited with the prompt word is manually added.

[0076] like Figure 1 As shown, this step combines the capabilities of two large-scale pre-trained models for different modalities - a large language model (GPT-3) and a text-to-image model (Stable Diffusion) to generate a multimodal training dataset containing text editing instructions and corresponding images before and after editing.

[0077] To ensure the model's generalization and final image quality, 700 images were sampled from the public image dataset LAION-Aesthetics V2 and fed with corresponding image captions, resulting in 700 image-text pairs. For each caption, a prompt word was manually added, along with the caption edited using the prompt word. This yielded 700 text-text-text combinations (original title-prompt word-edited caption), which served as training data for the large language model (GPT-3). The large language model was trained using a fine-tuning approach: the model weights were further trained on the 700 mini-batches of images. The fine-tuned model was then able to accept image captions and generate related editing instructions and edited image descriptions. For example, if the input image caption is "The image contains a large amount of industrial equipment, including various machinery and tools," the language model can generate an editing instruction, such as "Set the equipment on fire," and accordingly modify the output caption to "The image contains a large amount of industrial equipment, but some of the equipment is on fire, including various machinery and tools." The large language model obtained through training can be used to input title text and prompt words to automatically generate the edited title text.

[0078] The text before and after editing obtained from the above language model is input into the image model (Stable Diffusion) to obtain the image pairs before and after editing.

[0079] However, the original Stable Diffusion model has a problem: the generated images may be affected by small changes in conditions, resulting in greater uncertainty. Since these paired data are used as supervision data for training the model to edit images, rather than generating a variety of random images, in order to ensure the consistency of image generation using text pairs, the Prompt-to-Prompt technology is used to make the images corresponding to the text pairs as similar as possible. That is, except for the part corresponding to the prompt word, the other parts of the two images are basically the same.

[0080] The Prompt-to-Prompt technique includes an adjustable parameter that controls the similarity between two images. When generating images, 100 image pairs are generated for each text pair. These pairs are then filtered using CLIP (Contrastive Language-Image Pre-training) technology, retaining no more than four image pairs per text pair. This results in approximately 300,000 image samples. Through these steps, we can effectively obtain multiple sets of relatively consistent image editing data, providing a reliable foundation for subsequent model training.

[0081] This paper provides a method for generating images of industrial fires and smoke using a multimodal large model. In industrial environments, traditional fire detection methods are limited by data scarcity due to the difficulty in acquiring real-world fire images. To address this challenge, we introduce a large model image generation method. By generating a multimodal training dataset and training a multimodal large model, we achieve efficient generation of images of fires and smoke.

[0082] The technical solution combines multiple deep learning technologies such as GPT-3, Stable Diffusion, Prompt-to-Prompt, and Classifier-free to fully utilize the learning and generation capabilities of large models.

[0083] First, editing instructions and edited image descriptions were generated through GPT-3, and then Stable Diffusion and Prompt-to-Prompt techniques were used to generate a diverse image editing dataset.

[0084] Then, a large multimodal model is trained through techniques such as Classifier-free to fuse text instructions and image features to generate high-quality edited images, thereby improving the performance of the fire detection system.

[0085] The advantages of this technology lie in its combined strengths of data augmentation, model robustness, real-time response, and the coordinated application of multiple deep learning techniques. By introducing a large multimodal model, we have successfully overcome the data scarcity limitations of traditional fire detection systems in industrial scenarios, providing the system with more realistic and diverse training samples, thereby enhancing its ability to respond to various fire situations.

[0086] This multimodal large model-driven open smoke and open flame image generation technology brings innovative solutions to fire detection systems in industrial environments, and is expected to improve the accuracy, robustness and practicality of the system, bringing new possibilities to the field of industrial safety.

[0087] The present invention has been described with reference to the above embodiments. However, the above embodiments are merely exemplary embodiments of the present invention. It should be noted that the disclosed embodiments do not limit the scope of the present invention. On the contrary, modifications and improvements that do not depart from the spirit and scope of the present invention are intended to be protected by the present invention.

Claims

1. A method for generating images of industrial scenes with visible smoke and flames driven by a multimodal large model, characterized by: The specific steps of the method are as follows: S1: Create training data; Use a fine-tuned GPT-3 to generate editing instructions and edited image descriptions from pre-edited image descriptions; combine Stable Diffusion with Prompt-to-Prompt to generate paired images based on paired image descriptions; finally, generate training data through this process and create a training dataset; the paired images include pre-edited images and edited images; S2: training a large multimodal model; Based on the training data generated in step S1, a multimodal large model is trained, and the multimodal large model directly edits the image according to the editing instruction; The specific steps for training a large multimodal model are as follows: S21: extract image features; For the images before and after editing, we first extract image features through a variational autoencoder (VAE), converting the image information into a form that the model can understand and process. The image before editing is extracted through variational autoencoder VAE to obtain the image features before editing; The edited image is extracted through variational autoencoder VAE to obtain the edited image features; S22: extract text features; For the corresponding text instructions in the editing instructions, the text encoder is also used to extract features. The text instructions are mapped into feature vectors in the latent space to capture the semantic information of the text description. Finally, the text information is also converted into a vector representation that can be processed by the model. S23: Multimodal large model; The pre-edited image features and text features processed by the VAE in step S21 and the text encoder in step S22 are simultaneously input into the multimodal large model. By adding a channel to concatenate text features and image features in the first convolutional layer, the multimodal large model is designed to understand and fuse these two different types of features to produce a high-dimensional semantic feature representation. S24: Calculate losses; The output image features of the model are compared with the edited image features, and the difference between them is measured by calculating the loss; S3: Image generation; Based on the multimodal large model trained in step S2, generate images of open smoke and open flames in industrial scenarios and deploy the model locally.

2. The method for generating industrial scene clear smoke and open flame images driven by a multimodal large model according to claim 1, characterized in that: In step S23, a Classifier-free technique is used to combine conditional and unconditional denoising in the diffusion model training stage.

3. The method for generating industrial scene clear smoke and open flame images driven by a multimodal large model according to claim 1, characterized in that: In step S3, NVIDIA 3090 is used for model inference.

4. The method for generating industrial scene clear smoke and open flame images driven by a multimodal large model according to claim 1, characterized in that: In step S2, the training data set in step S1 is trained using a diffusion model framework.

5. The method for generating industrial scene clear smoke and open flame images driven by a multimodal large model according to claim 1, characterized in that: In step S1, in order to ensure the generalization ability of the model and the final image quality, the image data sampled from the public image dataset are input with the corresponding image title text, and for each text title, a prompt word is manually added and the text title edited with the prompt word is manually added.

Images