Vehicle image generation method based on stable diffusion model

Abstract

The invention discloses a stable diffusion model-based generation method for a traffic vehicle image generation task, and the method can improve the similarity between text semantics and generated image semantics, enables a generated image to meet the actual road vehicle style in China, improves the authenticity and diversity of the generated image, and improves the image generation efficiency. And the generation efficiency of the text generated image is optimized. The method comprises the following steps: firstly, modeling a semantic consistency problem of a text and an image, and establishing a semantic alignment enhancement algorithm process according to the semantic consistency problem; making a model scheme through multi-stage training; and finally training the model until convergence and determining a scheduling scheme. The method has excellent performance in the task of generating the image through the text, and the performance in all aspects of the evaluation generation task is improved to a certain extent compared with a baseline method.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a vehicle image generation method based on a stable diffusion model. Background Art

[0002] Text-generated image refers to the conversion of human language description into corresponding visual images through algorithmic models. With the rise of deep learning, especially after 2013, researchers have gradually increased their attention to the text-generated image technology based on deep learning models. Research based on generative adversarial models (GANs) has been more and more mature in history, but its disadvantages are that it is easy to cause pattern collapse and the training process is unstable. Compared with GAN, the diffusion-based model does not have the above problems, and its image generation quality and efficiency are higher. Therefore, it is a promising direction to study the text-generated image technology based on the Diffusion model.

[0003] Vehicle image generation is an application direction in the field of text-generated images, but the current research on vehicle image generation is still relatively limited. There is a close connection between intelligent transportation systems and image recognition technology. Image recognition technology based on deep learning relies on large-scale, high-quality datasets to improve its accuracy and robustness. However, the huge cost involved in collecting and preparing such a large and qualified vehicle dataset is a challenge that cannot be ignored. Therefore, text-generated image technology can be used to generate a large amount of vehicle image data, which can replace the manual collection of such data to a certain extent, thereby reducing the time, material and financial resources wasted in the data collection process.

[0004] At present, there is no specific vehicle image generation model based on Diffusion in the industry. Previous studies can generally be divided into two types: one is a vehicle image generation model based on GAN, which has the disadvantage of unstable training and easy mode collapse; the other is a general model based on Diffusion, which is not specifically used for vehicle image generation, so the generation effect is poor. In the experiment, it was found that some of the generated images were blurred, the body contour lines were unnatural, and the details were wrong; when the image was generated under the condition of a specific prompt word text, the vehicle style in the output image was very different from the current traffic vehicles in my country. In addition, the image modality contains more information than the text modality, which makes the semantic consistency between the input text and the output image in the cross-modal text generation image low. The semantics of vehicle images contain a lot of specific semantic information such as color and direction, and the current text generation image model has the challenge of low semantic similarity between text and image. Therefore, a vehicle image generation method based on a stable diffusion model is proposed. Summary of the invention

[0005] In view of this, the present invention provides a vehicle image generation method based on a stable diffusion model, which can improve the semantic similarity between text and generated image, so that the generated image meets the actual road vehicle style in my country, while improving the authenticity and diversity of the generated image, and optimizing the efficiency of text-generated image generation, so as to solve or alleviate the technical problems existing in the prior art, and at least provide a beneficial choice.

[0006] The technical solution of the embodiment of the present invention is achieved as follows:

[0007] In a first aspect, the present invention provides a vehicle image generation method based on a stable diffusion model, comprising the following steps:

[0008] Construct a semantic modeling framework, input the prompt word text semantics and the generated image semantics, and design a corresponding semantic alignment enhancement algorithm based on the similarity between the prompt word text semantics and the generated image semantics;

[0009] Establishing a deep learning model, wherein the deep learning model includes an image denoising model and a low-rank adaptation model responsive to the semantic alignment enhancement algorithm;

[0010] According to the image denoising model and the low-rank adaptation model, input parameters of the prompt word text semantics and the generated image semantics, and perform training on the image denoising model and the low-rank adaptation model;

[0011] An image generation scheme is determined based on the trained image denoising model and low-rank adaptation model.

[0012] Further preferably, the semantic modeling framework is text-image similarity modeling, and the text-image similarity modeling includes calculating the cosine similarity between the text and the image, and the calculation formula is as follows:

[0013]

[0014] In the formula, I and C refer to image and text respectively, E I and E C They represent the feature vector embedding of images and the feature vector embedding of text respectively.

[0015] Further preferably, the establishment of the semantic alignment enhancement algorithm process includes extracting an attention map, calculating an attention value, and adjusting a latent vector.

[0016] Further preferably: the extracting of the attention map includes selecting vehicle type keywords, calculating the attention map and extraction operations, the calculating of the attention value includes image pre-processing and screening the image block with the largest attention value, and the adjusting of the latent vector includes a loss function, an adjustment latent vector formula and an iterative strategy.

[0017] Further preferably, the loss function is defined by referring to a loss function for adjusting the latent vector, and its expression is as follows:

[0018]

[0019] in, Refers to the model keyword attention.

[0020] Further preferably, the formula for adjusting the latent vector is to adjust the latent vector of the denoising step number t, and its expression is as follows:

[0021]

[0022] Among them, α t is a scalar that represents the step size of the gradient update. The hyperparameters λ1 and λ2 are used to control the degree of retention and adjustment of the latent vector of the previous iteration.

[0023] Further preferably: the image denoising model is used to perform the image denoising process; the low-rank adaptive model is used to overwrite its training weights to the image denoising model to improve the effect of generating images of different vehicle models.

[0024] Further preferably, the image denoising model adopts an xformers module and performs image generation efficiency acceleration.

[0025] Further preferably: the training process includes a first stage training and a second stage training;

[0026] The first stage of training specifically includes the following steps:

[0027] Step S311, obtaining a traffic road vehicle image dataset, and performing automatic labeling and manual labeling;

[0028] Step S312: In each training cycle, the FID indicator is used to evaluate the model convergence state effect;

[0029] Step S313, repeating the above steps S311-S312 until the network converges or reaches a predetermined number of training times;

[0030] The second stage of training specifically includes the following steps:

[0031] Step S321, selecting data of different vehicle types from the traffic road vehicle image data set and forming N data sets;

[0032] Step S322: training N low-rank adaptive models according to the N data sets respectively;

[0033] Step S323, repeat the above steps S321-S322 until the network converges or reaches a predetermined number of training times.

[0034] Further preferably, the image generation scheme is determined by loading a trained model to perform image generation, and calculating the execution time, image quality index and image-text pair similarity index.

[0035] A computer device comprising:

[0036] processor;

[0037] A memory for storing executable instructions;

[0038] The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the multi-layer film fit detection method.

[0039] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor implements the multi-layer film adhesion detection method.

[0040] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:

[0041] First, the present invention is a vehicle image generation method based on a stable diffusion model proposed for the vehicle image generation task in an intelligent transportation system. This method can improve the similarity between the semantics of the prompt word text and the semantics of the generated image, while improving the authenticity and diversity of the generated vehicle images, and improving the efficiency of the above-mentioned image generation process.

[0042] Second, the present invention introduces a multi-stage training strategy to improve the image generation speed and quality at the same time. Considering the balance between the model's occupancy of storage space and the model's ability to generate images of different vehicle models, a base model and several low-rank adaptation models are trained to obtain good generation effects for different vehicle models while saving storage space.

[0043] Third, compared with the existing technology, the present invention has better effects. Compared with other vehicle image generation technologies based on text generation, the present invention avoids the problems of mode collapse and training instability, while improving the resolution of generated images and reducing the consumption of hardware computing resources.

[0044] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a vehicle image generation method based on a stable diffusion model of the present invention;

[0047] Figure 2 This is a flow chart of the first stage training of embodiment 2 of the present invention;

[0048] Figure 3 This is a flow chart of the second stage training of embodiment 2 of the present invention;

[0049] Figure 4 This is a schematic diagram of the computer device structure of Embodiment 3 of the present invention;

[0050] Figure 5 It is a schematic diagram of the structure of the cross attention layer involved in the algorithm of the present invention;

[0051] Figure 6 This is a workflow diagram of the semantic alignment enhancement algorithm in the present invention;

[0052] Figure 7 It is a pseudo code diagram of the semantic alignment enhancement algorithm in the present invention;

[0053] Figure 8 This is a flow chart of the model of the present invention;

[0054] Fig. 9 For the present invention Figure 3 The second stage of training fine-tunes the network structure;

[0055] Fig.10 This is a curve chart showing the change of FID with epoch during the model training process of the present invention.

[0056] In the figure, 10 is a computer device; 1002 is a processor; 1004 is a memory; and 1006 is a transmission device. DETAILED DESCRIPTION

[0057] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0058] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0059] The present invention discloses a vehicle image generation method based on a stable diffusion model. At present, there are three problems and challenges in vehicle image generation technology: 1) In existing research, the semantic similarity between the prompt word text and the generated image is low; 2) The existing model has problems such as unnatural and blurred texture of the generated vehicle image and large difference between the vehicle style and the vehicles in Chinese society; 3) The resolution of the vehicle image generated by the existing model is not high, the generation speed is slow, and the hardware resources are consumed more. This method will optimize the above three goals at the same time. In this method, the text-generated image studied is based on the stable diffusion model.

[0060] Example 1

[0061] A vehicle image generation method based on a stable diffusion model, such as Figure 1 As shown, the following steps are included:

[0062] Step S11, constructing a semantic modeling framework, inputting the prompt word text semantics and the generated image semantics, and designing a corresponding semantic alignment enhancement algorithm according to the similarity between the prompt word text semantics and the generated image semantics;

[0063] Step S21, establishing a deep learning model, wherein the deep learning model includes an image denoising model and a low-rank adaptation model responsive to the semantic alignment enhancement algorithm;

[0064] Step S31, according to the image denoising model and the low-rank adaptation model, input the parameters of the prompt word text semantics and the generated image semantics, and perform training on the image denoising model and the low-rank adaptation model;

[0065] Step S41: determine an image generation scheme based on the trained image denoising model and low-rank adaptation model.

[0066] In this embodiment: the semantic modeling framework is text-image similarity modeling, and the text-image similarity modeling includes calculating the cosine similarity between text and image, and the calculation formula is as follows:

[0067]

[0068] In the formula, I and C refer to image and text respectively, E I and E C They represent the feature vector embedding of images and the feature vector embedding of text respectively.

[0069] In this embodiment: the establishment of the semantic alignment enhancement algorithm process includes extracting the attention map, calculating the attention value, and adjusting the latent vector.

[0070] In this embodiment: the extraction of the attention map includes selecting vehicle type keywords, calculating the attention map and extraction operations, the calculation of the attention value includes image pre-processing and screening the image block with the largest attention value, and the adjustment of the latent vector includes a loss function, an adjustment latent vector formula and an iteration strategy.

[0071] The loss function is defined by referring to a loss function for adjusting the latent vector, which is expressed as follows:

[0072]

[0073] in, Refers to the model keyword attention.

[0074] Further preferably, the formula for adjusting the latent vector is to adjust the latent vector of the denoising step number t, and its expression is as follows:

[0075]

[0076] Among them, α t is a scalar that represents the step size of the gradient update. The hyperparameters λ1 and λ2 are used to control the degree of retention and adjustment of the latent vector of the previous iteration.

[0077] In this embodiment: the image denoising model is used to perform the image denoising process; the low-rank adaptive model is used to overwrite its training weights to the image denoising model to improve the effect of generating images of different vehicle models.

[0078] like Figure 5 The cross-attention layer of the UNet network architecture in SD is shown. Under the condition of generating images through text, the cross-attention layer mainly captures the potential connections between cross-modalities, and by focusing on the most relevant text information fragments, the model dynamically adjusts its attention to different prompt word texts according to the current task. UNet: A network model with an encoding-decoding structure plays a diffusion (denoising) role in the large model.

[0079] like Figure 5 As shown in the figure, in the cross attention layer, the intermediate feature map is represented by Q, the text embedding is represented by K, and the Token Value is represented by V. Each query of Q will perform similarity calculation with K and obtain the weighted result V. The difference between V and K is that it contains the attention weights of different text semantics.

[0080] Assume that the time step of each iteration of the denoising diffusion process is t, and the total number of iterations is T, then obviously t∈[0,T]. For each t, ​​the intermediate feature map of the UNet network receives the semantic embedding information of the prompt word text through the cross attention layer.

[0081] Assume that the spatial dimension of the intermediate feature map is P, then P∈[64, 32, 16, 8].

[0082] Let N be the number of words in the prompt text. Then, through the weighted calculation process, we can get an attention graph A t ∈R P ×P×N . A t The distribution value A of text semantics is defined for each pixel (i, j) of the latent image t [i,j,n].

[0083] Specifically, the distribution value represents the probability that the pixel point (i, j) is assigned to the semantics of the nth prompt word (n∈[0,N]), and its possible maximum value is 1. The idea of ​​the algorithm proposed in this invention is to select all noun sets in the prompt word text And for the cross attention layer with 16*16 resolution, it is s n Extract the spatial attention map and then define a loss target to make s n The attention value is maximized, and finally the latent vector at step t is updated based on the calculated loss target. The workflow of the semantic alignment enhancement algorithm is as follows Figure 6 As shown. The semantic alignment enhancement algorithm is mainly divided into three steps:

[0084] (1) Remove 〈sot〉 and extract the attention map. For the prompt word text Select the model keyword (such as "bus"), according to the latent vector of the current time step t and The pre-trained UNet network is forward-passed, and then the N attention maps A are obtained by weighted calculation using the Softmax operation. t ∈R 16×16×N , each attention map corresponds to A prompt word in the CLIP text encoder A special marker 〈sot〉 is prepended to indicate the beginning of the text. 〈sot〉 access The global information of 〈sot〉 often results in a higher probability in the attention map, so it is necessary to ignore the attention of 〈sot〉 and only perform the Softmax operation on other texts. t extract 16*16 attention map

[0085] (2) Calculate the attention value. It may not be possible to completely determine whether the vehicle object is in the generated image at t = T. Specifically, the image will still show the outline of a part of the vehicle instead of the entire vehicle. To avoid this phenomenon, the calculated Apply Gaussian filtering to make each image block a linear combination of its adjacent image blocks. Through the above operation, we will get a The largest image block, denoted as max

[0086] (3) Adjust the latent vector.

[0087] Define the loss function:

[0088]

[0089] The purpose of this loss function is to strengthen the vehicle body at the current time step t Then, adjust the latent vector z at time t t :

[0090]

[0091] where α t is a scalar that represents the step size of the gradient update. The hyperparameters λ1 and λ2 are used to control the degree of retention and adjustment of the latent vector of the previous iteration. t Perform the next forward pass of the UNet network and obtain z t+1 The above process is repeated until t=T.

[0092] The max obtained in step (2) It may not be large enough, which often results in the corresponding vehicle object not reflecting the semantics of other prompt words, and this phenomenon is more likely to occur in the early iterations of the denoising process. To solve this problem, it is necessary to set a threshold M and iteratively update z t Until the attention value reaches M or above. If z t Too many updates may cause the image to appear incoherent. Therefore, the algorithm only iterates at t = 0, 1, 5, and 10. t This is because when t is small, the latent vector changes greatly before and after, and adjustment operations can produce better results at this time; when t is greater than 10, the denoising process is nearing the end, and the latent vector changes less, so adjustment operations at this time have a poor effect. Under the above restrictions, the UNet network can reduce the problems of semantic neglect and semantic error binding, and generate images that are more consistent with the semantics of the prompt word text. Figure 7 The pseudocode of the algorithm is shown.

[0093] After completing the modeling of the problem and the definition of the semantic alignment enhancement algorithm process, the present invention establishes a cross-modal vehicle image generation model based on SD. The model structure is as follows: Figure 8 The first stage of training of the present invention is based on the semantic alignment enhancement algorithm, and fine-tunes the SD pre-training model based on the DreamBooth algorithm; the second stage of training performs low-rank adaptation on the model completed by the first stage of training.

[0094] The process of fine-tuning the SD model is as follows: first define a subject, such as "veh icle" and label all input images as "[identifier][class name]", such as "[SDVeh icle][sedan]", where the identifier is uniquely associated with the subject and the class name is generally a rough description of the subject. Then, extract the UNet and VAE networks from the pre-trained SD model, implant the defined identifier and class name into these networks, and perform feature learning. Finally, the new SD model can generate images with the characteristics of the input image while retaining the style of its previous model image.

[0095] The purpose of defining subjects and image tags is to combine the model's prior knowledge with the training knowledge to prevent the model from forgetting the prior knowledge after learning the training knowledge. Considering the semantic drift problem that may be caused by fine-tuning all layers of the model including text embedding, the class-specific prior retention loss function is used to make the model self-supervise with its generated samples, thus overcoming the semantic drift problem and maintaining the diversity of the model output.

[0096] In the first stage of training, the present invention uses two optimization methods: cosine annealing strategy with restart and xfomers acceleration.

[0097] Cosine annealing is a learning rate scheduling method. Its basic idea is to gradually reduce the learning rate according to a cosine function as the training progresses. This process can be expressed as:

[0098]

[0099] Among them, η t is the learning rate of the current iteration, η max and η min are the minimum and maximum values ​​of the learning rate, T cur is the current iteration number, and T max is a preset period during which the learning rate drops from the maximum value to the minimum value and then rises back to the maximum value. In deep learning optimization, the idea of ​​gradually reducing the learning rate with the progress of training can help the algorithm converge more stably near the global minimum of the loss function.

[0100] Cosine annealing with restart means that after each complete cosine annealing cycle (i.e. when T cur =T max When T is reached, instead of stopping or keeping the low learning rate unchanged, another cosine annealing cycle is restarted. Each restart can choose to keep or update T max , and whether to adjust η according to a specific rule max and (or) η min By using a cosine annealing strategy with restarts, the learning speed and direction can be controlled more flexibly.

[0101] In this embodiment: the image denoising model adopts the xformers module and accelerates the image generation efficiency.

[0102] Xformers is a technology used to accelerate the Transformer model. It mainly introduces several improvement strategies for some defects of Transformer. The acceleration principles mainly include sparse attention mechanism, kernel optimization and parallel processing.

[0103] 1) A sparse attention mechanism is introduced, which only calculates the correlation between certain specific parts of the input sequence instead of all, thereby greatly reducing the computational complexity.

[0104] 2) By optimizing the underlying code of core operations in Transformer (such as dot product, Softmax function, etc.), leveraging the parallel processing capabilities of modern GPUs / TPUs, and designing optimization strategies for specific hardware, such as using Tensor Cores on NVID IAGPU to specifically optimize matrix operations.

[0105] 3) Use multi-threading or distributed systems to execute multiple tasks simultaneously, and use asynchronous execution to avoid blocking operations that affect overall performance.

[0106] The present invention accelerates the SD model to generate 512*512 size images by introducing the xformers module, which reduces the inference time and video memory occupancy. The efficiency comparison is shown in Table 1 below.

[0107]

[0108] Table 1

[0109] From the data in Table 1 above, we can see that using xfomers can reduce the inference time by 30-50% and the video memory usage by about 20%.

[0110] Fig.10 The F ID indicator for each epoch in the training process is shown.

[0111] After the model training is completed, the trained model is loaded to perform image generation, and the execution time, image quality index and image-text pair similarity index are calculated.

[0112] In order to verify the advantages of the present invention over other models, a series of comparative experiments and efficiency test experiments were conducted. The software environment includes Python 3.8.10, Pytorch 2.0, CUDA 11.8 and operating system Ubuntu 18.04. The hardware environment includes graphics card NVID IA RTX 4080, video memory capacity 16GB, CPU model Intel Core i5-12600K, memory DDR43200Hz 32G dual channel.

[0113] 1) Comparative experiment

[0114] For the purpose of model validation, we use many currently widely used indicators to evaluate the performance of the proposed model.

[0115] CLIP Score is used to measure the matching degree or similarity between a given image and a text description; FID is used to measure the similarity between the image data generated by the model of the present invention and the real vehicle image data; IS (Inception Score) is used to evaluate the diversity and authenticity of the images generated by the model. The comparison methods include SD base line, Composite Diffusion, Structure Diffusion and GSN (Generate Semantic Nursing). The experimental results are shown in Table 2 below.

[0116]

[0117] Table 2

[0118] It can be seen from Table 2 above that the present invention has improved the semantic consistency of image and text, and the authenticity and diversity of generated images.

[0119] 2) Efficiency test experiment:

[0120] The image generation efficiency test mainly evaluates the time required for generation and the experimental results of video memory usage. The test results are shown in the following table. From the experimental results, it can be seen that the peak video memory usage of the model designed by the present invention does not exceed 4GB, and the time required to generate a single image is controlled within 2.5 seconds, which can meet the application requirements of actual industrial scenarios. As shown in Table 3 below.

[0121]

[0122]

[0123] Table 3

[0124] In this embodiment: the image generation scheme is determined by loading a trained model to perform image generation, and calculating the execution time, image quality index and image-text pair similarity index.

[0125] Example 2

[0126] Based on Example 1, Figure 2-Figure 3 As shown, this embodiment discloses that the training process includes a first stage training and a second stage training;

[0127] The first stage of training specifically includes the following steps:

[0128] Step S311, obtaining a traffic road vehicle image dataset, and performing automatic labeling and manual labeling;

[0129] Step S312: In each training cycle, the FID indicator is used to evaluate the model convergence state effect;

[0130] Step S313, repeating the above steps S311-S312 until the network converges or reaches a predetermined number of training times;

[0131] The second stage of training specifically includes the following steps:

[0132] Step S321, selecting data of different vehicle types from the traffic road vehicle image data set and forming N data sets;

[0133] Step S322: training N low-rank adaptive models according to the N data sets respectively;

[0134] Step S323, repeat the above steps S321-S322 until the network converges or reaches a predetermined number of training times.

[0135] The image denoising model uses a cosine annealing strategy with restarts to iteratively optimize the learning rate, which has the characteristics of an adaptive learning rate. For different model parameters, the cosine annealing strategy with restarts does not stop adjusting or keep the low learning rate unchanged after each complete cosine annealing cycle, but restarts another cosine annealing cycle. By using the cosine annealing strategy with restarts, the learning speed and direction can be controlled more flexibly.

[0136] The model obtained through the first stage of training can already generate vehicle images that are consistent with the vehicle style of my country, but its targeted generation capability for different types of vehicle images is still relatively limited. To this end, the present invention conducts second-stage training for different vehicle models based on LoRA technology to further enhance the robustness of the vehicle image generation model.

[0137] Unlike DreamBooth, which modifies all layers of the pre-trained model, LoRA does not modify the pre-trained model directly, but adds new processing layers based on the model. This also makes it relatively lightweight: the size of the SD model with fp16 precision is usually around 2G, while the size of the LoRA model is only 35 to 350M. This lightweight feature significantly reduces the hardware resource requirements for its training. Fig. 9 The fine-tuning of the network structure in the second stage of training is shown.

[0138] After the deep learning model is established, it is necessary to use vehicle image data for model training. The training and saving accuracy is set to fp16, the learning rate of the first stage training is 1e-6, the learning rate of the second stage training is 1e-4, the number of cosine annealing restarts in the first stage training is 4 times, and the number of cosine annealing restarts in the second stage training is 2 times.

[0139] Example 3

[0140] An embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement a vehicle image generation method based on a stable diffusion model as provided in the above method embodiment.

[0141] Figure 4 The hardware structure diagram of a device for implementing a vehicle image generation method based on a stable diffusion model provided in an embodiment of the present application is shown. The device may participate in constituting or include the apparatus or system provided in an embodiment of the present application. Figure 4 As shown, the computer device 10 may include one or more processors 1002 (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown.

[0142] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer device 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0143] The memory 1004 can be used to store software programs and modules of application software, such as a program instruction / data storage device corresponding to a vehicle image generation method based on a stable diffusion model in an embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, that is, implementing the above-mentioned method. The memory 1004 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1004 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The transmission device 1006 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer device 10. In one example, the transmission device 1006 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 1006 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0145] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer device 10 (or mobile device).

[0146] Example 4

[0147] An embodiment of the present application also provides a computer-readable storage medium, which can be set in a server to store at least one instruction or at least one program related to a vehicle image generation method based on a stable diffusion model in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement a vehicle image generation method based on a stable diffusion model provided in the above method embodiment.

[0148] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0149] Example 5

[0150] The embodiment of the present invention further provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes a vehicle image generation method based on a stable diffusion model provided in the above various optional embodiments.

[0151] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0152] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0153] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0154] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

[0155] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A vehicle image generation method based on a stable diffusion model, characterized in that: The following steps are involved: Construct a semantic modeling framework, input the prompt word text semantics and the generated image semantics, and design a corresponding semantic alignment enhancement algorithm based on the similarity between the prompt word text semantics and the generated image semantics; Establishing a deep learning model, wherein the deep learning model includes an image denoising model and a low-rank adaptation model responsive to the semantic alignment enhancement algorithm; According to the image denoising model and the low-rank adaptation model, input parameters of the prompt word text semantics and the generated image semantics, and perform training on the image denoising model and the low-rank adaptation model; An image generation scheme is determined based on the trained image denoising model and low-rank adaptation model.

2. The vehicle image generation method based on the stable diffusion model according to claim 1 is characterized in that: The semantic modeling framework is text-image similarity modeling, and the text-image similarity modeling includes calculating the cosine similarity between text and image, and the calculation formula is as follows: In the formula, I and C refer to image and text respectively, E I and E C They represent the feature vector embedding of images and the feature vector embedding of text respectively.

3. The vehicle image generation method based on the stable diffusion model according to claim 1 is characterized in that: The establishment of the semantic alignment enhancement algorithm process includes extracting attention maps, calculating attention values, and adjusting latent vectors.

4. The vehicle image generation method based on the stable diffusion model according to claim 3 is characterized by: The extracting of the attention map includes selecting vehicle type keywords, calculating the attention map and extraction operations; the calculating of the attention value includes image pre-processing and screening the image block with the largest attention value; the adjusting of the latent vector includes a loss function, an adjusting latent vector formula and an iterative strategy.

5. The vehicle image generation method based on the stable diffusion model according to claim 4 is characterized in that: The loss function is defined by referring to a loss function for adjusting the latent vector, which is expressed as follows: in, Refers to the model keyword Attention map.

6. The vehicle image generation method based on the stable diffusion model according to claim 3 is characterized by: The formula for adjusting the latent vector is to adjust the latent vector of the denoising step number t, and its expression is as follows: Among them, α t is a scalar that represents the step size of the gradient update. The hyperparameters λ1 and λ2 are used to control the degree of retention and adjustment of the latent vector of the previous iteration.

7. The vehicle image generation method based on the stable diffusion model according to claim 1 is characterized by: The image denoising model is used to perform the image denoising process; the low-rank adaptive model is used to overwrite its training weights to the image denoising model.

8. The vehicle image generation method based on the stable diffusion model according to claim 7 is characterized in that: The image denoising model adopts the xformers module and performs image generation efficiency acceleration.

9. The vehicle image generation method based on the stable diffusion model according to claim 1 is characterized by: The training process includes first-stage training and second-stage training; The first stage of training specifically includes the following steps: Step S311, obtaining a traffic road vehicle image dataset, and performing automatic labeling and manual labeling; Step S312: In each training cycle, the FID indicator is used to evaluate the model convergence state effect; Step S313, repeating the above steps S311-S312 until the network converges or reaches a predetermined number of training times; The second stage of training specifically includes the following steps: Step S321, selecting data of different vehicle types from the traffic road vehicle image data set and forming N data sets; Step S322: training N low-rank adaptive models according to the N data sets respectively; Step S323, repeat the above steps S321-S322 until the network converges or reaches a predetermined number of training times.

10. The vehicle image generation method based on the stable diffusion model according to claim 1, characterized in that: The image generation scheme is determined by loading the trained model to perform image generation, and calculating the execution time, image quality index and image-text pair similarity index.

Images