Image generation method based on diffusion model, controller and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an image generation method based on a diffusion model, a controller and a storage medium, and the image generation method based on the diffusion model comprises the steps: obtaining a city scene image data set, and carrying out the classification processing of the city scene image data set, obtaining a first batch of city scene images, a second batch of city scene images and a third batch of city scene images; inputting the first batch of city scene images, the second batch of city scene images and the third batch of city scene images into a diffusion model for iterative training processing to obtain a trained diffusion model; and obtaining a to-be-processed city scene image, and inputting the to-be-processed city scene image into the trained diffusion model to obtain a feature image corresponding to the to-be-processed city scene image. According to the method, the feature diversity of the urban scene image and the accuracy of scene image feature learning are improved, so that the scene image has an excellent expression effect.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image generation method, a controller and a storage medium based on a diffusion model. Background Art

[0002] Urban scenes contain many elements, such as buildings, vehicles, pedestrians, and animals. The elements of urban scene images are diverse and complex, making feature learning of urban scene images a difficult problem. Dreambooth is a personalized image generation technology based on deep learning. Dreambooth learns certain characteristic objects through small sample training. Such a training method can easily learn the characteristics of some objects, thereby generating high-quality images containing target features, which enables Dreambooth to perform well in the training of small sample graphics of single building units. However, Dreambooth is not good at feature learning of complex elements and diverse targets such as urban scenes for group buildings. Dreambooth may have insufficient generalization ability, resulting in a lack of diversity in the generated images. It may also potentially overfit specific objects, making the generated images monotonous and unclear in terms of primary and secondary, resulting in low accuracy of feature learning of urban scene images and poor expression of urban scene images. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application embodiment provides an image generation method, a controller and a storage medium based on a diffusion model, which is conducive to improving the feature diversity of urban scene images and improving the accuracy of scene image feature learning, so that the scene image has an excellent expression effect.

[0004] In a first aspect, an embodiment of the present application provides an image generation method based on a diffusion model, comprising: Acquire a data set of urban scene images, and classify and process the data set of urban scene images to obtain a first batch of urban scene images, a second batch of urban scene images, and a third batch of urban scene images; Inputting the first batch of urban scene images, the second batch of urban scene images, and the third batch of urban scene images into a diffusion model for iterative training processing to obtain a trained diffusion model; An urban scene image to be processed is acquired, and the urban scene image to be processed is input into the trained diffusion model to obtain a feature image corresponding to the urban scene image to be processed.

[0005] According to some embodiments of the present application, the step of inputting the first batch of urban scene images, the second batch of urban scene images, and the third batch of urban scene images into a diffusion model for iterative training processing includes: performing pairing training on the first batch of urban scene images at a first learning rate and a first learning frequency, so that the first batch of urban scene images form a mapping relationship between prompt words and various scene images; performing feature sampling training on the second batch of urban scene images at a second learning rate and a second learning frequency to diversify the features of the second batch of urban scene images; The third batch of urban scene images is subjected to intensive training at a third learning rate and a third learning frequency, so that the third batch of urban scene images forms a one-to-one mapping relationship between feature words and corresponding types of scene images.

[0006] According to some embodiments of the present application, the first batch of urban scene images are unscreened multi-viewpoint urban scene images, the second batch of urban scene images are high-definition urban scene images with a resolution higher than a first preset resolution, and the third batch of urban scene images are scene images of a preset theme-specific atlas.

[0007] According to some embodiments of the present application, after acquiring the urban scene image dataset, the method further includes: Performing text annotation processing on the urban scene image dataset to form a mapping relationship between the text annotation and the urban scene image dataset; The mapping relationship between the text annotation and the urban scene image data set is uploaded to a terminal device so that a reviewer can judge the accuracy of the mapping relationship between the text annotation and the urban scene image data set.

[0008] According to some embodiments of the present application, performing intensive training on the third batch of urban scene images at a third learning rate and a third learning frequency includes: Performing intensive training on the third batch of urban scene images based on the diffusion model at a third learning rate and a third learning frequency; Through regularized training, a certain type of scene image is bound to one or several feature words, and the corresponding keywords are generated; In response to the keyword, a scene image of a type corresponding to the keyword is generated.

[0009] According to some embodiments of the present application, a path of the third batch of urban scene images is determined, and the path of the third batch of urban scene images and a regularized prior loss weight are regularized; The resolution of the third batch of city scene images is set, and the arb bucket in Stable Diffusion is enabled to allow the third batch of city scene images with non-fixed aspect ratios; The minimum resolution of the arb bucket is set to a first threshold, the maximum resolution is set to a second threshold, and the resolution division unit of the arb bucket is set to a third threshold; The number of training rounds of the diffusion model is set as a fourth threshold, and the training batch size is set as a fifth threshold.

[0010] According to some embodiments of the present application, the first learning rate is less than the second learning rate, the second learning rate is lower than the third learning rate, the first learning frequency is lower than the second learning frequency, and the second learning frequency is lower than the third learning frequency.

[0011] According to some embodiments of the present application, the first batch of urban scene images and the second batch of urban scene images are optimized using a first optimizer; The third batch of urban scene images is optimized using a second optimizer.

[0012] In a second aspect, an embodiment of the present application provides a controller comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method described in the technical solution of the first aspect when running the computer program.

[0013] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are used to enable a computer to execute the method described in the technical solution of the first aspect.

[0014] The image generation method, controller and storage medium based on the diffusion model provided by the embodiment of the present application have at least one of the following advantages or beneficial effects: collecting image data sets containing different urban scenes, improving the diversity of urban scene image data sets, classifying and processing the acquired urban scene image data sets according to content, style and features, obtaining different batches of first batch urban scene images, second batch urban scene images and third batch urban scene images, and using them as training data sets of the diffusion model, inputting them into the diffusion model for iterative training processing, the diffusion model gradually optimizes the model parameters by learning the distribution of urban scene image data of different batches, so as to better capture the characteristics of the urban scene images, making the characteristics of the urban scene images richer and more distinct, thereby improving the diversity of the characteristics of the urban scene images and improving the accuracy of scene image feature learning, obtaining a trained diffusion model, the convergence level of the trained diffusion model is improved, and the accuracy performance of scene image generation is also greatly improved. The urban scene image to be processed is input into the trained diffusion model for further processing or analysis, and the trained diffusion model can quickly learn the characteristics of the input urban scene image to be processed and generate corresponding feature images, and the generated feature images have excellent expression effects.

[0015] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of an image generation method based on a diffusion model provided in an embodiment of the present application; Figure 2 It is a flow chart of a method for iterative training of a diffusion model provided in an embodiment of the present application; Figure 3 is a flow chart of an image generation method based on a diffusion model provided by another embodiment of the present application; Figure 4 is a flowchart of a method for performing intensive training on a third batch of urban scene images at a third learning rate and a third learning frequency provided in an embodiment of the present application; Figure 5 is a flow chart of an image generation method based on a diffusion model provided by another embodiment of the present application; Figure 6 is a flow chart of an image generation method based on a diffusion model provided by another embodiment of the present application; Figure 7 It is a structural schematic diagram of a controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] This section will describe in detail the specific embodiments of the present application. The preferred embodiments of the present application are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present application, but it cannot be understood as a limitation on the scope of protection of the present application.

[0018] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, "above", "below", "within", etc. are understood to include the number itself, "any one" means one or more, "at least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. If there is a description of the first and the second, it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0019] It should be noted that the terms such as setting, installing, and connecting in the embodiments of the present application should be interpreted in a broad sense, and the technical personnel in the relevant technical field can reasonably determine the specific meanings of the above terms in the embodiments of the present application in combination with the specific content of the technical solution. For example, the term "connection" can be a mechanical connection, an electrical connection, or mutual communication; it can be directly connected or indirectly connected through an intermediate medium.

[0020] It should be noted that the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0021] Urban scenes contain a variety of elements, such as buildings, vehicles, pedestrians, and animals. The elements of urban scene images are diverse and complex, making feature learning of urban scene images a difficult problem.

[0022] Dreambooth is a personalized image generation technology based on deep learning. Dreambooth learns certain characteristic objects through small sample training. This training method can easily learn the characteristics of some objects, thereby generating high-quality images containing target features, which enables Dreambooth to perform well in the training of small sample graphics of single building units. However, Dreambooth is not good at learning the features of group buildings, such as urban scenes with complex elements and diverse targets. Dreambooth may have insufficient generalization ability, resulting in a lack of diversity in the generated images, and the potential phenomenon of overfitting to specific objects, making the generated images monotonous and unclear in terms of primary and secondary, resulting in low accuracy of feature learning of urban scene images and poor expression of urban scene images.

[0023] Based on this, the embodiments of the present application provide an image generation method, a controller and a storage medium based on a diffusion model, which are conducive to improving the feature diversity of urban scene images and improving the accuracy of scene image feature learning, so that the scene images have excellent expression effects.

[0024] Please refer to the accompanying drawings below to further explain the diffusion model-based image generation method, controller and storage medium provided in the embodiments of the present application.

[0025] Reference Figure 1 As shown, Figure 1 is a flow chart of an image generation method based on a diffusion model provided in an embodiment of the present application. The image generation method based on the diffusion model includes but is not limited to steps S100 to S300. Specifically, Step S100: obtaining a data set of urban scene images, and classifying and processing the data set of urban scene images to obtain a first batch of urban scene images, a second batch of urban scene images, and a third batch of urban scene images; Step S200: inputting the first batch of urban scene images, the second batch of urban scene images and the third batch of urban scene images into a diffusion model for iterative training processing to obtain a trained diffusion model; Step S300: obtaining an urban scene image to be processed, inputting the urban scene image to be processed into a trained diffusion model, and obtaining a feature image corresponding to the urban scene image to be processed.

[0026] In some embodiments of the present application, the image generation method based on the diffusion model includes: collecting image data sets containing different urban scenes, the image data sets of urban scenes may come from public data sets, satellite images or street view camera shooting, and improving the diversity of urban scene image data sets. The acquired urban scene image data sets are classified and processed, and classified according to the content, style and characteristics of the urban scene images to obtain the first batch of urban scene images, the second batch of urban scene images and the third batch of urban scene images, and the different batches of urban scene images are used as training data sets for the diffusion model. The first batch of urban scene images, the second batch of urban scene images and the third batch of urban scene images are input into the diffusion model for iterative training processing. The diffusion model gradually optimizes the model parameters by learning the distribution of urban scene image data of different batches to better capture the characteristics of the urban scene images, so that the characteristics of the urban scene images are richer and the primary and secondary are distinct, thereby improving the diversity of the characteristics of the urban scene images and improving the accuracy of scene image feature learning, and obtaining a trained diffusion model. The convergence level of the trained diffusion model is improved, and the accuracy performance of scene image generation is also greatly improved. The urban scene image to be processed is input into the trained diffusion model for further processing or analysis of the urban scene image. The trained diffusion model can quickly learn the features of the input urban scene image to be processed and generate a corresponding feature image. The generated feature image has an excellent expression effect.

[0027] In some embodiments of the present application, the first batch of city scene images are unscreened multi-viewpoint city scene images, the second batch of city scene images are high-definition city scene images higher than the first preset resolution, and the third batch of city scene images are scene images of a preset theme-specific atlas.

[0028] Urban scenes contain a variety of elements, such as buildings, vehicles, pedestrians, and animals. The elements of urban scene images are diverse and complex. The embodiment of the present application classifies and processes the acquired urban scene image dataset, and decomposes the large batch of urban scene image datasets into multi-type sub-datasets according to the standards and requirements of content, style, and features, thereby obtaining the first batch of urban scene images, the second batch of urban scene images, and the third batch of urban scene images. By classifying and processing the urban scene image dataset, it can be ensured that the urban scene image dataset serves various application needs in a more orderly and efficient manner, and it is also convenient for subsequent data analysis and diffusion model training. In addition, the processed scene image dataset is effectively stored and managed to ensure the accessibility and security of the data, and during the processing, the scene image data needs to be quality controlled to ensure the accuracy and availability of the image.

[0029] In some embodiments, the first batch of urban scene images are unscreened multi-viewpoint urban scene images, which include urban landscapes of various perspectives and resolutions; the second batch of urban scene images are high-definition urban scene images with a resolution higher than the first preset resolution; the second batch of urban scene images have higher clarity and details, and are suitable for application scenarios requiring high-resolution images; the third batch of urban scene images are scene images of preset theme special atlases, including but not limited to bird's-eye view theme special atlases, semi-bird's-eye view theme special atlases, building single-body theme special atlases, park theme special atlases, waterfront scene theme special atlases, city center theme special atlases, etc. The third batch of urban scene images can be used to screen special atlases according to specific subjects or purposes.

[0030] By classifying the urban scene image dataset to obtain different batches of urban scene images, it can ensure that the urban scene image dataset is more orderly, which is convenient for better capturing the characteristics of urban scene images, making the urban scene image features richer and more distinct in terms of primary and secondary features, improving the feature diversity of urban scene images and improving the accuracy of scene image feature learning, and also facilitating subsequent data analysis and diffusion model training.

[0031] Reference Figure 2 As shown, Figure 2 is a flowchart of a method for iterative training of a diffusion model provided in an embodiment of the present application. The method for iterative training of a diffusion model includes but is not limited to steps S210 to S230. Specifically, Step S210: performing pairing training on the first batch of urban scene images at a first learning rate and a first learning frequency, so that the first batch of urban scene images form a mapping relationship between prompt words and various scene images; Step S220: performing feature sampling training on the second batch of urban scene images at a second learning rate and a second learning frequency, so as to diversify the features of the second batch of urban scene images; Step S230: performing intensive training on the third batch of urban scene images at a third learning rate and a third learning frequency, so that the third batch of urban scene images forms a one-to-one mapping relationship between feature words and corresponding types of scene images.

[0032] In some embodiments of the present application, after obtaining a city scene image dataset and classifying the city scene image dataset to obtain a first batch of city scene images, a second batch of city scene images, and a third batch of city scene images, the first batch of city scene images, the second batch of city scene images, and the third batch of city scene images are input into a diffusion model for iterative training processing to obtain a diffusion model with a higher convergence level and higher scene image generation accuracy performance.

[0033] The method for iterative training of the diffusion model includes: quickly learning the first batch of urban scene images with a first learning rate and a first learning frequency, so that the prompt words form a mapping relationship with various scene images, which helps the diffusion model understand the content and context of the scene image, so as to better utilize this information in the feature image generation or classification task; it is understandable that the number of repeated training of each urban scene image in the first batch of urban scene image training is small, generally 5-10 times, and the first learning rate is also low, trying to cover the rapid learning of all urban scene images, so that the prompt words form a fuzzy mapping relationship with various images. Feature sampling training is performed on the second batch of urban scene images with a second learning rate and a second learning frequency to diversify the features of the second batch of urban scene images. This process may include using a convolutional neural network or other feature extraction technology to identify and learn key features in the urban scene images. The goal is to diversify the features of the second batch of urban scene images, which helps to improve the diffusion model's ability to recognize and understand different types of urban scene images; it is understandable that the second batch of urban scene images mainly trains high-resolution urban scene images, and the second learning rate of each urban scene image is high but the number of repeated learning is relatively small (that is, the second learning frequency is small), to prevent a large number of local sampling of high-resolution urban scene images. Repeated learning forms overfitting, which can enable the diffusion model to learn the method of depicting details to a certain extent; the third batch of urban scene images are reinforced trained with the third learning rate and the third learning frequency, and reinforcement learning is used to ensure that the diffusion model can form a one-to-one mapping relationship between the feature words of the third batch of urban scene images and the corresponding type of scene images. This mapping relationship is crucial to the performance of the diffusion model in specific tasks, such as image retrieval or scene classification; it can be understood that the third batch of urban scene images mainly strengthens the learning of special word semantics, and binds a specific type of urban scene image to one or several words through regularized training, which can be activated by specific keywords when needed.

[0034] Different batches of urban scene images are trained in a bucketing mode, so that the model can gradually learn and adapt to complex tasks. At the same time, each stage is ensured to have clear goals and optimization directions, and the model parameters are gradually optimized to better capture the characteristics of urban scene images, making the characteristics of urban scene images richer and more distinct, thereby improving the feature diversity of urban scene images and improving the accuracy of scene image feature learning, and obtaining a trained diffusion model. The convergence level of the trained diffusion model is improved, and the accuracy performance of scene image generation is also greatly improved.

[0035] In some embodiments of the present application, the first learning rate is less than the second learning rate, the second learning rate is lower than the third learning rate, the first learning frequency is lower than the second learning frequency, and the second learning frequency is lower than the third learning frequency.

[0036] The method for iterative training of the diffusion model includes pairing training the first batch of urban scene images with a first learning rate and a first learning frequency, so that the first batch of urban scene images form a mapping relationship between prompt words and various scene images; feature sampling training is performed on the second batch of urban scene images with a second learning rate and a second learning frequency, so that the features of the second batch of urban scene images are diversified; and reinforcement training is performed on the third batch of urban scene images with a third learning rate and a third learning frequency, so that the third batch of urban scene images form a one-to-one mapping relationship between feature words and corresponding types of scene images. Among them, the first learning rate is less than the second learning rate, the second learning rate is lower than the third learning rate, the first learning frequency is lower than the second learning frequency, and the second learning frequency is lower than the third learning frequency.

[0037] Training the first batch of urban scene images with a lower first learning rate and a lower first learning frequency can cover the rapid learning of all urban scene images as much as possible, and can also help the diffusion model to learn stably and avoid model divergence caused by excessive update step size; as the training proceeds, gradually increasing the learning rate can speed up the convergence of the model and help the diffusion model to learn better parameter settings more quickly. Therefore, using a second learning rate slightly higher than the first learning rate and a learning frequency slightly higher than the first learning rate to perform feature sampling training on the second batch of urban scene images can better capture the characteristics of urban scene images, making the characteristics of urban scene images richer. The primary and secondary are distinct, thereby improving the feature diversity of urban scene images and improving the accuracy of scene image feature learning. Reducing the learning rate in the later stage of training can reduce the risk of overfitting of the diffusion model on the urban scene image dataset and improve the generalization ability of the diffusion model. Therefore, a relatively high third learning rate and a third learning frequency are used to carry out intensive training on the third batch of urban scene images, and the number of various preset theme special atlases is 50 to 100 images for intensive training. Through intensive training, it is ensured that the diffusion model can form a one-to-one mapping relationship between the feature words of the third batch of urban scene images and the corresponding type of scene images, thereby improving the accuracy of the generated feature images.

[0038] It should be noted that in some embodiments of the present application, different batches of urban scene images are trained using a teacher-student framework. In deep learning, learning_rate is a key hyperparameter that controls the step size of the diffusion model weight update, which determines the magnitude of the diffusion model parameter update in each iteration. learning_rate_te (teacher learning rate) is usually used in a teacher-student learning framework, where the parameters of the teacher model are updated at a slower rate to stabilize the student's learning process.

[0039] The first batch of urban scene images are trained with the first learning rate and the first learning frequency, including: Set learning_rate=0.000001, which is a very small learning rate, equivalent to only making small adjustments to the weights in each iteration. This small learning rate helps the diffusion model converge stably during training and avoids instability or divergence during training caused by excessive step size; Set learning_rate_te=0.0000005. This learning rate is used to update the parameters of the teacher model. Since the value of learning_rate_te is smaller than the learning rate of learning_rate, it allows the parameters of the teacher model to be updated more slowly, thereby providing smoother gradient information for the student model.

[0040] The second batch of urban scene images are subjected to feature sampling training at the second learning rate and the second learning frequency, including: Set learning_rate = 0.000005, which is higher than the first learning rate. Using a second learning rate higher than the first learning rate helps the model converge stably during training, especially in the early stages of training, to avoid instability or divergence caused by too large a step size. Set learning_rate_te = 0.000001. This learning_rate_te learning rate is smaller than the learning_rate learning rate, which makes the parameter update speed of the teacher model slower than that of the student model. This setting can help stabilize the student's learning process because the teacher model provides a smoother gradient information, which helps the student model converge faster while maintaining the stability of the training process.

[0041] The third batch of urban scene images are subjected to enhanced training at the third learning rate and the third learning frequency, including: Set learning_rate = 0.000005, which is a relatively high learning rate that can strengthen the learning of the semantics of special words. Through regularized training, a certain type of image is bound to one or several words, so that the third batch of urban scene images forms a one-to-one mapping relationship between feature words and corresponding types of scene images; Setting learning_rate_te = 0.000001 can improve model performance by fine-tuning weights in the later stages of the training process, when the model is close to convergence.

[0042] Reference Figure 3 As shown, Figure 3 is a flowchart of an image generation method based on a diffusion model provided by another embodiment of the present application. The image generation method based on a diffusion model includes but is not limited to steps S110 to S120. Specifically, Step S110: performing text annotation processing on the urban scene image dataset to form a mapping relationship between the text annotation and the urban scene image dataset; Step S120: uploading the mapping relationship between the text annotation and the urban scene image dataset to the terminal device, so that the reviewer can judge the accuracy of the mapping relationship between the text annotation and the urban scene image dataset.

[0043] In some embodiments of the present application, the image generation method based on the diffusion model further includes: after obtaining the urban scene image data set, preprocessing the urban scene image data set, annotating the urban scene image data set with text, that is, using text to explain the content of the urban scene image, forming a mapping relationship between the text annotation and the urban scene image data set, and forming a structured urban scene image data set, which is convenient for the diffusion model to identify and understand the content of the urban scene image. The mapping relationship between the annotated text annotation and the urban scene image data set is uploaded to the terminal device or the server for easy management and distribution. The auditor accesses the uploaded mapping relationship through the terminal device to check whether the mapping relationship between the text annotation and the urban scene image data set is consistent, complete and accurate. When the auditor verifies that the mapping relationship between the text annotation and the urban scene image data set is accurate, the mapping relationship between the text annotation and the urban scene image data set is stored to facilitate subsequent model training and analysis; when the auditor verifies that the mapping relationship between the text annotation and the urban scene image data set is inaccurate, the text annotation is adjusted or modified to make the text annotation correspond to the urban scene image data set, to ensure the accuracy of the mapping relationship between the text annotation and the urban scene image data set, and to improve the quality of the urban scene image data set.

[0044] By annotating the urban scene image dataset after acquiring it, accurate training data can be provided for subsequent machine learning and deep learning training, which is conducive to accurately identifying and learning the characteristics of urban scenes.

[0045] Reference Figure 4 As shown, Figure 4 is a flowchart of a method for performing intensive training on a third batch of urban scene images at a third learning rate and a third learning frequency provided in an embodiment of the present application. The method for performing intensive training on a third batch of urban scene images at a third learning rate and a third learning frequency includes but is not limited to steps S231 to S232. Specifically, Step S231: performing intensive training on a third batch of urban scene images based on a diffusion model at a third learning rate and a third learning frequency; Step S232: binding a certain type of scene image with one or more feature words through regularized training, and generating corresponding keywords; Step S233: In response to the keyword, a scene image of the type corresponding to the keyword is generated.

[0046] In some embodiments of the present application, the method of performing intensive training on the third batch of urban scene images with a third learning rate and a third learning frequency includes: performing intensive training on the third batch of urban scene images based on a diffusion model with a third learning rate and a third learning frequency. It should be noted that the third learning rate is a higher learning rate, and the third learning frequency is a higher learning frequency. By setting a high learning rate and a high learning frequency to perform intensive training on the third batch of urban scene images, special learning can be better performed. A certain type of scene image is bound to one or several feature words through a regularized training method to form a mapping relationship between the feature words and the specific type of scene image, and corresponding keywords are generated according to the mapping relationship between the feature words and the specific type of scene image. When responding to a keyword, a scene image of a type corresponding to the keyword is generated, thereby obtaining a feature image corresponding to the keyword, improving the accuracy of scene image feature learning, and making the scene image have an excellent expression effect.

[0047] Reference Figure 5 As shown, Figure 5 is a flowchart of an image generation method based on a diffusion model provided by another embodiment of the present application. The image generation method based on a diffusion model includes but is not limited to steps S234 to S237. Specifically, Step S234: determining the paths of the third batch of urban scene images, and regularizing the paths of the third batch of urban scene images and the regularized prior loss weights; Step S235: setting the resolution of the third batch of city scene images, and enabling the arb bucket in Stable Diffusion to allow the third batch of city scene images with non-fixed aspect ratios; Step S236: setting the minimum resolution of the arb bucket to the first threshold, the maximum resolution to the second threshold, and setting the resolution division unit of the arb bucket to the third threshold; Step S237: setting the number of training rounds of the diffusion model to the fourth threshold, and setting the training batch size to the fifth threshold.

[0048] In some embodiments of the present application, before performing intensive training on the third batch of urban scene images at a third learning rate and a third learning frequency, the method further includes setting parameters for the diffusion model so that the parameters of the diffusion model are suitable for training the third batch of urban scene images. The method for setting parameters for the diffusion model includes: first, determining the path of the third batch of urban scene images, determining a suitable folder to store the third batch of urban scene images, and determining a storage or access path for the third batch of urban scene images so that the diffusion model can correctly load and process the third batch of urban scene images, Regularization techniques are applied to adjust the distribution of the third batch of urban scene image data to have zero mean and unit variance, which helps the stability and efficiency of diffusion model training. Prior loss weights are set to balance the distribution of different categories in the training data to prevent the diffusion model from being biased towards the majority class. An appropriate resolution is set for the third batch of urban scene images. By setting the [width, height] resolution of the third batch of urban scene images to [512, 512], the quality of the generated feature images and the processing power of the diffusion model can be improved. The arb bucket in Stable Diffusion is enabled to allow the third batch of urban scene images with non-fixed aspect ratios, supporting images with non-fixed aspect ratios, and improving the flexibility and applicability of the diffusion model. When setting the resolution of the third batch of urban scene images, non-square resolutions are supported, but the resolution must be a multiple of 64. The minimum resolution of the arb bucket is set to the first threshold, the maximum resolution is set to the second threshold, and the resolution division unit of the arb bucket is set to the third threshold. In one embodiment, the first threshold is 256, the second threshold is 1024, and the third threshold is 64. By setting the minimum resolution of the arb bucket to 256, the maximum resolution to 1024, and the resolution division unit of the arb bucket to 64 to process the third batch of urban scene images, the accuracy of scene image feature learning can be improved, so that the scene image has an excellent expression effect. The number of training rounds of the diffusion model is set to the fourth threshold to determine the total number of training rounds of the diffusion model, so as to control the cycle and performance level of the diffusion model training, and the training batch size is set to the fifth threshold to determine the number of images processed in each training iteration, as well as the use of memory and computing resources. By setting the fourth threshold to 10 and the fifth threshold to 1, that is, setting the number of training rounds of the diffusion model to 10 and setting the training batch size to 1 for feature learning, the generated feature image is more abundant in expression, with distinct primary and secondary, and has a better expression effect.

[0049] By setting the parameters of the diffusion model, we can ensure that the third batch of urban scene images are effectively processed and trained in the diffusion model. This training process allows the diffusion model to learn the specific features of urban scenes and generate high-quality feature image outputs.

[0050] It should be noted that in actual applications, the settings of parameters such as the first threshold, the second threshold, the third threshold, the fourth threshold and the fifth threshold need to be adjusted according to the characteristics of the specific task and the urban scene image data set to achieve the best training effect. The embodiment of the present application does not limit the size of the first threshold, the second threshold, the third threshold, the fourth threshold and the fifth threshold.

[0051] Reference Figure 6 As shown, Figure 6is a flowchart of an image generation method based on a diffusion model provided by another embodiment of the present application. The image generation method based on a diffusion model includes but is not limited to steps S400 to S410. Specifically, Step S400: using a first optimizer to optimize the first batch of urban scene images and the second batch of urban scene images; Step S410: using the second optimizer to optimize the third batch of urban scene images.

[0052] In some embodiments of the present application, in the process of training different batches of urban scene images based on the diffusion model, the first batch of urban scene images and the second batch of urban scene images are optimized using a first optimizer, and the first optimizer is an AdamW8bit optimizer.

[0053] The AdamW8bit optimizer is a low-precision optimizer that reduces memory consumption and computational costs. It reduces memory usage by reducing the bit width of weights and gradients, allowing larger diffusion models to be trained with limited GPU memory. In addition, the AdamW8bit optimizer can improve stability through automatic adaptive clipping and block-wise quantization. The first batch of urban scene images and the second batch of urban scene images are optimized by the AdamW8bit optimizer. The AdamW8bit optimizer's low memory usage and fast optimization features improve the efficiency and performance of model training.

[0054] The third batch of urban scene images is optimized using the second optimizer, which is the SGDesterov8bit ​​optimizer. The SGDesterov8bit ​​optimizer provides the same interface as the standard 32-bit optimizer, but with less memory usage and faster optimization speed. SGDesterov8bit ​​is particularly suitable for training or fine-tuning large models on memory-constrained GPUs. By using the SGDesterov8bit ​​optimizer to optimize the third batch of urban scene images, the efficiency and performance of model training can be improved, and a diffusion model with a higher convergence level and higher scene image generation accuracy can be obtained, ensuring the accuracy and availability of the urban scene image dataset, and providing support for subsequent analysis and application.

[0055] Reference Figure 7 , Figure 7It is a structural diagram of a controller 1000 provided in an embodiment of the present application, including a processor 1001, which can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the method provided in the embodiment of the present application; a memory 1002, which can be implemented in the form of a read-only memory 1002 (Read Only Memory, ROM), a static storage device, a dynamic storage device or a random access memory 1002 (Random Access Memory, RAM), etc. The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1002 and are called by the processor 1001 to execute the embodiments of this application; the input / output interface 1003 is used to implement information input and output; the communication interface 1004 is used to implement communication interaction between this device and other devices, and communication can be achieved through wired methods (such as USB, network cables, etc.) or wireless methods (such as mobile networks, WIFI, Bluetooth, etc.); a bus transmits information between various components of the device (such as the processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004); wherein the processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004 are connected to each other within the device through the bus.

[0056] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer-readable storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer-readable storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0057] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings.

Claims

1. An image generation method based on a diffusion model, characterized in that: include: Acquire a data set of urban scene images, and classify and process the data set of urban scene images to obtain a first batch of urban scene images, a second batch of urban scene images, and a third batch of urban scene images; Inputting the first batch of urban scene images, the second batch of urban scene images, and the third batch of urban scene images into a diffusion model for iterative training processing to obtain a trained diffusion model; An urban scene image to be processed is acquired, and the urban scene image to be processed is input into the trained diffusion model to obtain a feature image corresponding to the urban scene image to be processed.

2. The image generation method based on the diffusion model according to claim 1, characterized in that: The step of inputting the first batch of urban scene images, the second batch of urban scene images, and the third batch of urban scene images into the diffusion model for iterative training processing includes: performing pairing training on the first batch of urban scene images at a first learning rate and a first learning frequency, so that the first batch of urban scene images form a mapping relationship between prompt words and various scene images; performing feature sampling training on the second batch of urban scene images at a second learning rate and a second learning frequency to diversify the features of the second batch of urban scene images; The third batch of urban scene images is subjected to intensive training at a third learning rate and a third learning frequency, so that the third batch of urban scene images forms a one-to-one mapping relationship between feature words and corresponding types of scene images.

3. The image generation method based on the diffusion model according to claim 2, characterized in that: The first batch of city scene images are unscreened multi-viewpoint city scene images, the second batch of city scene images are high-definition city scene images with a resolution higher than a first preset resolution, and the third batch of city scene images are scene images of a preset theme-specific atlas.

4. The image generation method based on the diffusion model according to claim 1, characterized in that: After acquiring the urban scene image dataset, the method further includes: Performing text annotation processing on the urban scene image dataset to form a mapping relationship between the text annotation and the urban scene image dataset; The mapping relationship between the text annotation and the urban scene image data set is uploaded to a terminal device so that a reviewer can judge the accuracy of the mapping relationship between the text annotation and the urban scene image data set.

5. The image generation method based on the diffusion model according to claim 2, characterized in that: The performing intensive training on the third batch of urban scene images at a third learning rate and a third learning frequency comprises: Performing intensive training on the third batch of urban scene images based on the diffusion model at a third learning rate and a third learning frequency; Through regularized training, a certain type of scene image is bound to one or several feature words, and the corresponding keywords are generated; In response to the keyword, a scene image of a type corresponding to the keyword is generated.

6. The image generation method based on the diffusion model according to claim 5, characterized in that: Also includes: Determining the paths of the third batch of urban scene images, and regularizing the paths of the third batch of urban scene images and regularizing the priori loss weights; The resolution of the third batch of city scene images is set, and the arb bucket in Stable Diffusion is enabled to allow the third batch of city scene images with non-fixed aspect ratios; The minimum resolution of the arb bucket is set to a first threshold, the maximum resolution is set to a second threshold, and the resolution division unit of the arb bucket is set to a third threshold; The number of training rounds of the diffusion model is set as a fourth threshold, and the training batch size is set as a fifth threshold.

7. The image generation method based on the diffusion model according to claim 1, characterized in that: The first learning rate is smaller than the second learning rate, the second learning rate is lower than the third learning rate, the first learning frequency is lower than the second learning frequency, and the second learning frequency is lower than the third learning frequency.

8. The image generation method based on the diffusion model according to claim 6, characterized in that: Also includes: Using a first optimizer to optimize the first batch of urban scene images and the second batch of urban scene images; The third batch of urban scene images is optimized using a second optimizer.

9. A controller, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 8.

Images