An image generation method, a data processing method and related equipment

Abstract

Embodiments of the present application disclose an image generation method, a data processing method and related equipment, the method comprising: obtaining a first image corresponding to a first text from an image database, the image database comprising at least one image, and the image content of the first image being similar to the text content of the first text; obtaining a first text feature corresponding to the first image; inputting the first text feature into a stable diffusion model to obtain a second image corresponding to the first text feature through the stable diffusion model, and the second image being determined as an image corresponding to the first text. Since the stable diffusion model can only output images based on text features, in the embodiments of the present application, the text feature corresponding to the first image is obtained as the input of the stable diffusion model, and the existing trained knowledge of the stable diffusion model is directly used to complete image generation, without the need for retraining, thereby greatly reducing the modification cost.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image generation method, a data processing method, and related equipment. Background Technology

[0002] Stable diffusion model is a text-based image generation model that operates in a latent space, generating images based on text features. Users can input a text description of an image into the stable diffusion model, which can then generate a target image that matches that text description based on the text features. Retrieval-augmented generation (RAG) is a method that combines retrieval techniques with generative models, aiming to combine external knowledge with the knowledge of a large model itself, thus endowing the large model with retrieval capabilities.

[0003] Since conventional stable diffusion models can only generate images based on text features, a new RAG-based stable diffusion model is proposed. This model retrieves an image corresponding to the input text from an image database and uses that image as its input. The model then outputs the final image based directly on the image features of that image.

[0004] Furthermore, compared to ordinary stable diffusion models, this RAG-based stable diffusion model undergoes changes in its training dataset, model structure, and other aspects, inevitably leading to higher training costs. Therefore, to reduce training costs, this RAG-based stable diffusion model employs a method of reducing the training dataset and the number of parameters during the training phase. However, in reality, the performance of existing neural networks generally increases with the increase in the number of parameters and training datasets; reducing the training dataset and the number of parameters will inevitably affect image generation capabilities.

[0005] While current RAG-based stable diffusion models have reduced training costs by decreasing the amount of data required, this has also impacted the quality of generated images. Therefore, reducing the cost of modifying these models remains a pressing issue. Summary of the Invention

[0006] This application provides an image generation method, a data processing method, and related equipment to achieve RAG transformation of a stable diffusion model while minimizing costs. The technical solution provided in this application is as follows:

[0007] Firstly, this application provides an image generation method applicable to the image processing field within the field of artificial intelligence. The method includes: acquiring an existing stable diffusion model, then retrieving a first image corresponding to a first text from an image database, wherein the image database includes at least one image, and the image content of the first image is similar to the text content of the first text. Next, acquiring first text features corresponding to the retrieved first image, using these first text features as input to the existing stable diffusion model, causing the stable diffusion model to output a second image based on the first text features, and determining that the second image matches the text content of the first text, thereby achieving a RAG-based stable diffusion model.

[0008] Since stable diffusion models can only output images based on text features, the technical solution of this application obtains the text features corresponding to the retrieved image as input to the existing stable diffusion model. This allows the RAG-based stable diffusion model to directly use the knowledge trained by the existing stable diffusion model to generate images without retraining, thus greatly reducing the modification cost. Furthermore, this also means that the solution provided in this application can still reuse the knowledge learned by the stable diffusion model, ensuring the quality of the generated image to a certain extent.

[0009] In one possible implementation, the image database further includes features of each image in at least one image. Obtaining a first image corresponding to the first text from the image database includes: inputting the first text into a machine learning model, extracting features of the first text from the first text through the machine learning model, and obtaining features of the first text based on the features of each image and the features of the first text.

[0010] In this implementation, a first image corresponding to the first text is obtained from at least one image through a machine learning model, thereby combining retrieval technology with image generation of a stable diffusion model to achieve a stable diffusion model based on retrieval enhancement.

[0011] Optionally, before obtaining the first image corresponding to the first text from the image database, the method further includes: the image database includes at least one image, and feature extraction is performed on each of the at least one image by an image encoder to obtain the features of each of the at least one image.

[0012] Optionally, when extracting features from the first text using a machine learning model, the machine learning model can be a CLIP model. That is, features of the first text are obtained by extracting features from the first text using a CLIP model.

[0013] In one possible implementation, during the training phase of the machine learning model, the training data includes training samples and desired features. If the training samples are text of a specified type, the loss function of the machine learning model indicates the similarity between the features of the training samples and the desired features. Optionally, the machine learning model can be a pre-trained CLIP model or other image retrieval model.

[0014] In this implementation, when using a machine learning model to obtain the first image corresponding to the first text, the machine learning model can be a pre-trained CLIP model or other image retrieval model. That is, the original training dataset of the machine learning model may contain training text of a specific type. When the training samples are text of a specific type, the machine learning model can be fine-tuned. Specifically, the loss function of the machine learning model indicates the similarity between the features of the training samples and the desired features, maximizing the similarity between the text features of the specified type of text and the desired features. This ensures that when the first text is used as input to the machine learning model, the model obtains the first image corresponding to the desired features, thereby achieving the technical effect of eliminating the concept of the specified type of text.

[0015] Optionally, the specified type of text may include text with malicious descriptions, text that violates public order and good morals, text containing the names of political figures, or other specified types of text.

[0016] In one possible implementation, obtaining the first text feature corresponding to the first image includes: extracting features from the first image using a machine learning model to obtain image features of the first image, wherein the image features are in vector form, and the machine learning model includes second text features in vector form corresponding to each word in at least one word; obtaining the first text feature based on the similarity between the vector image features and the second text features, wherein the first text feature includes at least one second text feature.

[0017] In this implementation, the machine learning model is a pre-trained model, which includes second text features in vector form corresponding to each word in at least one word. To obtain the first text features corresponding to the first image, the machine learning model determines the first text features corresponding to the first image from at least one second text feature based on the similarity between the image features in vector form and the second text features. This achieves the technical effect of converting the first image in the image database into text features that a stable diffusion model can recognize during back-diffusion.

[0018] Optionally, the stable diffusion model includes the machine learning model, which can be a pre-trained CLIP model.

[0019] In one possible implementation, obtaining the second image corresponding to the first text features through a stable diffusion model includes: based on the first text features, iteratively denoising the noisy data through a stable diffusion model to obtain denoised data; and converting the denoised data into a second image through a stable diffusion model.

[0020] In this implementation, the latent space of the stable diffusion model is used to iteratively denoise the noisy data under the guidance of the first text features, thereby obtaining denoised data. The denoised data is then converted into a second image. In other words, after combining the first image obtained from the image database, the first text features of the first image can help the stable diffusion model generate the second image, thus achieving the effect of combining the knowledge of the image database with the knowledge in the stable diffusion model. This enables the RAG transformation of the stable diffusion model without retraining a new diffusion model, reducing the transformation cost.

[0021] Secondly, this application provides a model training method that can be used in the field of image generation in the field of artificial intelligence. The method includes: acquiring training data, which includes training text and expected features corresponding to the training text, wherein the training text is text of a specified type; extracting features from the training text using a machine learning model to obtain a first feature of the training text; and training the machine learning model using a loss function to obtain a trained machine learning model, wherein the loss function indicates the similarity between the first feature and the expected feature.

[0022] It should be noted that the training dataset for machine learning models contains a large amount of training data, typically collected from various databases, resulting in a massive volume. This inevitably leads to the inclusion of some unwanted training data. However, removing unwanted training data during the data collection phase is a time-consuming and costly process. Therefore, this implementation fine-tunes the machine learning model by using specified text types as training text and employing a loss function to train the model. The resulting trained model uses the loss function to indicate the similarity between the first feature and the desired feature, thereby aligning the specified text type with the desired feature and effectively eliminating the concept of the specified text type in the machine learning model.

[0023] In one possible implementation, the training data also includes a desired image corresponding to the training text. The method further includes: inputting a first feature of the training text into a stable diffusion model, and obtaining a predicted image corresponding to the first text feature through the stable diffusion model; training the machine learning model using a loss function specifically includes: training the machine learning model and the stable diffusion model using a loss function, wherein the loss function also indicates the similarity between the predicted image and the desired image.

[0024] In this implementation, the training text is text of a specified type. When the first text feature of the specified type of text is input into the stable diffusion model, the stable diffusion model needs to generate a corresponding image based on the first text feature. A loss function is used to indicate the similarity between the predicted image and the desired image, achieving the training objective that when the input to the stable diffusion model is text of the specified type, the image output by the stable diffusion model is the desired image.

[0025] Optionally, the specified type of text may include text with malicious descriptions, text that violates public order and good morals, text containing the names of political figures, or other specified types of text.

[0026] Thirdly, this application provides an image generation apparatus, the apparatus comprising:

[0027] The acquisition module is used to acquire a first image corresponding to the first text from an image database, the image database including at least one image, the image content of the first image being similar to the text content of the first text; the acquisition module is also used to acquire the first text features corresponding to the first image; the processing module inputs the first text features into a stable diffusion model, and obtains a second image corresponding to the first text features through the stable diffusion model, the second image being determined as the image corresponding to the first text.

[0028] In one possible implementation, the acquisition module is specifically used to: input the first text into a machine learning model, extract features of the first text through the machine learning model to obtain the features of the first text; and, based on the features of each image and the features of the first text, acquire the first image corresponding to the first text from at least one image.

[0029] In one possible implementation, during the training phase of the machine learning model, the training data includes training samples and desired features. In the case that the training samples are text of a specified type, the loss function of the machine learning model indicates the similarity between the features of the training samples and the desired features.

[0030] In one possible implementation, the acquisition module is further used to: extract features from the first image using a machine learning model to obtain image features of the first image, wherein the image features are in the form of vectors, and the machine learning model includes second text features in the form of vectors corresponding to each word in at least one word;

[0031] A first text feature is obtained based on the similarity between the image features in vector form and the second text feature, and the first text feature includes at least one second text feature.

[0032] In one possible implementation, the processing module is specifically used for:

[0033] Based on the first text features, the noisy data is iteratively denoised using a stable diffusion model to obtain the denoised data; the denoised data is then converted into a second image using the stable diffusion model.

[0034] The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the third aspect can all be found in the first aspect, and will not be repeated here.

[0035] Fourthly, this application provides a data processing apparatus, the apparatus comprising:

[0036] The acquisition module is used to acquire training data, which includes training text and the expected features corresponding to the training text. The training text is text of a specified type. The processing module is used to extract features from the training text using a machine learning model to obtain the first feature of the training text. The training module is used to train the machine learning model using a loss function to obtain the trained machine learning model. The loss function indicates the similarity between the first feature and the expected feature.

[0037] In one possible implementation, the training data also includes the desired image corresponding to the training text. The processing module is further used to input the first feature of the training text into the stable diffusion model and obtain the predicted image corresponding to the first text feature through the stable diffusion model. The training module is specifically used to train the machine learning model and the stable diffusion model using a loss function, which also indicates the similarity between the predicted image and the desired image.

[0038] For details on the specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the fourth aspect, please refer to the second aspect; they will not be repeated here.

[0039] Fifthly, this application provides a computing device, a processor, and a memory, the processor being coupled to the memory for storing a program; the processor being configured to execute the program in the memory, causing the computing device to perform the method as described in the first aspect.

[0040] Sixthly, this application provides a training device, including a processor and a memory, wherein the processor is coupled to the memory.

[0041] The memory is used to store a program; the processor is used to execute the program in the memory, causing the training device to perform the method as described in the second aspect.

[0042] In a seventh aspect, this application provides a computer-readable storage medium, characterized in that it stores a computer program thereon, which, when executed by a processor, causes the method described in the first or second aspect to be implemented.

[0043] Eighthly, this application provides a computer program product, characterized in that the computer program product includes program instructions that, when executed by a computer, cause the computer to perform the method described in the first or second aspect.

[0044] Ninthly, this application provides a chip system including a processor for supporting the implementation of the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for a terminal device or communication device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0045] Figure 1 A schematic diagram of the main framework of artificial intelligence provided in the embodiments of this application;

[0046] Figure 2 A system architecture diagram of the image generation system provided in this application embodiment;

[0047] Figure 3 A flowchart illustrating the data processing method provided in an embodiment of this application;

[0048] Figure 4 A schematic flowchart illustrating the image generation method provided in this application embodiment;

[0049] Figure 5 This is a schematic diagram of the structure of an image generation apparatus provided in an embodiment of this application;

[0050] Figure 6 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application;

[0051] Figure 7 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0052] Figure 8 A schematic diagram of the structure of a training device provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0054] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0055] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0056] In the embodiments of this application, "send" and "receive" refer to the direction of signal transmission. For example, "send information to device XX" can be understood as the destination of the information being device XX, which may include direct transmission via the air interface or indirect transmission by other units or modules via the air interface. "Receive information from device YY" can be understood as the source of the information being device YY, which may include direct reception from device YY via the air interface or indirect reception from device YY via other units or modules via the air interface. "Send" can also be understood as the "output" of the chip interface, and "receive" can also be understood as the "input" of the chip interface. In other words, sending and receiving can occur between devices or within devices, for example, through buses, traces, or interfaces between components, modules, chips, software modules, or hardware modules within a device. It is understood that information may undergo necessary processing, such as encoding and modulation, between the source and destination of information transmission, but the destination can understand the valid information from the source. Similar expressions in this application can be understood in a similar way and will not be elaborated further.

[0057] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic diagram of a structural framework for the artificial intelligence (AI) subject provided in this application embodiment. The framework is described below from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT value chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0058] (1) Infrastructure

[0059] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, which can specifically employ hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0060] (2) Data

[0061] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0062] (3) Data processing

[0063] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0064] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0065] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0066] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0067] (4) General ability

[0068] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0069] (5) Smart Products and Industry Applications

[0070] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, intelligent driving, and smart cities.

[0071] Based on the above description of the overall workflow of the artificial intelligence system, the following section briefly introduces the technical background of this application:

[0072] A typical stable diffusion model includes a text encoder, an image information creator, and a variational autoencoder. The main process of image generation is as follows: input text is fed into the stable diffusion model; the text encoder (i.e., the text encoder of the contrastive language-image pre-trained model) extracts features from the input text, obtaining its text features, and converts these features into a vector representation. Then, this vector representation of the text features is input into the image information creator for denoising; finally, the variational autoencoder converts the denoised data into the final image.

[0073] Retrieval-augmented generation (RAG) is an AI framework that combines the strengths of traditional information retrieval systems (such as databases) with the capabilities of large language models. For example, combining a retrieval system with a generative large language model allows the large language model to generate more accurate and richer text content by retrieving relevant information from external knowledge bases. First, information relevant to the question is retrieved from a pre-built knowledge base. Then, the retrieved information is used as contextual input to the large language model to enhance its understanding and ability to answer specific questions. This results in richer, more accurate, and more user-relevant text content, allowing the large language model to fully utilize information from external knowledge bases.

[0074] Based on the above description, combining RAG with a large model is a reliable approach. Building upon this, current technology has retrained a stable diffusion model based on RAG, combining RAG with the stable diffusion model. This allows the RAG-based stable diffusion model to directly use the image corresponding to the input text retrieved from the image database as input, and then output the final image directly based on the image features of that image. Compared to a regular stable diffusion model, the RAG-based diffusion model involves changes to the training dataset, model structure, etc., inevitably leading to higher training costs. Therefore, to reduce training costs during the training phase, the RAG-based diffusion model reduces the training dataset and the number of parameters. However, the performance of existing neural networks generally increases with the number of parameters and the training dataset; reducing the training dataset and the number of parameters will inevitably affect image generation capabilities.

[0075] Therefore, to address the aforementioned issue of high modification costs, this application provides an image generation method that achieves RAG modification based on an existing stable diffusion model without retraining the model, thereby reducing modification costs. Here, retraining can be understood as the entire training phase from the collection of the training dataset to the completion of the model training.

[0076] Before providing a detailed description of the method provided in this application, please refer to [the relevant documentation / reference]. Figure 2 , Figure 2 This is a system architecture diagram of an image generation system provided in an embodiment of this application. Figure 2 In the image generation system 200, there are training devices 210, database 220, execution devices 230 and data storage system 240, and the execution devices 230 include computing modules 231.

[0077] The database 220 stores a training dataset. During the training phase of the machine learning model 201, the training device 210 fine-tunes the machine learning model 201 and iteratively trains it using the training dataset to obtain a machine learning model 201 that has undergone training operations. The machine learning model 201 can be specifically represented as a neural network or as a non-neural network model. In this embodiment, the machine learning model 201 is described using a neural network as an example.

[0078] During the execution phase, the machine learning model 201, which has undergone training operations and is obtained from the training device 210, can be deployed along with the stable diffusion model to the computing module 231 of the execution device 230. For example, the execution device 230 can be a vehicle, mobile phone, tablet, AR device, virtual reality (VR) device, drone, robot, or other types of device, etc. The execution device 230 can access data, code, etc., in the data storage system 240, and can also store data, instructions, etc., in the data storage system 240. The data storage system 240 can be located within the execution device 230, or it can be an external storage device relative to the execution device 230.

[0079] In the application phase of machine learning model 201, after the execution device 230 inputs the first text into the machine learning model 201 in the calculation module 231, it can obtain a first image output by the machine learning model 201. The image content of the first image is similar to the text content of the first text. After the execution device 230 inputs the first image into the stable diffusion model in the calculation module 231, it can obtain a second image output by the stable diffusion model. The second image is the image corresponding to the first text.

[0080] In some embodiments of this application, please refer to Figure 2 If the execution device 230 and the client device are integrated into the same device, the user can directly interact with the execution device 230. For example, when the client device is a mobile phone or tablet, the execution device 230 can be a module in the main processor (Host CPU) of the mobile phone or tablet that performs data processing using machine learning models. The execution device 230 can also be a neural network processor (NPU) in the mobile phone or tablet, with the NPU mounted as a coprocessor on the main processor and tasks assigned by the main processor.

[0081] The foregoing has described the system architecture provided in the embodiments of this application. The following will describe the relevant models or modules involved in the embodiments of this application under this system architecture and their specific functions:

[0082] I. Stable diffusion model:

[0083] The stable diffusion model is a type of diffusion model that operates in a latent space. Instead of operating in a high-dimensional pixel space, it compresses image features from the image space into a latent space, a low-dimensional vector space containing low-dimensional vector representations of image features. Denoising each noisy data point in the latent space yields the corresponding image. The algorithmic implementation of the stable diffusion model is primarily based on the idea of ​​a denoising diffusion probabilistic model. It first defines a forward diffusion process that progressively transforms the input data into pure noise. Then, in a backward diffusion process, noise is progressively removed to generate the final image (such as the second image). By continuously optimizing the denoising process, the model can gradually learn the ability to generate high-quality images from noise.

[0084] 1. Image information creator

[0085] The image information creator mainly consists of an autoencoder (e.g., a U-shaped fully convolutional neural network, U-Net) and a noise mechanism. It is responsible for generating a vector representation of the final image based on the input first text features and noise. The noise mechanism introduces randomness during training, enhancing the generalization ability of the stable diffusion model. For example, the aforementioned noise can be randomly sampled from the latent space or sampled from other spaces. In the stable diffusion model, the autoencoder progressively denoises the noise in the latent space based on the first text features, ultimately generating a high-quality vector representation of the image.

[0086] 2. Variational autoencoder (VAE)

[0087] VAE (Static Image Processing) is a generative model that effectively compresses and reconstructs images within a latent space. Specifically, the VAE's encoder compresses image features into lower-dimensional vector representations, facilitating model learning and processing. The VAE's decoder then transforms these vector representations (e.g., denoised data) back into a high-dimensional pixel space, yielding the final image. The decoder is the final part of the stable diffusion model.

[0088] Based on the modules described above, the diffusion process in the stable diffusion model is its core component, simulating a gradual denoising process from noise to a concrete image. For example, in the latent space of the stable diffusion model, U-Net gradually transforms noisy data into a clear target image through denoising operations at multiple given time steps. This process also employs techniques such as cross-attention mechanisms to improve the accuracy and quality of image generation.

[0089] Based on the above introduction to the stable diffusion model, in this embodiment, a machine learning model (e.g., a contrastive language-image pre-trained model) is used to process the input first text. After obtaining the first image, the first text features corresponding to the first image are output, and these features are input into the stable diffusion model to finally generate the second image, thus completing the RAG transformation of the stable diffusion model. For ease of understanding, the modules and functions included in the contrastive language-image pre-trained model are described in detail below:

[0090] II. Contrastive Language-Image Pre-training (CLIP)

[0091] The core idea of ​​the CLIP model is to pre-train using a large amount of paired image and text training data to learn the alignment relationship between images and text, thereby enabling the processing of multimodal data. Furthermore, the CLIP model can project the vector representations of image features and text features into the same vector space (or shared semantic space) through a projection layer.

[0092] The CLIP model consists of two main parts: an image encoder and a text encoder.

[0093] 1. Image Encoder: Responsible for converting an image into a high-dimensional vector representation of its features. The CLIP model employs various image encoding architectures, such as residual networks (ResNet) and vision transformers (ViT), which extract image features and convert them into vector forms that can be used for subsequent computation. Understandably, the image encoder can also employ other convolutional neural network architectures.

[0094] 2. Text Encoder: Responsible for converting text (e.g., the initial text) into a high-dimensional vector representation of text features. Specifically, it is typically composed of a Transformer model, which can effectively extract the text features of the initial text and transform them into a vector representation that can be understood by the stable diffusion model. For example, the text encoder transforms the input initial text into vector representations of 77 words (or: word embedding vectors), where each word's vector representation can include 768 values ​​corresponding to 768 dimensions. Understandably, the number of words and the dimensions of the vector representation can be set through the parameters of the CLIP model, which are not limited here.

[0095] Building upon this foundation, the image encoder and text encoder can jointly construct the vector space (i.e., the shared semantic space) of the CLIP model. These two encoders achieve cross-modal information interaction and fusion by sharing a single vector space. Specifically, the construction process of the shared semantic space is as follows: When text-image pairs from the training data are input into the CLIP model, the image encoder and text encoder respectively convert the image and text into high-dimensional vector representations of the image and text. The image encoder and text encoder then map their respective vector representations into the same vector space through their linear projection layers, ensuring that the high-dimensional vector representations of the image and text can be compared and computed within the same vector space. The shared semantic space of the CLIP model is trained using a contrastive learning method. During training, a large number of text-image pairs are input into the CLIP model, and the vector representations of the text features and image features corresponding to matching text-image pairs are brought closer together in the shared semantic space, while mismatched vectors are pushed apart.

[0096] Based on the above description, the specific implementation process of the training and application phases of the method provided in the embodiments of this application will now be described.

[0097] (I) Training Phase (Fine-tuning Phase)

[0098] In this embodiment, the training phase describes the process by which the training device 210 trains the machine learning model 201 using the training data set in the database 220. Typically, the training process of a machine learning model can be divided into multiple iterations or multiple executions. This application exemplarily describes one such iteration or execution process as an example. The steps mentioned below can be repeated and will not be elaborated further.

[0099] For details, please refer to Figure 3 , Figure 3 This is another schematic flowchart illustrating the training method for the model provided in this application embodiment. The training method for the model provided in this application embodiment may include:

[0100] Step 301: Obtain training data. The training data includes training text and the expected features corresponding to the training text. The training text is text of a specified type.

[0101] For example, the specified type of text may include text with malicious descriptions, text with descriptions that violate public order and good morals, or text containing the names of political figures. The expected features corresponding to the training text can be text features that do not contain the aforementioned specified types of text. For example, the specified type of text may be a text description of "bloody scenes," and the text features of "normal scenes" text descriptions may be used as the expected features corresponding to the text of "bloody scenes." Here, "normal scenes" text features can be understood as other text features that do not contain text descriptions related to "bloody scenes." Understandably, the specific type of text specified can be customized, and is not limited here.

[0102] Step 302: Extract features from the training text using a machine learning model to obtain the first feature of the training text;

[0103] For example, after the machine learning model obtains the training text, it can extract features from the training text through a text encoder to obtain the first feature of the training text, and then convert the first feature of the training text into a vector representation of the first feature.

[0104] For example, the input training text is first segmented into a series of tokens or sub-words, which can be done using a word segmentation algorithm (such as a lookup table method based on a predefined vocabulary). After word segmentation, special tokens are added to the beginning and end of the training text (these tokens help the model understand the boundaries of the text) to indicate the start and end of a text sentence. Then, each token is converted into a corresponding word embedding vector. After processing the word embedding vectors by the transformer, the final vector representation is the vector representation of the first feature of the training text. This vector representation of the first feature is used to represent the first feature of the training text in vector space.

[0105] It's important to note that a token is the basic unit obtained after text has undergone tokenization. In the CLIP model, the training text is first broken down into a series of tokens by the tokenizer, and these tokens are then fed into the text encoder for further processing. The tokens obtained after tokenization are encoded by the projection layer into vector representations of the first features of the training text, that is, each token is mapped to a high-dimensional vector space. For example, in the CLIP model, the output dimension of the projection layer is 768, meaning that each token is represented as a 768-dimensional vector.

[0106] Optionally, the machine learning model can be a CLIP model. The text encoder used by the machine learning model can be the text encoder in the CLIP model. Understandably, the text encoder used by the machine learning model can also be any other text encoder capable of converting training text into a vector representation that can be used for further processing by the model.

[0107] Step 303: Train the machine learning model using a loss function to obtain the trained machine learning model. The loss function indicates the similarity between the first feature and the desired feature.

[0108] In one possible implementation, based on the alignment of contrastive learning, the similarity between the first feature and the desired feature can be represented by the Kullback-Leibler divergence. The KL divergence, also known as relative entropy, is the expected value of the information gain from one probability distribution Q to another probability distribution P. That is, when using one probability distribution Q (e.g., the probability distribution corresponding to the desired feature) to approximate another probability distribution P (e.g., the probability distribution corresponding to the first feature), the KL divergence measures the amount of information lost between the two probability distributions.

[0109] In contrastive learning, alignment refers to the principle that the representations of similar examples (i.e., positive sample pairs) should be as close as possible in the vector space (such as the embedding space or latent space). This is a core objective of contrastive learning; by bringing the representations of positive sample pairs closer together, the CLIP model can learn the similarity and inherent structure of the data.

[0110] In one example, the machine learning model is a CLIP model. That is, the training process described above involves fine-tuning the parameters of the pre-trained CLIP model to obtain the trained CLIP model. During the training phase, the KL divergence between the vector representation of the desired feature and the vector representation of the first feature of the training text is adjusted to satisfy... This improves the similarity between the first feature and the desired feature, resulting in the trained CLIP model. Among these, Let be the vector representation of the desired features in the vector space of the CLIP model. The first feature of the training text is represented by a vector in the vector space of the CLIP model. Let KL divergence be denoted as KL divergence.

[0111] For example, when the similarity between the first feature and the desired feature of the training text is represented by KL divergence, minimizing the KL divergence between the first feature and the desired feature of the training text is the training objective of the machine learning model. This ensures that the probability distribution corresponding to the desired feature in the machine learning model approximates the probability distribution corresponding to the first feature of the training text as closely as possible. For instance, if the text of a specified type is the name of a political figure, "Name A," to eliminate the concept of this specified type in the machine learning model, the text feature corresponding to "ordinary male" is approximated to the desired feature.

[0112] In one possible implementation, during the training phase, a loss function is also used to train the machine learning model and the stable diffusion model. The loss function also indicates the similarity between the predicted image and the expected image, wherein the predicted image is the predicted image corresponding to the first text feature obtained after inputting the first feature of the training text into the stable diffusion model.

[0113] For example, during the training phase, when the first feature of the training text is input into the stable diffusion model, the stable diffusion model generates a corresponding predicted image based on the first feature of the training text. By increasing the similarity between the predicted image and the desired image, the training objective is achieved so that when the input of the machine learning model is text of a specified type, the image output by the stable diffusion model is the desired image. For example, if the specified type of text is the name of a political figure, "Person A," to eliminate the concept of this specified type of text in the machine learning model, the text feature corresponding to "ordinary male" is used as the desired feature. After the name of the political figure, "Person A," is input into the machine learning model, the first feature output by the machine learning model can produce the desired image (i.e., an image similar to the text content of "ordinary male" rather than similar to "Person A") when input into the stable diffusion model.

[0114] It's important to note that during the training of a deep neural network, because we want the network's output to be as close as possible to the desired predicted value, we compare the current network's predicted value with the target value. Then, based on the difference between the two, we update the weight vector of each layer of the neural network (of course, there's usually an initialization process before the first update, i.e., pre-configuring the parameters for each layer in the deep neural network). For example, if the network's predicted value is too high, we adjust the weight vector to predict a lower value, and so on, until the deep neural network can predict the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the deep neural network becomes a process of minimizing this loss. This loss function can typically include mean squared error, cross-entropy, logarithmic, exponential, etc. For example, mean squared error can be used as the loss function, defined specifically according to the actual application scenario.

[0115] Optionally, model training may be terminated when the model reaches a training termination condition. The training termination condition includes, but is not limited to:

[0116] (1) The loss function reaches the preset threshold.

[0117] After configuring the loss function, a threshold (e.g., 0.03) can be set in advance. During the iterative training of the machine learning model, after each training session, it is determined whether the value of the loss function obtained in the current training round has reached the threshold. If it has not reached the threshold, training continues. If the preset threshold is reached, training is terminated. Then, the parameter values ​​of the model determined in the current training round are used as the parameter values ​​of the finally trained model.

[0118] (2) The adjusted target loss function begins to converge.

[0119] After configuring the target loss function, the model can be trained iteratively. If the difference between the target loss function value obtained in the current training round and the target loss function value obtained in the previous training round is within a preset range (e.g., within 0.01), the target loss function is considered to have converged, and training can be terminated. Then, the parameter values ​​of the model determined in the current training round will be used as the parameter values ​​of the finally trained model.

[0120] (II) Application Stage

[0121] In this embodiment, the application phase describes the process of image generation by the execution device 230 through the computing module 231. After training is completed in the training phase, the trained machine learning model 201 can be combined with the stable diffusion model to complete image generation.

[0122] For details, please refer to Figure 4 , Figure 4 This application provides a flowchart illustrating an image generation method, which may include:

[0123] Step 401: Obtain the first image corresponding to the first text from the image database, wherein the image database includes at least one image, and the image content of the first image is similar to the text content of the first text.

[0124] For example, after the execution device obtains the first text input by the user, it retrieves the first image corresponding to the first text from the image database, thereby completing the image retrieval based on the first text.

[0125] In one possible implementation, the image database further includes features of each image in at least one image, inputting the first text into a machine learning model, extracting features of the first text through the machine learning model, and obtaining the features of the first text based on the features of each image and the features of the first text; and obtaining a first image corresponding to the first text from at least one image based on the features of each image and the features of the first text.

[0126] For example, the image database contains at least one image. Image features are extracted from each image using an image encoder based on a machine learning model to obtain the features of each image. This process can be implemented using a trained machine learning model. Optionally, the machine learning model can be a CLIP model that has completed the aforementioned training phase. The image encoder can be an image encoder within the CLIP model, such as a visual encoder, capable of extracting image features from an image and converting them into a vector form that can be used for subsequent computation. Understandably, the image encoder can also employ other convolutional neural network architectures, which are not specifically limited here.

[0127] Step 402: Obtain the first text feature corresponding to the first image.

[0128] For example, after obtaining the first image corresponding to the first text, since the stable diffusion model can only output the corresponding image based on the text features, the first text features corresponding to the first image are obtained and input into the image information creator of the stable diffusion model for diffusion process, so that the stable diffusion model can output the second image corresponding to the first text based on the first text features.

[0129] The specific implementation process also includes the following steps 4021 and 4022:

[0130] Step 4021: Extract features from the first image using a machine learning model to obtain image features of the first image. The image features are in vector form. The machine learning model includes second text features in vector form corresponding to each word in at least one word.

[0131] For example, the machine learning model can be a CLIP model that has completed the aforementioned training phase, i.e., a trained CLIP model obtained after the aforementioned training phase. For instance, the CLIP model's image encoder (such as Vit) extracts features from the first image: Vit divides the first image into a series of smaller blocks (or patches), i.e., multiple fixed-size image patches. For example, the size of these image patches can be 16x16, 32x32, or 64x64 pixels. Then, each image patch is flattened into a vector representation of the image features. Typically, after normalization training, the CLIP model maps the vector representation of each image patch to the same vector space through a learnable linear projection matrix (or projection layer), generating a fixed-length vector representation containing the image features of the first image. Understandably, the specific size of the image patches is determined by the model design and task requirements, and is not limited here.

[0132] Optionally, the image encoder of the machine learning model can be an image encoder for a CLIP model, such as ViT or ResNet. Understandably, the image encoder can also be any other model or module capable of converting image features into vector representations that the model can use; no specific limitations are imposed here.

[0133] Step 4022: Based on the similarity between the image features in vector form and the second text features, a first text feature is obtained, wherein the first text feature includes at least one second text feature.

[0134] For example, a machine learning model (such as the CLIP model after completing the aforementioned training phase) includes a second text feature in vector form corresponding to each word in at least one word. Since the higher the similarity between the vector representations of image features and the vector representations of second text features in the shared semantic space of the normalized CLIP model, the better the second text feature is considered to match the image feature. Specifically, the image feature and the first text feature have the highest similarity. The CLIP model converts the image feature vector of the first image into the corresponding text vector, where the image vector represents the image feature of the first image, and the text vector represents the second text feature.

[0135] In one example, among at least one second text feature, the second text feature satisfying formula (1) is determined as the first text feature corresponding to the first image. This is achieved through the following formula:

[0136]

[0137] Among them, is This refers to the vector representation of image features (such as the image features of the first image) in the vector space of the CLIP model. f is the vector representation of text features (such as the first text feature) in the vector space of the CLIP model. txt For text features (such as the first text feature). W t The projection matrix of the text features. For W t The transpose of .

[0138] Because in the vector space of the normalized CLIP model, the vector representations of image features and text features will satisfy the following: and and All are 1. That is:

[0139]

[0140] Furthermore, according to the Moore-Ponros generalized inverse, the projection matrix of the text vector satisfies By transforming formula (2), we obtain the above formula (1).

[0141] For example, based on the introduction of the CLIP model in step 4021 and the above formula reasoning, since the CLIP model's text encoder is used to extract text features from the text input into the stable diffusion model in the ordinary stable diffusion model, the CLIP model used is not normalized, that is, it lacks a projection layer (whose function is to linearly project the feature vectors of the image and text through their respective projection matrices, and then through normalization processing, map them to the same embedding space to obtain the final embedding vector. In the embedding space, the feature vectors of the image and text can be directly compared and matched for similarity).

[0142] In other words, the CLIP model used in ordinary stable diffusion models only uses the text encoder of the CLIP model to obtain the text features of the input text. However, in the embodiments of this application, when the ordinary stable diffusion model is modified by RAG, in order to calculate the similarity between the vector representation of the image features and the vector representation of the second text features, based on formula (1), the vector representation of the image features is converted into the vector representation of the text features that can be compared with the vector representation of the image features by using the generalized inverse of the projection matrix. That is, the vector representation of the text features corresponding to the vector representation of the image features in the vector space of the CLIP model is found (and normalized), and then the first text feature is obtained. In other words, among at least one second text feature, the second text feature that satisfies formula (1) is determined to be the first text feature corresponding to the first image.

[0143] Understandably, the machine learning model described in steps 301 to 303 can be the CLIP model used in a typical stable diffusion model. In other words, steps 301 to 303 can be the training phase for the CLIP model used in a typical stable diffusion model, or it can be the training phase for a CLIP model independent of the typical stable diffusion model. The typical stable diffusion model is trained using... Figure 3 , Figure 4 The described implementation method yields a stable diffusion model based on RAG.

[0144] For example, the first text feature with the highest similarity is determined by calculating the similarity between the image features in vector form and the second text features. The similarity calculation can use the cosine similarity between the image features in vector form and the second text features. Understandably, other metrics, such as Euclidean distance, can also be used in the vector space of the CLIP model to measure the similarity between the image features in vector form and the second text features; specific methods are not limited here.

[0145] Step 403: Input the first text feature into the stable diffusion model, and obtain the second image corresponding to the first text feature through the stable diffusion model. The second image is determined to be the image corresponding to the first text.

[0146] In one possible implementation, after processing the first image to obtain the first text features in step 402, the execution device inputs the first text features into the image information creator of the stable diffusion model (e.g., U-Net and a noise mechanism). The noise data randomly sampled in the latent space is iteratively denoised using U-Net and the noise mechanism to obtain denoised data. The decoder of the VAE of the stable diffusion model converts the denoised data into a second image, which is then determined to be the image corresponding to the first text. The first text features are represented as a vector representation obtained from the first image in step 402. The implementation process is described in detail below:

[0147] Step 4031: Randomly generate noisy data;

[0148] For example, the backdiffusion process of a stable diffusion model is performed in the latent space. The first text features are input into the stable diffusion model as an embedding vector. A noise data (or a vector or random tensor in the latent space) is randomly sampled from the latent space. This noise data is generated in the latent space and is usually a multidimensional tensor containing the initial information of the final image (i.e., the second image).

[0149] It should be noted that the latent space of the stable diffusion model is a low-dimensional vector space containing multiple noisy data points (i.e., low-dimensional vector representations of image features). Denoising each noisy data point yields the corresponding image. The noisy data in the latent space is fed into a decoder, which decodes these vector representations into the corresponding images.

[0150] Step 4032: Based on the first text feature, iteratively denoise the noisy data to obtain the denoised data;

[0151] For example, noise prediction is performed by inputting noisy data into a U-Net in a stable diffusion model, and the vector representation of the first text feature is mapped to each layer of the U-Net in the latent space. Based on the predicted noise and the vector representation of the first text feature, the noisy data is denoised, that is, the noise is removed from the noisy data to obtain new noisy data.

[0152] Then, the new noise data and the vector representation of the first text feature are input into U-Net, and the new noise is predicted. Based on this, the new noise data is removed to obtain the noise data after a second iteration. After repeating this iteration a set number of times (e.g., a given time step), at time step t=1, the model will output the final data (i.e., the denoised data). This image data is usually already very similar to the first text feature.

[0153] U-Net is a neural network architecture for image segmentation, used as a noise predictor in the stable diffusion model. Through a series of convolutional layers, upsampling, and downsampling operations, U-Net can progressively remove noise from noisy data. During backdiffusion, the vector representation of the first text feature is injected into each layer of U-Net, and a cross-attention mechanism ensures that the final image generated after progressively removing noise from the noisy data is similar to the text content corresponding to the first text feature.

[0154] Step 4033: Convert the denoised data into a second image.

[0155] For example, the vector representation of the image obtained in step 4032 (i.e., the vector representation of the denoised data in the latent space) is transformed back into the pixel space by the decoder of the variational autoencoder to obtain the final image (i.e., the second image). The decoder typically contains multiple fully connected layers or convolutional layers to output the image that transforms the vector representation in the latent space back into the pixel space.

[0156] It should be noted that, in this embodiment, by inputting the first text into a machine learning model (e.g., a CLIP model), the machine learning model's text compiler extracts features from the first text and converts it into a vector representation of the text features of the first text. Based on the vector representation of the text features of the first text, the vector representation of the image feature with the highest similarity to the vector representation of the text features of the first text is determined from the image database, and the first image corresponding to the vector representation of the image feature is obtained. After the first image is converted into the first text features by the machine learning model, the first text features are input into the image information creator of the stable diffusion model, and the implementation process as in step 403 continues.

[0157] To better implement the above-described solutions of the embodiments of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 5 and Figure 6 , Figure 5 This is a schematic diagram of an image generation apparatus provided in an embodiment of this application. The image generation apparatus 500 includes:

[0158] The acquisition module 501 is used to acquire a first image corresponding to the first text from an image database, the image database including at least one image, the image content of the first image being similar to the text content of the first text; the acquisition module 501 is also used to acquire the first text features corresponding to the first image; the processing module 502 inputs the first text features into a stable diffusion model, and obtains a second image corresponding to the first text features through the stable diffusion model, the second image being determined as the image corresponding to the first text.

[0159] In one possible implementation, the acquisition module 501 is specifically used to: input the first text into a machine learning model, extract features of the first text through the machine learning model to obtain the features of the first text; and acquire the first image corresponding to the first text from at least one image based on the features of each image and the features of the first text.

[0160] In one possible implementation, the training data includes training samples and desired features during the training phase of the machine learning model, wherein, in the case that the training samples are text of a specified type, the loss function of the machine learning model indicates the similarity between the features of the training samples and the desired features.

[0161] In one possible implementation, the acquisition module 501 is further configured to: extract features from the first image using a machine learning model to obtain image features of the first image, wherein the image features are in the form of vectors, and the machine learning model includes second text features in the form of vectors corresponding to each word in at least one word; and obtain first text features based on the similarity between the vector image features and the second text features, wherein the first text features include at least one second text feature.

[0162] In one possible implementation, the processing module 502 is specifically used to: based on the first text features, iteratively denoise the noisy data using a stable diffusion model to obtain denoised data; and convert the denoised data into a second image using a stable diffusion model.

[0163] See Figure 6 , Figure 6 This application provides a schematic diagram of the structure of a data processing apparatus 600, which includes:

[0164] The acquisition module 601 is used to acquire training data, which includes training text and expected features corresponding to the training text. The training text is text of a specified type. The processing module 602 is used to extract features from the training text using a machine learning model to obtain the first feature of the training text. The training module 603 is used to train the machine learning model using a loss function to obtain the trained machine learning model. The loss function indicates the similarity between the first feature and the expected feature.

[0165] In one possible implementation, the training data also includes the desired image corresponding to the training text. The processing module 602 is further used to input the first feature of the training text into the stable diffusion model and obtain the predicted image corresponding to the first text feature through the stable diffusion model. The training module 603 is specifically used to train the machine learning model and the stable diffusion model using a loss function, which also indicates the similarity between the predicted image and the desired image.

[0166] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of an execution device provided in an embodiment of this application. Specifically, the execution device 700 includes: a receiver 701, a transmitter 702, a processor 703, and a memory 704 (wherein the execution device 700 may have one or more processors 703). Figure 7 (Taking a processor as an example), the processor 703 may include an application processor 7031 and a communication processor 7032. In some embodiments of this application, the receiver 701, transmitter 702, processor 703, and memory 704 may be connected via a bus or other means.

[0167] Memory 704 may include read-only memory and random access memory, and provides instructions and data to processor 703. A portion of memory 704 may also include non-volatile random access memory (NVRAM). Memory 704 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0168] Processor 703 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0169] The methods disclosed in the embodiments of this application can be applied to processor 703, or implemented by processor 703. Processor 703 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 703 or by instructions in software form. Processor 703 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 703 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 704, and processor 703 reads the information from memory 704 and, in conjunction with its hardware, completes the steps of the above method.

[0170] Receiver 701 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 702 can be used to output digital or character information through the first interface; transmitter 702 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 702 may also include a display device such as a display screen.

[0171] In this embodiment of the application, the processor 703 is used to execute... Figure 4 The image generation method executed by the execution device in the corresponding embodiment. It should be noted that the specific manner in which the application processor 7031 in processor 703 executes the aforementioned steps differs from that in this application. Figure 4 The various method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figure 4 The corresponding method embodiments are the same, and for details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0172] This application also provides a training device; please refer to [link / reference]. Figure 8 , Figure 8 This is a schematic diagram of a training device provided in an embodiment of this application. Specifically, the training device 800 is implemented by one or more servers. The training device 800 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 822 (e.g., one or more processors) and a memory 832, and one or more storage media 830 (e.g., one or more mass storage devices) for storing application programs 842 or data 844. The memory 832 and storage media 830 can be temporary or persistent storage. The program stored in the storage media 830 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the training device. Furthermore, the CPU 822 may be configured to communicate with the storage media 830 and execute the series of instruction operations in the storage media 830 on the training device 800.

[0173] The training device 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input / output interfaces 858, and / or one or more operating systems 841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0174] In this embodiment, the central processing unit 822 is used to execute... Figure 3 The training method for the model executed by the training device in the corresponding embodiment. It should be noted that the specific manner in which the central processing unit 822 executes the above steps differs from that in this application. Figure 3 The various method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figure 3 The corresponding method embodiments are the same, and for details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0175] This application embodiment also provides a computer-readable storage medium storing a program for performing signal processing, which, when run on a computer, causes the computer to perform the aforementioned actions. Figure 3 The steps performed by the training device in the method described in the illustrated embodiment, or causing the computer to perform the steps as described above. Figure 4 The steps performed by the execution device in the method described in the illustrated embodiment.

[0176] This application also provides a computer program product, which includes a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 3The steps performed by the training device in the method described in the illustrated embodiment, or causing the computer to perform the steps as described above. Figure 4 The steps performed by the execution device in the method described in the illustrated embodiment.

[0177] The execution device and training device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip to perform the above-mentioned operations. Figure 3 The training method for the model described in the illustrated embodiment, or, to cause the chip to perform the above... Figure 4 The image generation method described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register, cache, etc. The storage unit can also be a storage unit located outside the chip within the wireless access device, such as read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM), etc.

[0178] For example, please refer to Figure 9 , Figure 9 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 90. The NPU 90 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 903, which is controlled by a controller 904 to extract matrix data from the memory and perform multiplication operations.

[0179] In some implementations, the arithmetic circuit 903 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 903 is a two-dimensional pulsating array. The arithmetic circuit 903 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 903 is a general-purpose matrix processor.

[0180] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 902 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 901 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 908.

[0181] Unified memory 906 is used to store input and output data. Weight data is directly transferred to weight memory 902 via Direct Memory Access Controller (DMAC) 905. Input data is also transferred to unified memory 906 via DMAC.

[0182] BIU stands for Bus Interface Unit 910, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 909.

[0183] The Bus Interface Unit (BIU) 910 is used by the instruction fetch memory 909 to fetch instructions from external memory, and also by the memory access controller 905 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0184] The DMAC is mainly used to move input data from external memory DDR to unified memory 906, or to weight data to weight memory 902, or to input data to input memory 901.

[0185] The vector computation unit 907 includes multiple arithmetic processing units that further process the output of the computation circuit as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0186] In some implementations, the vector computation unit 907 can store the processed output vector in the unified memory 906. For example, the vector computation unit 907 can apply linear and / or nonlinear functions to the output of the computation circuit 903, such as performing linear interpolation on feature planes extracted by a convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 907 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 903, for example, for use in subsequent layers of the neural network.

[0187] The instruction fetch buffer 909 connected to the controller 904 is used to store the instructions used by the controller 904;

[0188] Unified memory 906, input memory 901, weighted memory 902, and instruction fetch memory 909 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0189] In the machine learning model described above, the operations of each layer can be performed by the operation circuit 903 or the vector calculation unit 907.

[0190] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0191] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0193] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0194] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An image generation method, characterized in that, The method includes: Obtain a first image corresponding to the first text from an image database, wherein the image database includes at least one image and the image content of the first image is similar to the text content of the first text; Obtain the first text feature corresponding to the first image; The first text feature is input into a stable diffusion model, and a second image corresponding to the first text feature is obtained through the stable diffusion model. The second image is determined to be the image corresponding to the first text.

2. The method according to claim 1, characterized in that, The image database also includes features of each image in the at least one image, and the step of obtaining the first image corresponding to the first text from the image database includes: The first text is input into a machine learning model, and the machine learning model extracts features from the first text to obtain the features of the first text. Based on the features of each image and the features of the first text, the first image corresponding to the first text is obtained from the at least one image.

3. The method according to claim 2, characterized in that, During the training phase of the machine learning model, the training data includes training samples and desired features. When the training samples are text of a specified type, the loss function of the machine learning model indicates the similarity between the features of the training samples and the desired features.

4. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining the first text feature corresponding to the first image includes: The first image is subjected to feature extraction by a machine learning model to obtain image features of the first image, wherein the image features are in vector form, and the machine learning model includes second text features in vector form corresponding to each word in at least one word; The first text feature is obtained based on the similarity between the image feature in vector form and the second text feature, wherein the first text feature includes at least one of the second text features.

5. The method according to any one of claims 1 to 4, characterized in that, The process of obtaining the second image corresponding to the first text feature through the stable diffusion model includes: Based on the first text feature, the noise data is iteratively denoised using the stable diffusion model to obtain the denoised data. The denoised data is converted into the second image using the stable diffusion model.

6. A data processing method, characterized in that, The method includes: Acquire training data, which includes training text and expected features corresponding to the training text, wherein the training text is text of a specified type; The training text is used to extract features through a machine learning model to obtain the first feature of the training text; The machine learning model is trained using a loss function to obtain a trained machine learning model, wherein the loss function indicates the similarity between the first feature and the desired feature.

7. The method according to claim 6, characterized in that, The training data also includes a desired image corresponding to the training text, and the method further includes: The first feature of the training text is input into the stable diffusion model, and the predicted image corresponding to the first text feature is obtained through the stable diffusion model. The step of training the machine learning model using a loss function includes: The machine learning model and the stable diffusion model are trained using the loss function, which also indicates the similarity between the predicted image and the desired image.

8. An image generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire a first image corresponding to the first text from an image database, wherein the image database includes at least one image and the image content of the first image is similar to the text content of the first text. The acquisition module is further configured to acquire the first text feature corresponding to the first image; The processing module inputs the first text features into a stable diffusion model, and obtains a second image corresponding to the first text features through the stable diffusion model. The second image is determined to be the image corresponding to the first text.

9. The apparatus according to claim 8, characterized in that, The acquisition module is specifically used for: The first text is input into a machine learning model, and the machine learning model extracts features from the first text to obtain the features of the first text. Based on the features of each image and the features of the first text, the first image corresponding to the first text is obtained from the at least one image.

10. The apparatus according to claim 9, characterized in that, During the training phase of the machine learning model, the training data includes training samples and desired features. When the training samples are text of a specified type, the loss function of the machine learning model indicates the similarity between the features of the training samples and the desired features.

11. The apparatus according to any one of claims 8 to 10, characterized in that, The acquisition module is further specifically used for: The first image is subjected to feature extraction by a machine learning model to obtain image features of the first image, wherein the image features are in vector form, and the machine learning model includes second text features in vector form corresponding to each word in at least one word; The first text feature is obtained based on the similarity between the image feature in vector form and the second text feature, wherein the first text feature includes at least one of the second text features.

12. The apparatus according to any one of claims 8 to 11, characterized in that, The processing module is specifically used for: Based on the first text feature, the noise data is iteratively denoised using the stable diffusion model to obtain the denoised data. The denoised data is converted into the second image using the stable diffusion model.

13. A data processing apparatus, characterized in that, The device includes: An acquisition module is used to acquire training data, the training data including training text and expected features corresponding to the training text, wherein the training text is text of a specified type; The processing module is used to extract features from the training text using a machine learning model to obtain the first feature of the training text. The training module is used to train the machine learning model using a loss function to obtain a trained machine learning model, wherein the loss function indicates the similarity between the first feature and the desired feature.

14. The apparatus according to claim 13, characterized in that, The training data also includes the expected image corresponding to the training text. The processing module is further configured to input the first feature of the training text into the stable diffusion model and obtain the predicted image corresponding to the first text feature through the stable diffusion model. The training module is specifically used to: train the machine learning model and the stable diffusion model using the loss function, wherein the loss function also indicates the similarity between the predicted image and the expected image.

15. A computing device, characterized in that, It includes a processor and a memory, wherein the processor is coupled to the memory. The memory is used to store programs; The processor is configured to execute a program in the memory, causing the computing device to perform the method as described in any one of claims 1 to 5.

16. A training device, characterized in that, It includes a processor and a memory, wherein the processor is coupled to the memory. The memory is used to store programs; The processor is configured to execute a program in the memory, causing the training device to perform the method as described in any one of claims 6 to 7.

17. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, causes the method described in any one of claims 1 to 7 to be implemented.

18. A computer program product, characterized in that, The computer program product includes program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

Images