Image generation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image generation method and device, equipment and a storage medium, and relates to the technical field of image processing and artificial intelligence. In the method, based on a text condition vector of text condition information, at least one target retrieval vector matched with the text condition vector is determined from at least one candidate retrieval vector stored in a pre-constructed retrieval library, and target retrieval images corresponding to the at least one target retrieval vector are obtained; each candidate retrieval vector is an image retrieval vector or a text retrieval vector; based on the text condition vector, respectively carrying out content correction processing on the image feature vector of the at least one target retrieval image to obtain a corresponding prior image semantic vector; and generating a target generated image corresponding to the text condition information based on the at least one priori image semantic vector in combination with the text condition vector. In the process of generating the image, the high-quality external image is introduced, so that the generated image is prevented from being distorted, and the image generation quality is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, apparatus, device and storage medium. Background Art

[0002] With the development of science and technology, artificial intelligence generated content (AIGC) is widely used in various industries; for example, images generated based on artificial intelligence are widely used in games, social networking, e-commerce and other industries.

[0003] In recent years, with the expansion of data resources, increased computing power, and the continuous optimization of image generation methods, conditional-guided diffusion models (also known as conditional diffusion models) have become the dominant model in the field of image generation. Conditional diffusion models can use information such as text and labels as guidance for image generation, thereby improving the quality and controllability of image generation.

[0004] However, when using the conditional diffusion model to drive the image generation process by inputting text prompts, the image generation is only assisted by the text prompts, and less conditional information is introduced. In addition, there is information missing in the introduced conditional information, which ultimately leads to distortion of the generated image. That is, the generated image is different from the desired image, and the generated image quality is poor.

[0005] Therefore, how to prevent distortion during image generation and improve image generation quality is a technical problem that needs to be solved at present. Summary of the Invention

[0006] The embodiments of the present application provide an image generation method, apparatus, device, and storage medium to prevent distortion of generated images and improve image generation quality.

[0007] In a first aspect, an embodiment of the present application provides an image generation method, the method comprising:

[0008] Based on the text condition vector of the text condition information, at least one target retrieval vector matching the text condition vector is determined from at least one candidate retrieval vector stored in a pre-built retrieval library, and a target retrieval image corresponding to each of the at least one target retrieval vectors is obtained; each candidate retrieval vector is: an image retrieval vector or a text retrieval vector;

[0009] Based on the text condition vector, content correction processing is performed on the image feature vector of at least one target retrieval image to obtain a corresponding prior image semantic vector;

[0010] Based on at least one prior image semantic vector and in combination with the text condition vector, a target generated image corresponding to the text condition information is generated.

[0011] In a second aspect, an embodiment of the present application provides an image generating device, the device comprising:

[0012] a determination unit configured to determine, based on the text condition vector of the text condition information, at least one target retrieval vector that matches the text condition vector from at least one candidate retrieval vector stored in a pre-built retrieval library, and obtain a target retrieval image corresponding to each of the at least one target retrieval vectors; each candidate retrieval vector is an image retrieval vector or a text retrieval vector;

[0013] a processing unit configured to perform content correction processing on the image feature vector of at least one target retrieval image based on the text condition vector to obtain a corresponding prior image semantic vector;

[0014] The generating unit is configured to generate a target generated image corresponding to the text condition information based on at least one prior image semantic vector and in combination with the text condition vector.

[0015] In a possible implementation, the search library is constructed as follows:

[0016] Acquire at least one historical candidate data, each historical candidate data being a historical image or a historical image-text pair, wherein the historical image-text pair includes: a historical image and text description information for describing the historical image;

[0017] Performing data cleaning on at least one historical candidate data to obtain at least one target candidate data;

[0018] Perform feature extraction on at least one target candidate data to obtain a candidate retrieval vector associated with each target candidate data; wherein the candidate retrieval vector and the text condition vector are in the same vector space;

[0019] Each target candidate data and the associated candidate search vector are stored in the search library.

[0020] In one possible implementation, performing data cleaning on at least one historical candidate data to obtain at least one target candidate data includes:

[0021] For at least one historical candidate data, respectively obtain an image evaluation value of a corresponding historical image; wherein the image evaluation value includes at least one of an interest evaluation value and a visual quality evaluation value, and the visual quality evaluation value is determined based on image resolution and image content;

[0022] Based on at least one image evaluation value, historical candidate data having an image evaluation value reaching an evaluation value threshold is selected from at least one historical candidate data, and the selected historical candidate data is used as target candidate data.

[0023] In a possible implementation, the retrieval library includes at least one image style set, and each image style set includes at least one target candidate data belonging to the same image style.

[0024] In a possible implementation, the determining unit is specifically configured to:

[0025] For each image style set, perform the following operations:

[0026] Perform similarity matching between the candidate retrieval vector of each target candidate data in the image style set and the text condition vector to obtain the corresponding similarity matching results;

[0027] Based on the similarity matching result, at least one target retrieval vector that meets the similarity condition is selected from the image style set.

[0028] In a possible implementation, if the text condition information includes image style indication information, the determining unit is specifically configured to:

[0029] Selecting a target image style set that matches the image style indication information from the at least one image style set;

[0030] Perform similarity matching on the candidate retrieval vector of each target candidate data in the selected target image style set and the text condition vector respectively to obtain the corresponding similarity matching results;

[0031] Based on the similarity matching result, at least one target retrieval vector that meets the similarity condition is selected from an image style set.

[0032] In a possible implementation, the processing unit is specifically configured to:

[0033] For each target image, perform the following operations:

[0034] Obtain the image feature vector of the target retrieval image, and perform noise processing on each image feature vector to obtain the corresponding image noise vector;

[0035] The image noise vector and the text condition vector are spliced ​​to obtain a spliced ​​vector;

[0036] Based on the weight of each vector element in the splicing vector, the image feature vector is modified to obtain the corresponding prior image semantic vector.

[0037] In a possible implementation, the generating unit is specifically configured to:

[0038] For each prior image semantic vector, determining an image key matrix and an image value matrix corresponding to the prior image semantic vector based on a preset image key weight matrix and a preset image value weight matrix, respectively; determining a semantic vector weight based on a preset query matrix and the image key matrix, and determining a corresponding first prediction vector based on the semantic vector weight and the corresponding image value matrix;

[0039] For the text condition vector, based on a preset text key weight matrix and a preset text value weight matrix, a text key matrix and a text value matrix are determined; based on a preset query matrix and the text key matrix, a text condition vector weight is determined, and based on the text condition vector weight and the corresponding text value matrix, a second prediction vector is determined;

[0040] At least one first prediction vector and a second prediction vector are fused to obtain a fused vector, and a target generated image corresponding to the text condition information is generated based on the fused vector.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the steps of the image generation method provided in the embodiment of the present application.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the image generation method provided in the embodiment of the present application are implemented.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the image generation method provided in the embodiment of the present application.

[0044] The beneficial effects of this application are as follows:

[0045] The present application provides an image generation method, apparatus, device, and storage medium, which relate to the field of image processing technology and artificial intelligence, and can be applied to cloud technology, smart transportation, assisted driving, and other scenarios. In the present application, when generating an image based on text condition information:

[0046] First, based on the text condition vector of the text condition information, at least one target retrieval vector matching the text condition vector is determined from at least one candidate retrieval vector stored in a pre-built retrieval library, and then a target retrieval image corresponding to each of the at least one target retrieval vectors is obtained; an image retrieval is performed based on the text condition information, and the retrieved target retrieval image is matched with the text condition information, thereby ensuring the accuracy of the target retrieval image introduced in the text-to-image process;

[0047] Then, based on the text condition vector, the image feature vector of at least one target retrieval image is subjected to content correction processing to obtain the corresponding prior image semantic vector; information is extracted and corrected from the image features of the target retrieval image to make the prior image semantic vector closer to the text condition information, thereby ensuring the quality and accuracy of the prior image semantic vector;

[0048] Finally, based on at least one prior image semantic vector and the text conditional vector, a target generated image corresponding to the text conditional information is generated. In addition to the text conditional information, high-quality external image knowledge (i.e., the prior image semantic vector of the target retrieval image) is incorporated into the image generation process as prior information for the text-generated image. This enhanced reference information improves the controllability and stability of image generation, avoids distortion in the generated image, and enhances the quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0051] Figure 2 A flowchart of an image generation method provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of constructing a search library provided in an embodiment of the present application;

[0053] Figure 4 A data cleaning schematic diagram provided in an embodiment of the present application;

[0054] Figure 5 Another data cleaning schematic diagram provided in an embodiment of the present application;

[0055] Figure 6A schematic diagram of the specific structure of a priori model provided in an embodiment of the present application;

[0056] Figure 7 A structural diagram of a cultural graph model provided in an embodiment of the present application;

[0057] Figure 8 A schematic diagram of a U-Net network provided in an embodiment of the present application;

[0058] Figure 9 A schematic diagram of a specific implementation method for image generation provided in an embodiment of the present application;

[0059] Figure 10 A schematic diagram of another specific implementation method for image generation provided in an embodiment of the present application;

[0060] Figure 11 A flow chart of a model training method provided in an embodiment of the present application;

[0061] Figure 12 A schematic diagram of a model training provided in an embodiment of the present application;

[0062] Figure 13 A structural diagram of an image generating device provided in an embodiment of the present application;

[0063] Figure 14 A structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and beneficial effects of this application more clear and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0065] The following explains some of the terms used in the embodiments of the present application to facilitate understanding by those skilled in the art.

[0066] The prior image semantic vector is the image semantic vector obtained by performing content correction processing on the image feature vector of the target retrieval image. The image semantic vector serves as the prior information introduced in the text-generating image process.

[0067] An image semantic vector includes visual information, object information, and conceptual information. Visual information typically refers to the underlying elements of an image, such as color, texture, and shape. Object information typically refers to the intermediate elements of an image, typically containing object attributes or characteristics, i.e., the state of an object at a given moment. Conceptual information typically refers to the higher-level elements of an image, representing the overall content of the image. For example, if an image contains sand, blue sky, and seawater, the visual information represents the image's individual segments. Object information represents the specific objects corresponding to these segments, i.e., sand, blue sky, and seawater. Conceptual information represents the overall content of the image: the beach.

[0068] Prior information can be understood as some image information about the generated image that has been obtained before the text-generating image operation is performed.

[0069] The retrieval library is a database used for image retrieval, which includes multiple data to be retrieved (i.e., target candidate data in this application) and candidate retrieval vectors corresponding to each data to be retrieved, and each data to be retrieved is an image or an image-text pair; when the data to be retrieved is an image, the candidate retrieval vector is an image retrieval vector; when the data to be retrieved is an image-text pair, the candidate retrieval vector is an image retrieval vector and / or a text retrieval vector.

[0070] The diffusion model, also known as the text-graph diffusion model, is a deep learning model used for text-to-image generation. A text-graph diffusion model, trained by inverting the diffusion process of natural images, can gradually generate new natural images from a completely random noise image, guided by text. Noise images are generated by random signal interference during capture or transmission, manifesting as random variations in image information or pixel brightness.

[0071] The word “exemplary” is used hereinafter to mean “serving as an example, example, or illustration.” Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0072] The terms "first" and "second" are used for descriptive purposes only and should not be construed as explicitly or implicitly indicating relative importance or the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0073] With the development of science and technology, artificial intelligence generated content (AIGC) is widely used in various industries; for example, images generated based on artificial intelligence are widely used in games, social networking, e-commerce and other industries.

[0074] In recent years, with the expansion of data resources, increased computing power, and the continuous optimization of image generation methods, conditional-guided diffusion models (also known as conditional diffusion models) have become the dominant model in the field of image generation. Conditional diffusion models can use information such as text and labels as guidance for image generation, thereby improving the quality and controllability of image generation.

[0075] However, when using the conditional diffusion model to drive the image generation process by inputting text prompts, the image generation is only assisted by the text prompts, and less conditional information is introduced. In addition, there is information missing in the introduced conditional information, which ultimately leads to distortion of the generated image. That is, the generated image is different from the desired image, and the generated image quality is poor.

[0076] Therefore, how to prevent distortion during image generation and improve image generation quality is a technical problem that needs to be solved at present.

[0077] In view of this, in order to introduce more information into the text generation process, the inventors of this application considered applying retrieval augmentation technology. Retrieval augmentation technology enables the model to dynamically obtain and utilize relevant information during the generation process by introducing an external knowledge database. That is, it can find the required information in a large-scale knowledge database and inject the found information into the generation model to improve the generation quality and efficiency of the model.

[0078] At the same time, the inventors of this application have learned that retrieval enhancement technology in related art is widely used in large prediction models, but has not been applied to cross-modal image generation processes such as text-based image generation. Therefore, if retrieval enhancement technology in related art is directly introduced into the text-generated image process, only the text prompt information input in the text-generated image process will be enhanced. Even if the text prompt information is enhanced, image generation is still assisted by text prompt words. This may cause distortion of the generated image due to missing information, that is, the generated image may differ from the desired image and the generated image quality may be poor.

[0079] Therefore, to address the issues of differences between the generated image and the desired image, distortion in the generated image, and poor image quality, the present application proposes the use of retrieval enhancement technology to introduce prior image information. To implement retrieval enhancement technology, a retrieval library must first be constructed. The pre-constructed retrieval library is then used to retrieve prior image information. Finally, an image is generated based on the retrieved prior image information and text conditional information. This allows the image information from the external database to be stably and reliably integrated into the text-generated image diffusion model, thereby comprehensively improving the generation capabilities of the related art text-generated image diffusion model.

[0080] In summary, based on the literary graph model and retrieval enhancement technology, the embodiments of the present application provide an image generation method, device, equipment and storage medium, which relate to the field of image processing technology and artificial intelligence, and can be applied to cloud technology, smart transportation, assisted driving and other scenarios.

[0081] In the embodiment of the present application, when generating an image based on text condition information:

[0082] First, based on the text condition vector of the text condition information, at least one target retrieval vector matching the text condition vector is determined from at least one candidate retrieval vector stored in a pre-built retrieval library, and then a target retrieval image corresponding to each of the at least one target retrieval vectors is obtained; an image retrieval is performed based on the text condition information, and the retrieved target retrieval image is matched with the text condition information, thereby ensuring the accuracy of the target retrieval image introduced in the text-to-image process;

[0083] Then, based on the text condition vector, the image feature vector of at least one target retrieval image is subjected to content correction processing to obtain the corresponding prior image semantic vector; information is extracted and corrected from the image features of the target retrieval image to make the prior image semantic vector closer to the text condition information, thereby ensuring the quality and accuracy of the prior image semantic vector;

[0084] Finally, based on at least one prior image semantic vector and the text conditional vector, a target generated image corresponding to the text conditional information is generated. In addition to the text conditional information, high-quality external image knowledge (i.e., the prior image semantic vector of the target retrieval image) is incorporated into the image generation process as prior information for the text-generated image. This enhanced reference information improves the controllability and stability of image generation, avoids distortion in the generated image, and enhances the quality of the generated image.

[0085] In the embodiments of the present application, the image generation process involves artificial intelligence (AI) and machine learning technologies, and is designed based on speech technology, natural language processing technology and machine learning (ML) in artificial intelligence.

[0086] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in ways similar to human intelligence.

[0087] Artificial intelligence (AI) is the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technologies primarily encompass computer vision, natural language processing, and machine learning / deep learning. With the advancement of AI technology, research and application are expanding across a wide range of fields, including smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, driverless and autonomous driving, robotics, and smart healthcare. As technology advances, AI will be applied in even more areas and play an increasingly important role.

[0088] Machine learning is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Compared to data mining, which seeks to identify patterns within large datasets, machine learning focuses more on algorithmic design, enabling computers to automatically "learn" patterns from data and use these patterns to make predictions about unknown data.

[0089] Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. Reinforcement learning (RL), also known as reinforcement learning, evaluation learning, or enhanced learning, is a paradigm and methodology within machine learning that describes and solves problems in which intelligent agents learn strategies to maximize rewards or achieve specific goals during their interactions with their environment.

[0090] The following is a brief description of the application scenarios set by this application. It should be noted that the following scenarios are only used to illustrate the embodiments of this application and are not limiting. During specific implementation, the technical solutions provided in the embodiments of this application can be flexibly applied according to actual needs.

[0091] See also Figure 1 , Figure 1This is a schematic diagram of an application scenario provided by an embodiment of the present application. The application scenario includes a terminal device 110 and a server 120, and the terminal device 110 and the server 120 can communicate with each other through a communication network.

[0092] In an optional implementation, the communication network can be a wired network or a wireless network. Therefore, the terminal device 110 and the server 120 can be directly or indirectly connected via wired or wireless communication. For example, the terminal device 110 can be indirectly connected to the server 120 via a wireless access point, or the terminal device 110 can be directly connected to the server 120 via the Internet, which is not limited in this application.

[0093] The terminal device 110 includes, but is not limited to, mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, and in-vehicle terminals. Various clients may be installed on the terminal device, and the client may be an online platform or application that supports text input and generates images based on the input text, or a webpage or mini-program. In other words, the client supports the application of the text-based graph model. For example, the client may be an intelligent creation system that supports the application of the text-based graph model and can provide users with personalized image customization capabilities through the text-based graph model.

[0094] The server 120 is a background server corresponding to the client installed in the terminal device 110; for example, the server 120 is a background service function of the intelligent creation system, which is used to implement the steps of generating images provided in the embodiments of the present application. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0095] In a possible application scenario, the image generation method proposed in the embodiment of the present application can be implemented solely by a terminal device, or solely by a server, or by a server and a terminal device in cooperation to implement the image generation method proposed in the embodiment of the present application.

[0096] Exemplarily, the terminal device obtains text condition information and transmits the text condition information to the server. The server obtains the target retrieval image from the retrieval library based on the text condition information, and continues to generate an image based on the target retrieval image and the text condition information, and then sends the generated image to the terminal device for display to the user through the terminal device.

[0097] Take the scenario where an intelligent creation system is installed in a terminal device and the server provides background services to the intelligent creation system as an example:

[0098] When the intelligent creation system provides users with personalized image customization functions through the text-based image model, the terminal device obtains the demand information (i.e., text condition information) entered by the user in the intelligent creation system, and then uploads the demand information to the server, which further generates an image based on the demand information.

[0099] In one possible scenario, the intelligent creation system can be applied to interior design and home customization, fashion matching and clothing design, advertising creativity and brand customization, gift customization and personalized product, game design, social media content generation, e-commerce product display, and so on. Therefore, the image generation method provided in the embodiment of this application can also realize personalized image customization based on the images in the search library and the input text condition information, meeting the creation needs and customization requirements in various scenarios.

[0100] In one possible application scenario, the relevant data (such as images and text) and model parameters involved in the embodiments of this application can be stored using cloud storage technology. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (or storage nodes) in the network through application software or application interfaces to work together and provide external data storage and service access functions.

[0101] It should be noted that Figure 1 The examples shown are for illustration only. In practice, the number of terminal devices 110 and servers 120 is not limited and is not specifically limited in the embodiments of the present application. In the embodiments of the present application, when there are multiple servers 120, the multiple servers 120 can form a blockchain, and the servers 120 are nodes on the blockchain.

[0102] To further illustrate the technical solution provided by the embodiment of the present application, the image generation method provided by the exemplary implementation of the present application is described below by taking the server alone as an example and combining with the accompanying drawings.

[0103] See also Figure 2 , Figure 2 A flowchart of an image generation method provided in an embodiment of the present application includes the following steps:

[0104] Step S200: Based on the text condition vector of the text condition information, at least one target retrieval vector that matches the text condition vector is determined from at least one candidate retrieval vector stored in a pre-built retrieval library, and a target retrieval image corresponding to each of the at least one target retrieval vectors is obtained; each candidate retrieval vector is: an image retrieval vector or a text retrieval vector.

[0105] In one possible implementation, when matching candidate retrieval vectors stored in a retrieval library with text condition vectors to determine at least one target retrieval vector, the similarity between the candidate retrieval vector and the text condition vector is first determined, and then, based on the similarity, at least one target retrieval vector matching the text condition vector is retrieved from the retrieval library.

[0106] For example, inner product, Euclidean distance, cosine similarity, etc. are used as similarity measurement methods between candidate search vectors and text condition vectors; when searching for at least one target search vector that matches the text condition vector, a variety of factors such as the amount of data in the application scenario, search accuracy requirements, search speed requirements, and memory requirements can be comprehensively considered to select an appropriate search matching method; wherein, the search matching method includes but is not limited to: K-Nearest Neighbors (KNN) method, Inverted File System Product Quantization (IVF-PQ), and Hierarchical Navigating Small World (HNSW) method. Of course, the simplest search matching method can also be used, that is, the candidate search vector whose similarity reaches the similarity threshold is used as the target search vector, or when the similarities are sorted in order of size, the candidate search vector with the similarity association in the top TOPK is used as the target search vector.

[0107] In order to ensure that the target retrieval image can be accurately obtained from the retrieval library, the target retrieval image is introduced as prior knowledge during the image generation process, so that the generated image is closer to the desired image, the generated image is prevented from being distorted, and the quality of the generated image is guaranteed. Before image generation, the embodiment of the present application pre-builds a retrieval library that supports image retrieval.

[0108] See also Figure 3 , Figure 3 A schematic diagram of constructing a search library provided in an embodiment of the present application, corresponding to steps A1 to A4.

[0109] Step A1: Acquire at least one historical candidate data. Each historical candidate data is a historical image or a historical image-text pair. The historical image-text pair includes: a historical image and text description information for describing the historical image.

[0110] Step A2: performing data cleaning on the acquired at least one historical candidate data to obtain at least one target candidate data.

[0111] After obtaining at least one historical candidate data, in order to ensure the quality of the retrieval image in the retrieval library, the embodiment of the present application proposes an implementation method for data cleaning of the obtained historical candidate data. The standards for data cleaning are mainly divided into two dimensions: image aesthetics (i.e., image visual quality) and interest preference (i.e., interest level).

[0112] For example, see Figure 4 , Figure 4 A data cleaning diagram provided for an embodiment of the present application, corresponding to steps B1 to B4.

[0113] Step B1: Input each historical candidate data into the interest evaluation model and the visual quality evaluation model respectively, obtain the interest evaluation value output by the interest evaluation model, and obtain the visual quality evaluation value output by the visual quality evaluation model; wherein the visual quality evaluation value is determined based on the image resolution and image content.

[0114] In order to ensure that the original images in the retrieval library have high visual quality, that is, good image aesthetics and high resolution, before obtaining the visual evaluation value of the historical candidate data through the visual quality assessment model, in an embodiment of the present application, the visual quality assessment model to be trained is trained based on the images annotated with visual quality to obtain a trained visual quality assessment model, so as to output the visual quality assessment value for the images in the candidate historical data based on the trained visual quality assessment model.

[0115] Similarly, in order to better measure the interestingness of an image, before obtaining the interestingness evaluation value of the historical candidate data through the interestingness evaluation model, in an embodiment of the present application, the interestingness evaluation model is obtained by training the interest preference data set to output the interestingness evaluation value for the image in the candidate historical data based on the trained interestingness evaluation model.

[0116] It should be noted that in the embodiments of the present application, the interest evaluation model and the visual quality evaluation model may not be trained, but a more mature model algorithm in the industry may be used; for example, the visual quality uses the aesthetic score (AestheticScore) as a measure of the overall beauty of the image; the interest uses the image reward (Image Reward), the selection score (PickScore), etc. as a measure of interest preference.

[0117] Step B2: for at least one historical candidate data, obtain an image evaluation value of a corresponding historical image based on the corresponding interest evaluation value and visual quality evaluation value.

[0118] In a possible implementation, after the interest evaluation value and the visual quality evaluation value are obtained, weighted processing is performed on the interest evaluation value and the visual quality evaluation value to obtain an image evaluation value.

[0119] It should be noted that the weights of the interest evaluation value and the visual quality evaluation value can be set according to actual conditions, as long as the sum of the weight values ​​is 1.

[0120] Step B3: Based on at least one image evaluation value, select historical candidate data whose image evaluation value reaches an evaluation value threshold from at least one historical candidate data, and use the selected historical candidate data as target candidate data.

[0121] For another example, see Figure 5 , Figure 5 A data cleaning schematic diagram provided for an embodiment of the present application, corresponding to steps C1 to C4.

[0122] Step C1: Input each historical candidate data into the visual quality assessment model to obtain the visual quality assessment value output by the visual quality assessment model.

[0123] In order to ensure that the original images in the retrieval library have high visual quality, that is, good image aesthetics and high resolution, before obtaining the visual evaluation value of the historical candidate data through the visual quality assessment model, in an embodiment of the present application, the visual quality assessment model to be trained is trained based on the images annotated with visual quality to obtain a trained visual quality assessment model, so as to output the visual quality assessment value for the images in the candidate historical data based on the trained visual quality assessment model.

[0124] It should be noted that visual quality can also be determined using relatively mature model algorithms in the industry, such as using an aesthetic score as a measure of the overall beauty of an image.

[0125] Step C2: Based on at least one visual quality evaluation value, select historical candidate data whose visual quality evaluation value reaches a quality threshold from at least one historical candidate data, and use the selected historical candidate data as screening candidate data.

[0126] Step C3: Input each screening candidate data into the interest evaluation model to obtain the interest evaluation value output by the interest evaluation model.

[0127] In order to better measure the interestingness of an image, before obtaining the interestingness evaluation value of the historical candidate data through the interestingness evaluation model, in an embodiment of the present application, the interestingness evaluation model is obtained by training the interest preference data set to output the interestingness evaluation value for the image in the candidate historical data based on the trained interestingness evaluation model.

[0128] It should be noted that the interest level can also be determined using a relatively mature model algorithm in the industry, such as using an image reward, a pick score, etc. as a measure of interest preference.

[0129] Step C4: Based on at least one interest evaluation value, select historical candidate data whose interest evaluation value reaches a quality threshold from at least one screening candidate data, and use the selected screening candidate data as screening candidate data.

[0130] Step A3: Perform feature extraction on at least one target candidate data obtained after data cleaning to obtain a candidate retrieval vector associated with each target candidate data; wherein the candidate retrieval vector and the text condition vector are in the same vector space.

[0131] In order to enable the retrieval library to support the implementation method of obtaining the target retrieval image based on the text condition vector, that is, to facilitate vector retrieval, the target retrieval image is obtained by vector matching. Therefore, after obtaining the target candidate data, the target candidate data is feature extracted and converted into a candidate retrieval vector.

[0132] Exemplarily, when the target candidate data in the retrieval library is an image, the candidate retrieval vector is an image retrieval vector, so that the text condition vector is matched with the image retrieval vector to obtain the target retrieval image.

[0133] In another exemplary embodiment, when the target candidate data in the search library is an image-text pair, the candidate search vector can be an image search vector or a text search vector. When the candidate search vector is a text search vector, the text search vector is matched with the text condition vector to obtain the matching text, and based on the image-text pair, the image corresponding to the matching text is used as the target search image. When the candidate search vector is an image search vector, the text condition vector is matched with the image search vector to obtain the target search image.

[0134] Therefore, in order to enable the retrieval library provided in the embodiment of the present application to simultaneously support homomodal retrieval (text search for text) and cross-modal retrieval (text search for image), it is necessary to ensure that the image retrieval vector and the text condition vector are located in the same feature space; in one possible implementation method, when extracting features of the target candidate data, the Constrastive Language-Image Pre-training (CLIP-L) model is used as a feature extractor for the target candidate data and text condition information, and the extracted features are subjected to L2 normalization.

[0135] Step S201 : Based on the text condition vector, content correction processing is performed on the image feature vectors of at least one target retrieval image to obtain a corresponding priori image semantic vector.

[0136] In an embodiment of the present application, after obtaining at least one target retrieval image, in order to accurately use the content in the target retrieval image in the image generation process and ensure the accuracy of the generated image, after obtaining at least one target retrieval image from the retrieval library, the image feature vector of at least one target retrieval image is subjected to content correction processing.

[0137] In one possible implementation, when performing content correction processing on the image feature vector of at least one target retrieval image to obtain the corresponding prior image semantic vector, the image feature vector of the target retrieval image is first obtained, and each image feature vector is noise-added to obtain the corresponding image noise vector; then, the image noise vector and the text condition vector are spliced ​​to obtain the corresponding splicing vector, and the weight of each vector element in the splicing vector is determined; finally, based on the weight of each vector element, the image feature vector is content-corrected to obtain the corresponding prior image semantic vector.

[0138] In order to accurately perform content correction processing on the image feature vector of the target retrieval image, the embodiment of the present application proposes a priori model for content correction processing, see Figure 6 , Figure 6 A schematic diagram of the specific structure of a priori model provided in an embodiment of the present application; Figure 6 As shown: The prior model includes: image encoding network, noise addition network, text encoding network, transformer network.

[0139] An example is given to obtain the corresponding prior image semantic vector for a target retrieval image: for the target retrieval image x, the image feature vector (Z) is obtained through the image encoding network (E). Then, the image feature vector (Z) is diffused and noised and projected into the latent space to obtain the latent space vector, i.e., the image noise vector (Z)T ), at the same time, for the text condition information (Text), the text condition vector is obtained through the text encoding network (τ); then, the condition vector and the image noise vector (Z T ) is spliced ​​and input into the transformer network. The spliced ​​vector is processed T times by the self-attention mechanism of the transformer network to determine the weight of each vector element in the spliced ​​vector. Based on the weight of each vector element, the image feature vector is content-corrected to generate a priori image semantic vector Z'.

[0140] Step S202 : generating a target generated image corresponding to the text condition information based on at least one prior image semantic vector and in combination with the text condition vector.

[0141] For example, referring to steps D1 to D3, generating a target generated image corresponding to the text condition information is described in detail.

[0142] Step D1, for each prior image semantic vector, respectively performs: based on a preset image key weight matrix and a preset image value weight matrix, determining the image key matrix and image value matrix corresponding to the prior image semantic vector; based on a preset query matrix and the image key matrix, determining the semantic vector weight; and based on the semantic vector weight and the corresponding image value matrix, determining the corresponding first prediction vector. Specifically, it can be expressed as:

[0143]

[0144] Among them, Z i Represents the first prediction vector; Q = ZW q Corresponding to the image query matrix in the cross attention, Z is the query feature of the given input, W q is the preset query weight matrix; Corresponding to the image key matrix in the cross attention, c i is the image sequence feature obtained by mapping the prior image semantic vector, is the preset image key weight matrix; Corresponding to the image value matrix in the cross attention, The weight matrix representing the preset image values ​​is also called a mapping matrix.

[0145] Step D2, for the text condition vector, based on a preset text key weight matrix and a preset text value weight matrix, determines a text key matrix and a text value matrix; based on a preset query matrix and the text key matrix, determines the text condition vector weight, and determines a second prediction vector based on the text condition vector weight and the corresponding text value matrix; specifically, it can be expressed as:

[0146]

[0147] Among them, Z t Represents the second prediction vector; Q = ZW q Corresponding to the text query matrix in the cross attention, Z is the query feature of the given input, W q is the preset query weight matrix; Corresponding to the text key matrix in the cross attention, c t is the text sequence feature of text condition information, is the preset text key weight matrix; Corresponding to the text value matrix in the cross attention, The weight matrix representing the preset text values ​​is also called a mapping matrix.

[0148] Step D3, fusing at least one first prediction vector and a second prediction vector to obtain a fused vector, and generating a target generated image corresponding to the text condition information based on the fused vector; specifically, it can be expressed as:

[0149] Z out =Z t +λ*(Z i1 +Z i2 +…+Z in )

[0150] Among them, Z out represents the fusion vector; Z t represents the second prediction vector; Z i1 , Z i2 …Z in represents at least one first prediction vector; λ is a scaling factor used to control the weights of text and image conditions.

[0151] It should be noted that, in the embodiment of the present application, the prior image semantic vector processing process and the text condition vector processing process share a preset query matrix.

[0152] In order to accurately generate the target generated image, the embodiment of the present application proposes a text-based graph model for generating images, see Figure 7 , Figure 7 A structure diagram of a cultural graph model provided for the implementation of this application; Figure 7 As shown: The text-based graph model includes: linear mapping network, text encoding network, decoupled cross attention network, U-Net network and image decoding network.

[0153] An example is given of generating a target generated image based on a prior image semantic vector combined with a text conditional vector: the prior image semantic vector is mapped to image sequence features through a linear mapping network; the text conditional information is obtained through a text encoding network to obtain text sequence features; the image sequence features and text sequence features are applied to the U-Net network through a decoupled cross-attention network. In the transformer layer of the U-Net network, given the input query features, cross-attention operations can be performed on the text sequence features and the image sequence features respectively to obtain the output of the decoupled cross-attention, which is a fusion vector of text and image. For details, please refer to the above steps D1 to D3, which will not be discussed here; finally, the fusion vector is decoded through the image encoding network to output the target generated image.

[0154] In one possible implementation, the linear mapping network includes a linear layer and a normalization layer.

[0155] In one possible implementation, the U-Net network includes multiple cross-attention (QKV) modules (also known as cross-attention layers) for modeling the connection between input and output. In the embodiment of the present application, the target generated image is obtained based on text and prior images.

[0156] See also Figure 8 , Figure 8 Schematic diagram of a U-Net network in an embodiment of the present application, wherein: Figure 8 The structure of different layers of the U-Net network is listed, and due to space limitations, Figure 8 Only some examples are given. Figure 8 As can be seen, the cross-attention layer is divided into three parts: input part (IN) 81, intermediate part (MID) 82, and output part (OUT) 83. In addition, a text encoder (BASE) 84 is added. In addition to the text encoder (BASE), the input part (IN), intermediate part (MID), and output part (OUT) can all be understood as the cross-attention layer in the text-based graph model.

[0157] like Figure 8 As shown, the input part 81 simply illustrates four layers, namely the residual module 811, the attention module 812, the residual module 813 and the attention module 814; the middle part 82 simply illustrates three layers, namely the residual module 821, the attention module 822 and the residual module 823; the output part 83 simply illustrates four layers, namely the residual module 831, the attention module 832, the residual module 833 and the attention module 834. It should be noted that Figure 8 The U-Net structures listed are just simple examples.

[0158] Among them, the U-Net network can also include a skip connection structure. Each downsampling can have a skip connection cascaded with the corresponding upsampling, so that the U-Net network can fuse the features of the corresponding position of the encoder on the channel in each upsampling, and improve the detection accuracy by fusing features of different sizes.

[0159] In actual application scenarios, a specific retrieval library can be constructed according to needs. For example, if the goal is to improve the quality of generated images, a retrieval library containing high-quality images can be collected and constructed. If the goal is to achieve image style control (such as two-dimensional, XX artist style, etc.), a retrieval library containing different image styles can be constructed. Therefore, the retrieval library proposed in the embodiment of the present application can also include at least one image style set, and each image style set includes at least one target candidate data belonging to the same image style.

[0160] In this application, the application of a retrieval library combined with generated images makes the image generation method proposed in this application highly scalable and low-cost. For example, by expanding the data in the retrieval library and constructing a database containing different styles, style control of a small number of samples can be achieved, expanding the stylized generation capability. At the same time, multiple target retrieval images can be obtained from the retrieval library through retrieval, thereby introducing external diverse image information into the image generation process, thereby improving the diversity of image generation. Furthermore, the retrieval library is highly controllable and secure, so the application of the retrieval library combined with the generated images enhances the controllability and stability of image generation, prevents distortion of the generated images, and improves the quality of the generated images.

[0161] In order to more clearly understand the implementation of the present application, the image generation proposed in the present application is described in detail below with reference to the accompanying drawings.

[0162] See also Figure 9 , Figure 9 This is a schematic diagram of a specific implementation method for image generation provided in an embodiment of the present application, from Figure 9 It can be seen from the figure that in order to generate a target image of a bird standing on a flower, the text condition information that should be input at this time is: a bird standing on a flower; and when generating an image based on the text condition information "a bird standing on a flower":

[0163] First, the text condition vector of “a bird standing on a flower” is obtained through the text feature extractor.

[0164] Secondly, based on the text condition vector, a search is performed from a pre-built retrieval library to obtain a matching target retrieval image; if the retrieval library includes multiple image style sets, at this time, the candidate retrieval vector of each target candidate data in each image style set is similarly matched with the text condition vector to obtain a corresponding similarity matching result, and based on the similarity matching result, at least one target retrieval vector that meets the similarity condition is selected from the image style set, and the target retrieval image is obtained based on the target retrieval vector.

[0165] for example

[0166] It should be noted that during the retrieval process, retrieval can be performed cross-modally or homomodally. Cross-modal retrieval uses text condition vectors to directly retrieve images. Homomodal retrieval, on the other hand, first uses text condition vectors to search for text in the retrieval database (only for image-text pairs), and then retrieves the retrieval image corresponding to the retrieved text.

[0167] Next, the target retrieval image is input into the feature extractor to obtain the image feature vector of the target retrieval image, and a partial noise addition process is performed on the image feature vector of the target retrieval image.

[0168] The noised image feature vector and the text condition vector are then fed into the prior model to generate the image prior semantic vector. During this process, the text condition is used as a guide, and a modified image prior semantic vector is obtained through embedding to embedding, which serves as the image prior. The prior model can extract and modify information from the retrieved image, processing it into high-level semantic information suitable for subsequent text-to-image models. This improves the stability of the generated information and enhances the model's contextual capabilities.

[0169] Finally, the image prior semantic vector generated by the prior model and the text vector of the text condition information are used as image generation conditions to guide the text-based graph model to generate the corresponding target generated image.

[0170] See also Figure 10 , Figure 10 This is another schematic diagram of a specific implementation method for image generation provided in an embodiment of the present application, from Figure 10 It can be seen that in order to generate a target image of a bird standing on a flower in a specific style, such as a simple style, the text condition information that should be input is: a bird standing on a flower in a simple style; and when generating an image based on the text condition information "a bird standing on a flower in a simple style":

[0171] First, the text condition vector of “a bird standing on a flower in a simple style” is obtained through the text feature extractor.

[0172] Secondly, based on the text condition vector, a search is performed from a pre-built retrieval library to obtain a matching target retrieval image; if the retrieval library includes multiple image style sets, then a target image style set (simple style) that matches the image style indication information is selected from the multiple image style sets; the candidate retrieval vector of each target candidate data in the selected target image style set is respectively matched with the text condition vector for similarity to obtain a corresponding similarity matching result; based on the similarity matching result, at least one target retrieval vector that meets the similarity condition is selected from an image style set, and a target retrieval image is obtained based on the target retrieval vector.

[0173] It should be noted that during the retrieval process, retrieval can be performed cross-modally or homomodally. Cross-modal retrieval uses text condition vectors to directly retrieve images. Homomodal retrieval, on the other hand, first uses text condition vectors to search for text in the retrieval database (only for image-text pairs), and then retrieves the retrieval image corresponding to the retrieved text.

[0174] Next, the target retrieval image is input into the feature extractor to obtain the image feature vector of the target retrieval image, and a partial noise addition process is performed on the image feature vector of the target retrieval image.

[0175] The noised image feature vector and the text condition vector are then fed into the prior model to generate the image prior semantic vector. During this process, the text condition is used as a guide, and a modified image prior semantic vector is obtained through embedding to embedding, which serves as the image prior. The prior model can extract and modify information from the retrieved image, processing it into high-level semantic information suitable for subsequent text-to-image models. This improves the stability of the generated information and enhances the model's contextual capabilities.

[0176] Finally, the image prior semantic vector generated by the prior model and the text vector of the text condition information are used as image generation conditions to guide the text-based graph model to generate the corresponding target generated image.

[0177] It should be noted that when the data in the retrieval library is an image-text pair, when obtaining the target retrieval image from the retrieval library, the descriptive text associated with the target retrieval image can also be obtained synchronously, and the target retrieval image, descriptive text and text condition information can be used together as the guiding conditions of the text-generated image model to generate the target model.

[0178] In the present application, first, based on the text condition vector of the text condition information, at least one target retrieval vector that matches the text condition vector is determined from at least one candidate retrieval vector stored in a pre-built retrieval library, and then the target retrieval image corresponding to each of the at least one target retrieval vectors is obtained; image retrieval is performed based on the text condition information, so that the retrieved target retrieval image matches the text condition information, thereby ensuring the accuracy of the target retrieval image introduced in the process of the text-generated image; then, based on the text condition vector, the image feature vector of at least one target retrieval image is subjected to content correction processing to obtain the corresponding prior image semantic vector; information is extracted and corrected on the image features of the target retrieval image, so that the prior image semantic vector is closer to the text condition information, thereby ensuring the quality and accuracy of the prior image semantic vector; finally, based on at least one prior image semantic vector, the target generated image corresponding to the text condition information is generated in combination with the text condition vector. In the image generation process, in addition to the text condition information, high-quality external image knowledge (i.e., the prior image semantic vector of the target retrieval image) is also introduced as the prior information of the text-generated image, and the reference information is enhanced to improve the controllability and stability of the image generation, avoid distortion of the generated image, and improve the quality of the generated image.

[0179] In the embodiment of the present application, since the prior model is used in the image generation process to obtain the prior image semantic vector, and the Vincent graph model is used to generate the target generation image, in order to ensure the accuracy of the model output results, the model should be trained before using it. And since both the prior model and the Vincent graph model are used in the image generation process, a joint training method can be adopted, or a method of training the two models separately can be adopted. Below, the image generation model training process is explained by taking the joint training method as an example. It should be noted that the image generation model includes a prior model and a Vincent graph model.

[0180] In an embodiment of the present application, during the training of the image generation model, a total of multiple rounds (e.g., 100) of iterations are performed on all pairs of image-text samples (i.e., the training set of image-text sample pairs), wherein one round of iteration is defined as the complete training of all pairs of image-text samples in the image generation model to be trained. In each round of iteration, due to the limited video memory resources of the training machine, all pairs of samples cannot be input into the image generation model for training at one time. Therefore, all pairs of samples need to be trained in batches, and each batch of samples is generated by, for example, random partitioning, and each batch of samples is input into the image generation model for training such as forward calculation, backward calculation, and model parameter update.

[0181] Before the first round of training, the parameters of the image generation model to be trained need to be initialized. Furthermore, hyperparameters such as batch size, number of epochs, and learning rate are set. After these settings are completed, training begins to obtain a trained image generation model that supports image retrieval, image prior knowledge acquisition, and image generation. The learning rate is set to 1e-4.

[0182] Since the operations performed in each loop iteration are the same, we will take one loop iteration as an example to illustrate the training of the image generation model. Figure 11 , Figure 11 A flow chart of a model training method provided in an embodiment of the present application includes the following steps:

[0183] Step S1100 : selecting an image-text sample pair from an image-text sample pair training set; wherein the image-text sample pair includes: a sample image and a description text of the sample image.

[0184] Step S1101 : Based on the text feature vector of the description text, at least one target retrieval image matching the description text is determined from at least one candidate retrieval vector stored in a pre-built retrieval library.

[0185] Step S1102 : Based on the text feature vector describing the text, content correction processing is performed on the image feature vector of at least one target retrieval image to obtain a corresponding priori image semantic vector.

[0186] Step S1103 : generating a predicted image corresponding to the description text based on at least one prior image semantic vector and in combination with the text feature vector of the description text.

[0187] Step S1104: Adjust the model parameters based on the loss function constructed based on the difference between the predicted generated image and the sample image.

[0188] In order to understand the model training process more clearly, based on Figure 11 A schematic diagram of the specific implementation of model training is given, see Figure 12 ,from Figure 12As can be seen in the figure: the descriptive text of the sample image is input into the text feature extractor to obtain the text feature vector. Based on the text feature vector, a matching target retrieval image is obtained from the retrieval library. The target retrieval image and the text feature vector are input into the trained prior model of the image generation model to be trained to obtain the prior image semantic vector of the target retrieval image. The prior image semantic vector and the text feature vector are input into the trained text-image model of the image generation model to be trained to obtain the predicted generated image. A loss function is constructed based on the predicted generated image and the sample image. The loss function is used to reversely adjust the parameters in the model. It should be noted that after completing the training of a batch, the iteration process ends.

[0189] It should be noted that the model training method in the embodiments of the present application can be executed by a computing device, which can be a server or a terminal device. That is, the model training method can be executed by the server or the terminal device alone, or by the server and the terminal device together. The model training process in the embodiments of the present application is similar to the model application process. The detailed steps can be found in the aforementioned image generation method section and will not be repeated here.

[0190] In an embodiment of the present application, before adjusting the model parameters, it is also possible to determine whether the model convergence conditions are met. Exemplarily, the model convergence conditions may include at least one of the following conditions: the model loss is not greater than a preset loss value threshold; the number of iterations reaches a preset upper limit.

[0191] In the present application, the model in the image generation process is trained to obtain a trained model with accurate output, and the trained model is further used for image generation to ensure the accuracy of the generated image. In the present application, the retrieval enhancement framework corresponding to the retrieval library is well compatible with the image generation model, so that the image generation method based on the retrieval library proposed in the present application has strong scalability. That is, after the retrieval enhancement framework corresponding to the retrieval library is constructed, it is only necessary to expand the retrieval library to improve the image generation capability. For example, a database containing different image styles can be constructed to achieve style control of a small number of samples and expand the stylized generation capability of image generation. At the same time, based on the combination of the retrieval library and the image generation model, the image generation method proposed in the present application is safe, and the controllability and stability of the image generation model can be enhanced through retrieval. The images generated by the image generation model are diverse, and external diverse image information can be introduced through retrieval, thereby improving the diversity of image generation. By introducing high-quality external image knowledge and the proposed retrieval enhancement framework, generation distortion can be prevented and the quality of the generated image can be improved.

[0192] It should also be noted that although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0193] Based on the same inventive concept, the present embodiment further provides an image generating device 1300, see Figure 13 , the image generating device 1300 includes:

[0194] Determining unit 1301 is configured to determine, based on the text condition vector of the text condition information, at least one target retrieval vector that matches the text condition vector from at least one candidate retrieval vector stored in a pre-built retrieval library, and obtain a target retrieval image corresponding to each of the at least one target retrieval vectors; each candidate retrieval vector is either an image retrieval vector or a text retrieval vector;

[0195] A processing unit 1302 is configured to perform content correction processing on the image feature vectors of at least one target retrieval image based on the text condition vector to obtain a corresponding prior image semantic vector;

[0196] The generating unit 1303 is configured to generate a target generated image corresponding to the text condition information based on at least one prior image semantic vector and in combination with the text condition vector.

[0197] In a possible implementation, the search library is constructed as follows:

[0198] Acquire at least one historical candidate data, each historical candidate data being a historical image or a historical image-text pair, wherein the historical image-text pair includes: a historical image and text description information for describing the historical image;

[0199] Performing data cleaning on at least one historical candidate data to obtain at least one target candidate data;

[0200] Perform feature extraction on at least one target candidate data to obtain a candidate retrieval vector associated with each target candidate data; wherein the candidate retrieval vector and the text condition vector are in the same vector space;

[0201] Each target candidate data and the associated candidate search vector are stored in the search library.

[0202] In one possible implementation, performing data cleaning on at least one historical candidate data to obtain at least one target candidate data includes:

[0203] For at least one historical candidate data, respectively obtain an image evaluation value of a corresponding historical image; wherein the image evaluation value includes at least one of an interest evaluation value and a visual quality evaluation value, and the visual quality evaluation value is determined based on image resolution and image content;

[0204] Based on at least one image evaluation value, historical candidate data having an image evaluation value reaching an evaluation value threshold is selected from at least one historical candidate data, and the selected historical candidate data is used as target candidate data.

[0205] In a possible implementation, the retrieval library includes at least one image style set, and each image style set includes at least one target candidate data belonging to the same image style.

[0206] In a possible implementation, the determining unit 1301 is specifically configured to:

[0207] For each image style set, perform the following operations:

[0208] Perform similarity matching between the candidate retrieval vector of each target candidate data in the image style set and the text condition vector to obtain the corresponding similarity matching results;

[0209] Based on the similarity matching result, at least one target retrieval vector that meets the similarity condition is selected from the image style set.

[0210] In a possible implementation, if the text condition information includes image style indication information, the determining unit 1301 is specifically configured to:

[0211] Selecting a target image style set that matches the image style indication information from the at least one image style set;

[0212] Perform similarity matching on the candidate retrieval vector of each target candidate data in the selected target image style set and the text condition vector respectively to obtain the corresponding similarity matching results;

[0213] Based on the similarity matching result, at least one target retrieval vector that meets the similarity condition is selected from an image style set.

[0214] In a possible implementation, the processing unit 1302 is specifically configured to:

[0215] For each target image, perform the following operations:

[0216] Obtain the image feature vector of the target retrieval image, and perform noise processing on each image feature vector to obtain the corresponding image noise vector;

[0217] The image noise vector and the text condition vector are spliced ​​to obtain a spliced ​​vector;

[0218] Based on the weight of each vector element in the splicing vector, the image feature vector is modified to obtain the corresponding prior image semantic vector.

[0219] In a possible implementation, the generating unit 1303 is specifically configured to:

[0220] For each prior image semantic vector, determining an image key matrix and an image value matrix corresponding to the prior image semantic vector based on a preset image key weight matrix and a preset image value weight matrix, respectively; determining a semantic vector weight based on a preset query matrix and the image key matrix, and determining a corresponding first prediction vector based on the semantic vector weight and the corresponding image value matrix;

[0221] For the text condition vector, based on a preset text key weight matrix and a preset text value weight matrix, a text key matrix and a text value matrix are determined; based on a preset query matrix and the text key matrix, a text condition vector weight is determined, and based on the text condition vector weight and the corresponding text value matrix, a second prediction vector is determined;

[0222] At least one first prediction vector and a second prediction vector are fused to obtain a fused vector, and a target generated image corresponding to the text condition information is generated based on the fused vector.

[0223] It should be noted that although several units (or modules) of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units (or modules) described above can be concretized in one unit (or module). Conversely, the features and functions of one unit (or module) described above can be further divided into multiple units (or modules) for concretization. Of course, when implementing the present application, the functions of each unit (or module) can also be implemented in the same or multiple software or hardware.

[0224] In the embodiments of the present application, the term unit (or module) refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0225] After introducing the image generation method and apparatus according to an exemplary embodiment of the present application, another exemplary embodiment of the present application, an electronic device, will be introduced next.

[0226] Those skilled in the art will appreciate that various aspects of the present application may be implemented as systems, methods, or program products. Therefore, various aspects of the present application may be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0227] In one possible implementation, the electronic device provided in the embodiments of the present application may include at least a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform any step of the image generation method in the various exemplary embodiments of the present application.

[0228] In this embodiment, the structure of the electronic device can be as follows Figure 14 As shown, it includes a memory 1401 , a communication module 1403 and one or more processors 1402 .

[0229] Memory 1401 is used to store computer programs executed by processor 1402. Memory 1401 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and programs required for running instant messaging functions, while the data storage area may store various instant messaging messages and operating instruction sets.

[0230] Memory 1401 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing a desired computer program in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1401 may be a combination of the aforementioned memories.

[0231] The processor 1402 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1402 is configured to implement the above-mentioned image generation method when calling the computer program stored in the memory 1401 .

[0232] The communication module 1403 is used to communicate with the target device and other servers.

[0233] The specific connection medium between the memory 1401, the communication module 1403 and the processor 1402 is not limited in the embodiment of the present application. Figure 14 In the embodiment, the memory 1401 and the processor 1402 are connected via a bus 1404. Figure 14 The connections between the other components are shown in bold lines for illustration only and are not intended to be limiting. The bus 1404 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 14 The diagram shows a single thick line, but this does not indicate that there is only one bus or one type of bus.

[0234] The memory 1401 stores a computer storage medium, which stores computer executable instructions for implementing the image generation method of the embodiment of the present application. The processor 1402 is used to execute the above-mentioned image generation method.

[0235] In some possible embodiments, various aspects of the image generation method provided in the present application can also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the image generation method according to various exemplary embodiments of the present application described above in this specification.

[0236] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0237] The program product of the embodiment of the present application may be a portable compact disc read-only memory (CD-ROM) and include a computer program, and can be run on a computing device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with a command execution system, device, or device.

[0238] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with a command execution system, apparatus, or device.

[0239] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0240] The computer program for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program can be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any variety of models, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0241] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0242] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to produce a machine, so that the commands executed by the processor of the computer or other programmable device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that specifies functions in a box or multiple boxes.

[0243] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device that implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 Specify the function in one or more boxes.

[0244] These computer program instructions can also be loaded onto a computer or other programmable device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the process in the computer or other programmable device. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0245] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0246] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. An image generation method, characterized in that: The method comprises: Based on a text condition vector of the text condition information, determining at least one target retrieval vector that matches the text condition vector from at least one candidate retrieval vector stored in a pre-built retrieval library, and obtaining a target retrieval image corresponding to each of the at least one target retrieval vectors; each candidate retrieval vector is an image retrieval vector or a text retrieval vector; Based on the text condition vector, performing content correction processing on the image feature vector of at least one of the target retrieval images to obtain a corresponding prior image semantic vector; Based on at least one of the prior image semantic vectors and in combination with the text condition vector, a target generated image corresponding to the text condition information is generated.

2. The method according to claim 1, wherein The search library is constructed in the following manner: Acquire at least one historical candidate data, each historical candidate data being a historical image or a historical image-text pair, wherein the historical image-text pair includes: a historical image and text description information for describing the historical image; Performing data cleaning on the at least one historical candidate data to obtain at least one target candidate data; Performing feature extraction on the at least one target candidate data to obtain a candidate retrieval vector associated with each target candidate data; wherein the candidate retrieval vector and the text condition vector are in the same vector space; The target candidate data and the associated candidate search vectors are correspondingly stored in the search library.

3. The method according to claim 2, wherein The step of performing data cleaning on the at least one historical candidate data to obtain at least one target candidate data includes: For the at least one historical candidate data, respectively obtain an image evaluation value of a corresponding historical image; wherein the image evaluation value includes at least one of an interest evaluation value and a visual quality evaluation value, and the visual quality evaluation value is determined based on image resolution and image content; Based on at least one of the image evaluation values, a historical candidate data whose image evaluation value reaches an evaluation value threshold is selected from the at least one historical candidate data, and the selected historical candidate data is used as the target candidate data.

4. The method according to any one of claims 1 to 3, characterized in that: The retrieval library includes at least one image style set, and each image style set includes at least one target candidate data belonging to the same image style.

5. The method according to claim 4, wherein The text condition vector based on the text condition information determines at least one target search vector that matches the text condition vector from at least one candidate search vector stored in a pre-built search library, including: For each of the image style sets, perform the following operations respectively: Performing similarity matching on the candidate retrieval vector of each target candidate data in the image style set and the text condition vector respectively to obtain corresponding similarity matching results; Based on the similarity matching result, at least one target retrieval vector that meets the similarity condition is selected from the image style set.

6. The method according to claim 4, wherein If the text condition information includes image style indication information, determining at least one target search vector that matches the text condition vector from at least one candidate search vector stored in a pre-built search library based on the text condition information includes: selecting, from the at least one image style set, a target image style set that matches the image style indication information; Performing similarity matching on the candidate retrieval vector of each target candidate data in the selected target image style set and the text condition vector respectively to obtain corresponding similarity matching results; Based on the similarity matching result, at least one target retrieval vector that meets the similarity condition is selected from the one image style set.

7. The method according to any one of claims 1 to 3, characterized in that: The step of performing content correction processing on the image feature vector of at least one target retrieval image based on the text condition vector to obtain a corresponding prior image semantic vector includes: For each target image, perform the following operations: Obtaining an image feature vector of the target retrieval image, and performing noise processing on each image feature vector to obtain a corresponding image noise vector; Splicing the image noise vector and the text condition vector to obtain a splicing vector; Based on the weight of each vector element in the splicing vector, content correction processing is performed on the image feature vector to obtain a corresponding priori image semantic vector.

8. The method according to any one of claims 1 to 3, characterized in that: The step of generating a target generated image corresponding to the text condition information based on at least one of the prior image semantic vectors and in combination with the text condition vector includes: For each of the prior image semantic vectors, determining an image key matrix and an image value matrix corresponding to the prior image semantic vector based on a preset image key weight matrix and a preset image value weight matrix, respectively; determining a semantic vector weight based on a preset query matrix and the image key matrix, and determining a corresponding first prediction vector based on the semantic vector weight and the corresponding image value matrix; For the text condition vector, based on a preset text key weight matrix and a preset text value weight matrix, a text key matrix and a text value matrix are determined; based on a preset query matrix and the text key matrix, a text condition vector weight is determined, and based on the text condition vector weight and the corresponding text value matrix, a second prediction vector is determined; At least one of the first prediction vector and the second prediction vector is fused to obtain a fused vector, and a target generated image corresponding to the text condition information is generated based on the fused vector.

9. An image generating device, characterized in that: The device comprises: a determination unit configured to determine, based on a text condition vector of the text condition information, at least one target retrieval vector that matches the text condition vector from at least one candidate retrieval vector stored in a pre-built retrieval library, and obtain a target retrieval image corresponding to each of the at least one target retrieval vectors; each candidate retrieval vector is an image retrieval vector or a text retrieval vector; a processing unit configured to perform content correction processing on the image feature vector of at least one target retrieval image based on the text condition vector to obtain a corresponding prior image semantic vector; A generating unit is configured to generate a target generated image corresponding to the text condition information based on at least one of the prior image semantic vectors and in combination with the text condition vector.

10. The device according to claim 9, wherein The retrieval library includes at least one image style set, and each image style set includes at least one target candidate data belonging to the same image style.

11. The device according to claim 10, wherein The determining unit is specifically configured to: For each of the image style sets, perform the following operations respectively: Performing similarity matching on the candidate retrieval vector of each target candidate data in the image style set and the text condition vector respectively to obtain corresponding similarity matching results; Based on the similarity matching result, at least one target retrieval vector that meets the similarity condition is selected from the image style set.

12. The device according to claim 10, wherein If the text condition information includes image style indication information, the determining unit is specifically configured to: selecting, from the at least one image style set, a target image style set that matches the image style indication information; Performing similarity matching on the candidate retrieval vector of each target candidate data in the selected target image style set and the text condition vector respectively to obtain corresponding similarity matching results; Based on the similarity matching result, at least one target retrieval vector that meets the similarity condition is selected from the one image style set.

13. A computing device, characterized in that The computing device comprises: a processor and a memory, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method according to any one of claims 1 to 8.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

15. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when executed by a processor.

Images