Method and apparatus for determining picture generation model, method and apparatus for picture generation, computing device, storage medium, and program product

Through the combination of noise prediction model and preselected model, the image generation model is optimized, which solves the problems of high cost and poor quality of sample acquisition in the field of image recognition and classification, and achieves efficient and good quality image generation, improving model training efficiency and performance.

WO2024239755A1PCT designated stage expired Publication Date: 2024-11-28TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Patent Information

Application Number
PCT/CN2024/079813
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-25
Filing Date
2024-03-04
Publication Date
2024-11-28

Smart Images

  • Figure CN2024079813_28112024_PF_FP_ABST
    Figure CN2024079813_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to the technical field of machine learning, particularly to a method and apparatus for determining a picture generation model, a method and apparatus for picture generation, a computing device, a storage medium, and a program product. The determination method comprises: acquiring first guidance information and second guidance information; combining the first guidance information and the second guidance information to obtain combined guidance information; inputting the first guidance information and a first noise-containing picture into a noise prediction model to identify a first noise feature from the first noise-containing picture; inputting the second guidance information and a second noise-containing picture into the noise prediction model to identify a second noise feature from the second noise-containing picture; inputting the combined guidance information and a third noise-containing picture into a pre-selection model with the same model structure as the noise prediction model to identify a third noise feature from the third noise-containing picture; determining a combined noise feature on the basis of the first noise feature and the second noise feature; and adjusting model parameters of the pre-selection model according to the difference between the combined noise feature and the third noise feature to update the pre-selection model.
Need to check novelty before this filing date? Find Prior Art

Description

Method and device for determining image generation model, image generation method and device, computing device, storage medium, and program product

[0001] This application claims priority to the Chinese patent application with application number 202310601130.X filed on May 25, 2023, and invention name “Method and device for determining image generation model, method and device for image generation”. Technical Field

[0002] The present disclosure relates to the field of machine learning, and in particular, to a method and apparatus for determining an image generation model, an image generation method and apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0003] With the rapid development of machine learning technology, various machine learning models are playing an increasingly important role in diverse fields. For example, in areas involving image recognition and classification, such as content moderation, corresponding machine learning models can replace manual labor, efficiently and accurately completing large amounts of image processing tasks. Before these machine learning models are put into use, they often require training on a large number of image samples to achieve the expected performance.

[0004] Summary of the Invention

[0005] In view of this, the present disclosure provides a method and apparatus for determining an image generation model, an image generation method and apparatus, a computing device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, a method for determining a picture generation model is provided, including: obtaining first guidance information and second guidance information, the first guidance information being used to characterize a first image feature, and the second guidance information being used to characterize a second image feature; inputting the first guidance information and a first noisy picture into a noise prediction model to identify a first noise feature from the first noisy picture; inputting the second guidance information and the second noisy picture into the noise prediction model to identify a second noise feature from the second noisy picture; inputting merged guidance information including the first guidance information and the second guidance information and a third noisy picture into a preselected model having the same model structure as the noise prediction model to identify a third noise feature from the third noisy picture; merging the first noise feature and the second noise feature to obtain a merged noise feature; and adjusting model parameters of the preselected model according to a difference between the merged noise feature and the third noise feature to update the preselected model.

[0007] According to another aspect of the present disclosure, a picture generation method is provided, including: obtaining guidance information; inputting the guidance information into a picture generation model, the picture generation model being determined by the method for determining the picture generation model described in the aforementioned aspect; and obtaining a picture corresponding to the guidance information based on the output of the picture generation model.

[0008] According to another aspect of the present disclosure, a device for determining a picture generation model is provided, comprising: an acquisition module configured to acquire first guidance information and second guidance information, the first guidance information being used to characterize a first image feature, and the second guidance information being used to characterize a second image feature; a first generation module configured to input the first guidance information and a first noisy picture into a noise prediction model to identify a first noise feature from the first noisy picture; a second generation module configured to input the second guidance information and the second noisy picture into the noise prediction model to identify a second noise feature from the second noisy picture; a third generation module configured to input merged guidance information including the first guidance information and the second guidance information and a third noisy picture into a preselected model having the same model structure as the noise prediction model to identify a third noise feature from the third noisy picture; a determination module configured to merge the first noise feature and the second noise feature to obtain a merged noise feature; and an update module configured to adjust model parameters of the preselected model according to a difference between the merged noise feature and the third noise feature to update the preselected model.

[0009] According to yet another aspect of the present disclosure, a computing device is provided, comprising: a memory configured to store computer-executable instructions; and a processor configured to perform the method described according to the aforementioned aspects when the computer-executable instructions are executed by the processor.

[0010] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed, the method described in the above aspects is executed.

[0011] According to yet another aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, which implement the steps of the method described in the above aspects when executed by a processor.

[0012] According to the method for determining an image generation model provided by the present disclosure, a noise prediction model can be used to obtain a first noise feature and a second noise feature based on the first and second guidance information, respectively. A preselected model with the same structure as the noise prediction model can be used to obtain a third noise feature based on the merged guidance information. The model parameters of the preselected model are then updated based on the difference between the merged noise feature obtained based on the first and second noise features and the third noise feature. By iteratively executing the above method for determining an image generation model, the model parameters of the preselected model can be continuously optimized so that when the determined image generation model outputs image data, it can reflect the global information involved in the merged guidance information while not blurring or ignoring the local information involved in the first and second guidance information. Thus, when used, the determined image generation model can quickly generate a large number of image samples that accurately reflect the content of the guidance information and have the expected labels by inputting the guidance information, which helps reduce the cost of acquiring image samples and improve the quality of the image samples. When these high-quality image samples are used to train image recognition models or image classification models, the training efficiency of these models can be improved, computing resources can be saved, and the processing performance of these models in fields such as content review can be optimized.

[0013] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0015] FIG1 schematically illustrates an example application scenario in which the technical solution provided by the present disclosure can be applied;

[0016] FIG2A schematically shows an example flowchart of a method for determining an image generation model according to some embodiments of the present disclosure;

[0017] FIG2B schematically shows another example flowchart of a method for determining an image generation model according to some embodiments of the present disclosure;

[0018] FIG3 schematically shows an example picture according to some embodiments of the present disclosure;

[0019] FIG4 schematically illustrates an example training process of a text generation model according to some embodiments of the present disclosure;

[0020] FIG5 schematically illustrates an example model structure of a noise prediction model and a preselected model according to some embodiments of the present disclosure;

[0021] FIG6 schematically illustrates an example training architecture of a text-to-image conversion model according to some embodiments of the present disclosure;

[0022] FIG7A schematically illustrates an example of a diffusion process according to some embodiments of the present disclosure;

[0023] FIG7B schematically illustrates an example of a noise prediction process according to some embodiments of the present disclosure;

[0024] FIG8 schematically illustrates an example training architecture for determining an image generation model according to some embodiments of the present disclosure;

[0025] FIG9 schematically illustrates an example internal structure of a preselected model according to some embodiments of the present disclosure;

[0026] FIG10 schematically shows an example flow chart of a method for generating a picture according to some embodiments of the present disclosure;

[0027] FIG11 schematically shows an example block diagram of an apparatus for determining an image generation model according to some embodiments of the present disclosure;

[0028] FIG12 schematically shows an example block diagram of a picture generating apparatus according to some embodiments of the present disclosure;

[0029] FIG13 schematically illustrates an example block diagram of a computing device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] Before introducing the embodiments of the present disclosure in detail, some related concepts are first explained.

[0031] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0032] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0033] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing them to produce images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0034] Key technologies in speech technology include automatic speech recognition (ASR), text-to-speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech becoming one of the most promising methods of human-computer interaction.

[0035] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0036] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0037] Autonomous driving technology usually includes high-precision maps, environmental perception, behavioral decision-making, path planning, motion control and other technologies. Autonomous driving technology has broad application prospects.

[0038] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0039] The solutions provided in the embodiments of this application involve artificial intelligence machine learning, natural language processing, computer vision and other technologies, which will be specifically explained through examples below.

[0040] In addition, in this disclosure, the "guidance information" mentioned can be text information used to guide the model to generate the corresponding image. For example, the guidance information can be provided manually by the user, or can be automatically generated by an algorithm or model, or can be provided by a combination of the two, etc.

[0041] In the present disclosure, the "frozen model" mentioned may refer to a model whose parameters have been frozen. For example, at least a portion of the model parameters of the frozen model are fixed and will not be updated as the method for determining the image generation model described in the embodiments of the present disclosure is executed. Alternatively, the frozen model may be an existing model or a pre-trained model. More specifically, in the present disclosure, the frozen model may be a trained model that generates image data based on input text data, and the image represented by the output image data may include the object depicted by the input text data. For example, the input text data may depict objects such as the sun, a house, etc., and the image represented by the output image data may include objects such as the sun, a house, etc. Exemplarily, the frozen model may be an SD (Stable Diffusion) model, such as the SD V2.0 model, or other image generation models may be used as frozen models.

[0042] In this disclosure, the "text generation model" referred to may refer to a model that can feedback output text based on input (e.g., input text). For example, it can feedback output text by expanding or rewriting the input text according to preset rules. Exemplarily, the text generation model can be a ChatGPT (Chat Generative Pre-trained Transformer) model, or other types of text generation models may also be used.

[0043] FIG1 schematically illustrates an example application scenario 100 to which the technical solution provided by the present disclosure can be applied.

[0044] As shown in FIG1 , scenario 100 may include a server 110. Server 110 may be a single server or a server cluster, on which an application for executing the method for determining an image generation model and / or an image generation method according to various embodiments of the present disclosure may be run, and illustratively, related data may also be stored. Server 110 may also run other applications and store other data. For example, server 110 may include multiple virtual hosts to run different applications and provide different services.

[0045] The scenario 100 may also include a terminal device 120. The terminal device 120 may be various types of devices, such as a mobile phone, a tablet computer, a laptop computer, a wearable device such as a smart watch, an in-vehicle device, etc. Exemplarily, the method for determining an image generation model or the image generation method according to various embodiments of the present disclosure may also be executed by the terminal device 120, for example, by an application deployed on the terminal device 120. Alternatively, the method for determining an image generation model or the image generation method according to various embodiments of the present disclosure may also be executed in combination by the terminal device 120 and the server 110. For example, the steps included in the method for determining an image generation model or the image generation method according to various embodiments of the present disclosure may be partially executed at the terminal device 120 and partially executed at the server 110.

[0046] Exemplarily, the user can input data such as text information through the input interface of the server 110 or the terminal device 120 (such as a keyboard, a microphone, a data interface, etc.), or the data can also be pre-stored in the storage device of the server 110 or the terminal device 120 or other external storage devices and automatically read when needed. The server 110 and / or the terminal device 120 can execute the method for determining the image generation model or the image generation method provided in the present disclosure. The user can view the various processing data, update parameters, etc. in the process of determining the image generation model and view the images generated by the determined image generation model through the output interface of the server 110 or the terminal device 120 (such as a display, a touch screen, etc.).

[0047] In addition, the scene 110 may also include a database device 130. The database device 130 can be regarded as an electronic file cabinet, that is, a place for storing electronic files, and users can perform operations such as adding, querying, updating, and deleting data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application. Exemplarily, the database device 130 can be used to store data or files such as text and pictures. For example, the server 110 or the terminal device 120 can obtain the required data from the database 130, or can send (for example, upload) the generated and updated data to the database 130.

[0048] The server 110, the terminal device 120, and the database 130 can communicate with each other via a network 140. The network 140 can be a wired network connected via a cable, an optical fiber, or a wireless network such as 2G, 3G, 4G, 5G, Wi-Fi, Bluetooth, ZigBee, or Li-Fi.

[0049] Through analysis of relevant technologies, the applicant discovered that with the rapid development of machine learning technology, various machine learning models are increasingly being used to perform tasks such as image recognition and classification in application areas involving these tasks. While this helps improve task processing efficiency and saves manpower and time, the performance of these machine learning models is closely related to the quality and quantity of training samples. However, due to limitations in the application area, obtaining large quantities of high-quality training samples is often difficult and requires significant manpower and time. Specifically, in image processing applications such as content review, problems such as missing data, incomplete labeling, poor quality, and insufficient data utilization are often encountered. Missing data often manifests itself in limited data sources or difficulties in data collection. For example, available data may be significantly dependent on data provided by the model user. If the data available is limited, it will be difficult to obtain sufficient training sample data to optimize the model to the desired performance. Incomplete labeling is mainly manifested in data annotation standards that vary from different modalities or perspectives. For supervised training, it is generally necessary to label the initial data obtained, that is, to assign corresponding labels to each sample. This type of annotation is often done manually, and due to differences in annotation standards and the influence of the perspective of the objects in the image, incomplete labels are often produced. This poor quality stems from the fact that much of the data is collected from publicly available sources online, resulting in varying image quality. For example, some images have very low pixel count, which significantly impacts model performance. Insufficient data utilization primarily manifests itself in methods such as image classification, where only the image itself is considered, while other multimodal information, such as the underlying text, is ignored. In summary, conventional sample acquisition methods generally require a significant amount of time to accumulate and annotate relevant data. This significant time consumption is inconsistent with the rapid rollout and iteration requirements of relevant application domains. Furthermore, the quality of samples obtained using this method is difficult to guarantee, and the inadequate utilization of multidimensional information can negatively impact the quality and efficiency of model training.

[0050] Based on the above considerations, the applicant has proposed a scheme for determining an image generation model and an image generation scheme, so as to allow a large number of high-quality images with expected labels to be quickly generated for the input text. These images can be used as samples for the training process of relevant models. This can greatly shorten the acquisition speed of image samples, while ensuring that the image samples have the expected image quality, thereby helping to improve the overall efficiency of model launch or optimization, and helping to improve model performance. The scheme for determining an image generation model and the image generation scheme proposed in the present disclosure will be described in detail below with reference to the accompanying drawings.

[0051] FIG2A schematically illustrates an example flow chart of a method 200 for determining an image generation model according to some embodiments of the present disclosure. FIG2B schematically illustrates an example flow chart of a method 200 for determining an image generation model according to some embodiments of the present disclosure. Exemplarily, the method 200 may be applied to the scenario 100 shown in FIG1 . For example, the method 200 may be deployed on the server 110, the terminal device 120, or a combination thereof in a form such as an application. As shown in FIG2A in conjunction with FIG2B , the method 200 may include steps 210 to 260, as follows.

[0052] In step 210, first guidance information and second guidance information may be acquired. The first guidance information is used to characterize the first image feature, and the second guidance information is used to characterize the second image feature.

[0053] Among them, the first guidance information may include description information of the first image feature. The second guidance information may include description information of the second image feature. The first guidance information and the second guidance information may be description information for different image features of the same picture. The same picture is, for example, a picture to be generated. The first image feature and the second image feature are different image features in the picture to be generated. The first guidance information and the second guidance information may be obtained from manually provided data, or may be obtained from data automatically generated by algorithms, models, etc., or may be obtained from data manually provided and processed by algorithms, models, etc. Exemplarily, the first guidance information and the second guidance information may be obtained from information provided in the form of text data, or may be obtained from information provided in other data forms (such as voice and other data forms).

[0054] Exemplarily, the first guidance information and the second guidance information may respectively include descriptions of one or several image features in the same picture or different pictures. The same picture or different pictures are, for example, pictures to be generated. In the present disclosure, the image features described by the guidance information may refer to, for example, objects, backgrounds, or other features related to the pictures. For example, the first guidance information may include description information of a first image feature in a picture (e.g., a plate, a cup), and the second guidance information may include description information of a second image feature in a picture (e.g., an apple, a pear), etc. In addition, exemplarily, in addition to the first and second guidance information, more guidance information may be obtained, such as third guidance information, fourth guidance information, etc.

[0055] At step 220, the first guidance information and the first noisy image may be input into a noise prediction model to identify a first noise feature from the first noisy image. The first noisy image includes a first image feature. The noise prediction model is a pre-trained model with at least some model parameters frozen. Frozen parameters mean that the model parameters are fixed and will not be updated.

[0056] In an embodiment of the present application, the noise prediction model is, for example, a frozen model. A first noise feature (or first prediction) can be generated by the noise prediction model based on the first guidance information. The noise prediction model is configured to output an image based on the input guidance information, such that the output image includes the image features described by the input guidance information. The output image is an image obtained by removing the noise feature.

[0057] Exemplarily, the obtained first guidance information can be directly input into the noise prediction model, or the first guidance information can be pre-processed and then input into the noise prediction model. Exemplarily, the first noise feature can be obtained based on the output of the noise prediction model, or the first noise feature can be obtained based on the intermediate processed data of the noise prediction model. And, exemplarily, the first noise feature can be obtained directly based on the output or intermediate processed data of the noise prediction model, or the output or intermediate processed data can be processed to obtain the first noise feature. Exemplarily, the noise prediction model can be a trained image generation model that can output image data based on the input guidance information. The output image data can describe an image that includes the object described by the guidance information. For example, if the input guidance information includes information related to an apple, the image described by the output image data can include the object apple. Exemplarily, a pre-trained image generation model can be selected as the noise prediction model, such as the SD model mentioned above. As mentioned above, at least some model parameters of the noise prediction model are frozen and are not updated as method 200 is executed.

[0058] At step 230, the second guidance information and the second noisy picture are input into a noise prediction model to identify a second noise feature from the second noisy picture. The second noisy picture includes a second image feature.

[0059] In an embodiment of the present application, a second noise feature (or second prediction) can be generated based on the second guidance information through a noise prediction model. For example, the second noise feature can be generated based on the second guidance information through a process similar to step 220. The noise prediction model used in step 230 and the noise prediction model used in step 220 can be a unified model, or two models with the same structure and parameters. For example, if there is other guidance information, other noise features corresponding to the other guidance information can be obtained in the same manner as step 220.

[0060] At step 240, the combined guidance information including the first guidance information and the second guidance information and the third noisy picture are input into a preselected model having the same model structure as the noise prediction model to identify a third noise feature from the third noisy picture. The third noisy picture includes the first image feature and the second image feature.

[0061] The combined guidance information, the first guidance information, and the second guidance information may be different descriptive information for the same image. For example, the same image may be an image to be generated by an image generation model. In one example, the combined guidance information may include descriptive information for the first image feature and the second image feature, or in other words, the combined guidance information includes the first guidance information and the second guidance information. In another example, the combined guidance information may include descriptive information for the first image feature and the second image feature as well as additional information, or in other words, the combined guidance information includes the first guidance information, the second guidance information, and the additional information.

[0062] The merged guidance information can include all descriptions of image features included in the first and second guidance information, as well as additional information not included in the first or second guidance information. The additional information can, for example, describe the spatial relationship between two or more image features. If the first guidance information may include descriptions of a first image feature in an image (e.g., a plate or a cup), and the second guidance information may include descriptions of a second image feature in the image (e.g., an apple or a pear), the merged guidance information can include both information that is identical or similar to the description of the first image feature (e.g., the plate or cup) in the first guidance information, information that is semantically identical or similar to the description of the second image feature (e.g., the apple or pear) in the second guidance information, and additional information. This additional information can, for example, specify that the plate is below the apple. Exemplarily, the first and second guidance information can be combined to generate the merged guidance information. The combination can include combining information that is semantically identical or similar to the description of the first and second image features to generate the merged guidance information. During the aforementioned combination process, the additional information can also be combined. The same or similar semantics can be determined by semantic similarity.

[0063] Exemplarily, preset rules can be used to constrain the specific forms of the first guidance information, the second guidance information, and the merged guidance information, and to constrain the specific relationship between the merged guidance information and the first guidance information and the second guidance information. In addition, exemplarily, in addition to the first and second guidance information, more guidance information can be obtained, such as the third guidance information, the fourth guidance information, etc. Accordingly, the merged guidance information can include combined information and additional information of the first and second guidance information and the third and fourth guidance information. Although the description of this disclosure mainly describes the case of using two guidance information, the first and second guidance information, and one merged guidance information, the technical solution provided by this disclosure is not only applicable to this case, but also applicable to the case of using three or more guidance information and one merged guidance information.

[0064] In an embodiment of the present application, a third noise feature (or a third prediction) can be generated based on the merged guidance information through a preselected model. The preselected model and the noise prediction model have the same model structure. Exemplarily, the third noise feature can be generated based on the merged guidance information through a process similar to steps 220 and 230. The difference is that the preselected model used in step 240 has the same model structure as the noise prediction model used in steps 220 and 230, but some or all of the model parameters of the preselected model are adjustable during the execution of method 200, rather than frozen.

[0065] At step 250 , the first noise signature and the second noise signature may be combined to obtain a combined noise signature.

[0066] For example, the first noise feature and the second noise feature may be merged according to a pre-set merging rule, such as determining a merged noise feature (or a merged prediction) based on the sum, weighted sum, mean, weighted mean, etc. of the first noise feature and the second noise feature. For example, if there are other noise features corresponding to other guidance information, the first noise feature, the second noise feature, the other noise features, etc. may be merged to obtain a merged noise feature.

[0067] At step 260, model parameters of the preselected model may be adjusted based on the difference between the combined noise feature and the third noise feature to update the preselected model. The updated preselected model may be used as the image generation model. For example, the combined noise feature may be considered as an expected value or a true value, and the difference between the third noise feature and the combined noise feature may be measured using the distance between the two. At least some model parameters of the preselected model may be updated based on this distance. For example, the updating of the model parameters of the preselected model may be implemented based on gradient descent or other principles.

[0068] It should be understood that the various steps of the above method 200 do not have to be performed in the order described, and at least some of the steps can be performed in parallel, or in an order opposite to the order shown or described. In addition, the above method for determining the image generation model can be iteratively executed until, in a certain iteration, the difference between the combined noise feature and the third noise feature is lower than or equal to the difference threshold, and illustratively, other performance indicators also reach the expected level, then the iteration is stopped, and the model obtained in the last iteration is put into use as the trained and perfected image generation model. Alternatively, the iteration can be stopped when the number of iterations reaches an upper limit.

[0069] Through the above-mentioned method 200, the model parameters of the pre-selected model can be optimized so that the image generation model finally determined can reflect the global information involved in the merged guidance information when outputting the image data, and will not blur or ignore the local information involved in the first guidance information and the second guidance information. Thus, when the determined image generation model is used, a large number of image samples that accurately reflect the content of the guidance information and have the expected labels can be quickly generated by inputting the guidance information, which helps to reduce the acquisition cost of the image samples and improve the quality of the image samples. When these high-quality image samples are used to train image recognition models or image classification models, the training efficiency of these models can be improved, computing resources can be saved, and the processing performance of these models in fields such as content review can be optimized.

[0070] In some embodiments, step 210 may include: obtaining first basic information and second basic information; inputting the first basic information and the second basic information into a text generation model, respectively, so that the text generation model performs at least one of expansion and modification on the first basic information and the second basic information according to preset rules, and generates first guide information and second guide information. In other embodiments, step 210 may include: obtaining first basic information, second basic information, and merged basic information; inputting the first basic information, the second basic information, and the merged basic information into a text generation model, respectively, so that the text generation model performs at least one of expansion and modification on the first basic information, the second basic information, and the merged basic information according to preset rules, and generates first guide information, second guide information, and merged guide information.

[0071] The text generation model is configured to perform at least one of expansion and modification on the input basic information according to preset rules, and output corresponding guidance information, so that the output guidance information includes richer descriptive information than the input basic information. Exemplarily, when the third and fourth guidance information are also used, the third and fourth basic information can be obtained in a similar manner, and the corresponding third and fourth guidance information can be obtained through the text generation model. In this embodiment, "richer descriptive information" can include at least one of the following two items: adding descriptive information that did not originally exist on the basis of the basic information, and expanding the expression of the basic information so that the same sentence can be expressed in different ways.

[0072] Exemplarily, the first basic information, the second basic information and the merged basic information can be obtained through an input interface or from an internal or external storage device. For example, the operator can provide the basic information through an input interface such as a keyboard or a microphone, or the basic information can be pre-written into a file and stored in an internal or external storage device, so that the corresponding basic information can be obtained by reading and parsing the file. Exemplarily, in the above embodiment, various trained text generation models can be used. Exemplarily, ChatGPT or a similar model can be used as the above text generation model. By expanding, rewriting, etc. the basic information with the help of a text generation model to obtain the corresponding guidance information, the efficiency of obtaining the guidance information can be further improved and the cost can be reduced. Specifically, the operator only needs to provide a brief basic description information without spending more time describing many details. With the help of a text generation model, tasks such as expansion and modification can be completed quickly, and many versions of guidance information can be obtained in a short time based on a set of basic information, which not only helps to improve efficiency, but also helps to improve the generalization performance of the model.

[0073] Exemplarily, in order to obtain guidance information that meets the requirements, description rules can be set in advance for the first guidance information, the second guidance information, and the merged guidance information. For example, one or more of the following description rules can be set. (1) There should be a basic description of the image to be generated to ensure that the generated guidance information contains description information of at least one expected label (expected labeled object) in the image to be generated. (2) For the generated first guidance information and the second guidance information, the spatial relationship is not considered, for example, the description of the spatial relationship between the image features (objects) in the image to be generated is not considered. (3) For the generated merged guidance information, the spatial relationship should be included, for example, the description of the spatial relationship between the image features (objects) in the image to be generated should be included. (4) The guidance information should be generated in one sentence as much as possible, and no more than two sentences. (5) For each generated guidance information, corresponding data augmentation should be performed, specifically including: generating a sentence similar to the existing sentence expression, and changing the expression of the positional relationship between the expected labels contained in the image to be generated, for example, the original expression is "knife on fork", which can be augmented to "fork under knife". By pre-setting description rules, text generation models such as ChatGPT can be better utilized to obtain the expected guidance information. More specifically, by requiring the guidance information to have a basic description of the image to be generated and ensuring the presence of descriptive information for the desired label in the image to be generated, it can be ensured that the subsequently generated images can correspond to such expected labels (including the objects corresponding to the expected labels), so that they can be directly used for model training tasks. For example, for an image classification model, the expected label is used as the expected category of the image. In addition, by expanding the description requirements, it can be ensured that the final image generation model has sufficient robustness and that the images generated by the image generation model have sufficient generalization so as to provide more diverse generated images. It should be understood that the above description rules are only exemplary, and different description rules can be designed according to actual application requirements.

[0074] To further facilitate understanding, a set of example basic information is given by taking the picture shown in Figure 3 as an example. As shown in Figure 3, the picture also contains a knife, a fork, and many foods such as radish, fish, green vegetables, and steamed buns. Suppose that for a certain business, the key objects of concern are objects such as knives and forks, that is, the labels can be knives and forks, and at the same time, the associated effects brought by food are not desired to be ignored. To this end, the first guidance information c0 that describes the knife and fork, the second guidance information c1 that integrates background information such as food, and the combined information of c0 and c1 and the merged guidance information c0+c1 of additional information are required. In this example or other similar cases, the first guidance information can also be called basic condition guidance information, and the second guidance information can also be called key detail condition guidance information. For example, the first basic information for c0 can be provided as: Please describe a picture that contains both a knife and a fork; the second basic information for c1 can be provided as: Please describe a picture that includes many objects such as radish, fish, green vegetables, steamed buns, and a dinner plate; the combined basic information for c0+c1 can be provided as: Please add many objects such as radish, fish, green vegetables, steamed buns, and a dinner plate based on c0, and the knife and fork should be placed scattered on the dinner plate. The above-mentioned first basic information, second basic information, and combined basic information can be input into a text generation model such as ChatGPT respectively. The model can expand these basic information, for example, further add descriptions about the color and material of the knife and fork, descriptions about the color and size of the radish, fish, etc., descriptions about the style of the picture, etc. These basic information can also be rewritten, for example, some of the expressions can be changed in terms of expression and description order.

[0075] For example, the text generation model described above, such as ChatGPT, can be obtained through the example training process 400 shown in Figure 4. As shown in Figure 4, the training process 400 can be divided into three stages. In the first stage, the supervision strategy model can be trained. Specifically, a question dataset can be collected, questions can be randomly selected from it, and annotators can provide expected answers. The model can then be trained or fine-tuned using this annotated data. In the second stage, the reward model can be trained. Specifically, questions can be randomly selected from the dataset, and the model obtained in the first stage can be used to generate multiple different answers for each question. The annotators can then rank these different answers, and the reward model can be trained using this ranking result. In the third stage, the strategy can be optimized through reinforcement learning. Specifically, questions can be randomly selected from the dataset, and the answers given by the model obtained in the first stage can be scored using the reward model obtained in the second stage. The model parameters can then be further updated using reinforcement learning based on the reward scores. The above process can be iteratively performed until a text generation model with the desired quality is obtained.

[0076] In some embodiments, the noise prediction model and the determined picture generation model can be a model that implements the picture generation process through a combination of a noise prediction process and a denoising process, such as the SD model mentioned above. For example, Figure 5 schematically shows an example model structure of a noise prediction model 500 according to some embodiments of the present disclosure. The preselected model and the determined picture generation model can also have the same structure. As shown in the figure, the model 500 can include an encoder ε, a decoder D, a diffusion process, a denoising process, etc.

[0077] Specifically, the encoder ε can be used to encode the input image data x from the pixel space to the latent space, obtaining z = ε (x), which can be achieved by the downsampling part of the U-net. The decoder D can be used to decode the denoised image data z from the latent space to the pixel space, obtaining This can be achieved, for example, by the upsampling part of U-net.

[0078] Both the diffusion process and the denoising process can be completed in the latent space, which can represent the high-dimensional information of the image. Diffusion and denoising in the latent space are conducive to achieving higher processing efficiency. In addition, the model 500 also includes a conditioning module, which can process data such as text, semantics, representations, and images so that the model can generate corresponding images based on the information contained in these data. For example, for text, it can be converted into a corresponding embedding vector through a text-to-image conversion model such as CLIP (Contrastive Language-Image Pre-training).

[0079] Figure 6 schematically illustrates an example training architecture of a text-to-image conversion model 600 according to some embodiments of the present disclosure. As shown in the figure, the model 600 may include an image encoder and a text encoder. Exemplarily, the model 600 can be trained through the process shown in the figure. First, a picture and a text are randomly taken from the training set, where the text and the picture may not match. The task of the model 600 is to predict whether the picture and the text match, thereby starting training. After randomly taking out the text and the picture, the picture encoder and the text encoder can be used to compress the picture and the text into two embedding vectors, respectively, referred to as the picture embedding vector and the text embedding vector. Then, the similarity between the two can be compared using a similarity measurement method such as cosine similarity to predict whether the extracted text and picture match. Subsequently, the parameters of the two encoders can be reversely updated based on whether the prediction results and the labels are the same. By repeating the above back propagation process continuously, a pair of trained encoders can be obtained. For matching text and images, the two can output similar embedding vectors, resulting in a similarity close to 1. For mismatched text and images, the two can output completely different embedding vectors, resulting in a similarity close to 0. This allows two unrelated types of information—text for natural language and images for computer vision—to be connected and given a unified mathematical representation, enabling conversion between the two.

[0080] Figure 7A schematically illustrates an example diffusion process 700 according to some embodiments of the present disclosure. For example, the model diffusion process can control different noise levels (Noise Amount) at different times t (or time steps t), thereby obtaining different noisy images. The model utilizes a conditional u-net model to obtain predicted noise features, also known as noise samples (Predicted Noise Samples). Then, the image after t-step denoising can be obtained by calculating the difference between the input noisy image and the predicted noise features. Based on the above method embodiments, for example, after identifying a first noise feature from a first noisy image, the identified first noise feature can be removed from the first noisy image to obtain a denoised image containing the first image feature; after identifying a second noise feature from a second noisy image, the identified second noise feature can be removed from the second noisy image to obtain a denoised image containing the second image feature; and after identifying a third noise feature from a third noisy image, the identified third noise feature can be removed from the third noisy image to obtain a denoised image containing the third image feature.

[0081] For example, the process of predicting the desired image from the initial noise can be achieved through multiple rounds of denoising. In such a process, t can represent the number of denoising rounds or time steps, and the noise levels corresponding to different denoising time steps can be different. For example, assuming the noise level of the initial noise is 100% and the noise level of the final desired image is 0, the noise levels corresponding to the intermediate time steps can be uniformly distributed or distributed in other forms.

[0082] More specifically, for a piece of text, such as the guiding information mentioned above (which can be the first guiding information, the second guiding information, or the merged guiding information), it can be compressed into an embedding vector by the text encoder shown in Figure 6. In the denoising process of the u-net model of the noise prediction model or the pre-selected model, the embedding vector can be injected into the denoising process through the attention mechanism (i.e., the Attention module shown in the figure). For example, each ResNet in the U-Net module is no longer directly connected to the adjacent ResNet, but an Attention module is added between them. Through the attention mechanism, the embedding vector obtained based on the guiding information can be processed so that its semantic information can be continuously injected. As a result, the predicted noise features and the denoised image are associated with the semantics of the guiding information.

[0083] In the above embodiment in which the image generation process is implemented by a combination of noise prediction and denoising processes, the image generation model determination process may be implemented with the help of noise prediction.

[0084] In one embodiment, step 220 may include: inputting the first guidance information and the noisy image data into a noise prediction model, wherein the noisy image data is used to represent data of an image with noise (the image may be described by the first guidance information, the second guidance information, or the merged guidance information), and, based on the noise prediction model, obtaining a first noise prediction of the noisy image as a first noise feature. Step 230 may include: inputting the second guidance information and the noisy image data into the noise prediction model, and, based on the noise prediction model, obtaining a second noise prediction of the noisy image as a second noise feature. Step 240 may include: inputting the merged guidance information and the noisy image data into a preselected model, and, based on the preselected model, obtaining a third noise prediction of the noisy image as a third noise feature.

[0085] In another embodiment, step 220 may include: inputting first guidance information and at least one first noisy picture at at least one time step of the noise prediction model, to predict at least one first noise feature of the at least one first noisy picture based on the semantics of the first guidance information, wherein the at least one first noisy picture includes the first noisy picture, and the at least one first noise feature includes the first noise feature. Step 230 may include: inputting second guidance information and at least one second noisy picture at at least one time step of the noise prediction model, to predict at least one second noise feature of the at least one second noisy picture based on the semantics of the second guidance information, wherein the at least one second noisy picture includes the second noisy picture, and the at least one second noise feature includes the second noise feature. Step 240 may include: inputting merged guidance information and at least one third noisy picture at at least one time step of the preselected model, to predict at least one third noise feature of the at least one third noisy picture based on the semantics of the merged guidance information, wherein the at least one third noisy picture includes the third noisy picture, and the at least one third noise feature includes the third noise feature. The first noisy picture and the second noisy picture are input at the tth time step in the at least one time step of the noise prediction model; the third noisy picture is input at the tth time step in the at least one time step of the preselected model.

[0086] The above process can be referred to as an example of a noise prediction process according to some embodiments of the present disclosure shown in FIG. 7B .

[0087] As shown in FIG7B , the noise prediction model θ * and the preselected model θ both include T time steps. * From the 1st time step to the Tth time step, the first guidance information and at least one first noisy picture are input respectively, so as to predict at least one first noise feature of at least one first noisy picture based on the semantics of the first guidance information. Take the tth time step and the Tth time step as an example. At the tth time step, the time step t, the first guidance information c0, the first noisy picture x t The vector x(x t ,c0,t), to predict the first noisy image x based on the semantics of the first guide information c0 t The first noise characteristic In the noise prediction model θ * The Tth time step, input time step T, the first guide information c0, the first noisy image x T The vector x(x T ,c0,T), to predict the first noisy image x based on the semantics of the first guide information c0 T The first noise characteristic In addition, in the noise prediction model θ *From the first time step to the Tth time step, the second guidance information and at least one second noisy picture are input respectively, so as to predict at least one second noise feature of at least one second noisy picture based on the semantics of the second guidance information. Take the tth time step and the Tth time step as an example. At the tth time step, the time step t, the second guidance information c1, the second noisy picture x t The vector x(x t ,c1,t), to predict the second noisy image x based on the semantics of the second guide information c1 t The second noise characteristic In the noise prediction model θ * The Tth time step, input time step T, the second guide information c1, the second noisy image x T The vector x(x T ,c1,T), to predict the second noisy image x based on the semantics of the second guide information c1 T The second noise characteristic

[0088] In addition, at the 1st time step to the Tth time step of the preselected model θ, the merged guidance information and at least one third noisy image are respectively input to predict at least one third noise feature of the at least one third noisy image based on the semantics of the merged guidance information. Take the tth time step and the Tth time step as an example. At the tth time step, the time step t, the merged guidance information c0+c1, and the third noisy image x are input. t The vector x(x t ,c 0+ c1,t), to predict the third noisy image x based on the semantics of the combined guidance information c0+c1 t The third noise characteristic ε θ (x t ,c0+c1,t). At the Tth time step of the preselected model θ, input the time step T, the merged guidance information c0+c1, and the third noisy image x T The vector x(x T ,c 0+ c1,T), to predict the third noisy image x based on the semantics of the combined guidance information c0+c1 T The third noise characteristic ε θ (x T ,c0+c1,T).

[0089] In the above example, although the noise prediction model θ * At the t-th time step of the preselected model θ, the first noisy image, the second noisy image, and the third noisy image are all x t , but in other examples, different noisy images can also be used. Although in the noise prediction model θ *At the Tth time step of the preselected model θ, the first noisy image, the second noisy image, and the third noisy image are all x T ,In other examples, different noisy images can also be used.

[0090] As mentioned above, the first guidance information, the second guidance information, and the combined guidance information may be provided manually, by a model, or a combination thereof according to preset rules.

[0091] Exemplarily, the noisy image data may include first data and second data, wherein the first data may be configured to characterize pixel data of the noisy image, and the second data may be configured to characterize the noise level of the noisy image. As mentioned above, in some embodiments, for the noise prediction model and the determined image generation model, the image generation may be implemented by iterating multiple rounds of noise prediction and denoising processes. In such embodiments, the noisy image data may be divided into two parts, one part being image data for characterizing the image containing noise, and the other part being noise level data, or may also be referred to as time step data, etc.

[0092] Exemplarily, the above-mentioned noisy picture can be obtained in the following manner: obtaining a basic picture, which is used to represent the picture described by the first guidance information, the second guidance information and the combined guidance information; and obtaining the noisy picture by adding a preset level of noise to the basic picture.

[0093] Exemplarily, a first basic picture can be obtained, and by adding at least one preset level of noise to the first basic picture, at least one first noisy picture is obtained, and the first basic picture contains the first image feature; a second basic picture can be obtained, and by adding at least one preset level of noise to the second basic picture, at least one second noisy picture is obtained, and the second basic picture contains the second image feature; a third basic picture can be obtained, and by adding at least one preset level of noise to the third basic picture, at least one third noisy picture is obtained, and the third basic picture contains the first image feature and the second image feature.

[0094] Exemplarily, a first noisy picture input at a time step of at least one time step of a noise prediction model is obtained by adding a preset level of noise to a first noisy picture input at a time step before the time step; a second noisy picture input at the time step of the noise prediction model is obtained by adding the preset level of noise to a second noisy picture input at a time step before the time step; and a third noisy picture input at the time step of the preselected model is obtained by adding the preset level of noise to a third noisy picture input at a time step before the time step.

[0095] For example, let's continue with the example mentioned above. For any of the first, second, and third base images, assuming the noise level of the initialization noise (random noise) is 100%, the noise level of the final desired image is 0 (i.e., no noise), and assuming a total of 50 denoising steps, then for time step t=1, 100% noise (e.g., random noise) can be added to the base image to obtain a noisy image; for time step t=2, 98% noise can be added to the base image to obtain a noisy image; for time step t=3, 96% noise can be added to the base image to obtain a noisy image; and so on. Alternatively, the noise level corresponding to each time step can be determined by other methods. In this way, images with various levels of noise can be flexibly obtained to optimize the preselected model. In addition, for example, the noisy images can also be obtained by other methods, such as selecting an image that already contains noise as the noisy image, using an image that has been processed by a noise prediction model for several steps as the noisy image, and so on. The random noise can be Gaussian noise.

[0096] In some embodiments, the first guidance information can be used to describe key features in an image, and the second guidance information can be used to describe at least other non-key features in the same image. In such embodiments, step 250 may include determining a combined noise feature based on a weighted sum of the first noise feature and the second noise feature, wherein the weight of the first noise feature is greater than the weight of the second noise feature. In such embodiments, a key feature can be understood as a feature related to the image label. For example, in the image shown in FIG3 , according to the description of the example above, the expected label may be a knife and fork. The first guidance information can be used to describe the knife and fork, and the second guidance information can be used to describe radish, fish, green vegetables, steamed bun, dinner plate, etc. By setting the weight of the first noise feature to be greater than the weight of the second noise feature in the combined noise feature, and then updating the model parameters of the pre-selected model based on the difference between the combined noise feature and the third noise feature, the proportion of the first guidance information in the global information can be increased. This allows the trained pre-selected model (i.e., the determined image generation model) to describe the global information involved in the combined guidance information while not neglecting detailed information, particularly the desired key information. As a result, images generated based on the merged guidance information can correspond to the desired labels and can be directly used to train relevant image processing models. This helps significantly improve the efficiency of image generation, enhances the relevance of the generated images, and helps improve the efficiency and effectiveness of training relevant image processing models.

[0097] In some embodiments, step 260 may include: determining a combined prediction loss based on the combined noise feature and the third noise feature; and adjusting model parameters of the preselected model based on the combined prediction loss to reduce the combined prediction loss. For example, the combined prediction loss may be determined using a loss function such as an L2 loss to more accurately reflect the difference between the combined noise feature and the third noise feature, and to more efficiently update the model parameters of the preselected model.

[0098] In some embodiments, the initial model parameters of the preselected model can be the same as those of the noise prediction model. As mentioned above, the noise prediction model and the preselected model can have the same model structure, that is, they belong to the same model category. Exemplarily, the initial model parameters of the preselected model can be set arbitrarily. However, to save training costs, the initial model parameters of the preselected model can be consistent with those of the noise prediction model, that is, the initial preselected model can be completely consistent with the noise prediction model. Since the noise prediction model is a trained model that can generate output images based on input text, but the images it generates may not meet the desired effect, optimizing training based on it can save time and make it easier to obtain a model that meets the desired image generation requirements. Exemplarily, the noise prediction model and the initial preselected model can be the same SD model (e.g., SD V2.0 model). Exemplarily, the image generation model determined after fine-tuning based on the SD model can be called an ASD (Add Stable Diffusion) model.

[0099] For further understanding, FIG8 schematically illustrates an example training architecture 800 for determining an image generation model according to some embodiments of the present disclosure. As described in the previous embodiments, the basic description (such as the first basic information, the second basic information, the merged basic information, etc.) can be input into a text generation model such as ChatGPT to obtain corresponding guidance information (such as the first guidance information c0, the second guidance information c1, the merged guidance information c0+c1, etc.). The obtained guidance information can be compared with the noisy image x t The noise prediction model or preselected model, such as the model shown in Figure 5, is then provided to the noise prediction model or preselected model to obtain the corresponding noise prediction. The noisy image is provided to the noise prediction model or preselected model in the form of a tensor. The model parameters of the preselected model can then be updated by aligning the predicted noise obtained after the diffusion process of the original noise prediction model using the first guidance information c0 and the second guidance information c1 with the predicted noise obtained after the diffusion process of the preselected model using the combined guidance information c0+c1.

[0100] Specifically, as shown in FIG8 , the first guide information c0 and the first noisy image x t , the vector x(x t, c0, t) and represents the second guide information c1 and the second noisy picture x t , the vector x(x t ,c1,t) are input into the noise prediction model θ * To get the high-dimensional noise corresponding to the tth step and As the first noise feature and the second noise feature. and Get the combined noise As a merged noise feature, the weight η can be regarded as a conditional factor, which can characterize the proportion of c0 in the merged noise, and further characterize its proportion in the guide image generation process. For example, if c0 is a description involving knife and fork, and knife and fork are the main label information, that is, the key information of concern, then the weight of c0 can be higher than c1, for example, η can be set to 1.5, or it can be set to other values ​​greater than 1. For the merged guide information c0+c1, the merged guide information and the third noisy image x can be represented by t , the vector x(x t ,c0+c1,t) is input into the pre-selected model θ to obtain the same high-dimensional noise ε corresponding to the t-th step θ (x t ,c0+c1,t), as the third noise feature. The preselected model θ can have the same * The same model structure and its initial parameters can be compared with the noise prediction model θ * The same, in order to speed up the training. For example, the noise prediction model θ * And the initial pre-selected model θ can be the aforementioned SD V2.0 model. θ (x t ,c0+c1,t) and Determine the prediction loss, such as L2 loss (see L2Loss(t) in FIG7B ), and update one or more model parameters of the preselected model θ based on the loss so that the loss tends to be minimized. In other words, the training process can make the third noise prediction obtained based on the merged guidance information consistent with the merged noise prediction obtained based on the first guidance information and the second guidance information, thereby ensuring that the determined image generation model can depict global information while not ignoring local information, and exemplarily emphasizes more critical local information. In the above example, at the tth time step, the first noisy picture, the second noisy picture, and the third noisy picture are all x t, that is, the same noisy picture. In other examples, the first noisy picture, the second noisy picture, and the third noisy picture may be different noisy pictures, wherein the first noisy picture includes the first image feature, the second noisy picture includes the second image feature, and the third noisy picture includes the first image feature and the second image feature.

[0101] More specifically, the merged guidance information and the noisy image data can be input into the U-Net model as shown in FIG9 , and the U-Net model can be included in the denoising U-Net module in the model 500 shown in the figure. Exemplarily, the aforementioned merged guidance information c0+c1 can be encoded into a text embedding vector by a trained text encoder as shown in FIG6 , and the text embedding vector can be provided to each time step as a global condition and introduced through the attention mechanism. The first data in the noisy image data (i.e., the data representing the pixel data of the noisy image) can be provided to the input layer, and gradually undergo normalization, scaling, translation and other processing, and fused with the aforementioned text embedding vector through the attention mechanism. The second data in the noisy image data (i.e., the data representing the noise level of the noisy image) can be incorporated into the processing process as shown in the figure.

[0102] Figure 10 schematically illustrates an example flow chart of a method 1000 for generating an image according to some embodiments of the present disclosure. For example, method 1000 can be applied to scenario 100 shown in Figure 1 . For example, it can be deployed on server 110, terminal device 120, or a combination thereof, in a form such as an application. As shown, method 1000 may include steps 1010 to 1030, as detailed below.

[0103] In step 1010, guidance information may be obtained. For example, the guidance information may be derived from manual input, automatic generation by an algorithm or model, or a combination thereof. This may be the same as the method for obtaining guidance information described above in relation to the embodiment of method 200.

[0104] In step 1020 , the guidance information may be input into an image generation model, where the image generation model is determined by the method 200 described according to various embodiments described above.

[0105] In step 1030 , a picture corresponding to the guidance information may be obtained based on the output of the picture generation model.

[0106] Based on the guidance information in the form of merged guidance information, an image containing global information and all local information can be generated by the image generation model determined according to method 200. In other words, the generated image can contain both the object described by the local information and richer additional information. This image can correspond to one or more specific tags (the one or more specific tags can be included in the provided guidance information), has image quality that meets the requirements, and because it is directly generated based on text through the model, it does not involve privacy or sensitive information. In addition, when using the determined image generation model, it is only necessary to provide guidance information (i.e., the aforementioned merged guidance information) according to preset rules, which requires less professional skills from the operator and can generate a large number of images with the required labels in a short time. Therefore, in actual business scenarios, through the above scheme, more data that is beneficial to business training can be better generated, thereby ensuring that the business can be quickly iterated and optimized. Moreover, by generating a large amount of data in a short period of time, the recognition ability of some current models for business images can also be improved by increasing the amount of data, thereby helping to ensure the effectiveness of online business.

[0107] In some embodiments, step 1010 may include: obtaining basic information; inputting the basic information into a text generation model, and obtaining guidance information based on the output of the text generation model, wherein the text generation model is configured to perform at least one of expansion and modification on the input basic information according to preset rules, and output corresponding guidance information, such that the output guidance information includes richer descriptive information than the input basic information. The text generation model and the process of obtaining guidance information therethrough have been described above in the embodiment of step 210 of method 200 and will not be repeated here.

[0108] As mentioned above, in some embodiments, the aforementioned text generation model can be implemented using ChatGPT or a similar model, and the image generation model can be obtained by optimizing and training based on SD or a similar model. Regarding ChatGPT and SD, the former belongs to the field of natural language processing (NLP), and the latter belongs to the field of computer vision (CV). Both are products of big data and large model training, and have received widespread attention from academia and industry for their excellent performance. However, the applicant has found that there is currently no relevant technology that can unify the fields and capabilities of the two, nor is there a relevant technology that can enable the two to be implemented in actual business and produce performance effects. However, through the technical solution provided by the present disclosure, the fields and capabilities of the two can be integrated, and the integration problem of the two at the data level (such as the text data involved in ChatGPT and the image data involved in SD) can be solved, ensuring that it has a better portrayal effect on the key information in the global information, and enabling it to play its excellent performance under big data and large scale in actual business (such as content review), and better contribute to the development of actual business.

[0109] Figure 11 schematically illustrates an example block diagram of an apparatus 1100 for determining an image generation model according to some embodiments of the present disclosure. As shown, apparatus 1100 may include an acquisition module 1110, a first generation module 1120, a second generation module 1130, a third generation module 1140, a determination module 1150, and an update module 1160.

[0110] According to one embodiment, the acquisition module 1110 may be configured to acquire first guidance information and second guidance information. The first guidance information is used to characterize first image features, and the second guidance information is used to characterize second image features. The first guidance information, the second guidance information, and the combined guidance information may be different descriptive information for the same image, and the combined guidance information includes a combination of the first and second guidance information and additional information.

[0111] The first generation module 1120 may be configured to input the first guidance information and the first noisy image into a noise prediction model to identify a first noise feature from the first noisy image. The first noisy image includes the first image feature, and the noise prediction model is a pre-trained model with at least some model parameters frozen. Based on the first guidance information, the first generation module 1120 generates the first noise feature using the noise prediction model. The noise prediction model is configured to output an image based on the input guidance information, such that the output image includes the feature described by the input guidance information.

[0112] The second generation module 1130 may be configured to input the second guidance information and the second noisy image into the noise prediction model to identify a second noise feature from the second noisy image. The second noisy image includes the second image feature. The second generation module 1130 generates the second noise feature using the noise prediction model based on the second guidance information.

[0113] The third generation module 1140 may be configured to input the combined guidance information including the first guidance information and the second guidance information and a third noisy picture into a preselected model having the same model structure as the noise prediction model, to identify a third noise feature from the third noisy picture. The third noisy picture includes the first image feature and the second image feature.

[0114] The determination module 1150 may be configured to combine the first noise feature and the second noise feature to obtain a combined noise feature.

[0115] The updating module 1160 may be configured to adjust the model parameters of the preselected model according to the difference between the combined noise feature and the third noise feature, so as to update the preselected model.

[0116] FIG12 schematically shows an example block diagram of a picture generation apparatus 1200 according to some embodiments of the present disclosure. As shown in the figure, the apparatus 1200 may include a first acquisition module 1210 , an input module 1220 , and a second acquisition module 1230 .

[0117] Specifically, the first acquisition module 1210 can be configured to acquire guidance information; the input module 1220 can be configured to input the guidance information into the image generation model, which is determined by the method 200 described in accordance with the various embodiments above; the second acquisition module 1230 can be configured to acquire an image corresponding to the guidance information based on the output of the image generation model.

[0118] It should be understood that the apparatus 1100 and the apparatus 1200 can be implemented in software, hardware, or a combination of software and hardware. Multiple different modules can be implemented in the same software or hardware structure, or a module can be implemented by multiple different software or hardware structures.

[0119] Furthermore, apparatus 1100 can be used to implement method 200 described above, and apparatus 1200 can be used to implement method 1000 described in the previous embodiment. The relevant details have been described in detail above and will not be repeated here for the sake of brevity. Apparatus 1100 and apparatus 1200 can have the same features and advantages as those described with respect to the aforementioned methods.

[0120] Figure 13 schematically shows an example block diagram of a computing device 1300 according to some embodiments of the present disclosure, which may represent, for example, the server 110, the terminal device 120 in Figure 1, or other types of computing devices that may be used to deploy the apparatuses 1100 and 1200 provided in the present disclosure.

[0121] As shown, the example computing device 1300 includes a processing system 1301, one or more computer-readable media 1302, and one or more I / O interfaces 1303 that are communicatively coupled to each other. Although not shown, the computing device 1300 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus can include any one or a combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures, or may also include control and data lines.

[0122] Processing system 1301 represents functionality that uses hardware to perform one or more operations. Thus, processing system 1301 is illustrated as including hardware elements 1304 that can be configured as processors, functional blocks, and the like. This can include implementing application-specific integrated circuits (ASICs) or other logic devices formed using one or more semiconductors in hardware. Hardware elements 1304 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor can be comprised of semiconductor(s) and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor-executable instructions can be electronically executable instructions.

[0123] Computer-readable media 1302 is illustrated as including memory / storage 1305. Memory / storage 1305 represents a memory / storage associated with one or more computer-readable media. Memory / storage 1305 may include volatile storage media (such as random access memory (RAM)) and / or non-volatile storage media (such as read-only memory (ROM), flash memory, optical disk, magnetic disk, etc.). Memory / storage 1305 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Exemplarily, memory / storage 1305 can be used to store the various texts, pictures, parameters, etc. mentioned in the above embodiments. Computer-readable media 1302 can be configured in various other ways as further described below.

[0124] One or more input / output interfaces 1303 represent functionality that allows a user to type commands and information into the computing device 1300 and also allows information to be presented to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, a touch function (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., that can use visible or invisible wavelengths (such as infrared frequencies) to detect motion that does not involve touch as gestures), a network card, a receiver, and the like. Examples of output devices include a display device (e.g., a monitor or projector), a speaker, a printer, a tactile response device, a network card, a transmitter, and the like. Exemplarily, in the embodiments described above, the input device can allow the user to provide the input basic information, etc., and the output device can allow the user to view the generated pictures, the updated model parameters, and the like.

[0125] The computing device 1300 also includes an application for determining an image generation model or an image generation application 1306. The application for determining an image generation model or the image generation application 1306 can be stored as computer program instructions in the memory / storage device 1305. The application for determining an image generation model or the image generation application 1306 can, together with the processing system 1301 and the like, implement all the functions of the various modules of the apparatus 1100 or the apparatus 1200 described with respect to FIG. 11 or FIG. 12 .

[0126] Various techniques may be described herein in the general context of software, hardware, elements, or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. As used herein, the terms "module," "function," etc. generally refer to software, firmware, hardware, or a combination thereof. A feature of the techniques described herein is that they are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0127] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 1300. By way of example and not limitation, computer-readable media may include "computer-readable storage media" and "computer-readable signal media."

[0128] As opposed to a mere signal transmission, carrier wave, or signal itself, "computer-readable storage media" refers to media and / or devices, and / or tangible storage devices, capable of persistently storing information. Thus, computer-readable storage media refers to non-signal-bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-executable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, hard disks, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or other storage devices, tangible media, or articles of manufacture suitable for storing desired information and accessible by a computer.

[0129] "Computer-readable signal media" refers to signal-bearing media configured to transmit instructions to the hardware of computing device 1300, such as via a network. Signal media typically embodies computer-executable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also includes any information transmission media. By way of example, and not limitation, signal media include wired media such as a wired network or direct connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0130] As previously described, hardware elements 1304 and computer-readable media 1302 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements can include integrated circuits or systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or other hardware devices. In this context, hardware elements can be used as processing equipment for executing program tasks defined by the instructions, modules, and / or logic embodied by the hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media previously described.

[0131] The aforementioned combinations may also be used to implement the various techniques and modules described herein. Thus, software, hardware, or program modules and other program modules may be implemented as one or more instructions and / or logic on some form of computer-readable storage medium and / or embodied by one or more hardware elements 1304. The computing device 1300 may be configured to implement specific instructions and / or functions corresponding to the software and / or hardware modules. Thus, for example, by using a computer-readable storage medium and / or hardware elements 1304 of a processing system, a module may be implemented as a module executable by the computing device 1300 as software, at least in part, in hardware. Instructions and / or functions may be executed / operable by, for example, one or more computing devices 1300 and / or processing systems 1301 to implement the techniques, modules, and examples described herein.

[0132] The techniques described herein may be supported by these various configurations of computing device 1300 and are not limited to the specific examples of the techniques described herein.

[0133] It should be understood that, for the sake of clarity, embodiments of the present disclosure have been described with reference to different functional units. However, it will be apparent that, without departing from the present disclosure, the functionality of each functional unit can be implemented in a single unit, in multiple units, or as a part of other functional units. For example, the functionality described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to the appropriate unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present disclosure can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0134] The present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed, the method for determining an image generation model or the image generation method described above is implemented.

[0135] The present disclosure provides a computer program product or computer program, which includes computer-executable instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computing device to perform the method for determining an image generation model or the image generation method provided in the various embodiments described above.

[0136] Variations on the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A method for determining an image generation model, comprising: Acquire first guidance information and second guidance information, where the first guidance information is used to characterize a first image feature, and the second guidance information is used to characterize a second image feature; inputting the first guidance information and the first noisy picture into a noise prediction model to identify a first noise feature from the first noisy picture; inputting the second guidance information and a second noisy picture into the noise prediction model to identify a second noise feature from the second noisy picture; inputting the combined guidance information including the first guidance information and the second guidance information and a third noisy picture into a preselected model having the same model structure as the noise prediction model, so as to identify a third noise feature from the third noisy picture; Combining the first noise feature and the second noise feature to obtain a combined noise feature; According to the difference between the combined noise feature and the third noise feature, the model parameters of the preselected model are adjusted to update the preselected model.

2. The method according to claim 1, wherein The first noisy picture includes the first image feature, the second noisy picture includes the second image feature, and the third noisy picture includes the first image feature and the second image feature.

3. The method according to claim 1 or 2, Inputting the first guidance information and the first noisy picture into a noise prediction model to identify a first noise feature from the first noisy picture includes: inputting the first guidance information and at least one first noisy picture into the noise prediction model at at least one time step, so as to predict at least one first noise feature of the at least one first noisy picture based on semantics of the first guidance information, wherein the at least one first noisy picture includes the first noisy picture, and the at least one first noise feature includes the first noise feature; Inputting the second guidance information and the second noisy picture into the noise prediction model to identify a second noise feature from the second noisy picture includes: inputting the second guidance information and at least one second noisy picture into the noise prediction model at the at least one time step, so as to predict at least one second noise feature of the at least one second noisy picture based on semantics of the second guidance information, wherein the at least one second noisy picture includes the second noisy picture, and the at least one second noise feature includes the second noise feature; Inputting the merged guidance information and the third noisy picture into a preselected model having the same model structure as the noise prediction model to identify a third noise feature from the third noisy picture includes: At at least one time step of the preselected model, the merged guidance information and at least one third noisy picture are input to predict at least one third noise feature of the at least one third noisy picture based on the semantics of the merged guidance information, the at least one third noisy picture including the third noisy picture, and the at least one third noise feature including the third noise feature.

4. The method according to claim 3, wherein: The first noisy picture and the second noisy picture are input at the t-th time step in the at least one time step of the noise prediction model; The third noisy image is input at the t-th time step in the at least one time step of the preselected model.

5. The method according to any one of claims 1 to 4, wherein: The acquiring of the first guidance information and the second guidance information includes: Acquire first basic information and second basic information; The first basic information and the second basic information are respectively input into a text generation model, so that the text generation model performs at least one of expansion and modification on the first basic information and the second basic information according to preset rules to generate the first guide information and the second guide information.

6. The method according to any one of claims 1 to 5, further comprising: Acquire a first basic image, and obtain the at least one first noisy image by adding at least one preset level of noise to the first basic image, wherein the first basic image includes the first image feature; Acquire a second basic image, and obtain the at least one second noisy image by adding the at least one preset level of noise to the second basic image, wherein the second basic image includes the second image feature; A third basic image is obtained, and the at least one third noisy image is obtained by adding the at least one preset level of noise to the third basic image, where the third basic image includes the first image feature and the second image feature.

7. The method according to any one of claims 1 to 6, wherein: The first image feature is a feature related to a picture label, the second image feature is a feature not related to a picture label, and determining the merged noise feature based on the first noise feature and the second noise feature includes: The combined noise feature is determined based on a weighted sum of the first noise feature and the second noise feature, wherein the weight of the first noise feature is greater than the weight of the second noise feature.

8. The method according to any one of claims 1 to 7, wherein: The adjusting the model parameters of the preselected model according to the difference between the combined noise feature and the third noise feature to update the preselected model includes: determining a combined prediction loss based on the combined noise feature and the third noise feature; Based on the combined prediction loss, model parameters of the preselected model are adjusted to reduce the combined prediction loss.

9. The method according to any one of claims 1 to 8, wherein: The initial model parameters of the preselected model are the same as the model parameters of the noise prediction model.

10. A method for generating an image, comprising: Get guidance information; Inputting the guidance information into an image generation model, wherein the image generation model is determined by the method according to any one of claims 1 to 9; Based on the output of the picture generation model, a picture corresponding to the guidance information is obtained.

11. The method according to claim 10, wherein: The obtaining of guidance information includes: Obtain basic information; Input the basic information into a text generation model, and obtain the guidance information based on the output of the text generation model, The text generation model is configured to perform at least one of expansion and modification on the input basic information according to preset rules, and output corresponding guidance information.

12. A device for determining an image generation model, comprising: The acquisition module is configured to acquire first guidance information and second guidance information, wherein the first guidance information is used to represent the first image characteristics. The second guidance information is used to represent the second image feature; a first generating module configured to input the first guidance information and a first noisy picture into a noise prediction model to identify a first noise feature from the first noisy picture; a second generating module configured to input the second guidance information and a second noisy picture into the noise prediction model to identify a second noise feature from the second noisy picture; a third generating module configured to input the combined guidance information including the first guidance information and the second guidance information and a third noisy picture into a preselected model having the same model structure as the noise prediction model, so as to identify a third noise feature from the third noisy picture; a determining module configured to combine the first noise feature and the second noise feature to obtain a combined noise feature; An updating module is configured to adjust the model parameters of the preselected model according to the difference between the combined noise feature and the third noise feature to update the preselected model.

13. A picture generating device, comprising: A first acquisition module is configured to acquire guidance information; an input module configured to input the guidance information into an image generation model, wherein the image generation model is determined by the method according to any one of claims 1 to 9; The second acquisition module is configured to acquire a picture corresponding to the guidance information based on the output of the picture generation model.

14. A computing device comprising: a memory configured to store computer-executable instructions; A processor configured to perform the method according to any one of claims 1 to 11 when the computer executable instructions are executed by the processor. 15 . A computer-readable storage medium storing computer-executable instructions, which, when executed, perform the method according to claim 1 .

16. A computer program product comprising computer executable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Citation Information

Patent Citations

  • Model training method, picture generating method and device

    CN113537487A

  • Training method of image recognition model, image recognition method, device and medium

    CN114255381A

  • Image generation method, and training method and device of image denoising model

    CN115222630A

  • Picture generation model determination method and device and picture generation method and device

    CN116977459A

  • Realistic depth image generation using generative adversarial nets

    WO2023018752A1

Cited By

  • Training method of multimedia resource generation model and multimedia resource generation method

    CN120494015A