Method and apparatus for training text-graph model, device, and storage medium

Abstract

A method and apparatus for training a text-graph model, a device, and a storage medium, relating to the artificial intelligence. The method comprises: on the basis of a training set of graph-text sample pairs, performing iterative training on a model to be trained, to obtain a text-graph model; in a training process: selecting a graph-text sample pair from the training set of graph-text sample pairs, the graph-text sample pair comprising a sample image and a description text, and the sample image comprising at least two objects; obtaining mask images corresponding to the at least two objects and associated object class names, the mask images being used for distinguishing position regions of the objects in the sample image; inputting the sample image and the description text to a model to be trained, to obtain an image prediction noise of the sample image, and inputting at least two mask images and the associated object class names to the model to be trained to obtain object prediction noises respectively associated with the at least two mask images; and constructing a loss function on the basis of the image prediction noise and the object prediction noises, and by using the loss function, adjusting a parameter for the model to be trained.

Description

A training method, device, equipment and storage medium for a cultural graph model

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 17, 2023, application number 202311044371.5, and application name “A training method, device, equipment and storage medium for a cultural graph model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to the training technology of cultural graph models. Background Art

[0003] The long-term development of artificial intelligence (AI) has significantly improved text-based graph models, enabling them to produce high-quality and diverse image outputs based on given text prompts. However, further fine-tuning of these models is necessary to ensure their output accuracy.

[0004] Currently, there are at least two ways to fine-tune the text-graph model: one is to fine-tune the text-graph model based on images and object concepts; and the other is to fine-tune the text-graph model based on prompt words converted from object concept images.

[0005] However, regardless of whether method 1 or method 2 is adopted, when fine-tuning the text graph model, the text graph model is focused on embedding a single object, ignoring the complexity of multi-object scenes. In addition, during the fine-tuning process, the training samples used contain complex background information, which will interfere with the training of the model and lead to inaccurate training of the text graph model.

[0006] Therefore, how to obtain accurate text-based graph models in multi-object scenes is a technical problem that needs to be solved.

[0007] Summary of the Invention

[0008] The embodiments of the present application provide a training method, apparatus, device, and storage medium for a text-based graph model, which are used to obtain an accurate text-based graph model in a multi-object scene, thereby expanding the application scenarios of the text-based graph model.

[0009] In a first aspect, an embodiment of the present application provides a method for training a text graph model, the method being executed by a computing device and comprising:

[0010] Based on the training set of image and text samples, the training model is trained iteratively to obtain a text-image model. The following steps are performed in one iterative process:

[0011] Selecting an image-text sample pair from an image-text sample pair training set; wherein the image-text sample pair includes: a sample image and a description text of the sample image, and the sample image includes at least two objects;

[0012] Obtaining mask images corresponding to at least two objects and associated object class names; wherein the mask images are used to distinguish the location areas of the corresponding objects in the sample image;

[0013] Inputting a sample image and a description text into the model to be trained to obtain image prediction noise of the sample image; and inputting at least two mask images and associated object class names into the model to be trained to obtain at least two object prediction noises, wherein one object prediction noise corresponds to one mask image;

[0014] A loss function is constructed based on the image prediction noise and the at least two object prediction noises, and the parameters of the training model are adjusted based on the loss function.

[0015] In a second aspect, an embodiment of the present application provides a training device for a text graph model, which is deployed on a computing device and includes: a training unit, which includes: an acquisition subunit, an acquisition subunit, a prediction subunit, and a parameter adjustment subunit; wherein:

[0016] The training unit is used to perform cyclic iterative training on the training model based on the image and text samples to obtain a text-image model. In one cyclic iteration process, the following subunits perform corresponding steps:

[0017] An acquisition subunit is used to select an image-text sample pair from an image-text sample pair training set; wherein the image-text sample pair includes: a sample image and a description text of the sample image, and the sample image includes at least two objects;

[0018] An acquisition subunit is used to obtain mask images corresponding to at least two objects and associated object class names; wherein the mask images are used to distinguish the location areas of the corresponding objects in the sample image;

[0019] The prediction subunit is configured to input a sample image and a description text into the model to be trained to obtain image prediction noise of the sample image; and input at least two mask images and associated object class names into the model to be trained to obtain at least two object prediction noises, wherein one object prediction noise corresponds to one mask image;

[0020] The parameter adjustment subunit is used to construct a loss function based on the image prediction noise and the at least two object prediction noises, and adjust the parameters of the training model based on the loss function.

[0021] In a third aspect, an embodiment of the present application provides a computing device comprising: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the steps of the training method of the cultural graph model provided in the embodiment of the present application.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the training method of the cultural graph model provided in the embodiment of the present application.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when the processor of a computing device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the computing device executes the steps of the training method of the literary graph model provided in the embodiment of the present application.

[0024] The beneficial effects of this application are as follows:

[0025] The embodiments of the present application provide a training method, apparatus, device, and storage medium for a text graph model, which relate to the fields of artificial intelligence and image processing. Considering that text graph models in related technologies are only applicable to single-object embedding in images, they ignore the complexity of multi-object scenes, limiting the ability of text graph models to generate multi-object embedding in images. Therefore, the embodiments of the present application propose a training method for a text graph model applicable to multi-object scenes. To ensure the accuracy of model training, a training set based on image and text samples is subjected to cyclic iterative training on the training model to obtain a text graph model.

[0026] During training, the first step is to obtain input information for the model to be trained. Specifically, image-text pairs are selected from the training set. These pairs consist of a sample image and a descriptive text. Since the text-based graph model is trained for multi-object scenes, enabling it to handle complex multi-object scenes and improve its multi-object generation capabilities, it is important to ensure that the sample images contain at least two objects. At the same time, considering that in addition to at least two objects, the sample image also contains complex background information, the background information interferes with the model training, making the model training inaccurate, and considering that when there are at least two objects in the sample image, the accurate correspondence between the object and the text is also the main factor for accurate model training. Therefore, after selecting the image-text sample pair, the mask image corresponding to at least two objects in the sample image and the associated object class name can be obtained. The mask image is used to distinguish the position area of ​​the object in the sample image, distinguish the object from the background, prevent the background information from interfering with the model training, and obtain the correspondence between the mask image and the object class name, which helps to enhance the object relationship between the object and the text, so that the model can better understand the text description during training and accurately map it to the corresponding object, thereby ensuring the accuracy of the text-image model.

[0027] The sample image and description text are then input into the model to be trained to obtain the image prediction noise of the sample image, and at least two mask images and associated object class names are input into the model to be trained to obtain the object prediction noise associated with each of the at least two mask images. A loss function is then constructed based on the image prediction noise and the at least two object prediction noises. Finally, the constructed loss function is used to adjust the parameters of the model to be trained. Since this application refers to the multi-object local area increase loss, this loss can separate objects from other areas, allowing the model to focus more on the details and boundaries of the objects, thereby reducing the impact of background interference on the text-based graph model, improving the accuracy of the text-based graph model in multi-object scenes, and further improving the consistency and accuracy of images generated based on the text-based graph model.

[0028] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0030] FIG1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0031] FIG2 is a schematic diagram of the structure of a cultural graph model provided in an embodiment of the present application;

[0032] FIG3 is a schematic diagram of a denoising network provided in an embodiment of the present application;

[0033] FIG4 is a schematic diagram of the specific structure of a cultural graph model provided in an embodiment of the present application;

[0034] FIG5 is a schematic diagram of an object annotation and mask image determination method according to an embodiment of the present application;

[0035] FIG6 is a flow chart of a method for training a text graph model according to an embodiment of the present application;

[0036] FIG7 is a schematic diagram of noise prediction provided by an embodiment of the present application;

[0037] FIG8 is a schematic diagram of a text graph model training provided in an embodiment of the present application;

[0038] FIG9 is a flow chart of a method for generating an image provided by an embodiment of the present application;

[0039] FIG10 is a schematic diagram of generating an image based on a cultural graph model according to an embodiment of the present application;

[0040] FIG11 is a schematic diagram of generating an image based on a cultural graph model according to an embodiment of the present application;

[0041] FIG12 is a structural diagram of a training device for a cultural graph model provided in an embodiment of the present application;

[0042] FIG13 is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and beneficial effects of this application more clear and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0044] The following explains some of the terms used in the embodiments of the present application to facilitate understanding by those skilled in the art.

[0045] The text-based graph model, also known as the text-based graph diffusion model, is a deep learning model used for text-to-image generation. After training by inverting the diffusion process of natural images, a text-based graph model can gradually generate new natural images from a completely random noise image, guided by text. Noise images are generated by random signal interference during capture or transmission, manifesting as random variations in image information or pixel brightness.

[0046] Variational AutoEncoder (VAE) is a probabilistic model based on variational inference and belongs to the generative model. Its architecture design includes an encoder and a decoder.

[0047] The encoder is used to map the original high-dimensional data into a low-dimensional feature space. This feature dimension is generally smaller than the original data dimension, which serves the purpose of compression or dimensionality reduction. This low-dimensional feature often becomes an intermediate latent representation; the decoder is used to reconstruct the original data based on the compressed low-dimensional features.

[0048] The mask image has the same size as the original image and is used to distinguish the location of the object in the original image. The mask image contains only 0 or 1, where 1 represents part of the region of interest or the object area.

[0049] Low-Rank Adaptation (LoRA) weights, lora is the low-rank adaptation of large language models. It freezes the weights of the pre-trained model and injects the trainable rank decomposition matrix into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. In an embodiment of the present application, lora mainly injects trainable network parameters into the denoising network in the text graph model. The denoising network layer is used to link the image with the descriptive text, and the lora weights affect the network parameters corresponding to the denoising network layer, such as the weight matrix part of the denoising network layer.

[0050] Fine-tuning is the process of using a pre-trained model to customize training for certain tasks, modifying the network for specific tasks. The model to be trained in the embodiments of this application is a pre-trained model. After adjusting the parameters using the methods in the embodiments of this application, a text-to-image model can be obtained, which is used to achieve the task of generating images based on text conditions.

[0051] The word “exemplary” is used hereinafter to mean “serving as an example, example, or illustration.” Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0052] The terms "first" and "second" are used for descriptive purposes only and should not be construed as explicitly or implicitly indicating relative importance or the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0053] The long-term development of artificial intelligence (AI) has significantly improved text-based graph models, enabling them to produce high-quality and diverse image outputs based on given text prompts. However, further fine-tuning of these models is necessary to ensure their output accuracy.

[0054] Currently, there are at least two ways to fine-tune the text graph model: one is to use DreamBooth to fine-tune the text graph model based on the provided images and object concepts; the other is to use Textual Inversion to fine-tune the text graph model based on the provided object concept images.

[0055] When using DreamBooth to fine-tune a text-based graph model (such as the open-source Stable Diffusion (SD) model) using images and object concepts, given several images and the text "a[V][class name]" associated with them, where [class name] is the object class name and [V] is a special identifier, fine-tuning results in a text-based graph diffusion model that includes [V] bound to the given object. This special identifier can then be used for image generation during inference.

[0056] Using Textual Inversion, we use the provided 3-5 object concept images to represent these concepts by learning pseudo-words in the text embedding space of the text graph model. These pseudo-words are then combined into natural language sentences to guide the generation of new objects.

[0057] However, regardless of whether method 1 or method 2 is adopted, when fine-tuning the text graph model, the text graph model is focused on embedding a single object, ignoring the complexity of multi-object scenes. In addition, during the fine-tuning process, the training samples used contain complex background information, which will interfere with the training of the model and lead to inaccurate training of the text graph model.

[0058] Therefore, in the related art, there are the following shortcomings in the training process of the cultural graph model:

[0059] Single-object embedding limitation: Focusing only on the embedding of a single object, it ignores the complexity of multi-object scenes and limits the ability of the text-based graph model to generate multi-object embeddings.

[0060] Background interference in training samples: If the training samples contain complex background information, it will interfere with the learning of the text-based graph model. The text-based graph model may mistakenly identify details in the background as part of the object, resulting in inaccurate training of the text-based graph model and further blurred boundaries between objects and backgrounds in images generated based on the text-based graph model.

[0061] In summary, how to obtain accurate text-based graph models in multi-object scenes is a technical problem that needs to be solved at present.

[0062] In view of this, embodiments of the present application provide a training method, apparatus, device, and storage medium for a text-based graph model. Considering that text-based graph models in related technologies are only applicable to single-object embedding in images, they ignore the complexity of multi-object scenes, limiting their ability to generate multi-object embeddings in images. Therefore, embodiments of the present application propose a training method for a text-based graph model applicable to multi-object scenes. To ensure the accuracy of model training, the training model is iteratively trained based on a training set of image and text samples to obtain a text-based graph model.

[0063] During training, the first step is to obtain input information for the model to be trained. Specifically, image-text pairs are selected from the training set. These pairs consist of a sample image and a descriptive text. Since the text-based graph model is trained for multi-object scenes, enabling it to handle complex multi-object scenes and improve its multi-object generation capabilities, it is important to ensure that the sample images contain at least two objects. At the same time, considering that in addition to at least two objects, the sample image also contains complex background information, the background information interferes with the model training, making the model training inaccurate, and considering that when there are at least two objects in the sample image, the accurate correspondence between the object and the text is also a major factor in the accuracy of the model training, therefore, after selecting the image-text sample pair, the mask image and the associated object class name corresponding to at least two objects in the sample image are obtained. The mask image is used to distinguish the position area of ​​the object in the sample image, distinguish the object from the background, and prevent the background information from interfering with the model training. The correspondence between the mask image and the object class name is obtained, which helps to enhance the object relationship between the object and the text, so that the model training process can better understand the text description and accurately map it to the corresponding object, thereby ensuring the accuracy of the text-image model. Therefore, in addition to the sample image and the description text, the input information also includes the mask image and the associated object class name.

[0064] After obtaining the input information, the input information is input into the model to be trained, and the model parameters are adjusted according to the output results of the text map. Specifically, the sample image and the description text are first input into the model to be trained to obtain the image prediction noise of the sample image, and at least two mask images and the associated object class names are input into the model to be trained to obtain the object prediction noise associated with each of the at least two mask images; then, a loss function is constructed based on the image prediction noise and the at least two object prediction noises; finally, the constructed loss function is used to adjust the parameters of the model to be trained. It can be seen that in the embodiment of the present application, the multi-object local area increase loss is referenced. This loss can separate the object from other areas, so that the model is more focused on the details and boundaries of the object, thereby reducing the impact of background interference on the model to be trained, improving the accuracy of the text map model in multi-object scenes, and further improving the consistency and accuracy of the images generated based on the model to be trained.

[0065] The embodiments of the present application relate to artificial intelligence (AI) and machine learning technologies, and are designed based on speech technology, natural language processing technology, and machine learning (ML) in artificial intelligence.

[0066] The following is a brief description of the application scenarios set by this application. It should be noted that the following scenarios are only used to illustrate the embodiments of this application and are not limiting. During specific implementation, the technical solutions provided in the embodiments of this application can be flexibly applied according to actual needs.

[0067] 1 is a schematic diagram of an application scenario provided by an embodiment of the present application, wherein the application scenario includes a terminal device 110 and a server 120, and the terminal device 110 and the server 120 can communicate with each other via a communication network.

[0068] In an optional implementation, the communication network can be a wired network or a wireless network. Therefore, the terminal device 110 and the server 120 can be directly or indirectly connected via wired or wireless communication. For example, the terminal device 110 can be indirectly connected to the server 120 via a wireless access point, or the terminal device 110 can be directly connected to the server 120 via the Internet, which is not limited in this application.

[0069] The terminal device 110 includes, but is not limited to, mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, and in-vehicle terminals. Various clients may be installed on the terminal device, and the client may be an online platform or application that supports text input and generates images based on the input text, or a webpage or mini-program. In other words, the client supports the application of the text-based graph model. For example, the client may be an intelligent creation system that supports the application of the text-based graph model and can provide users with personalized image customization capabilities through the text-based graph model.

[0070] When the intelligent creation system provides users with personalized image customization functions through the text-based image model, users are required to first upload an image and annotate the objects in the image to obtain the object's mask image. At the same time, the object class name of each mask image associated with the object must be set to further guide the intelligent creation system to customize the image. For example:

[0071] Interior design and home customization scenarios: Users can upload photos of their interior spaces and precisely specify objects such as furniture and decorations using mask images and associated object class names. The intelligent creation system generates personalized interior design plans based on the user's needs and style preferences, helping them customize their homes.

[0072] Fashion matching and clothing design scenarios: Users can upload photos and specify clothing, accessories, and other objects using the object's mask image and associated object class name. The system provides personalized fashion matching suggestions based on the user's body shape, style preferences, and other information, helping users design and match clothing.

[0073] Advertising Creativity and Brand Customization: Advertising agencies or brands can use this technology to provide clients with personalized advertising creatives and brand customization services. By uploading brand-related images, the company uses object mask images and associated object class names to specify products or elements to highlight. This generates customized advertising materials consistent with the brand's image, helping brands improve their advertising effectiveness and brand awareness.

[0074] Gift customization and personalized product scenarios: Users can upload a specific image and, using the object's mask image and associated object class name, specify the specific elements of the gift or product to be personalized. Based on the user's specifications, the intelligent creation system generates a personalized gift customization plan, helping users create a unique gift or personalized product.

[0075] Therefore, the text-based image model provided by the embodiment of the present application can realize personalized image customization based on the provided image, the mask image of the object and the associated object class name, so as to meet the creative needs and customization requirements in various scenarios.

[0076] The server 120 is a background server corresponding to the client installed in the terminal device 110. It can provide background service functions of the intelligent creation system, such as implementing the training method of the text graph model provided in the embodiment of the present application and the steps of generating images based on the text graph model. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0077] In a possible application scenario, a terminal device obtains an image and text information, transmits the image and text information to a server, and the server generates an image based on a text-based graph model, and then sends the generated image to the terminal device for display to the user through the terminal device.

[0078] In one possible application scenario, the relevant data (such as image and text sample pairs) and model parameters involved in the embodiments of this application can be stored using cloud storage technology. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (or storage nodes) in the network through application software or application interfaces to work together and provide external data storage and service access functions.

[0079] It should be noted that FIG1 is merely an example. In practice, the number of terminal devices 110 and servers 120 is not limited and is not specifically limited in the embodiments of this application. In the embodiments of this application, when there are multiple servers 120, the multiple servers 120 can form a blockchain, and the servers 120 are nodes on the blockchain.

[0080] It should be noted that the training method of the cultural graph model in the embodiment of the present application can be executed by a computing device, which can be a server 120 or a terminal device 110, that is, the method can be executed by the server 120 or the terminal device 110 alone, or can be executed jointly by the server 120 and the terminal device 110.

[0081] To further illustrate the technical solution provided by the embodiment of the present application, the following takes the execution of a server alone as an example, and describes the training method of the culture graph model provided by the exemplary implementation of the present application, as well as the application of the culture graph model in conjunction with the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not subject to any restrictions in this regard. In addition, although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0082] To enable the application of the text-based graph model to multi-object scenarios and ensure its accuracy in these scenarios, this embodiment of the present application provides a method for training the text-based graph model. Furthermore, to verify the consistency and accuracy of images generated based on the text-based graph model, this embodiment of the present application proposes a method for applying the model after obtaining the text-based graph model. Specifically, the method utilizes the text-based graph model in conjunction with specified text containing at least two target class names to generate an image associated with the specified text.

[0083] Below, the overall implementation method of the embodiment of the present application is explained from the model training process and the model application process respectively.

[0084] Example 1: Model training process

[0085] In the embodiment of the present application, the training process is a process of performing multiple cycles of iterative training on the training model using training samples, which can mainly include the model design stage, data preparation stage and iterative training stage. Each stage will be introduced below.

[0086] 1. Model design stage

[0087] See Figure 2, which is a schematic diagram of the structure of a text-based graph model provided in an embodiment of the present application. The text-based graph model includes: a noise addition network, a denoising network, a text encoding network, an image encoding network, and an image decoding network; wherein:

[0088] An image coding network is used to encode a random image (also referred to as a random seed) to obtain corresponding image features. In one possible implementation, the image coding network may adopt, but is not limited to, a variational autoencoder (VAE), which maps the random image to a latent feature space to obtain corresponding image features.

[0089] The denoising network is used to diffuse and add noise to the image features to obtain the corresponding noisy image features. In one possible implementation, the denoising network randomly adds Gaussian features to the image features. This process can be a fixed Markov chain process, which converts the original data distribution into a normal distribution by continuously adding Gaussian noise.

[0090] A text encoding network is used to perform text encoding on the acquired descriptive text (which may be a set of keywords, referred to as multi-label information) to obtain corresponding text features. In one possible implementation, the text encoding network may adopt, but is not limited to, a contrastive language-image pre-training (CLIP) model.

[0091] The denoising network is used to denoise the obtained noisy image features based on the obtained text features to obtain denoised image features. In one possible implementation, the denoising network converts Gaussian noise into content with a known data distribution through an iterative denoising process, such as using a neural network to restore data from a normal distribution to the original data distribution, so that the generated image has better diversity and realism.

[0092] In one possible implementation, the denoising network may adopt but is not limited to a U-Net network. The U-Net network is an encoding-decoding structure, and the U-Net network may adopt but is not limited to an attention mechanism for implementation.

[0093] Referring to FIG3 , FIG3 is a schematic diagram of a denoising network in an embodiment of the present application; illustratively, the denoising network is a U-Net network, which includes multiple cross-attention (QKV) modules (also known as cross-attention layers), and the cross-attention (QKV) modules included in the U-Net network can be named denoising network layers according to the function of the network. It is used to model the connection between text and image, and denoise the image based on the text.

[0094] Figure 3 illustrates the structure of the different layers of the U-Net network. Due to space limitations, only a portion of the structure is shown. As shown in Figure 3, the denoising network layer is divided into three main parts: input (IN) 31, middle (MID) 32, and output (OUT) 33. In addition, a text encoder (BASE) 34 is also included.

[0095] Except for the text encoder (BASE), the input part (IN), the intermediate part (MID) and the output part (OUT) can all be understood as the denoising network layers in the text graph model.

[0096] As shown in Figure 3, the input part 31 only briefly illustrates four layers, namely, residual module 311, attention module 312, residual module 313, and attention module 314; the middle part 32 only briefly illustrates three layers, namely, residual module 321, attention module 322, and residual module 323; the output part 33 only briefly illustrates four layers, namely, residual module 331, attention module 332, residual module 333, and attention module 334. It should be noted that the Unet structure listed in Figure 3 is only a simple example.

[0097] Among them, the U-Net network can also include a skip connection structure. Each downsampling can have a skip connection cascaded with the corresponding upsampling, so that the U-Net network can fuse the features of the corresponding position of the encoder on the channel in each upsampling, and improve the detection accuracy by fusing features of different sizes.

[0098] The image decoding network is used to decode the obtained denoised image features to obtain a set of predicted images corresponding to keywords, that is, the output image.

[0099] See Figure 4, which is a schematic diagram of the specific structure of a Wensheng graph model provided by an embodiment of the present application; as shown in Figure 4: for a random image x, the image feature (Z) is obtained through the image encoding network (E), and then the image feature (Z) is diffused and noised through the noise adding network and projected into the latent space to obtain the latent space vector, that is, the noise image feature (Z T ), at the same time, a description text (Text) is obtained through the text encoding network (τ) to obtain text features; then, the text features and the noise image features (Z T ) is input into the denoising network, the noise image feature (Z T ) is subjected to T denoising predictions under the constraints of text features through the denoising network, and finally generates a latent space prediction vector (Z'), that is, generates predicted image features; finally, the latent space prediction vector (Z') is decoded by the image decoding network (D) and the image is output. image This is the predicted image.

[0100] In the noise adding network, the image feature (Z) is subjected to T diffusion processes to generate the noise image feature (Z T ), Z T Represents the latent space value at time T. Correspondingly, in the denoising network, the noise image feature (Z T) performs T denoising predictions to obtain the predicted image features (Z'). Take the first denoising process as an example: the text features are used as KV in the QKV module, and the noise image features (Z T ) as Q in the QKV module, the text feature is used to constrain the noisy image feature (Z T ) so that the QKV module outputs predicted image features (Z') related to the input description text after T denoising.

[0101] It should be noted that FIG4 only shows one possible hierarchical relationship. In actual application, the number and connection relationship of QKV modules can be designed based on actual conditions.

[0102] In an embodiment of the present application, the features of a random image encoded by VAE are mapped to a latent space representation at time T through a diffusion network, and then a fitting of the noise representation (i.e., image prediction noise) is generated through denoising network learning, thereby subtracting the image prediction noise to obtain the actual image representation required, and then a predicted image is obtained through a decoder.

[0103] 2. Data Preparation Phase

[0104] Data collection is of paramount importance in machine learning, and can be said to be the most important link. The receipt preparation stage of the embodiment of the present application mainly includes: the preparation process of image and text sample pairs, and the object labeling process in the sample image of the image and text sample pairs.

[0105] Preparation process of picture and text sample pairs:

[0106] An image-text sample pair can be a paired sample consisting of a sample image and a descriptive text describing the sample image. The sample image can be an image used as a sample to train a text-based image model, and the descriptive text corresponding to the sample image is text describing the content of the sample image. In one possible implementation, the sample image and corresponding descriptive text in the image-text sample pair are uploaded by the user; the sample image contains at least two objects.

[0107] In another possible implementation, the image-text sample pair is targeted at an application scenario, and candidate images and associated descriptive text are obtained for that application scenario. This application scenario can be generating a corresponding film or TV series poster based on specified text, where the film or TV series poster is the sample image in the image-text sample pair, and the specified text is the descriptive text in the image-text sample pair. This application scenario can also be generating a corresponding short video cover, article cover, etc. based on the specified text.

[0108] Object labeling process:

[0109] Each object in the sample image is labeled and a mask image of the corresponding object is generated. The mask image can be an image that only shows the outline of the corresponding object and is used to distinguish the location area of ​​the corresponding object in the sample image. In one possible implementation, object annotation can be manually drawn or generated using an automatic annotation algorithm.

[0110] While labeling each object in the sample image, an object class name is also set for the object, so that the correspondence between the mask image of the object and the object class name can be determined.

[0111] See FIG5 , which is a schematic diagram of an object annotation and mask image determination method according to an embodiment of the present application;

[0112] Figure 5 exemplifies a sample image containing "flowers" and "birds": first, the "flowers" and "birds" are labeled separately to determine the positions of the "flowers" and "birds" in the image; then, the mask image of the "flower" and the mask image of the "bird" are determined, where the mask image of the "flower" only shows the outline of the "flower", and the position area of ​​the "flower" in the sample image is determined, and the mask image of the "bird" only shows the outline of the "bird", and the position area of ​​the "bird" in the sample image is determined; finally, the mask image and object class name of each object are determined.

[0113] In this way, in an embodiment of the present application, in addition to the image and text sample pairs, the sample data also includes mask images and object class names associated with each object in the sample image. Based on the setting of the mask image, the details and boundaries of the object can be distinguished, background interference can be reduced, and the accuracy of model training can be improved.

[0114] 3. Iterative training phase

[0115] In an embodiment of the present application, a text graph model is obtained by performing cyclic iterative training on the model to be trained. Cyclic iterative training is a technique commonly used in machine learning, deep learning or other algorithm optimization processes. Its core idea is to continuously improve the performance of the model by repeatedly executing a certain process or algorithm until a predetermined condition is met or a predetermined number of iterations is reached. During the model training process, a total of multiple rounds (such as 100) of cyclic iterations are performed on all pairs of graph and text samples (i.e., a training set of graph and text sample pairs), wherein all pairs of graph and text sample pairs are trained once in the model to be trained, which is called a round of iteration. In each round of iteration, due to the limited video memory resources of the training machine, all pairs of samples cannot be input into the model for training at one time, so all pairs of samples need to be trained in batches, and each batch of samples is generated by methods such as random division, and each batch of samples is input into the model separately for training such as forward calculation, backward calculation, and model parameter update.

[0116] In one possible implementation, image-text pairs are first selected from a training set of image-text pairs. Based on the sample images in the image-text pairs, mask images of at least two objects in the sample images and the associated object class names are obtained. Then, the image-text pairs and the correspondence between the object mask images and object types are used as training inputs. Multiple iterations of training are performed on the model to be trained in a multi-object embedding scenario. During each iteration, the text encoding network in the model to be trained and the LoRa module of the CrossAttention linear layer in the CrossAttention (QKV) module on the denoising network are adjusted based on the loss to obtain the text embeddings and LoRa weights for each object. The trained text-to-image model is then determined based on the object's text embeddings and LoRA weights. This enables the model to handle complex multi-object scenes and improves its ability to generate multiple objects.

[0117] Before the first round of training, the parameters of the model to be trained need to be initialized. Specifically, for model parameters that do not require adjustment (such as the model parameters of the image encoding network, denoising network, text encoding network, denoising network, and image decoding network), the model parameters of the corresponding pre-trained model are used; these parameters are continuously updated during each round of training, and the LoRA weights injected into the network are randomly initialized. Furthermore, hyperparameters such as batch, number of iterations (epochs), and learning rate are set. After the settings are completed, training begins to obtain the Wensheng graph model. The learning rate is set to 1e-4.

[0118] Since the operations performed in each loop iteration are the same, the training of the model to be trained is described by taking one loop iteration as an example. FIG6 shows a flow chart of a method for training a text graph model provided in an embodiment of the present application, which includes the following steps:

[0119] Step S601 : selecting an image-text sample pair from an image-text sample pair training set; wherein the image-text sample pair includes: a sample image and a description text of the sample image, and the sample image includes at least two objects.

[0120] The image-text sample pair training set may be a set consisting of multiple image-text sample pairs, used for training the text-image model.

[0121] Step S602 : Obtain mask images corresponding to at least two objects and associated object class names; wherein the mask images are used to distinguish the location areas of the corresponding objects in the sample image.

[0122] The object class name can be a label or name used to identify a specific object in a sample image. The object class name represents the category or type to which the object belongs. For example, "dog" is an object class name, "flower" is also an object class name, and so on.

[0123] In step S603, a sample image and a description text are input into the model to be trained to obtain image prediction noise of the sample image; and at least two mask images and associated object class names are input into the model to be trained to obtain at least two object prediction noises, where one object prediction noise corresponds to one mask image.

[0124] The model to be trained can be the basic model framework used to train the Wensheng graph model. The embodiment of the present application does not limit the model structure of the model to be trained. In one possible implementation method, the trained Wensheng graph model can be a Wensheng graph diffusion model, so the model structure of the model to be trained can also be an untrained diffusion model, as shown in Figures 2-4.

[0125] The image prediction noise can be the noise added to the sample image predicted by the model to be trained. The predicted value is a fit to the representation of the added noise. The object prediction noise can be the noise added to the mask image predicted by the model to be trained.

[0126] In one possible implementation, a sample image and a descriptive text are input into a model to be trained. When the image prediction noise of the sample image is obtained by the model to be trained, the following steps are performed: first, the original image features of the sample image are obtained through an image encoding network, and at the same time, the first text features of the descriptive text are obtained through a text encoding network; then, the original image features are subjected to noise processing by a denoising network to obtain first noise image features; finally, the first noise image features and the first text features are input into a denoising network of the model to be trained, and noise prediction is performed by the denoising network in combination with the first text features and the first noise image features to obtain image prediction noise of the sample image. The first text features may be a numerical vector obtained by converting the descriptive text into a computer-processable vector, which reflects the main content and characteristics of the descriptive text; the original image features may be a numerical vector obtained by converting the sample image into a computer-processable vector, which reflects the main content and characteristics of the sample image; and the first noise image features may be original image features with noise added.

[0127] Similarly, when obtaining the object prediction noise associated with each mask image: first, for each mask image, a corresponding mask image-text pair is constructed, which includes a mask image and an associated object class name; then, the mask image features of the mask image are obtained through the image encoding network, and the second text features of the object class name are obtained through the text encoding network; then, the mask image features are noised through the denoising network to obtain the second noise image features; finally, the second noise image features and the second text features are input into the denoising network, and the second text features and the second noise image features are combined to perform noise prediction through the denoising network to obtain the object prediction noise of the mask image. The second text features can be the object class name converted into a numerical vector that can be processed by a computer device, which is used to reflect the main content and characteristics of the object class name; the mask image features can be the mask image converted into a numerical vector that can be processed by a computer device, which is used to reflect the main content and characteristics of the mask image; and the second noise image features can be the mask image features with noise added.

[0128] During the noise addition process, the original image features are noised as a whole. When the mask image features are noised, only the part where the object is located is noised. In order to ensure the accuracy of the prediction, the noise added to the object area is consistent with the noise added to the image.

[0129] During the denoising process, the denoising network in the embodiments of the present application is primarily used to model the relationship between text and image. Thus, the image is denoised using the denoising network layers within the denoising network, conditional on each text. The denoising process first predicts noise, then subtracts the predicted noise from the noise image features obtained after the denoising process. Thus, the predicted noise can be determined using the denoising network.

[0130] In this embodiment of the present application, the Vincent graph diffusion model is used as the Vincent graph model to be trained. Because the Vincent graph diffusion model includes an iterative denoising process, it can convert Gaussian noise into samples that conform to a known data distribution. During this process, the model gradually adds noise to the sample image and gradually denoises and restores the sample image through a reverse process, thereby generating images with good diversity and realism. Using an untrained diffusion model as the model to be trained can obtain more accurate image prediction noise and object prediction noise, so that a Vincent graph model capable of generating images with good diversity and realism can be trained later.

[0131] In one possible implementation, the denoising network layer is a cross-attention layer included in the denoising network in the model to be trained; in an embodiment of the present application, the cross-attention layer is used to model the connection between text and image, and the LoRA weight in the cross-attention layer can pay attention to the trainable part in the cross-attention layer. Therefore, the network parameters of the cross-attention layer are controlled by the LoRA weight, and the image is denoised based on each text.

[0132] Taking image prediction noise as an example, see Figure 7, which is a noise prediction schematic diagram provided by an embodiment of the present application. The sample image in Figure 7 is an image describing "a brid in the flowers", and the description text is "a brid in the flowers". At this time, the sample image obtains the original image features (also known as latent space vectors) through the image encoding network (VAE_encoder), and the description text is divided into multiple prompt words. The prompt words pass through the text encoding network (CLIP) to obtain the first text features (also known as context text vectors); then, the original image features are forward-noised to obtain the first noise image features; then, the first noise image features and the first text features are input into the denoising network (U-Net) to obtain image prediction noise.

[0133] It should be noted that the image coding network is also called image encoder, and the text coding network is also called text encoder.

[0134] Step S604: construct a loss function based on the image prediction noise and the at least two object prediction noises, and adjust the parameters of the to-be-trained model based on the loss function.

[0135] In one possible implementation, when constructing a loss function based on image prediction noise and at least two object prediction noises: first difference information between the image prediction noise and the associated image target noise is obtained; and for each object prediction noise in the at least two object prediction noises, second difference information between the object prediction noise and the associated object target noise is obtained respectively; and a loss function is constructed based on the first difference information and the at least two second difference information.

[0136] Among them, the image target noise can be the real noise of the sample image, which can be used as the true value of the model training process; the object target noise can be the real noise of the mask image, which can be used as the true value of the model training process.

[0137] During the training process, in addition to the noise loss originally used for adding sample images (i.e., the first difference information), this application also introduces the added loss of local areas of multiple objects (i.e., the second difference information). Thus, on the basis of ensuring the image generation function of the Vincent graph model, the objects are separated from other areas, so that the model can focus more on the details and boundaries of the objects, thereby reducing the impact of background interference on the Vincent graph model, improving the accuracy of the Vincent graph model in multi-object scenes, and further improving the consistency and accuracy of images generated based on the Vincent graph model.

[0138] In an embodiment of the present application, the object target noise associated with each mask image is determined as follows: based on the image target noise and the mask image, a first target noise of the associated mask area is determined; then, based on the image prediction noise and the mask image, a second target noise excluding the mask area is determined; and then, based on the first target noise and the second target noise, the object target noise associated with the mask image is determined. The mask area can be a specific portion of the sample image defined and distinguished by a mask. The first target noise can be determined by performing a cross product calculation on the image target noise and the mask image, and the second target noise can be determined by performing a cross product calculation on the image prediction noise and the image excluding the mask image.

[0139] In the specific implementation process, the first target noise of the associated mask area can be determined according to the image target noise ∈ and the mask image m, that is, the first target noise is Predict noise ∈ from image θ And the mask image m, determine the second target noise except the mask area, that is, the second target noise is Based on the first target noise and the second target noise, the object target noise associated with the mask image is determined, so the object mask noise

[0140] In summary, the loss function constructed in the embodiment of the present application is:

[0141] in, is the object target noise associated with the mask image, c m，i is the object class name associated with the mask image, is the latent representation of the mask image i is the number of mask images, i=1,2...S.

[0142] The above method combines the loss of the masked area and the loss of the unmasked area to jointly determine the loss function, which can separate the object from other areas, allowing the model to focus more on the details and boundaries of the object, thereby reducing the impact of background interference on the cultural image model, improving the accuracy of the cultural image model in multi-object scenes, and further improving the consistency and accuracy of images generated based on the cultural image model.

[0143] In one possible implementation, a loss function is used to adjust the parameters of the training model. The parameters of the text encoder in the training model are adjusted based on the loss function; and the low-rank adaptive weights on each attention linear layer in the denoising network in the training model are adjusted.

[0144] See Figure 8, which is a schematic diagram of a text-based graph model training method provided in an embodiment of the present application. As shown in Figure 8, a sample image and descriptive text are input into the model to be trained to obtain image prediction noise for the sample image; at least two mask images and their associated object class names are input into the model to be trained to obtain object prediction noise associated with each of the at least two mask images; and a loss function is constructed based on the image prediction noise and image target noise, as well as the at least two object prediction noises and their associated object target noises, and the loss function is used to inversely adjust the parameters of the model to be trained.

[0145] In the embodiment of the present application, the parameters of the network of each part in the training model are adjusted separately. The parameters of each part of the network can be adjusted independently, thereby improving the independence of the module. The parameters of the networks of each part can be adjusted in parallel, thereby improving the parameter adjustment efficiency and reducing the computing resource requirements.

[0146] It should be noted that after completing a batch of training, an iteration process ends. In an embodiment of the present application, before adjusting the model parameters, it is also possible to determine whether the model convergence conditions are met. Exemplarily, the model convergence conditions may include at least one of the following conditions: the model loss is not greater than a preset loss value threshold; the number of iterations reaches a preset upper limit.

[0147] In the present application, since the input information includes sample images and descriptive text, and the sample image includes at least two objects, the model to be trained learns from multi-object scenes, so that the model to be trained can handle complex multi-object scenes and improve the generation ability of multiple objects; the input information also sets a mask image and an associated object class name, and the mask image is used to distinguish the position area of ​​the object in the sample image, so as to distinguish the object from the background and prevent the background information from interfering with the model training, and obtain the correspondence between the mask image and the object class name, which helps to enhance the object relationship between the object and the text, so that the text description can be better understood and accurately mapped to the corresponding object during the model training process, thereby ensuring the accuracy of the model to be trained; at the same time, the multi-object local area increase loss is referenced, which can separate the object from other areas, so that the model can focus more on the details and boundaries of the object, thereby reducing the impact of background interference on the model to be trained, improving the accuracy of the text image model in multi-object scenes, and further improving the consistency and accuracy of images generated based on the model to be trained.

[0148] Example 2: Model application process

[0149] In one possible implementation, the text-based graph model obtained through model training can also be used to generate an image corresponding to a specified text. Specifically, the target subject first obtains the specified text input, where the specified text contains at least two target class names. Then, based on the specified text, the text-based graph model is used, combined with the historical reference objects associated with the at least two target class names, to obtain an image corresponding to the specified text.

[0150] In this embodiment of the present application, the input method for the specified text includes, but is not limited to, voice or text. If the target object inputs the specified text via voice, voice technology can be used to convert the input voice information into the specified text. The target class name can be the object class name entered by the user and expected to be included in the generated image.

[0151] In an embodiment of the present application, based on a specified text, a text graph model is used in conjunction with historical reference objects associated with at least two target class names to obtain an image corresponding to the specified text: first, the specified text is segmented to obtain at least one keyword contained in the specified text, wherein the at least one keyword includes all target class names in the specified text; then, the obtained at least one keyword is input into the text graph model, and the text graph model is used in conjunction with historical reference objects associated with at least two target class names to obtain an image corresponding to the specified text. In this embodiment of the present application, the method for extracting keywords is not limited and will not be described in detail here.

[0152] In an embodiment of the present application, the text-generated image model can generate multiple candidate images for a specified text. When generating multiple candidate images, based on the aesthetic evaluation information corresponding to each candidate image, the top K candidate generated images are selected from at least one candidate generated image in descending order of value as images.

[0153] It should be noted that the aesthetic evaluation information is determined using an aesthetic evaluation model; and when there are multiple candidate images, the number of images can be one or more, and there is no limitation on this.

[0154] In a possible implementation, before inputting the designated text into the text-image model, it is determined that the text-image sample pair training set and the historical reference images include historical reference objects associated with at least two target class names.

[0155] In another possible implementation, before inputting the specified text into the text-graph model, when it is determined that the training set of the image-text sample pair and the historical reference image do not include historical reference objects associated with at least two target class names, an image containing reference objects associated with the target class names and the specified text are input into the text-graph model. Through the text-graph model, based on the specified text and in combination with the reference objects associated with at least two target class names in the image, an image corresponding to the specified text is obtained.

[0156] By determining whether the image-text sample pair training set and the historical reference image include at least two historical reference objects associated with the target class names, the most suitable method is selected to generate the image corresponding to the specified text, thereby ensuring the accuracy of the generated image.

[0157] The following describes how to generate an image using the text graph model obtained by the above-mentioned model training method, taking a server alone as an example. Referring to FIG9 , FIG9 is a flow chart of a method for generating an image provided in an embodiment of the present application, comprising the following steps:

[0158] Step S900: obtaining a specified text; wherein the specified text contains at least two target class names;

[0159] Step S901, determining whether the image-text sample pair training set and the historical reference image include at least two historical reference objects associated with the target class names; if so, executing step S902, otherwise executing step S903;

[0160] Step S902: input the specified text into the text-based graph model. The text-based graph model obtains an image corresponding to the specified text based on the specified text and the historical reference objects associated with at least two target class names.

[0161] Referring to FIG10 , FIG10 is a schematic diagram of an image generation method based on a text graph model provided by an embodiment of the present application. As can be seen from FIG10 , the server obtains the specified text of at least two target class names input by an object (e.g., a user) through the client. The specified text is "a man with headphone, holding a cup". After determining that the image-text sample pair training set and the historical reference image include: the historical reference object associated with the target class name (headphone "headphone"), the historical reference object associated with the target class name (cup "cup"), and the historical reference object associated with the target class name (man "man"), the specified text is input into the text graph model to obtain the image corresponding to the specified text. The server then returns the generated image to the client for presentation.

[0162] Step S903: Input the image containing the reference object associated with the target class name and the specified text into the text-graph model. The text-graph model obtains the image corresponding to the specified text based on the specified text and the reference objects associated with at least two target class names in the image.

[0163] In one possible implementation, if the image-text sample pair training set and the historical reference image do not include all reference objects associated with the target class name in the specified text, it is necessary to obtain an image that includes at least the objects that appear in the specified text but do not appear in the image-text sample pair training set and the historical reference image, and then perform image prediction based on the image and the specified text.

[0164] Referring to Figure 11, Figure 11 is a schematic diagram of an image generation method based on a text graph model provided by an embodiment of the present application. As shown in Figure 11, the server obtains the specified text of at least two target class names input by the object through the client. The specified text is "a man with headphone, holding a cup". After determining that the image-text sample pair training set and historical reference images do not include: the historical reference object associated with the target class name (headphone) and the historical reference object associated with the target class name (cup), a target reference image containing both the cup "cup" and the headphone is obtained. The target reference image containing both the cup "cup" and the headphone and the specified text are input into the text graph model. The text graph model, based on the specified text, combines the reference objects associated with the cup "cup" in the image, the reference objects associated with the headphone in the image, and the historical reference objects associated with the man "man" to obtain the image corresponding to the specified text. The server then returns the generated image to the client for presentation.

[0165] In this application, due to the improvement of the input data and loss function during the training process, the Wensheng graph model can be applied to multi-object scenes and the accuracy of the Wensheng graph model is guaranteed. Therefore, based on the Wensheng graph model, images containing multiple objects can be accurately obtained, and the consistency and accuracy of the images can be improved.

[0166] Based on the same inventive concept, an embodiment of the present application further provides a training device for a text graph model; as shown in FIG12 , the training device 1200 for the text graph model includes:

[0167] The training unit 1201 includes: an acquisition subunit 12010, an acquisition subunit 12011, a prediction subunit 12012, and a parameter adjustment subunit 12013; wherein:

[0168] The training unit 1201 is configured to perform cyclic iterative training on the training model based on the image and text samples to obtain a text-image model. In one cyclic iterative process, the following subunits perform corresponding steps:

[0169] The acquisition subunit 12010 is configured to select an image-text sample pair from the image-text sample pair training set; wherein the image-text sample pair includes: a sample image and a description text of the sample image, and the sample image includes at least two objects;

[0170] The obtaining subunit 12011 is used to obtain mask images corresponding to at least two objects and associated object class names; wherein the mask images are used to distinguish the location areas of the corresponding objects in the sample image;

[0171] The prediction subunit 12012 is configured to input a sample image and a description text into the model to be trained to obtain image prediction noise of the sample image; and input at least two mask images and associated object class names into the model to be trained to obtain object prediction noise associated with each of the at least two mask images;

[0172] The parameter adjustment subunit 12013 is used to construct a loss function based on image prediction noise and at least two object prediction noises, and adjust the parameters of the training model based on the loss function.

[0173] In one possible implementation, the prediction subunit 12012 specifically performs the following operations using the model to be trained:

[0174] Obtaining original image features of the sample image and first text features describing the text;

[0175] Performing noise processing on the original image feature to obtain a first noise image feature;

[0176] The first text feature and the first noise image feature are combined to perform noise prediction through a denoising network in the to-be-trained model to obtain image prediction noise of the sample image.

[0177] In one possible implementation, the prediction subunit 12012 is specifically configured to:

[0178] For each mask image in the at least two mask images, construct a corresponding mask image-text pair; wherein the mask image-text pair includes a mask image and an associated object class name;

[0179] Using the model to be trained, perform the following operations on the mask image-text pair:

[0180] Obtaining the mask image features of the mask image and the second text features of the object class name;

[0181] Performing noise processing on the mask image feature to obtain a second noise image feature;

[0182] The second text feature and the second noise image feature are combined to perform noise prediction through the denoising network in the to-be-trained model to obtain object prediction noise of the mask image.

[0183] In one possible implementation, the parameter adjustment subunit 12013 is specifically configured to:

[0184] obtaining first difference information between image prediction noise and associated image target noise;

[0185] For each of the at least two object prediction noises, obtaining second difference information between the object prediction noise and the associated object target noise;

[0186] A loss function is constructed based on the first difference information and at least two pieces of second difference information.

[0187] In one possible implementation, the object target noise associated with each mask image is determined as follows:

[0188] Determining a first target noise of an associated mask region according to the image target noise and the mask image;

[0189] Determining a second target noise excluding the masked area based on the image prediction noise and the mask image;

[0190] An object target noise associated with the mask image is determined based on the first target noise and the second target noise.

[0191] In one possible implementation, the parameter adjustment subunit 12013 is specifically configured to:

[0192] Based on the loss function, adjust the parameters of the text encoder in the training model; and

[0193] Adjust the low-rank adaptive weights on each attention linear layer in the denoising network in the training model.

[0194] In a possible implementation, the apparatus further includes a generating unit 1202, and the generating unit 1202 is specifically configured to:

[0195] Get the specified text; the specified text contains at least two target class names;

[0196] Through the text-generated graph model, based on the specified text, the image corresponding to the specified text is obtained in combination with the historical reference objects associated with at least two target class names.

[0197] In a possible implementation, the generating unit 1202 is further configured to:

[0198] Before inputting the specified text into the text-image model, it is determined that the image-text sample pair training set and the historical reference image include at least two historical reference objects associated with the target class names.

[0199] It should be noted that although several units (or modules) of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the implementation of the present application, the features and functions of two or more units (or modules) described above can be concretized in one unit (or module). Conversely, the features and functions of one unit (or module) described above can be further divided into multiple units (or modules) for concretization. Of course, when implementing the present application, the functions of each unit (or module) can also be implemented in the same or multiple software or hardware.

[0200] After introducing the training method and device of the cultural graph model of the exemplary implementation of the present application, another exemplary implementation computing device of the present application is introduced next.

[0201] Those skilled in the art will appreciate that various aspects of the present application may be implemented as systems, methods, or program products. Therefore, various aspects of the present application may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0202] In one possible implementation, the computing device provided in the embodiments of the present application may include at least a processor and a memory. The memory stores program code that, when executed by the processor, causes the processor to perform any step of the training method for the text graph model in various exemplary implementations of the present application.

[0203] In this embodiment, the structure of the computing device may be as shown in FIG. 13 , including a memory 1301 , a communication module 1303 , and one or more processors 1302 .

[0204] Memory 1301 is used to store computer programs executed by processor 1302. Memory 1301 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and programs required for running instant messaging functions, while the data storage area may store various instant messaging messages and operating instruction sets.

[0205] Memory 1301 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing a desired computer program in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1301 may be a combination of the aforementioned memories.

[0206] The processor 1302 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1302 is configured to implement the above-mentioned training method of the text graph model when calling the computer program stored in the memory 1301 .

[0207] The communication module 1303 is used to communicate with terminal devices and other servers.

[0208] The specific connection medium between the memory 1301, communication module 1303, and processor 1302 is not limited in the embodiments of the present application. In Figure 13, the embodiment of the present application shows that the memory 1301 and the processor 1302 are connected via a bus 1304. The bus 1304 is depicted as a bold line in Figure 13. The connection methods between other components are merely schematic and are not intended to be limiting. The bus 1304 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 13 only uses a bold line to depict, but does not represent only one bus or one type of bus.

[0209] The memory 1301 stores a computer storage medium, which stores a computer program for implementing the training method of the text graph model of the embodiment of the present application. The processor 1302 is used to execute the training method of the text graph model.

[0210] In addition, it should be noted that in the specific implementation of this application, user-related data is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0211] In some possible implementations, various aspects of the method for training a text graph model provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the computer program is run on a computing device, the computer program is used to enable the computing device to execute the steps of the method for training a text graph model according to various exemplary implementations of the present application described above in this specification.

[0212] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0213] The program product of the implementation of the present application may be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a computing device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with a command execution system, device, or device.

[0214] A readable signal medium may include a data signal transmitted in baseband or as part of a carrier wave, which carries readable program code. Such a transmitted data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with a command execution system, apparatus, or device.

[0215] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0216] The program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0217] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0218] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program commands. These computer program commands can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable device to produce a machine, so that the command executed by the processor of the computer or other programmable device produces a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0219] These computer program commands may also be stored in a computer-readable memory that can direct a computer or other programmable device to operate in a specific manner, so that the commands stored in the computer-readable memory produce a manufactured product including a command device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0220] These computer program commands can also be loaded onto a computer or other programmable device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0221] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0222] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for training a text graph model, the method being executed by a computing device, the method comprising: Based on the training set of the image and text samples, the training model is trained iteratively to obtain a text-generated image model; wherein the following steps are performed in one iterative process: Selecting a picture-text sample pair from the picture-text sample pair training set; wherein the picture-text sample pair comprises: a sample image and a description text of the sample image, and the sample image comprises at least two objects; Obtaining mask images corresponding to the at least two objects and associated object class names; wherein the mask images are used to distinguish the location areas of the corresponding objects in the sample images; Inputting the sample image and the description text into the model to be trained to obtain the image prediction noise of the sample image; and inputting at least two mask images and associated object class names into the model to be trained to obtain at least two object prediction noises, wherein one object prediction noise corresponds to one mask image; A loss function is constructed based on the image prediction noise and the at least two object prediction noises, and parameters of the model to be trained are adjusted based on the loss function.

2. The method according to claim 1, inputting the sample image and the description text into the model to be trained to obtain the image prediction noise of the sample image, comprising: The following operations are performed using the model to be trained: Obtaining original image features of the sample image and first text features of the description text; Performing noise processing on the original image feature to obtain a first noise image feature; The first text feature and the first noise image feature are combined to perform noise prediction through the denoising network in the model to be trained to obtain the image prediction noise of the sample image.

3. The method according to claim 1 or 2, wherein the at least two mask images and the associated object class names are input into the model to be trained to obtain at least two object prediction noises, comprising: For each mask image in at least two of the mask images, construct a corresponding mask image-text pair respectively; wherein the mask image-text pair includes a mask image and an associated object class name; Using the model to be trained, the following operations are performed on the mask image-text pair: Obtaining a mask image feature of the mask image and a second text feature of the object class name; Performing noise processing on the mask image feature to obtain a second noise image feature; The second text feature and the second noise image feature are combined to perform noise prediction through the denoising network in the model to be trained to obtain the object prediction noise of the mask image.

4. The method according to any one of claims 1 to 3, wherein constructing a loss function based on the image prediction noise and the at least two object prediction noises comprises: Obtaining first difference information between the image prediction noise and the associated image target noise; For each of the at least two object prediction noises, respectively obtain second difference information between the object prediction noise and the associated object target noise; A loss function is constructed based on the first difference information and at least two second difference information.

5. The method of claim 4, wherein the object target noise associated with each of the mask images is determined by: Determining a first target noise of an associated mask area according to the image target noise and the mask image; Determining a second target noise excluding the mask area according to the image prediction noise and the mask image; Based on the first target noise and the second target noise, an object target noise associated with the mask image is determined.

6. The method according to any one of claims 1 to 5, wherein adjusting parameters of the model to be trained based on the loss function comprises: Based on the loss function, adjusting parameters of the text encoder in the model to be trained; And adjust the low-rank adaptive weights on each attention linear layer in the denoising network in the model to be trained.

7. The method according to any one of claims 1 to 6, after obtaining the cultural graph model, further comprising: Obtaining a specified text; wherein the specified text contains at least two target class names; Through the text graph model, based on the designated text and in combination with the historical reference objects associated with the at least two target class names, an image corresponding to the designated text is obtained.

8. The method according to claim 7, before inputting the specified text into the text graph model, further comprising: Determine the historical reference objects associated with the at least two target class names in the image-text sample pair training set and the historical reference image.

9. A training device for a text graph model, the device being deployed on a computing device, the device comprising: A training unit, wherein the training unit includes: an acquisition subunit, an acquisition subunit, a prediction subunit, and a parameter adjustment subunit; wherein: The training unit is used to perform cyclic iterative training on the training model based on the image and text samples to obtain the text-generated graph model; wherein, in one cyclic iterative process, the following sub-units perform corresponding steps: The acquisition subunit is used to select a picture-text sample pair from the picture-text sample pair training set; wherein the picture-text sample pair includes: a sample image and a description text of the sample image, and the sample image includes at least two objects; The obtaining subunit is used to obtain the mask images corresponding to the at least two objects and the associated object class names; wherein the mask images are used to distinguish the location areas of the corresponding objects in the sample image; The prediction subunit is used to input the sample image and the description text into the model to be trained to obtain the image prediction noise of the sample image; and input at least two mask images and associated object class names into the model to be trained to obtain at least two object prediction noises, wherein one object prediction noise corresponds to one mask image; The parameter adjustment subunit is used to construct a loss function based on the image prediction noise and at least two object prediction noises, and adjust the parameters of the model to be trained based on the loss function.

10. The device according to claim 9, wherein the parameter adjustment subunit is specifically used for: Obtaining first difference information between the image prediction noise and the associated image target noise; For each object prediction noise of at least two object prediction noises, respectively obtain second difference information between the object prediction noise and the associated object target noise; A loss function is constructed based on the first difference information and at least two second difference information.

11. The apparatus of claim 10, wherein the object target noise associated with each of the mask images is determined by: Determining a first target noise of an associated mask area according to the image target noise and the mask image; Determining a second target noise excluding the mask area according to the image prediction noise and the mask image; Based on the first target noise and the second target noise, an object target noise associated with the mask image is determined.

12. The device according to any one of claims 9 to 11, wherein the parameter adjustment subunit is specifically used for: Based on the loss function, the parameters of the text encoder in the model to be trained are adjusted; and the low-rank adaptive weights on each attention linear layer in the denoising network in the model to be trained are adjusted.

13. A computing device, the computing device comprising: A processor and a memory, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to implement the method according to any one of claims 1 to 8.

14. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images