Text and graph generation model training method and device, text and graph generation method and device and storage medium

Abstract

The invention provides a text and graph generation model training method and device, a text and graph generation method and device and a storage medium, and relates to the technical field of neural networks. The text and graph generation model training method comprises the following steps: introducing a structured representation model into a text and graph generation model, and performing structured processing on fine-grained semantics in a text based on the structured representation model to obtain each semantic vector corresponding to a sample text so as to obtain a sample semantic vector set; and text and graph matching is carried out based on each semantic vector, so that a target matching image corresponding to each semantic vector can be obtained, and a matching image set is generated. Based on the incidence relation between the sample semantic vector set and the initial sample text set, the text graph generation model can be trained to accurately recognize the text, and the semantic vector optimally matched with the text is determined; and based on the association relationship between the sample semantic vector set and the matching image set, the text and graph generation model can be trained to accurately search the optimal matching image corresponding to the semantic vector, so that the accuracy of the text and graph generation result is improved.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a text-image generation model training method, a text-image generation method, a device, and a storage medium. Background Art

[0002] With the continuous development of neural network technology, text-to-image generation methods can be widely used in e-commerce, virtual and augmented reality, and gaming. Text-to-image generation methods refer to generating images that are highly relevant to the input text based on the input text.

[0003] Currently, all text-image generation methods use text representation models to convert text descriptions into feature vectors, and then use complex network structures to fuse the feature vectors and image vectors to obtain image vectors. Based on the image vectors, the image corresponding to the text is generated.

[0004] However, the currently used text representation models can only extract coarse-grained semantics from the text and cannot accurately identify the true semantics in the text, resulting in poor accuracy of images generated based on the extracted semantics. Summary of the Invention

[0005] The purpose of this application is to provide a text-image generation model training method, a text-image generation method, a device and a storage medium, so as to improve the text-image generation model's analysis of fine-grained semantics and improve the accuracy of the text-image generation results.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a text-image generation model training method, comprising:

[0008] Inputting an initial sample text set into an initial text-image generation model, performing semantic structuring processing on each sample text in the initial sample text set based on a structured representation model in the initial text-image generation model, and obtaining a sample semantic vector set corresponding to the initial sample text set, wherein the sample semantic vector set includes a semantic vector group corresponding to each sample text, the semantic vector group includes at least one semantic vector, and each semantic vector corresponds to a different semantic combination;

[0009] According to the sample semantic vector set, a matching image set corresponding to the sample semantic vector set is matched in a preset image-text library, wherein the matching image set includes a target matching image corresponding to each semantic vector;

[0010] According to the sample semantic vector set, the matching image set, and the initial sample text set, the initial text-image generation model is trained to obtain a text-image generation model.

[0011] In a second aspect, an embodiment of the present application further provides a text-image generation method, which is applied to the text-image generation model trained in the model training method described in the first aspect. The text-image generation method includes: obtaining a text to be processed;

[0012] Inputting the text to be processed into the trained text-image generation model, performing semantic structuring processing on the text to be processed based on the structured representation model to obtain at least one semantic vector corresponding to the text to be processed;

[0013] According to the text to be processed, determining a target semantic vector corresponding to the text to be processed from each semantic vector;

[0014] According to the target semantic vector corresponding to the text to be processed, a target matching image corresponding to the text to be processed is determined, and the target matching image is output.

[0015] In a third aspect, an embodiment of the present application provides a text-image generation model training device comprising: a processing module, a matching module, and a training module;

[0016] The processing module is configured to input an initial sample text set into an initial text-image generation model, perform semantic structuring processing on each sample text in the initial sample text set based on a structured representation model in the initial text-image generation model, and obtain a sample semantic vector set corresponding to the initial sample text set, wherein the sample semantic vector set includes a semantic vector group corresponding to each sample text, the semantic vector group includes at least one semantic vector, and each semantic vector corresponds to a different semantic combination;

[0017] The matching module is used to match the matching image set corresponding to the sample semantic vector set in a preset image-text library according to the sample semantic vector set, wherein the matching image set contains target matching images corresponding to each semantic vector;

[0018] The training module is used to obtain a text-image generation model by training the initial text-image generation model based on the sample semantic vector set, the matching image set, and the initial sample text set.

[0019] In a fourth aspect, an embodiment of the present application provides a text-image generation device, comprising: an acquisition module, a processing module, and a determination module;

[0020] The acquisition module is used to acquire the text to be processed;

[0021] The processing module is configured to input the text to be processed into the trained text-image generation model, perform semantic structuring processing on the text to be processed based on the structured representation model, and obtain at least one semantic vector corresponding to the text to be processed;

[0022] The determination module is configured to determine, based on the text to be processed, a target semantic vector corresponding to the text to be processed from each semantic vector;

[0023] The determination module is used to determine a target matching image corresponding to the text to be processed based on a target semantic vector corresponding to the text to be processed, and output the target matching image.

[0024] In a fifth aspect, an embodiment of the present application provides an electronic device comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the method provided in the first aspect or the second aspect.

[0025] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which executes the method provided in the first aspect or the second aspect when the computer program is executed by a processor.

[0026] The beneficial effects of this application are:

[0027] The present application provides a text-image generation model training method, a text-image generation method, a device and a storage medium. Among them, the text-image generation model training method can introduce a structured representation model into the text-image generation model, and can obtain semantic vectors under each semantic combination corresponding to the sample text based on the structured representation model for structural processing of fine-grained semantics in the text, thereby obtaining a sample semantic vector set; and based on each semantic vector, text-image matching can be performed to obtain target matching images corresponding to each semantic vector, and a matching image set can be generated. Therefore, based on the association between the sample semantic vector set and the initial sample text set, the text-image generation model can be trained to accurately recognize the text and determine the semantic vector that best matches the text; and based on the association between the sample semantic vector set and the matching image set, the text-image generation model can be trained to accurately search for the best matching image corresponding to the semantic vector, so that the trained text-image generation model can accurately recognize the semantics of the text to be processed, and based on the recognized semantics, perform accurate image search, so that the accuracy of the image generated based on the text is higher.

[0028] In addition, by introducing reinforcement learning methods to iteratively correct the network parameters of the trained model, the accuracy of the model can be further improved, and better processing effects can be achieved for fine-grained semantics.

[0029] Based on the text-image generation model obtained through the above training, executing the text-image generation method can greatly improve the matching degree between the generated image and the text to be processed, and the text-image generation method has higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 A flowchart of a text-image generation model training method provided in an embodiment of the present application;

[0032] Figure 2 A flowchart of another text-image generation model training method provided in an embodiment of the present application;

[0033] Figure 3 A flowchart of another text-image generation model training method provided in an embodiment of the present application;

[0034] Figure 4 A flowchart of another text-image generation model training method provided in an embodiment of the present application;

[0035] Figure 5 A flowchart of another text-image generation model training method provided in an embodiment of the present application;

[0036] Figure 6 A flowchart of another text-image generation model training method provided in an embodiment of the present application;

[0037] Figure 7 A flowchart of another text-image generation model training method provided in an embodiment of the present application;

[0038] Figure 8 A flowchart of another text-image generation model training method provided in an embodiment of the present application;

[0039] Figure 9 A flowchart of a method for generating text and images provided in an embodiment of the present application;

[0040] Figure 10 A schematic diagram of a text-image generation model training device provided in an embodiment of the present application;

[0041] Figure 11 A schematic diagram of a text-image generating device provided in an embodiment of the present application;

[0042] Figure 12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0044] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0045] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0046] The text-image generation method is based on artificial intelligence technology, which converts the input text into a corresponding image for output. The usual text-image generation method is that given a specific text, the corresponding text-image generation model will generate an image that matches the text based on the structure of the model. Taking the well-known Stable diffusion (text-to-image potential diffusion model) structure as an example, the model will first use a text encoder structure to convert the input text into a text vector representation, and then use a UNet (deep learning model) type complex network to fuse the text representation vector and the image vector to obtain an image vector that fuses the text representation vector. The final generated image is in good accordance with the description of the text. Such an effective design greatly improves the effect of text-image generation, and has therefore become the mainstream text-image generation method. However, the current method still has certain defects:

[0047] The text encoder model first tokenizes the text model and then encodes the words without discrimination, using the structure of self-attention to extract the entire semantics. Such a text modeling method is completely non-discriminative in some cases. For example, "astronaut rides a horse" and "the astronaut riding a horse" have the same characters and can be represented as ["yu", "hang", "yuan", "qi", "ma"]. Therefore, they are completely non-discriminative under such text representations, which ultimately leads to a completely wrong output of the text-to-image generation model when the text description is "the horse is riding the astronaut".

[0048] That is to say, the existing text-to-image generation models can only recognize the general semantics of the text, cannot perform fine-grained characterization, cannot accurately identify the relationships between the objects represented by the text description, and the representation of fine-grained semantics is not sufficient. As a result, texts with the same characters but different semantics may have the same output when generating images, and the accuracy of text-to-image generation is poor.

[0049] Based on this, this solution provides a training method for a text-to-image generation model. By introducing a structured representation model into the text-to-image generation model, fine-grained structured processing is performed on the sample text to identify different semantics contained in the sample text. Based on the similarity scores between different semantics and the target images corresponding to the sample text, training sample data for the text-to-image generation model is constructed, so that the trained text-to-image generation model can accurately identify the best semantics corresponding to the text.

[0050] Furthermore, through different semantics, image retrieval is performed in a massive image library to obtain matching images corresponding to the semantics, and training sample data for the text-to-image generation model is constructed. As a result, the trained text-to-image generation model can match the best matching image corresponding to the text based on the best semantics identified for the text, thus achieving the accuracy of text-to-image generation.

[0051] Figure 1 It is a schematic flowchart of a training method for a text-to-image generation model provided by an embodiment of this application; the execution subject of this method can be a computer device such as a terminal or a processor. As Figure 1 shown, the method may include:

[0052] S101. Input the initial sample text set into the initial text-to-image generation model, and perform semantic structuring processing on each sample text in the initial sample text set based on the structured representation model in the initial text-to-image generation model to obtain a sample semantic vector set corresponding to the initial sample text set.

[0053] Among them, the sample semantic vector set includes a semantic vector group corresponding to each sample text, and the semantic vector group includes at least one semantic vector, and each semantic vector corresponds to a different semantic combination.

[0054] The initial text-image generation model refers to the text-image generation model when the network parameters of the model are the initial default parameters. After the network parameters of the initial text-image generation model are trained based on the training sample data in this solution, the resulting model is the text-image generation model that can be used in this solution.

[0055] In some embodiments, a structured representation model can be introduced into the text-image generation model, that is, the structured representation model is embedded in the initial text-image generation model to perform semantic structuring on the input sample file. In this case, the input of the text-image generation model is text.

[0056] Of course, the structured representation model can also be deployed at the front end of the text-image generation model. In this case, the input of the text-image generation model is the output of the structured representation model.

[0057] Optionally, the initial sample text set may include multiple sample texts and target images corresponding to each sample text. The initial sample text set may be input as input data into the initial text-image generation model. First, the structured representation model in the initial text-image generation model performs semantic structuring on each sample text in the initial sample text set to parse and obtain at least one semantic vector corresponding to each sample text, thereby obtaining a semantic vector group corresponding to each sample text. The sample semantic vector group corresponding to each sample text is used to obtain a sample semantic vector set corresponding to the initial sample text set.

[0058] Among them, the structured representation model can be used to perform fine-grained analysis of the sample text and accurately identify various possible semantic combinations corresponding to the sample text.

[0059] For example, if a user inputs "astronaut riding a horse," the structured representation model performs semantic structuring and produces the following word segmentation results: astronaut, horse, and ride. However, the word segmentation results do not represent a single semantic combination. By swapping the positions of the words, different combinations can be achieved, resulting in multiple semantic combinations. For example, "astronaut riding a horse," "horse riding an astronaut," and so on can be obtained. Each semantic combination corresponds to a semantic vector. In other words, by encoding the semantic combination through the encoder, the text semantics can be converted into corresponding semantic vectors.

[0060] S102 . According to the sample semantic vector set, a matching image set corresponding to the sample semantic vector set is matched in a preset image-text library, where the matching image set includes a target matching image corresponding to each semantic vector.

[0061] Next, we can search for target matching images corresponding to each semantic vector in the sample semantic vector set in the preset image library, thereby obtaining a matching image set corresponding to the sample semantic vector set. The target matching image corresponding to the semantic vector is the image with the highest matching degree of the semantic vector.

[0062] S103: Based on the sample semantic vector set, the matching image set, and the initial sample text set, an initial text-image generation model is trained to obtain a text-image generation model.

[0063] Optionally, based on the association relationship between the sample semantic vector set, the matching image set and the initial sample text set obtained above, a text-image generation model can be trained.

[0064] Among them, based on the association between the sample semantic vector set and the initial sample text set, the text-image generation model can be trained to accurately recognize the text and determine the semantic vector that best matches the text; and based on the association between the sample semantic vector set and the matching image set, the text-image generation model can be trained to accurately search for the best matching image corresponding to the semantic vector, so that the trained text-image generation model can accurately recognize the semantics of the text to be processed, and based on the recognized semantics, perform accurate image search, so that the accuracy of the images generated based on the text is higher.

[0065] In summary, the text-image generation model training method provided in this embodiment, by introducing a structured representation model into the text-image generation model, can obtain semantic vectors under each semantic combination corresponding to the sample text based on the structured representation model for structural processing of fine-grained semantics in the text, thereby obtaining a sample semantic vector set; and based on each semantic vector, text-image matching can be performed to obtain target matching images corresponding to each semantic vector, thereby generating a matching image set. Therefore, based on the association between the sample semantic vector set and the initial sample text set, the text-image generation model can be trained to accurately recognize the text and determine the semantic vector that best matches the text; and based on the association between the sample semantic vector set and the matching image set, the text-image generation model can be trained to accurately search for the best matching image corresponding to the semantic vector, thereby accurately recognizing the semantics of the text to be processed based on the trained text-image generation model, and accurately searching for images based on the recognized semantics, so that the accuracy of the images generated based on the text is higher.

[0066] Figure 2 A flowchart of another text-image generation model training method provided in an embodiment of the present application; optionally, in step S101, semantic structuring is performed on each sample text in the initial sample text set based on the structured representation model in the initial text-image generation model to obtain a sample semantic vector set corresponding to the initial sample text set, which may include:

[0067] S201: Parse a sample text according to a structured representation model to obtain attribute elements contained in the sample text.

[0068] Text is usually composed of elements with different attributes, such as verbs, nouns, and adjectives; or object elements, relationship elements, etc. The structured representation model can be used to parse sample text and obtain the various attribute elements contained in the sample text.

[0069] For example, in the sample text "I want to watch TV," "I" is a personal pronoun, "watch" is a verb, and "TV" is a noun. By parsing the sample text, we can break it down into the attribute elements "I, want, watch, TV."

[0070] For example, the sample text is "Xiao Ming is reading a book", in which "Xiao Ming" and "book" are both objects, and "reading" represents the relationship between them. That is, through the object-relationship-object triple method, the sample text can be segmented into attribute elements such as "Xiao Ming, reading, book".

[0071] S202: Generate at least one semantic combination corresponding to the sample text according to the association relationship between the attribute elements.

[0072] The attribute elements obtained by the above analysis are independent, and by exchanging associations between the independent attribute elements, at least one combination can be obtained, and each combination can produce a semantic combination, thereby obtaining at least one semantic combination corresponding to the sample text.

[0073] For example, taking the sample text "You are my friend" as an example, after word segmentation, semantic combination is performed, and the semantic combinations that can be obtained include, for example: "You are my friend", "I am your friend", "Friends are you and me", etc.

[0074] Taking the sample text "Xiao Ming is reading a book" as an example, after word segmentation, semantic combination is performed, and the semantic combinations that can be obtained include, for example: "Xiao Ming is reading a book", "The book is reading Xiao Ming", etc.

[0075] S203 : Determine a semantic vector corresponding to each semantic combination according to each semantic combination and the coding information corresponding to each element.

[0076] Optionally, when parsing text, the structured representation model first maps the characters in the text to a vocabulary, mapping each character to a corresponding code. That is, each character has corresponding code information, which can be understood as identification information. Then, based on the code information of each character, the code information corresponding to each element can be obtained.

[0077] According to the encoding information corresponding to each semantic combination and each element, the semantic combination can be converted into a corresponding semantic vector.

[0078] Assume that the semantic combination is "Xiao Ming is reading a book", where the encoding information corresponding to "Xiao Ming" is a, the encoding information corresponding to "in" is b, the encoding information corresponding to "see" is c, and the encoding information corresponding to "book" is d. Then, convert "Xiao Ming is reading a book" into a semantic vector, and the resulting semantic vector is "abcd".

[0079] Of course, in actual applications, the encoding information of characters may not be represented by characters such as a and b. This is just an example, but each character corresponds to unique encoding information to accurately distinguish different semantic vectors.

[0080] S204 : Obtain a semantic vector group corresponding to the sample text according to the semantic vectors corresponding to each semantic combination.

[0081] A sample text can be parsed to obtain at least one semantic vector. Then, a semantic vector group corresponding to the sample text can be obtained from each semantic vector corresponding to the sample text.

[0082] Figure 3 A flowchart of another text-image generation model training method provided in an embodiment of the present application; optionally, in step S202, generating at least one semantic combination corresponding to the sample text based on the association relationship between the attribute elements may include:

[0083] S301: Determine at least one combination mode between attribute elements according to a structured representation model and at least one association relationship between attribute elements.

[0084] Optionally, based on the structured representation model's ability to represent structured knowledge, a fine-grained analysis can be performed on the semantic structure of the sample text to determine at least one association relationship between attribute elements, and each association relationship can correspond to a combination of attribute elements.

[0085] In a feasible way, the triple relationship in the sample text can be parsed and more finely divided into object elements, object-attribute tuples, and object-relationship-object triples. Taking "astronaut riding a horse" as an example, "astronaut" and "horse" are both objects, and "riding" is the relationship. By constructing two objects associated with an exchange relationship, we can obtain two descriptions: "astronaut riding a horse" and "horse riding an astronaut", that is, two semantic combinations.

[0086] S302: Generate at least one semantic combination corresponding to the sample text according to at least one combination mode between attribute elements.

[0087] According to the combination mode between each attribute element, each attribute element can be arranged and combined, so as to obtain a corresponding semantic combination based on each attribute element after the arrangement and combination and the encoding information corresponding to each attribute element.

[0088] Figure 4 A flowchart of another text-image generation model training method provided in an embodiment of the present application; optionally, in step S102, based on the sample semantic vector set, a matching image set corresponding to the sample semantic vector set is matched in a preset image-text library, where the matching image set contains target matching images corresponding to each semantic vector, and may include:

[0089] S401 : Perform image matching search based on each semantic vector in the semantic vector group corresponding to the sample text to obtain a target matching image matched by each semantic vector.

[0090] Different semantic vectors may correspond to the best matching images. In this embodiment, based on the semantic vectors corresponding to the sample text, the images corresponding to the semantic vectors may be searched in the preset image and text library to obtain the target matching images matched by each semantic vector.

[0091] S402: Determine a matching image group corresponding to the sample text according to the target matching image matched by the semantic vector.

[0092] Then, each semantic vector and the target matching image corresponding to each semantic vector can constitute a matching image group corresponding to the sample text.

[0093] S403 : Determine a matching image set corresponding to the sample semantic vector set according to the matching image group corresponding to each sample text.

[0094] A sample text corresponds to a matching image group, and then a matching image set can be obtained from the matching image group corresponding to each sample text.

[0095] Based on the sample semantic vector set and the matching image set corresponding to the sample semantic vector set, high-quality image-text pairs that conform to fine-grained semantics can be obtained.

[0096] Figure 5 A flowchart of another text-image generation model training method provided in an embodiment of the present application; optionally, in step S401, performing an image matching search based on each semantic vector in the semantic vector group corresponding to the sample text to obtain a target matching image matched by each semantic vector may include:

[0097] S501: Calculate the similarity between each semantic vector and the image vector of each candidate image.

[0098] This embodiment describes a method for image-text matching search. When performing image search based on text semantic vectors, candidate images can usually be converted into image vectors. The candidate images here can refer to any image in the above-mentioned preset image-text library.

[0099] By performing similarity calculations on each semantic vector and each candidate image in the preset image library, the similarity between each semantic vector and each candidate image can be determined. The similarity calculation here is a conventional calculation and will not be described in detail.

[0100] S502 : Determine a target matching image that matches the semantic vector from each candidate image based on the similarity between the semantic vector and the image vector of each candidate image.

[0101] Optionally, the candidate image with the highest similarity to the semantic vector can be used as the target matching image matched by the semantic vector. Through this embodiment, the structured representation model performs semantic structured representation to obtain a fine-grained semantic vector. Image search and matching is performed based on the fine-grained semantic vector to obtain the target matching image corresponding to the semantic vector. As a result, a high-quality image-text pair that conforms to fine-grained semantics is obtained, consisting of the semantic vector and the target matching image corresponding to the semantic vector. This can be used to train the text-image generation model.

[0102] Figure 6 A flowchart of another text-image generation model training method provided in an embodiment of the present application; optionally, in step S103, the text-image generation model is obtained by training the initial text-image generation model based on the sample semantic vector set, the matching image set, and the initial sample text set, which may include:

[0103] S601. Generate multiple sets of training sample data based on the semantic vector group corresponding to each sample text in the sample semantic vector set, the target image corresponding to the sample text in the initial sample text set, and the target matching image corresponding to each semantic vector in the semantic vector group corresponding to the sample text in the matching image set. The target image is annotated with a similarity label, and the similarity label is used to identify the degree of matching between each semantic vector in the semantic vector group and the target image.

[0104] This embodiment describes a training method for a text-image generation model. Based on the sample semantic vector set, matching image set, and initial sample text set obtained in the above embodiment, training sample data can be constructed first, and then model training can be performed based on the constructed training sample data.

[0105] Optionally, the semantic vector group corresponding to a sample text in the sample semantic vector set, the target image corresponding to the sample text in the initial sample text set, and the target matching image corresponding to each semantic vector in the semantic vector group corresponding to the sample text in the matching image set are used as training sample data.

[0106] That is to say, one sample text can generate a set of training sample data. Then, multiple sample texts can generate multiple sets of training sample data.

[0107] In each set of sample data, the target image corresponding to the sample text may be annotated with a similarity label, and the similarity label is used to identify the degree of matching between each semantic vector in the semantic vector group of the sample text and the target image.

[0108] S602: Obtain a training sample data set based on multiple sets of training sample data.

[0109] Then, a training sample data set can be formed by multiple sets of training sample data.

[0110] S603: Based on the initial text-image generation model and the training sample data set, a text-image generation model is trained.

[0111] Optionally, the training sample data set can be used as input data and input into the initial text-image generation model, so as to train the text-image generation model.

[0112] Figure 7 A flowchart of another text-image generation model training method provided in an embodiment of the present application; optionally, in step S603, training the text-image generation model based on the initial text-image generation model and the training sample dataset may include:

[0113] S701: Input the training sample data set into the initial text-image generation model, and construct a correspondence between the sample text and the target semantic vector corresponding to the sample text based on the semantic vector group corresponding to each sample text and the target image corresponding to each sample text.

[0114] Optionally, the initial text-image generation model can first learn the similarity between each semantic vector and the target image based on the semantic vector group corresponding to each sample text, the target image corresponding to each sample text, and the similarity label annotated by the target image, thereby constructing a correspondence between the sample text and the target semantic vector corresponding to the sample text, so that the initial text-image generation model can filter out the target semantic vector corresponding to the sample text from the semantic vectors of the sample text based on the similarity between the semantic vectors of the sample text and the target image corresponding to the sample text, that is, by learning the relationship between each semantic vector and the target image, it can accurately determine the target semantic vector that best matches the sample text from the multiple possible semantic vectors of the sample text.

[0115] S702: Based on each semantic vector in the semantic vector group corresponding to each sample text and the target matching image corresponding to each semantic vector, a correspondence between the target semantic vector corresponding to the sample text and the target matching image is constructed to obtain a text-image generation model.

[0116] Because high-quality image-text pairs with fine-grained semantics are constructed based on each semantic vector and the target matching image corresponding to each semantic vector, the initial text-image generation model can then construct a correspondence between the target semantic vector corresponding to the sample text and the target matching image based on each semantic vector and the target matching image corresponding to each semantic vector. This allows the initial text-image generation model to accurately match images based on the semantic vectors, thereby matching the sample text with the target semantic vector corresponding to the sample text to obtain the best matching image.

[0117] Based on the above training process, the trained text-image generation model can be used to obtain the target semantic vector corresponding to the sample text based on sample text analysis, and then perform image search based on the target semantic vector to determine the target image matching the target semantic vector, thereby converting the sample text into an image.

[0118] Figure 8 A flowchart of another text-image generation model training method provided in an embodiment of the present application; optionally, in step S603, training the text-image generation model based on the initial text-image generation model and the training sample dataset may include:

[0119] S801: Input a test text into the current text-image generation model to obtain an image corresponding to the test text output by the current text generation model.

[0120] In a feasible way, during the training of the text-image generation model, we also introduced the idea of ​​reinforcement learning to adjust the network parameters of the text-image generation model.

[0121] Optionally, test text can be collected. Multiple test texts can be used, and fine-grained text descriptions can also be selected. The test text is input into the current text-to-image generation model, and the current text generation model performs text-to-image conversion to obtain an image corresponding to the test text output by the current text generation model.

[0122] S802: Calculate the degree of matching between the test text and the image corresponding to the test text.

[0123] Next, the degree of matching between the test text and the image corresponding to the test text can be calculated based on the structured representation model in the current text generation model. The degree of matching can be obtained by calculating a similarity score between the test text and the image corresponding to the test text.

[0124] S803: If the matching degree does not meet the preset conditions, the network parameters of the current text-image generation model are modified until the matching degree meets the preset conditions, and then the modification of the current text-image generation model is stopped.

[0125] The similarity score between the test text and the image corresponding to the test text can be used as a reward score for reinforcement learning, and the reward score can be used to reversely update the network parameters of the model.

[0126] Optionally, it can be determined whether the similarity score between the test text and the image corresponding to the test text meets a preset condition. The preset condition here can be a pre-set similarity score threshold. If the similarity score between the test text and the image corresponding to the test text does not meet the threshold, the network parameters of the current text-image generation model are corrected, and based on the corrected text-image generation model, reinforcement learning is re-iterated until the similarity score between the test text and the image corresponding to the test text meets the set threshold, and the correction is stopped.

[0127] In another feasible method, instead of setting the similarity score threshold, a maximum number of iterations can be set. When the number of iterations of reinforcement learning meets the set maximum number of iterations, the correction of the model can be stopped. At this time, the network parameters of the model in the round with the highest output result accuracy among multiple iterations can be selected as the final network parameters of the model.

[0128] In summary, the text-image generation model training method provided in this embodiment, by introducing a structured representation model into the text-image generation model, can obtain semantic vectors under each semantic combination corresponding to the sample text based on the structured representation model for structural processing of fine-grained semantics in the text, thereby obtaining a sample semantic vector set; and based on each semantic vector, text-image matching can be performed to obtain target matching images corresponding to each semantic vector, thereby generating a matching image set. Therefore, based on the association between the sample semantic vector set and the initial sample text set, the text-image generation model can be trained to accurately recognize the text and determine the semantic vector that best matches the text; and based on the association between the sample semantic vector set and the matching image set, the text-image generation model can be trained to accurately search for the best matching image corresponding to the semantic vector, thereby accurately recognizing the semantics of the text to be processed based on the trained text-image generation model, and accurately searching for images based on the recognized semantics, so that the accuracy of the images generated based on the text is higher.

[0129] In addition, by introducing reinforcement learning methods to iteratively correct the network parameters of the trained model, the accuracy of the model can be further improved, and better processing effects can be achieved for fine-grained semantics.

[0130] The above describes the training method of the text-image generation model based on multiple embodiments. The following describes the application method of the text-image generation model.

[0131] Figure 9 A flowchart of a text-image generation method provided in an embodiment of the present application; this method can be applied to the text-image generation model trained in the above-mentioned text-image generation model training method, and can be used as follows Figure 9 The method shown in FIG, applies the trained text-image generation model to perform text-image generation processing. Figure 9 As shown, the method may include:

[0132] S901. Obtain the text to be processed.

[0133] The text to be processed can be text generated in actual application scenarios, such as clothing design scenarios, virtual reality scenarios, or game scenarios.

[0134] In a game scene, when designing a virtual object image and a game scene, corresponding images can be generated based on the text to be processed input by the user to perform virtual object image design and game scene design.

[0135] S902: Input the text to be processed into the trained text-image generation model, perform semantic structural processing on the text to be processed based on the structured representation model, and obtain at least one semantic vector corresponding to the text to be processed.

[0136] After the text to be processed is input into the text-image generation model, the text to be processed can be semantically structured based on the structured representation model in the text-image generation model to obtain at least one semantic vector corresponding to the text to be processed. That is, the text to be processed is analyzed in a fine-grained manner to obtain the semantic vectors of the text to be processed under different semantic combinations.

[0137] S903 : According to the text to be processed, determine a target semantic vector corresponding to the text to be processed from each semantic vector.

[0138] Next, based on the correspondence between the text to be processed and the target semantic vectors corresponding to the sample text, the target semantic vector corresponding to the text to be processed can be determined from each semantic vector. The target semantic vector is the vector that is closest to the true semantics of the text to be processed.

[0139] S904 : Determine a target matching image corresponding to the text to be processed according to the target semantic vector corresponding to the text to be processed, and output the target matching image.

[0140] Based on the correspondence between the target semantic vector corresponding to the text to be processed and the target semantic vector corresponding to the sample text and the target matching image, the target matching image corresponding to the text to be processed can be matched, thereby finally outputting the target matching image corresponding to the text to be processed.

[0141] Taking the text to be processed as "Xiao Ming is looking at monkeys" as an example, the text-to-image generation model uses "Xiao Ming is looking at monkeys" as input. After structural processing using the structured representation model, the word segmentation can be performed to obtain the attribute elements "Xiao Ming," "at," "look," and "monkey." Based on the associations between these attribute elements, at least one semantic combination can be obtained, such as "Xiao Ming is looking at monkeys" and "Monkeys are looking at Xiao Ming." Based on the correspondence between the text constructed by the text-to-image generation model and the target semantic vector, it can be determined that "Xiao Ming is looking at monkeys" has the highest similarity to the true semantics of the text to be processed. Therefore, the target semantics can be determined to be "Xiao Ming is looking at monkeys" from the multiple possible semantic combinations. Encoding based on the target semantics can then be performed to obtain the target semantic vector corresponding to the target semantics.

[0142] Next, based on the correspondence between the semantic vector constructed by the text generation model and the image, image matching retrieval can be performed on the target semantic vector to search for the target image that best matches the target semantics "Xiao Ming is looking at the monkey" from the massive image library, and then the searched target image is output as the image corresponding to the text to be processed.

[0143] In summary, based on the text-image generation model obtained through the above training, executing the text-image generation process can greatly improve the matching degree between the generated image and the text to be processed, and the text-image generation method has higher accuracy.

[0144] The following describes the devices, equipment, storage media, etc. used to execute the text-image generation model training method provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0145] Figure 10 This is a schematic diagram of a text-image generation model training device provided in an embodiment of the present application. The functions implemented by the text-image generation model training device correspond to the steps performed by the above-mentioned text-image generation model training method. The device can be understood as the above-mentioned server, or the server's processor, or can be understood as a component independent of the above-mentioned server or processor that implements the functions of the present application under the control of the server. Optionally, Figure 10 As shown, the apparatus may include: a processing module 110, a matching module 120, and a training module 130;

[0146] Processing module 110 is configured to input the initial sample text set into the initial text-image generation model, perform semantic structuring on each sample text in the initial sample text set based on the structured representation model in the initial text-image generation model, and obtain a sample semantic vector set corresponding to the initial sample text set, wherein the sample semantic vector set includes a semantic vector group corresponding to each sample text, each semantic vector group includes at least one semantic vector, and each semantic vector corresponds to a different semantic combination;

[0147] A matching module 120 is configured to match a matching image set corresponding to the sample semantic vector set in a preset image-text library according to the sample semantic vector set, wherein the matching image set includes a target matching image corresponding to each semantic vector;

[0148] The training module 130 is used to train the initial text-image generation model based on the sample semantic vector set, the matching image set, and the initial sample text set to obtain the text-image generation model.

[0149] Optionally, the processing module 110 is specifically configured to parse the sample text according to the structured representation model to obtain various attribute elements contained in the sample text;

[0150] Generate at least one semantic combination corresponding to the sample text according to the association relationship between the attribute elements;

[0151] Determine the semantic vector corresponding to each semantic combination according to the encoding information corresponding to each semantic combination and each element;

[0152] According to the semantic vectors corresponding to each semantic combination, a semantic vector group corresponding to the sample text is obtained.

[0153] Optionally, the processing module 110 is specifically configured to determine at least one combination mode between the attribute elements according to the structured representation model and at least one association relationship between the attribute elements;

[0154] At least one semantic combination corresponding to the sample text is generated according to at least one combination mode between the attribute elements.

[0155] Optionally, the matching module 120 is specifically configured to perform an image matching search based on each semantic vector in the semantic vector group corresponding to the sample text, and obtain a target matching image matched by each semantic vector;

[0156] Determine a matching image group corresponding to the sample text according to the target matching image matched by the semantic vector;

[0157] According to the matching image groups corresponding to the sample texts, the matching image sets corresponding to the sample semantic vector sets are determined.

[0158] Optionally, the matching module 120 is specifically configured to calculate the similarity between each semantic vector and the image vector of each candidate image;

[0159] According to the similarity between the semantic vector and the image vector of each candidate image, a target matching image that matches the semantic vector is determined from each candidate image.

[0160] Optionally, the training module 130 is specifically configured to generate multiple sets of training sample data based on the semantic vector group corresponding to each sample text in the sample semantic vector set, the target image corresponding to the sample text in the initial sample text set, and the target matching image corresponding to each semantic vector in the semantic vector group corresponding to the sample text in the matching image set, wherein the target image is annotated with a similarity label, and the similarity label is used to identify the degree of matching between each semantic vector in the semantic vector group and the target image;

[0161] According to multiple groups of training sample data, a training sample data set is obtained;

[0162] Based on the initial text-image generation model and the training sample data set, a text-image generation model is trained.

[0163] Optionally, the training module 130 is specifically configured to input the training sample data set into the initial text-image generation model, and construct a correspondence between the sample text and the target semantic vector corresponding to the sample text based on the semantic vector group corresponding to each sample text and the target image corresponding to each sample text;

[0164] According to each semantic vector in the semantic vector group corresponding to each sample text and the target matching image corresponding to each semantic vector, a correspondence between the target semantic vector corresponding to the sample text and the target matching image is constructed to obtain a text-image generation model.

[0165] Optionally, the training module 130 is specifically configured to input the test text into the current text-image generation model to obtain an image corresponding to the test text output by the current text generation model;

[0166] Calculate the degree of matching between the test text and the image corresponding to the test text;

[0167] If the matching degree does not meet the preset conditions, the network parameters of the current text-image generation model are modified until the matching degree meets the preset conditions, and then the modification of the current text-image generation model is stopped.

[0168] Figure 11 This is a schematic diagram of a text-image generation device provided in an embodiment of the present application. The functions implemented by the text-image generation device correspond to the steps performed by the above-mentioned text-image generation method. Figure 11 As shown, the apparatus may include: an acquisition module 210, a processing module 220, and a determination module 230;

[0169] An acquisition module 210 is used to acquire the text to be processed;

[0170] The processing module 220 is configured to input the text to be processed into the trained text-image generation model, perform semantic structural processing on the text to be processed based on the structured representation model, and obtain at least one semantic vector corresponding to the text to be processed;

[0171] A determination module 230 is configured to determine, based on the text to be processed, a target semantic vector corresponding to the text to be processed from among the semantic vectors;

[0172] The determination module 230 is configured to determine a target matching image corresponding to the text to be processed based on a target semantic vector corresponding to the text to be processed, and output the target matching image.

[0173] The above modules can be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), one or more digital singnal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0174] The above modules can be connected or communicate with each other via a wired connection or a wireless connection. The wired connection may include a metal cable, an optical cable, a hybrid cable, etc., or any combination thereof. The wireless connection may include a connection in the form of a LAN, a WAN, Bluetooth, ZigBee, or NFC, or any combination thereof. Two or more modules can be combined into a single module, and any module can be divided into two or more units. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application.

[0175] Figure 12A schematic diagram of the structure of an electronic device provided in an embodiment of the present application includes: a processor 801, a storage medium 802, and a bus 803. The storage medium 802 stores machine-readable instructions executable by the processor 801. When the electronic device runs a text-image generation model training method as described in the embodiment, the processor 801 communicates with the storage medium 802 via the bus 803. The processor 801 executes the machine-readable instructions to perform the following steps:

[0176] Inputting the initial sample text set into the initial text-image generation model, performing semantic structuring processing on each sample text in the initial sample text set based on the structured representation model in the initial text-image generation model, and obtaining a sample semantic vector set corresponding to the initial sample text set, wherein the sample semantic vector set includes a semantic vector group corresponding to each sample text, each semantic vector group includes at least one semantic vector, and each semantic vector corresponds to a different semantic combination;

[0177] According to the sample semantic vector set, a matching image set corresponding to the sample semantic vector set is matched in a preset image-text library, where the matching image set contains target matching images corresponding to each semantic vector;

[0178] According to the sample semantic vector set, the matching image set, and the initial sample text set, the initial text-image generation model is trained to obtain the text-image generation model.

[0179] In one feasible embodiment, when the processor 801 performs semantic structuring processing on each sample text in the initial sample text set based on the structured representation model in the initial text-image generation model to obtain a sample semantic vector set corresponding to the initial sample text set, the processor 801 is specifically configured to: parse the sample text according to the structured representation model to obtain each attribute element contained in the sample text;

[0180] Generate at least one semantic combination corresponding to the sample text according to the association relationship between the attribute elements;

[0181] Determine the semantic vector corresponding to each semantic combination according to the encoding information corresponding to each semantic combination and each element;

[0182] According to the semantic vectors corresponding to each semantic combination, a semantic vector group corresponding to the sample text is obtained.

[0183] In one feasible embodiment, when executing the process of generating at least one semantic combination corresponding to the sample text based on the association relationship between the attribute elements, the processor 801 is specifically configured to: determine at least one combination mode between the attribute elements based on the structured representation model and at least one association relationship between the attribute elements;

[0184] At least one semantic combination corresponding to the sample text is generated according to at least one combination mode between the attribute elements.

[0185] In one feasible embodiment, when the processor 801 matches a matching image set corresponding to the sample semantic vector set in a preset image-text library according to the sample semantic vector set, and the matching image set includes a target matching image corresponding to each semantic vector, the processor 801 is specifically configured to: perform an image matching search according to each semantic vector in the semantic vector group corresponding to the sample text, and obtain a target matching image matched by each semantic vector;

[0186] Determine a matching image group corresponding to the sample text according to the target matching image matched by the semantic vector;

[0187] According to the matching image groups corresponding to the sample texts, the matching image sets corresponding to the sample semantic vector sets are determined.

[0188] In one feasible embodiment, when the processor 801 performs an image matching search based on each semantic vector in the semantic vector group corresponding to the sample text and obtains a target matching image matched by each semantic vector, the processor 801 is specifically configured to: calculate the similarity between each semantic vector and the image vector of each candidate image;

[0189] According to the similarity between the semantic vector and the image vector of each candidate image, a target matching image that matches the semantic vector is determined from each candidate image.

[0190] In a feasible embodiment, when the processor 801 executes the training of the initial text-image generation model to obtain the text-image generation model based on the sample semantic vector set, the matching image set, and the initial sample text set, it is specifically used to: generate multiple sets of training sample data based on the semantic vector group corresponding to each sample text in the sample semantic vector set, the target image corresponding to the sample text in the initial sample text set, and the target matching image corresponding to each semantic vector in the semantic vector group corresponding to the sample text in the matching image set, the target image is annotated with a similarity label, and the similarity label is used to identify the degree of matching between each semantic vector in the semantic vector group and the target image;

[0191] According to multiple groups of training sample data, a training sample data set is obtained;

[0192] Based on the initial text-image generation model and the training sample dataset, a text-image generation model is trained.

[0193] In one feasible embodiment, when executing the training to obtain the text-image generation model based on the initial text-image generation model and the training sample data set, the processor 801 is specifically configured to: input the training sample data set into the initial text-image generation model, and construct a correspondence between the sample text and the target semantic vector corresponding to the sample text based on the semantic vector group corresponding to each sample text and the target image corresponding to each sample text;

[0194] According to each semantic vector in the semantic vector group corresponding to each sample text and the target matching image corresponding to each semantic vector, a correspondence between the target semantic vector corresponding to the sample text and the target matching image is constructed to obtain a text-image generation model.

[0195] In one feasible embodiment, when the processor 801 executes the training based on the initial text-image generation model and the training sample data set to obtain the text-image generation model, it is specifically configured to: input the test text into the current text-image generation model to obtain the image corresponding to the test text output by the current text generation model;

[0196] Calculate the degree of matching between the test text and the image corresponding to the test text;

[0197] If the matching degree does not meet the preset conditions, the network parameters of the current text-image generation model are modified until the matching degree meets the preset conditions, and then the modification of the current text-image generation model is stopped.

[0198] Through the above-mentioned method, by introducing a structured representation model into the text-to-image generation model, the structured representation model can be used to perform structural processing on the fine-grained semantics in the text, and semantic vectors corresponding to each semantic combination of the sample text can be obtained, thereby obtaining a sample semantic vector set; and by performing text-to-image matching based on each semantic vector, target matching images corresponding to each semantic vector can be obtained, thereby generating a matching image set. Therefore, based on the association between the sample semantic vector set and the initial sample text set, the text-to-image generation model can be trained to accurately recognize the text and determine the semantic vector that best matches the text; and based on the association between the sample semantic vector set and the matching image set, the text-to-image generation model can be trained to accurately search for the best matching image corresponding to the semantic vector, thereby accurately recognizing the semantics of the text to be processed based on the trained text-to-image generation model, and accurately searching for images based on the recognized semantics, thereby achieving higher accuracy in images generated based on the text.

[0199] When the electronic device executes a text-image generation method in the embodiment, the processor 801 communicates with the storage medium 802 via the bus 803, and the processor 801 executes machine-readable instructions to perform the following steps:

[0200] Get the text to be processed;

[0201] Inputting the text to be processed into the trained text-image generation model, performing semantic structural processing on the text to be processed based on the structured representation model, and obtaining at least one semantic vector corresponding to the text to be processed;

[0202] According to the text to be processed, determining the target semantic vector corresponding to the text to be processed from each semantic vector;

[0203] According to the target semantic vector corresponding to the text to be processed, a target matching image corresponding to the text to be processed is determined, and the target matching image is output.

[0204] Among them, the storage medium 802 stores program code. When the program code is executed by the processor 801, the processor 801 executes the various steps in the text-image generation model training method or the text-image generation method according to various exemplary embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0205] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0206] The storage medium 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory can include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disc, etc. The memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The storage medium 802 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0207] Optionally, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor performs the following steps:

[0208] Inputting the initial sample text set into the initial text-image generation model, performing semantic structuring processing on each sample text in the initial sample text set based on the structured representation model in the initial text-image generation model, and obtaining a sample semantic vector set corresponding to the initial sample text set, wherein the sample semantic vector set includes a semantic vector group corresponding to each sample text, each semantic vector group includes at least one semantic vector, and each semantic vector corresponds to a different semantic combination;

[0209] According to the sample semantic vector set, a matching image set corresponding to the sample semantic vector set is matched in a preset image-text library, where the matching image set contains target matching images corresponding to each semantic vector;

[0210] According to the sample semantic vector set, the matching image set, and the initial sample text set, the initial text-image generation model is trained to obtain the text-image generation model.

[0211] In one feasible embodiment, when the processor 801 performs semantic structuring processing on each sample text in the initial sample text set based on the structured representation model in the initial text-image generation model to obtain a sample semantic vector set corresponding to the initial sample text set, the processor 801 is specifically configured to: parse the sample text according to the structured representation model to obtain each attribute element contained in the sample text;

[0212] Generate at least one semantic combination corresponding to the sample text according to the association relationship between the attribute elements;

[0213] Determine the semantic vector corresponding to each semantic combination according to the encoding information corresponding to each semantic combination and each element;

[0214] According to the semantic vectors corresponding to each semantic combination, a semantic vector group corresponding to the sample text is obtained.

[0215] In one feasible embodiment, when executing the process of generating at least one semantic combination corresponding to the sample text based on the association relationship between the attribute elements, the processor 801 is specifically configured to: determine at least one combination mode between the attribute elements based on the structured representation model and at least one association relationship between the attribute elements;

[0216] At least one semantic combination corresponding to the sample text is generated according to at least one combination mode between the attribute elements.

[0217] In one feasible embodiment, when the processor 801 matches a matching image set corresponding to the sample semantic vector set in a preset image-text library according to the sample semantic vector set, and the matching image set includes a target matching image corresponding to each semantic vector, the processor 801 is specifically configured to: perform an image matching search according to each semantic vector in the semantic vector group corresponding to the sample text, and obtain a target matching image matched by each semantic vector;

[0218] Determine a matching image group corresponding to the sample text according to the target matching image matched by the semantic vector;

[0219] According to the matching image groups corresponding to the sample texts, the matching image sets corresponding to the sample semantic vector sets are determined.

[0220] In one feasible embodiment, when the processor 801 performs an image matching search based on each semantic vector in the semantic vector group corresponding to the sample text and obtains a target matching image matched by each semantic vector, the processor 801 is specifically configured to: calculate the similarity between each semantic vector and the image vector of each candidate image;

[0221] According to the similarity between the semantic vector and the image vector of each candidate image, a target matching image that matches the semantic vector is determined from each candidate image.

[0222] In a feasible embodiment, when the processor 801 executes the training of the initial text-image generation model to obtain the text-image generation model based on the sample semantic vector set, the matching image set, and the initial sample text set, it is specifically used to: generate multiple sets of training sample data based on the semantic vector group corresponding to each sample text in the sample semantic vector set, the target image corresponding to the sample text in the initial sample text set, and the target matching image corresponding to each semantic vector in the semantic vector group corresponding to the sample text in the matching image set, the target image is annotated with a similarity label, and the similarity label is used to identify the degree of matching between each semantic vector in the semantic vector group and the target image;

[0223] According to multiple groups of training sample data, a training sample data set is obtained;

[0224] Based on the initial text-image generation model and the training sample data set, a text-image generation model is trained.

[0225] In one feasible embodiment, when executing the training to obtain the text-image generation model based on the initial text-image generation model and the training sample data set, the processor 801 is specifically configured to: input the training sample data set into the initial text-image generation model, and construct a correspondence between the sample text and the target semantic vector corresponding to the sample text based on the semantic vector group corresponding to each sample text and the target image corresponding to each sample text;

[0226] According to each semantic vector in the semantic vector group corresponding to each sample text and the target matching image corresponding to each semantic vector, a correspondence between the target semantic vector corresponding to the sample text and the target matching image is constructed to obtain a text-image generation model.

[0227] In one feasible embodiment, when the processor 801 executes the training based on the initial text-image generation model and the training sample data set to obtain the text-image generation model, it is specifically configured to: input the test text into the current text-image generation model to obtain the image corresponding to the test text output by the current text generation model;

[0228] Calculate the degree of matching between the test text and the image corresponding to the test text;

[0229] If the matching degree does not meet the preset conditions, the network parameters of the current text-image generation model are modified until the matching degree meets the preset conditions, and then the modification of the current text-image generation model is stopped.

[0230] Through the above-mentioned method, by introducing a structured representation model into the text-to-image generation model, the structured representation model can be used to perform structural processing on the fine-grained semantics in the text, and semantic vectors corresponding to each semantic combination of the sample text can be obtained, thereby obtaining a sample semantic vector set; and based on each semantic vector, text-to-image matching can be performed to obtain target matching images corresponding to each semantic vector, thereby generating a matching image set. Therefore, based on the association between the sample semantic vector set and the initial sample text set, the text-to-image generation model can be trained to accurately recognize the text and determine the semantic vector that best matches the text; and based on the association between the sample semantic vector set and the matching image set, the text-to-image generation model can be trained to accurately search for the best matching image corresponding to the semantic vector, thereby accurately recognizing the semantics of the text to be processed based on the trained text-to-image generation model, and based on the recognized semantics, accurately searching for images, thereby making the images generated based on the text more accurate.

[0231] Optionally, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor performs the following steps:

[0232] Get the text to be processed;

[0233] Inputting the text to be processed into the trained text-image generation model, performing semantic structural processing on the text to be processed based on the structured representation model, and obtaining at least one semantic vector corresponding to the text to be processed;

[0234] According to the text to be processed, determining the target semantic vector corresponding to the text to be processed from each semantic vector;

[0235] According to the target semantic vector corresponding to the text to be processed, a target matching image corresponding to the text to be processed is determined, and the target matching image is output.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0237] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0238] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0239] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (English: Read-Only Memory, abbreviated: ROM), a random access memory (English: Random Access Memory, abbreviated: RAM), a disk or an optical disk, and other media that can store program code.

Claims

1. A text-image generation model training method, characterized in that: include: Inputting an initial sample text set into an initial text-image generation model, performing semantic structuring processing on each sample text in the initial sample text set based on a structured representation model in the initial text-image generation model, and obtaining a sample semantic vector set corresponding to the initial sample text set, wherein the sample semantic vector set includes a semantic vector group corresponding to each sample text, the semantic vector group includes at least one semantic vector, and each semantic vector corresponds to a different semantic combination; According to the sample semantic vector set, a matching image set corresponding to the sample semantic vector set is matched in a preset image-text library, wherein the matching image set includes a target matching image corresponding to each semantic vector; According to the sample semantic vector set, the matching image set, and the initial sample text set, the initial text-image generation model is trained to obtain a text-image generation model.

2. The text-image generation model training method according to claim 1, characterized in that: The structured representation model in the initial text-image generation model performs semantic structuring processing on each sample text in the initial sample text set to obtain a sample semantic vector set corresponding to the initial sample text set, including: Parsing the sample text according to the structured representation model to obtain various attribute elements contained in the sample text; generating at least one semantic combination corresponding to the sample text according to the association relationship between the attribute elements; Determine the semantic vector corresponding to each semantic combination according to the encoding information corresponding to each semantic combination and each element; According to the semantic vectors corresponding to the semantic combinations, a semantic vector group corresponding to the sample text is obtained.

3. The text-image generation model training method according to claim 2, characterized in that: Generating at least one semantic combination corresponding to the sample text according to the association relationship between the attribute elements includes: Determining at least one combination of the attribute elements based on the structured representation model and at least one association relationship between the attribute elements; At least one semantic combination corresponding to the sample text is generated according to at least one combination mode between the attribute elements.

4. The text-image generation model training method according to claim 1, characterized in that: The method of matching a matching image set corresponding to the sample semantic vector set in a preset image-text library according to the sample semantic vector set, wherein the matching image set includes a target matching image corresponding to each semantic vector, includes: Perform image matching search based on each semantic vector in the semantic vector group corresponding to the sample text to obtain a target matching image matched by each semantic vector; Determining a matching image group corresponding to the sample text according to a target matching image matched by the semantic vector; According to the matching image groups corresponding to the sample texts, the matching image sets corresponding to the sample semantic vector sets are determined.

5. The text-image generation model training method according to claim 4, characterized in that: The performing of image matching search based on each semantic vector in the semantic vector group corresponding to the sample text to obtain a target matching image matched by each semantic vector includes: Calculating the similarity between each of the semantic vectors and the image vector of each candidate image; According to the similarity between the semantic vector and the image vectors of each candidate image, a target matching image that matches the semantic vector is determined from each candidate image.

6. The text-image generation model training method according to claim 1, characterized in that: The method of obtaining a text-image generation model by training the initial text-image generation model based on the sample semantic vector set, the matching image set, and the initial sample text set includes: Generate multiple sets of training sample data based on the semantic vector group corresponding to each sample text in the sample semantic vector set, the target image corresponding to the sample text in the initial sample text set, and the target matching image corresponding to each semantic vector in the semantic vector group corresponding to the sample text in the matching image set, wherein the target image is annotated with a similarity label, and the similarity label is used to identify the degree of matching between each semantic vector in the semantic vector group and the target image; According to multiple groups of training sample data, a training sample data set is obtained; Based on the initial text-image generation model and the training sample data set, a text-image generation model is trained.

7. The text-image generation model training method according to claim 6, characterized in that: The training of the text-image generation model based on the initial text-image generation model and the training sample data set includes: Inputting the training sample data set into the initial text-image generation model, and constructing a correspondence between the sample text and the target semantic vector corresponding to the sample text based on the semantic vector group corresponding to each sample text and the target image corresponding to each sample text; According to each semantic vector in the semantic vector group corresponding to each sample text and the target matching image corresponding to each semantic vector, a correspondence between the target semantic vector corresponding to the sample text and the target matching image is constructed to obtain the text-image generation model.

8. The text-image generation model training method according to claim 6, characterized in that: The training of the text-image generation model based on the initial text-image generation model and the training sample data set includes: Input the test text into the current text-image generation model to obtain an image corresponding to the test text output by the current text generation model; Calculating a degree of matching between the test text and an image corresponding to the test text; If the matching degree does not satisfy the preset conditions, the network parameters of the current text-image generation model are modified until the matching degree satisfies the preset conditions, and then the modification of the current text-image generation model is stopped.

9. A method for generating text and images, characterized in that: The text-image generation model trained in the model training method according to any one of claims 1 to 8 is applied, wherein the text-image generation method comprises: Get the text to be processed; Inputting the text to be processed into the trained text-image generation model, performing semantic structuring processing on the text to be processed based on the structured representation model to obtain at least one semantic vector corresponding to the text to be processed; According to the text to be processed, determining a target semantic vector corresponding to the text to be processed from each semantic vector; According to the target semantic vector corresponding to the text to be processed, a target matching image corresponding to the text to be processed is determined, and the target matching image is output.

10. A text-image generation model training device, characterized in that: include: Processing module, matching module and training module; The processing module is configured to input an initial sample text set into an initial text-image generation model, perform semantic structuring processing on each sample text in the initial sample text set based on a structured representation model in the initial text-image generation model, and obtain a sample semantic vector set corresponding to the initial sample text set, wherein the sample semantic vector set includes a semantic vector group corresponding to each sample text, the semantic vector group includes at least one semantic vector, and each semantic vector corresponds to a different semantic combination; The matching module is used to match the matching image set corresponding to the sample semantic vector set in a preset image-text library according to the sample semantic vector set, wherein the matching image set contains target matching images corresponding to each semantic vector; The training module is used to obtain a text-image generation model by training the initial text-image generation model based on the sample semantic vector set, the matching image set, and the initial sample text set.

11. A text-image generating device, characterized in that: include: Acquisition module, processing module, determination module; The acquisition module is used to acquire the text to be processed; The processing module is configured to input the text to be processed into the trained text-image generation model, perform semantic structuring processing on the text to be processed based on the structured representation model, and obtain at least one semantic vector corresponding to the text to be processed; The determination module is configured to determine, based on the text to be processed, a target semantic vector corresponding to the text to be processed from each semantic vector; The determination module is used to determine a target matching image corresponding to the text to be processed based on a target semantic vector corresponding to the text to be processed, and output the target matching image.

12. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the program instructions to perform any one of the methods described in claims 1 to 9.

13. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which is used to execute the method according to any one of claims 1 to 9 when executed by a processor.

Images