Image generation method and device, text graph network architecture and electronic equipment

Abstract

The invention discloses an image generation method and device, a text graph network architecture and electronic equipment, and belongs to the technical field of artificial intelligence. The text graph network architecture comprises a text image feature extraction module used for generating a text image feature vector based on text graph requirement information; the text coding module is used for converting the text map requirement information into a text description feature vector; the image coding module is used for generating an image feature matrix based on the text map requirement information; the noise prediction module is used for obtaining a predicted image feature matrix based on the character image feature vector, the text description feature vector and the image feature matrix; and the image decoding module is used for generating an image matched with the text map requirement information based on the predicted image feature matrix.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image generation method, device, text-based image network architecture and electronic equipment. Background Art

[0002] With the development of technology, the text-based graph algorithm has been widely used in the fields of poster and advertising design, effectively improving the efficiency of image creation and design. However, since the conventional text-based graph model does not have the ability to generate text, the text content of image solutions such as posters and advertising design still requires manual participation, such as adding corresponding text content to the image through manual editing, resulting in low efficiency of text-based graphs. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image generation method, device, Vincent graph network architecture and electronic device, which can effectively improve the generation efficiency of Vincent graph images and reduce the time cost of image generation.

[0004] In a first aspect, an embodiment of the present application provides a text graph network architecture, including:

[0005] A text image feature extraction module is used to generate a text image feature vector based on text image requirement information;

[0006] A text encoding module, used for converting the text image requirement information into a text description feature vector;

[0007] An image encoding module, used for generating an image feature matrix based on the required information of the text image;

[0008] A noise prediction module, used for obtaining a predicted image feature matrix based on the text image feature vector, the text description feature vector and the image feature matrix;

[0009] The image decoding module is used to generate an image matching the required information of the Vincent graph based on the predicted image feature matrix.

[0010] In a second aspect, an embodiment of the present application provides an image generation method, including:

[0011] Generate text image feature vector and image feature matrix based on text image requirement information;

[0012] Converting the text graph requirement information into a text description feature vector;

[0013] Based on the text image feature vector, the text description feature vector and the image feature matrix, a predicted image feature matrix is ​​obtained;

[0014] Deconvolution processing is performed on the predicted image feature matrix to generate an image that matches the required information of the Vincent map.

[0015] In a third aspect, an embodiment of the present application provides an image generating device, including:

[0016] A generation module, used for generating a text image feature vector and an image feature matrix based on text image requirement information;

[0017] A conversion module, used for converting the text image requirement information into a text description feature vector;

[0018] A processing module, configured to obtain a predicted image feature matrix based on the text image feature vector, the text description feature vector and the image feature matrix;

[0019] The generation module is further used to perform deconvolution processing on the predicted image feature matrix to generate an image that matches the required information of the Vincent map.

[0020] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the second aspect are implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the second aspect are implemented.

[0022] In a sixth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the second aspect.

[0023] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the method described in the second aspect.

[0024] In an embodiment of the present application, by obtaining the text image feature vector, text description feature vector and image feature matrix of the required information of the vincent diagram, and interactively calculating the text image feature vector and the text description feature vector with the image feature matrix to realize noise prediction and obtain the predicted image feature matrix, the image generation process using the text description and the text image as control conditions is realized, and the automatic generation of images with text content is realized. Compared with the conventional vincent diagram generation scheme that requires manual operation to add text content to the image to generate an image with text content, the generation efficiency of the vincent diagram image can be effectively improved, and the time cost of image generation can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a structural diagram of a Wensheng graph network architecture provided by some embodiments of the present application;

[0026] Figure 2 is a flowchart of an image generation method provided by some embodiments of the present application;

[0027] Figure 3a is a schematic diagram of the operation of a Vincent diagram provided in some embodiments of the present application;

[0028] Figure 3b is a schematic diagram of the operation of a Vincent diagram provided in other embodiments of the present application;

[0029] Figure 3c is a schematic diagram of the operation of a Vincent diagram provided in other embodiments of the present application;

[0030] Figure 3d is a schematic diagram of the operation of a Vincent diagram provided in other embodiments of the present application;

[0031] Figure 4 is a structural diagram of an image generating device provided by some embodiments of the present application;

[0032] Figure 5 is a structural diagram of an electronic device provided by some embodiments of the present application;

[0033] Figure 6 It is a structural diagram of an electronic device provided in some other embodiments of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0036] In order to enable those skilled in the art to better understand the embodiments of the present application, some terms involved in the present application are first explained as follows.

[0037] The text-based graph network architecture can be understood as a neural network model of a text-based graph, which can output an image corresponding to the text content based on the input text content.

[0038] An image encoder can be understood as an image encoding network used to convert a real image into an image feature matrix.

[0039] The image decoder can be understood as an image encoding network used to convert the image feature matrix into a real image.

[0040] The transformer module can be understood as a deep learning model based on the attention mechanism, which can include self-attention operator, cross-attention operator, vector addition operator, normalization operator and forward feedback operator.

[0041] Among them, the above-mentioned attention mechanism includes three input vectors, which are expressed as Query, Key and Value respectively; moreover, in the calculation process of the attention mechanism based on the self-attention operator, the above-mentioned three input vectors are from the same input mapping; in the calculation process of the attention mechanism based on the cross-attention operator, Query is an independent input mapping, and Key and Value come from the same input mapping.

[0042] Among them, Query input can be understood as the query vector input of the attention mechanism, Key input can be understood as the key vector input of the attention mechanism, and Value input can be understood as the value vector input of the attention mechanism.

[0043] The convolution module can be understood as a module that performs convolution calculations on images based on the convolution operator, which includes parameters such as the convolution kernel, step size, and number of input channels.

[0044] The deconvolution module can be understood as a module that performs deconvolution calculations on image features based on the deconvolution operator, which includes parameters such as the convolution kernel, step size, and number of input channels.

[0045] The text-based graph network architecture provided in the embodiment of the present application can be applied to text-based graph scenarios in which corresponding images are generated based on text content; one specific application scenario is that text information input by a user can be input into the text-based graph network architecture so as to output an image with text content; another specific application scenario is that based on the voice information input by the user, the voice information can be first converted into text information, and then the converted text information can be input into the text-based graph network architecture so as to output an image with text content.

[0046] like Figure 1 As shown, the text-image network architecture provided in the embodiment of the present application includes: a text image feature extraction module 10, a text encoding module 20, an image encoding module 30, a noise prediction module 40 and an image decoding module 50, wherein:

[0047] The text image feature extraction module 10 is used to generate a text image feature vector based on the text image requirement information; wherein the text image requirement information can be understood as text information used to generate an image with text content;

[0048] A text encoding module 20, for converting text image requirement information into a text description feature vector;

[0049] An image encoding module 30 is used to generate an image feature matrix based on the Wensheng map requirement information;

[0050] A noise prediction module 40, for obtaining a predicted image feature matrix based on the text image feature vector, the text description feature vector and the image feature matrix;

[0051] The image decoding module 50 is used to generate an image matching the required information of the Vincent graph based on the predicted image feature matrix.

[0052] In some embodiments of the present application, text description information describing text features of the text to be displayed in the text image requirement information can be obtained, and then the text image of the text to be displayed is determined based on the text description information, and finally the text image feature vector corresponding to the text image is extracted based on the text image feature extraction module 10.

[0053] For example, when the text image requirement information is "a cartoon photo of a newlywed couple, with "Happy Wedding" written on the top in Songti font and black font color", the text description information describing the text features of the text to be displayed in the text image requirement information can be extracted, that is, the text description information "a. Text content: Happy Wedding; b. Font: Songti; c. Font color: Black" can be extracted.

[0054] For another example, when the Wensheng picture requirement information is "beautiful early autumn, the Chinese characters "Golden October" are written in the middle of the picture, the font is running script, the font color is white, the eaves of an ancient temple, ginkgo leaves are fallen, and there are posters and wallpapers", the text description information describing the text features of the text to be displayed in the Wensheng picture requirement information can be extracted, that is, the text description information "a. Text content: Golden October; b. Font: running script; c. Font color: white" can be extracted. Among them, the Wensheng picture requirement information "beautiful early autumn, the Chinese characters "Golden October" are written in the middle of the picture, the font is running script, the font color is white, the eaves of an ancient temple, ginkgo leaves are fallen, and there are posters and wallpapers" can be converted based on the voice information input by the user.

[0055] The extracted text description information can be used as input to a text image construction module to construct corresponding text images for each text.

[0056] In some embodiments of the present application, a text image of the word "new" can be constructed based on the following text dictionary information, and the text dictionary information specifically includes: {"text content":"new","text font":"Songti","font color":"yellow"}.

[0057] Correspondingly, the text images of "marriage", "happy" and "joy" can also be obtained through the same construction method.

[0058] The size of each text image can be set to 224*224 resolution, with 3 channels, where 3 represents red, green, and blue. That is, the obtained text image can be represented by a 224*224*3-dimensional matrix vector.

[0059] It is understandable that the resolution of the text image includes but is not limited to 224*224, and is not specifically limited here.

[0060] In some embodiments of the present application, self-attention calculation can be performed on the matrix vector used to represent the text image, so that the obtained text image feature vector can more accurately express the text description information describing the text features of the text to be displayed in the text image requirement information.

[0061] In some embodiments of the present application, the text encoding module 20 can be understood as a text encoder, which can convert text content such as document image requirement information into a text description feature vector.

[0062] In some embodiments of the present application, the text-graph requirement information may be first passed through a Tokenizer module, which may convert text into a numerical vector, such as converting the text-graph requirement information into a vector [128, 1024], where 128 represents the maximum number of character tokens and 1024 is the representation vector of each character;

[0063] Then, the [128, 1024]-dimensional numerical vector is input into the self-attention module to perform self-attention calculation to obtain the feature vector after self-attention calculation, and the feature vector after self-attention calculation is recorded as the text description feature vector, thereby realizing the conversion acquisition of the text description feature vector.

[0064] In some embodiments of the present application, a training image of the Vincent image requirement information can be determined based on the Vincent image requirement information, and then the training image is input into the image encoding module 30 to encode the training image into an image feature matrix to achieve the generation and acquisition of the image feature matrix of the Vincent image requirement information.

[0065] In some embodiments of the present application, for the predicted image feature matrix obtained based on the text image feature vector, the text description feature vector and the image feature matrix, the predicted image feature matrix can be decoded to generate an image that matches the required information of the text image.

[0066] Among them, the image encoding module 30 can be understood as the above-mentioned image encoder, which is used to convert the real image into an image feature matrix; the image decoding module 50 can be understood as the above-mentioned image decoder, which is used to convert the image feature matrix into a real image, that is, the encoding process performed by the image encoding module 30 and the decoding process performed by the image decoding module 50 can be understood as each other's inverse processes to achieve the conversion between the real image and the image feature matrix.

[0067] For example, a real image can be represented as a matrix of [1024,1024,3], and its corresponding image feature matrix is ​​a matrix of [128,128,16]. By calculating the number of elements in these two matrices, we can know that the image feature vector matrix is ​​smaller than the real image matrix. Therefore, to realize text-controlled image generation, we only need to perform it in the image feature space, treat the image feature vector as a real image, and then generate the image feature vector through the text control. In this way, the image feature vector can be converted into a real image through the image decoding network, which reduces the amount of calculation.

[0068] In some embodiments of the present application, the noise prediction module 40 can interactively calculate the text image feature vector and the text description feature vector with the image feature matrix to realize noise prediction and obtain the predicted image feature matrix, thereby realizing the image generation process with the text description and the text image as control conditions, so that the text content in the image generated based on the predicted image feature matrix and matching the required information of the vignette is consistent with the above-mentioned text description information, which can effectively improve the generation efficiency of the vignette image and reduce the time cost of image generation.

[0069] It should be noted that the Wensheng graph network architecture provided in the embodiment of the present application may also include a self-attention module, a cross-attention module, a noise module, a Wensheng graph model, etc.

[0070] Among them, the self-attention module can perform self-attention calculation on the feature vector so that the feature vector can more accurately express the required information of the text image; the cross-attention module can perform cross-attention calculation on the feature vector to realize the image generation process with text description and text image as control conditions; the noise module can perform noise processing on the image feature matrix so that the predicted image feature matrix obtained based on the image feature matrix after noise processing can more accurately express the required information of the text image; the text image model is used to generate corresponding training images based on the required information of the text image, so as to generate the image feature matrix of the required information of the text image based on the training image.

[0071] In some embodiments of the present application, the loss function Loss can be constructed using the random initial noise of the noise module. By training the loss function, the network weights of the text image feature extraction module 10, the text encoding module 20 and the image encoding module 30 in the Wensheng graph network architecture can be reversely adjusted so that the predicted image noise matrix output by the noise prediction module 40 is close to the random initial noise, so that the predicted image feature matrix obtained based on the image feature matrix after noise processing can more accurately express the required information of the Wensheng graph.

[0072] See also Figure 2 , Figure 2 is a flowchart of an image generation method provided by some embodiments of the present application. The image generation method provided by the embodiments of the present application can be based on Figure 1 The Wensheng graph network architecture shown is implemented, or applied to an electronic device including the Wensheng graph network architecture, such as Figure 2 As shown, the image generation method includes:

[0073] Step 201: Generate a text image feature vector and an image feature matrix based on text image requirement information.

[0074] In this step, the text image feature vector and the image feature matrix can be generated based on the text image requirement information.

[0075] In some embodiments of the present application, a text image of the text to be displayed may be determined based on text description information describing text features of the text to be displayed in the text image requirement information, and then a corresponding text image feature vector may be generated based on the text image of the text to be displayed.

[0076] In some embodiments of the present application, the Vincent diagram requirement information can be input into a pre-trained Vincent diagram training model to output a training image corresponding to the Vincent diagram requirement information; then the training image is encoded to obtain an image feature matrix of the Vincent diagram requirement information.

[0077] In some embodiments of the present application, a series of convolution processes may be performed on the training image to convert it into an image feature matrix that is compatible with the text image feature vector. For example, the training image may be converted into an image feature matrix that is compatible with the text image feature vector by the above-mentioned image encoding module 30.

[0078] In some embodiments of the present application, the calculation process of the latent, convolution modules (ConvBlock) 1, ConvBlock2 and ConvBlock3 in the image encoding module 30 is the same, except that the step size of the convolution kernel is different. Here, ConvBlock1 is used as an example to explain the calculation process:

[0079] Step 1: The training image [1024, 1024, 3] is first passed through the convolution layer Conv1_1 to calculate the [512, 512, 4] feature matrix. The convolution kernel size is 3x3, the step size is 2, and the number of output channels is 4;

[0080] Step 2: The feature matrix of [512,512,4] dimensions is processed by the activation function (Relu) 1_1 layer and is still a feature matrix of [512,512,4] dimensions;

[0081] Step3: Conv1_2 and Relu1_2 repeat the process of Step1 and Step2, except that the convolution step size of Conv1_2 is changed to 1, which ensures that the dimension of the output feature matrix remains [512, 512, 4].

[0082] Step 202: Convert the text-image requirement information into a text description feature vector.

[0083] In this step, the generation process of the text description feature vector is the same or similar to the conversion process of the text encoding module 20 in the above-mentioned text-generating graph network architecture, and will not be repeated here.

[0084] Step 203: Obtain a predicted image feature matrix based on the text image feature vector, the text description feature vector and the image feature matrix.

[0085] In this step, the text image feature vector and the text description feature vector can be interactively calculated with the image feature matrix to realize noise prediction and obtain the predicted image feature matrix, thereby realizing the image generation process with the text description and the text image as control conditions, so that the text content in the image generated based on the predicted image feature matrix and matching the required information of the vizine graph is consistent with the above text description information, which can effectively improve the generation efficiency of the vizine graph image and reduce the time cost of image generation.

[0086] Step 204: Deconvolution is performed on the predicted image feature matrix to generate an image that matches the required information of the Vincent image.

[0087] In this step, the predicted image feature matrix may be subjected to deconvolution processing to decode and obtain an image that matches the required information of the Vincent graph.

[0088] Specifically, for the predicted image feature matrix with a dimension of 128*128*16, a series of deconvolution processes can be adopted to generate a picture of 1024*1024*3. For example, the predicted image feature matrix can be decoded into an image feature matrix adapted to the text image feature vector by the above-mentioned image decoding module 50.

[0089] In some embodiments of the present application, the calculation process of the deconvolution modules (DeconvBlock) 1, DeconvBlock2 and DeconvBlock3 in the image decoding module 50 is the same, except that the deconvolution kernel, step size and number of channels are different. Here we take DeconvBlock2 as an example to explain the calculation process:

[0090] Step 1: The image 128*128*16 output by DeconvBlock1 is calculated through the deconvolution layer Deconv2_1 to obtain a 256*256*8 feature matrix. The deconvolution kernel size is 3x3, the step size is 2, and the number of output channels is 8;

[0091] Step 2: The 256*256*8-dimensional feature matrix is ​​still a 256*256*8-dimensional feature matrix after being processed by the Relu2_1 layer;

[0092] Step 3: Conv2_2, Relu2_2, Conv2_3, and Relu2_3 are the usual convolution and Relu operations. The dimension of the final output feature matrix of DeconvBlock2 is 256*256*8.

[0093] In this embodiment, by obtaining the text image feature vector, text description feature vector and image feature matrix of the required information of the text image, and interactively calculating the text image feature vector and the text description feature vector with the image feature matrix to realize noise prediction and obtain the predicted image feature matrix, the image generation process with the text description and the text image as control conditions is realized, and the automatic generation of images with text content is realized. Compared with the conventional text image generation scheme that requires manual operation to add text content to the image to generate an image with text content, the generation efficiency of the text image can be effectively improved, and the time cost of image generation can be reduced.

[0094] In some embodiments of the present application, obtaining a predicted image feature matrix based on the text image feature vector, the text description feature vector and the image feature matrix includes:

[0095] Based on a preset noise matrix, performing noise processing on the image feature matrix;

[0096] Perform matrix transformation on the image feature matrix after noise processing to obtain the image feature vector;

[0097] The image feature vector is used as a query vector input of a first cross-attention module of a text-graph network architecture, and the text image feature vector is used as a key vector input and a value vector input of the first cross-attention module, respectively, and input into the first cross-attention module to output a first predicted noise vector;

[0098] Using the image feature vector as a query vector input of a second cross-attention module of the text graph network architecture, and using the text description feature vector as a key vector input and a value vector input of the second cross-attention module, respectively inputting the second cross-attention module, and outputting a second predicted noise vector;

[0099] A predicted image feature matrix is ​​obtained based on the first predicted noise vector and the second predicted noise vector.

[0100] In some embodiments of the present application, cross-attention calculations can be performed on image feature vectors and text image feature vectors based on a first cross-attention module, and cross-attention calculations can be performed on image feature vectors and text description feature vectors based on a second cross-attention module to achieve noise prediction and obtain a predicted image feature matrix, thereby realizing an image generation process that uses text descriptions and text images as control conditions, so that the text content in an image generated based on the predicted image feature matrix and matching the required information of the Vincent diagram is consistent with the above-mentioned text description information, which can effectively improve the generation efficiency of the Vincent diagram image and reduce the time cost of image generation.

[0101] Moreover, by performing noise addition processing on the image feature matrix based on a preset noise matrix, the interference effect of other noises on the predicted image feature matrix can be reduced.

[0102] The above-mentioned preset noise matrix is ​​random noise, such as a randomly initialized Gaussian noise matrix.

[0103] Among them, the first cross-attention module and the second cross-attention module can be the same cross-attention module in the Wensheng graph network architecture, or can be different cross-attention modules in the Wensheng graph network architecture.

[0104] Among them, the first cross-attention module and the second cross-attention module can be understood as the cross-attention operator of the transformer module, that is, the calculation output of the first prediction noise vector can be realized based on the cross-attention operator of the transformer module, and the calculation output of the second prediction noise vector can be realized.

[0105] In some embodiments of the present application, obtaining a predicted image feature matrix based on the first predicted noise vector and the second predicted noise vector includes:

[0106] Obtaining a weighted average noise vector based on the first prediction noise vector and the second prediction noise vector;

[0107] Performing matrix conversion processing on the weighted average noise vector to obtain a predicted image noise matrix;

[0108] The predicted image noise matrix is ​​subjected to noise reduction processing by using the preset noise matrix to obtain a predicted image feature matrix.

[0109] In this embodiment, by performing weighted calculation on the first prediction noise vector and the second prediction noise vector to obtain a weighted average noise vector, and performing noise reduction processing on the prediction image noise matrix, the interference effect of other noises on the prediction image feature matrix can be reduced.

[0110] In some embodiments of the present application, obtaining a weighted average noise vector based on the first prediction noise vector and the second prediction noise vector includes:

[0111] Averaging the first prediction noise vector and the second prediction noise vector to obtain a third prediction noise vector;

[0112] The third predicted noise vector is used as the query vector input, key vector input and value vector input of the first self-attention module of the Wensheng graph network architecture, input into the first self-attention module, and outputs a weighted average noise vector.

[0113] In this embodiment, by performing self-attention calculation on the third predicted noise vector obtained after the averaging process, the predicted image feature matrix generated based on the weighted average noise vector can more accurately express the required information of the Vincent graph.

[0114] Among them, the first self-attention module can be understood as the self-attention operator of the transformer module, that is, the calculation output of the weighted average noise vector can be realized based on the self-attention operator of the transformer module.

[0115] In some embodiments of the present application, a predicted image feature matrix can be generated by the noise prediction module 40. In this process, random noise can be added to the image feature matrix, and then the random noise can be predicted through the network, so as to achieve the destruction and reconstruction of the image features, and in the process of image feature reconstruction, the text image feature vector and the text description feature vector can be interactively calculated with the image feature matrix, so as to achieve the prediction of noise, and then achieve the image generation process with the text description and the text image as the control conditions.

[0116] The above-mentioned image feature destruction process can be understood as the image feature noise addition process; and the above-mentioned image feature reconstruction process can be understood as the image feature denoising process.

[0117] In some embodiments of the present application, the text image feature vector can be represented as a 4*1024 two-dimensional feature vector, the text description feature vector can be represented as a 128*1024 two-dimensional feature vector, and the image feature matrix can be represented as a [128, 128, 16] feature matrix; and the noise prediction module 40 can implement the following process:

[0118] (1) The image feature matrix can be subjected to noise processing based on a randomly initialized Gaussian noise matrix to obtain a noise-processed image feature matrix, and the noise-processed image feature matrix can be expressed as a 128*128*16 feature matrix;

[0119] (2) Performing matrix transformation processing on the image feature matrix after the noise processing, such as performing a reshape operation, to obtain the image feature vector after the noise processing, and the image feature vector after the noise processing can be represented as a two-dimensional feature vector of 256*1024.

[0120] (3) A cross-attention module based on the text-graph network architecture can be used, and the image feature vector after noise processing is used as the query vector input of the cross-attention module, and the text image feature vector is used as the key vector input and value vector input of the cross-attention module, respectively input into the cross-attention module, perform cross-attention mechanism calculation, and output a first predicted noise vector, and the first predicted noise vector can be represented as a two-dimensional feature vector of 256*1024;

[0121] (4) A cross-attention module based on the text-graph network architecture can be used, and the image feature vector after noise processing is used as the query vector input of the cross-attention module, and the text description feature vector is used as the key vector input and value vector input of the cross-attention module, respectively input into the cross-attention module, perform cross-attention mechanism calculation, and output a second predicted noise vector, and the second predicted noise vector can be represented as a two-dimensional feature vector of 256*1024;

[0122] (5) The first prediction noise vector and the second prediction noise vector may be averaged to obtain a third prediction noise vector, and the third prediction noise vector may be represented as a two-dimensional feature vector of 256*1024;

[0123] (6) A self-attention module based on the Wensheng graph network architecture can be used, and the third predicted noise vector is used as the query vector input, key vector input and value vector input of the self-attention module, input into the self-attention module, perform self-attention calculation, and output a fourth predicted noise vector, and the fourth predicted noise vector can be represented as a two-dimensional feature vector of 256*1024;

[0124] (7) A matrix conversion operation, such as a reshape operation, may be performed on the fourth prediction noise vector to convert the two-dimensional fourth prediction noise vector into a prediction image noise matrix, and the prediction image noise matrix may be represented as a 128*128*16 noise matrix;

[0125] (8) The randomly initialized Gaussian noise matrix can be used to perform noise reduction processing on the predicted image noise matrix to obtain a predicted image feature matrix, and the predicted image feature matrix can be represented as a 128*128*16 feature matrix.

[0126] In some embodiments of the present application, generating a text image feature vector based on text image requirement information includes:

[0127] Obtaining text description information describing text features of the text to be displayed in the text image requirement information;

[0128] Inputting the text description information into a text image construction module, and outputting a text image matching the text description information;

[0129] Extracting text image features from the text image to obtain a text image feature vector.

[0130] In this embodiment, the text image matching the text description information can be output based on the text image construction module, and then the text image features in the text image can be extracted to obtain the text image feature vector.

[0131] In some embodiments of the present application, extracting the text image features in the text image to obtain the text image feature vector includes:

[0132] Performing convolution processing on the text image to obtain an initial feature vector;

[0133] Performing matrix conversion processing on the initial feature vector to obtain N visual feature vectors, where N is an integer greater than 1;

[0134] Using each of the N visual feature vectors as a query vector input, a key vector input, and a value vector input of a second self-attention module of the Vincent graph network architecture, inputting the second self-attention module, and outputting N self-attention noise vectors;

[0135] Averaging the N self-attention noise vectors to obtain an average feature vector;

[0136] The average feature vector is flattened to obtain a text image feature vector.

[0137] In this embodiment, by using the visual feature vector of the text image as the query vector input, key vector input and value vector input of the second self-attention module of the text graph network architecture, and inputting it into the second self-attention module for self-attention calculation, the final text image feature vector can more accurately express the text image.

[0138] Among them, the second self-attention module can be understood as the self-attention operator of the transformer module, that is, the calculation output of the above-mentioned self-attention noise vector can be realized based on the self-attention operator of the transformer module.

[0139] In some embodiments of the present application, extracting a text image feature vector from a text image based on the text image feature extraction module 10 specifically includes the following steps:

[0140] Step 1: The text image can be convolved based on the convolution module to obtain the initial feature vector. For example, for a convolution module with a convolution kernel of 3x3, a step size of 2, and an output channel dimension of 16, a 224*224*3 text image can be convolved to obtain an initial feature vector represented as 112*112*16.

[0141] Among them, the initial feature vector represented as 112*112*16 is still a feature vector of 112*112*16 after being processed by the ReLU layer.

[0142] Step 2: Reshape the initial feature vector to convert the 112*112*16 initial feature vector into 14*14 8*8*16 dimensional vectors, where 112=14*8.

[0143] Among them, a 8*8*16-dimensional vector can be recorded as a visual token, which can be used as a basic unit in the next transformer module for calculation.

[0144] Step 3: The transformer module can be composed of 10 layers of TransformerBlock stacked together, and the above 14*14 8*8*16-dimensional vectors can be used as query vector input, key vector input and value vector input of the self-attention module, and then participate in the self-attention calculation to obtain 14*14 self-attention noise vectors.

[0145] Step 4: The 14*14 self-attention noise vectors can be averaged to obtain an average feature vector; then the average feature vector is flattened to obtain a 1*1024-dimensional vector, and the text image feature vector can be obtained based on the feature vector corresponding to each text image.

[0146] For the case where the text image requirement information is "Please help me draw a picture, a cartoon photo of a newlywed couple, with "Happy Wedding" written on the top in Songti font and black font color", it is necessary to construct four text images of "new", "wedding", "happy" and "happy", and the text image feature vectors of the above four text images can be obtained respectively by the above method, and the text image feature vectors of the four text images can be expressed as 4*1024, and "4" represents the number of characters to be displayed.

[0147] It can be understood that, for duplicate characters, only the text image feature vector of one of the characters may be obtained.

[0148] It should be noted that the above-mentioned text image feature vector can be understood as the feature vector of a character in the above-mentioned text to be displayed, and can also be understood as the set of feature vectors of all characters in the above-mentioned text to be displayed.

[0149] In some embodiments of the present application, before generating the text image feature vector and the image feature matrix based on the text image requirement information, the method further includes:

[0150] Get the text and image request information through text input or voice input.

[0151] In this embodiment, the text and image requirement information can be obtained by text input or voice input to meet the input requirements of the text and image requirement information in different scenarios.

[0152] In some embodiments of the present application, before generating the text image feature vector and the image feature matrix based on the text image requirement information, the method further includes:

[0153] The training sample is divided into M sub-training samples based on the training parameters, where M is an integer greater than 1;

[0154] Based on the M sub-training samples, iteratively train the initial model to obtain a first text-generated graph model;

[0155] Based on Mi sub-training samples, iteratively train the i-th text graph model to obtain the i+1-th text graph model; when i is 1, the i-th text graph model is the first text graph model; when i is an integer greater than 1 and less than M, the i-th text graph model is a text graph model generated after i-1 iterations of training based on the first text graph model;

[0156] Wherein, in the case where i+1=M, the i+1th Vincent graph model is determined as the Vincent graph training model, and the Vincent graph training model is used to perform image prediction on the Vincent graph requirement information.

[0157] In this embodiment, the training sample is divided into M sub-training samples based on the training parameters, and the i-th text graph model is iteratively trained based on the Mi sub-training samples to obtain the i+1-th text graph model, that is, through stage-by-stage progressive training, the network weights of the text image feature extraction module 10, the text encoding module 20 and the image encoding module 30 in the text graph network architecture are adjusted in the text graph network architecture, thereby improving the accuracy of the text graph training model.

[0158] Among them, the text graph training model can be understood as the above-mentioned text graph network architecture.

[0159] The above training parameters may be image quality, that is, the training sample may be divided into M sub-training samples according to the image quality.

[0160] For example, the images in the training samples are divided into three levels: ultra-high quality, high quality, and qualified quality, to obtain sub-training samples of ultra-high quality images, sub-training samples of high quality images, and sub-training samples of qualified quality images.

[0161] In some embodiments of the present application, a training database may be constructed first, for example, public and open-source images on the Internet may be collected to obtain massive image data.

[0162] For example, for 2 billion image data, we can use open source OCR recognition, image description algorithms and other methods to label the images with text descriptions. The text description style of the images can be shown in Table 1:

[0163] Table 1

[0164] Sample identification image Image Description 1 sodfsf SJSJJS …… …… …… 2000000000 sksid KSDJFI

[0165] Among them, the aesthetic score can be used to score the quality of the 2 billion images. The scoring can be based on a 5-point system, with 3 points indicating acceptable, 4 points indicating high quality, and 5 points indicating super high quality. In this way, we divide the image quality into:

[0166] Dataset A: Ultra-high-quality dataset, assuming a dataset of 100 million image-text pairs;

[0167] Dataset B, high-quality dataset, assuming a dataset of 300 million image-text pairs;

[0168] Dataset C, qualified dataset, assuming a dataset of 600 million image-text pairs;

[0169] Furthermore, a phased progressive training strategy can be adopted to train data with scores above 3:

[0170] Step 1: One-stage training, using all images and texts with scores above 3 points to train the data, including the above three data sets A, B, and C, to produce the weights of the first-stage text-image model;

[0171] Step 2: Two-stage training. The training process is the same as step 1, except that all images and texts with scores above 4 are used for training, including the two datasets A and B mentioned above, to produce the weights of the two-stage text-image model.

[0172] Step 3: Three-stage training. The training process is the same as step 1, except that all the pictures and texts with full scores of 5 points are used for training, including data set A, to produce the final text-image model weights, that is, to obtain the text-image training model.

[0173] In some embodiments of the present application, the above-mentioned loss function Loss can be trained using the above-mentioned data set, and the network weights of the text image feature extraction module 10, the text encoding module 20 and the image encoding module 30 in the Wensheng graph network architecture are reversely adjusted, that is, the network weights of the text image feature extraction module 10, the text encoding module 20 and the image encoding module 30 in the Wensheng graph model are reversely adjusted, and the final Wensheng graph training model is obtained, that is, the network weights of the text image feature extraction module 10, the text encoding module 20 and the image encoding module 30 in the Wensheng graph network architecture after convergence are obtained.

[0174] For example, after the above-mentioned one-stage training, the network weights of the text image feature extraction module 10, the text encoding module 20 and the image encoding module 30 in the first-stage text graph model weight are 0.44, 0.23 and 0.33 respectively; and after the above-mentioned two-stage training, the network weights of the text image feature extraction module 10, the text encoding module 20 and the image encoding module 30 in the second-stage text graph model weight are 0.42, 0.24 and 0.34 respectively; finally, after the above-mentioned three-stage training, the network weights of the text image feature extraction module 10, the text encoding module 20 and the image encoding module 30 in the three-stage text graph model weight are 0.40, 0.25 and 0.35 respectively, that is, the network weights of the text image feature extraction module 10, the text encoding module 20 and the image encoding module 30 in the converged text graph network architecture are 0.40, 0.25 and 0.35 respectively.

[0175] In some embodiments of the present application, the user can Figure 3a In the first dialog box 32 of the artificial intelligence (AI) interface 31 shown in FIG. 1 , the text image requirement information of “a cartoon photo of a newlywed couple, with “Happy Wedding” written on the top in Songti font and black font color” is input, and the text image network architecture can generate the text image requirement information as shown in FIG. 1 based on the text image requirement information input by the user. Figure 3b The first image 33 shown includes not only the photo content of "cartoon photo of newlyweds", but also the text content of "Happy Wedding", thereby realizing the process of generating a text image with text content based on the text content input by the user.

[0176] In some embodiments of the present application, the user can Figure 3c In the second dialog box 34 of the AI ​​interface 31 shown in FIG. 1 , the text image requirement information of “a handsome boy’s head portrait with “Happy Birthday” written on the top, the font is Songti, and the font color is black” is input, and the above text image network architecture can generate the text image requirement information as shown in FIG. Figure 3dThe second image 35 shown includes not only the photo content of "a handsome boy's head portrait", but also the text content of "Happy Birthday", thereby realizing the process of generating an image with text content based on the text content input by the user.

[0177] Among them, the above-mentioned AI interface 31 can be an operating interface of a voice assistant of a terminal device such as a mobile phone.

[0178] The image generation method of the embodiment of the present application generates a text image feature vector and an image feature matrix based on the Vincent diagram requirement information; converts the Vincent diagram requirement information into a text description feature vector; obtains a predicted image feature matrix based on the text image feature vector, the text description feature vector and the image feature matrix; and performs deconvolution processing on the predicted image feature matrix to generate an image that matches the Vincent diagram requirement information. This can effectively improve the generation efficiency of the Vincent diagram image and reduce the time cost of image generation.

[0179] The image generation method provided in the embodiment of the present application can be executed by an image generation device. In the embodiment of the present application, the image generation device provided in the embodiment of the present application is described by taking the image generation method executed by the image generation device as an example.

[0180] See also Figure 4 , Figure 4 is a structural diagram of an image generating device provided by some embodiments of the present application. Figure 4 As shown, the image generating device 400 includes:

[0181] A generating module 401 is used to generate a text image feature vector and an image feature matrix based on text image requirement information;

[0182] A conversion module 402, used to convert the text image requirement information into a text description feature vector;

[0183] A processing module 403 is used to obtain a predicted image feature matrix based on the text image feature vector, the text description feature vector and the image feature matrix;

[0184] The generating module 401 is further used to perform a deconvolution process on the predicted image feature matrix to generate an image matching the required information of the Vincent map.

[0185] Optionally, the processing module 403 includes:

[0186] A processing unit, configured to perform noise addition processing on the image feature matrix based on a preset noise matrix;

[0187] A matrix conversion unit, used for performing matrix conversion processing on the image feature matrix after the noise addition processing to obtain an image feature vector;

[0188] A computing unit, configured to use the image feature vector as a query vector input of a first cross-attention module of a text-graph network architecture, and use the text image feature vector as a key vector input and a value vector input of the first cross-attention module, respectively input the first cross-attention module, and output a first predicted noise vector;

[0189] The computing unit is used to input the image feature vector as a query vector of a second cross-attention module of the text graph network architecture, and input the text description feature vector as a key vector input and a value vector input of the second cross-attention module, respectively, and output a second predicted noise vector;

[0190] The processing unit is used to obtain a predicted image feature matrix based on the first predicted noise vector and the second predicted noise vector.

[0191] Optionally, the processing unit is specifically configured to:

[0192] Obtaining a weighted average noise vector based on the first prediction noise vector and the second prediction noise vector;

[0193] Performing matrix conversion processing on the weighted average noise vector to obtain a predicted image noise matrix;

[0194] The predicted image noise matrix is ​​subjected to noise reduction processing by using the preset noise matrix to obtain a predicted image feature matrix.

[0195] Optionally, the processing unit is specifically configured to:

[0196] Averaging the first prediction noise vector and the second prediction noise vector to obtain a third prediction noise vector;

[0197] The third predicted noise vector is used as the query vector input, key vector input and value vector input of the first self-attention module of the Wensheng graph network architecture, input into the first self-attention module, and outputs a weighted average noise vector.

[0198] Optionally, the generating module 401 includes:

[0199] An acquisition unit, used for acquiring text description information describing text features of the text to be displayed in the text image requirement information;

[0200] An output unit, used to input the text description information into a text image construction module, and output a text image matching the text description information;

[0201] The feature extraction unit is used to extract the text image features in the text image to obtain the text image feature vector.

[0202] Optionally, the feature extraction unit is specifically used to:

[0203] Performing convolution processing on the text image to obtain an initial feature vector;

[0204] Performing matrix conversion processing on the initial feature vector to obtain N visual feature vectors, where N is an integer greater than 1;

[0205] Using each of the N visual feature vectors as a query vector input, a key vector input, and a value vector input of a second self-attention module of the Vincent graph network architecture, inputting the second self-attention module, and outputting N self-attention noise vectors;

[0206] Averaging the N self-attention noise vectors to obtain an average feature vector;

[0207] The average feature vector is flattened to obtain a text image feature vector.

[0208] Optionally, the image generating device 400 further includes:

[0209] The acquisition module is used to acquire the text image requirement information through text input or voice input.

[0210] Optionally, the image generating device 400 further includes:

[0211] A division module, used for dividing the training sample into M sub-training samples based on the training parameters, where M is an integer greater than 1;

[0212] A training module, used for iteratively training the initial model based on the M sub-training samples to obtain a first text graph model;

[0213] The training module is further used to iteratively train the i-th text graph model based on Mi sub-training samples to obtain the i+1-th text graph model; when i is 1, the i-th text graph model is the first text graph model; when i is an integer greater than 1 and less than M, the i-th text graph model is a text graph model generated after i-1 iterations of training based on the first text graph model;

[0214] Wherein, in the case where i+1=M, the i+1th Vincent graph model is determined as the Vincent graph training model, and the Vincent graph training model is used to perform image prediction on the Vincent graph requirement information.

[0215] The image generating device 400 in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0216] The image generation device 400 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0217] The image generation device 400 provided in the embodiment of the present application can achieve Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0218] See also Figure 5 , Figure 5 is a structural diagram of an electronic device provided by some embodiments of the present application. Figure 5 As shown, an embodiment of the present application further provides an electronic device 500, including a processor 501 and a memory 502, wherein the memory 502 stores programs or instructions that can be executed on the processor 501, and when the program or instructions are executed by the processor 501, the various steps of the above-mentioned image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0219] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0220] See also Figure 6 , Figure 6 1 is a structural diagram of an electronic device provided by another embodiment of the present application. Figure 6As shown, the electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610 and other components.

[0221] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0222] The processor 610 is configured to:

[0223] Generate text image feature vector and image feature matrix based on text image requirement information;

[0224] Converting the text graph requirement information into a text description feature vector;

[0225] Based on the text image feature vector, the text description feature vector and the image feature matrix, a predicted image feature matrix is ​​obtained;

[0226] Deconvolution processing is performed on the predicted image feature matrix to generate an image that matches the required information of the Vincent map.

[0227] Optionally, the processor 610 is configured to:

[0228] Based on a preset noise matrix, performing noise processing on the image feature matrix;

[0229] Perform matrix transformation on the image feature matrix after noise processing to obtain the image feature vector;

[0230] The image feature vector is used as a query vector input of a first cross-attention module of a text-graph network architecture, and the text image feature vector is used as a key vector input and a value vector input of the first cross-attention module, respectively, and input into the first cross-attention module to output a first predicted noise vector;

[0231] Using the image feature vector as a query vector input of a second cross-attention module of the text graph network architecture, and using the text description feature vector as a key vector input and a value vector input of the second cross-attention module, respectively inputting the second cross-attention module, and outputting a second predicted noise vector;

[0232] A predicted image feature matrix is ​​obtained based on the first predicted noise vector and the second predicted noise vector.

[0233] Optionally, the processor 610 is configured to:

[0234] Obtaining a weighted average noise vector based on the first prediction noise vector and the second prediction noise vector;

[0235] Performing matrix conversion processing on the weighted average noise vector to obtain a predicted image noise matrix;

[0236] The predicted image noise matrix is ​​subjected to noise reduction processing by using the preset noise matrix to obtain a predicted image feature matrix.

[0237] Optionally, the processor 610 is configured to:

[0238] Averaging the first prediction noise vector and the second prediction noise vector to obtain a third prediction noise vector;

[0239] The third predicted noise vector is used as the query vector input, key vector input and value vector input of the first self-attention module of the Wensheng graph network architecture, input into the first self-attention module, and outputs a weighted average noise vector.

[0240] Optionally, the processor 610 is configured to:

[0241] Obtaining text description information describing text features of the text to be displayed in the text image requirement information;

[0242] Inputting the text description information into a text image construction module, and outputting a text image matching the text description information;

[0243] Extracting text image features from the text image to obtain a text image feature vector.

[0244] Optionally, the processor 610 is configured to:

[0245] Performing convolution processing on the text image to obtain an initial feature vector;

[0246] Performing matrix conversion processing on the initial feature vector to obtain N visual feature vectors, where N is an integer greater than 1;

[0247] Using each of the N visual feature vectors as a query vector input, a key vector input, and a value vector input of a second self-attention module of the Vincent graph network architecture, inputting the second self-attention module, and outputting N self-attention noise vectors;

[0248] Averaging the N self-attention noise vectors to obtain an average feature vector;

[0249] The average feature vector is flattened to obtain a text image feature vector.

[0250] Optionally, the user input unit 607 is used to:

[0251] Get the text and image request information through text input or voice input.

[0252] Optionally, the processor 610 is configured to:

[0253] The training sample is divided into M sub-training samples based on the training parameters, where M is an integer greater than 1;

[0254] Based on the M sub-training samples, iteratively train the initial model to obtain a first text-generated graph model;

[0255] Based on Mi sub-training samples, iteratively train the i-th text graph model to obtain the i+1-th text graph model; when i is 1, the i-th text graph model is the first text graph model; when i is an integer greater than 1 and less than M, the i-th text graph model is a text graph model generated after i-1 iterations of training based on the first text graph model;

[0256] Wherein, in the case where i+1=M, the i+1th Vincent graph model is determined as the Vincent graph training model, and the Vincent graph training model is used to perform image prediction on the Vincent graph requirement information.

[0257] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0258] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0259] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.

[0260] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0261] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0262] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0263] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0264] An embodiment of the present application provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0265] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0266] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0267] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A text graph network architecture, characterized in that: include: A text image feature extraction module is used to generate a text image feature vector based on text image requirement information; A text encoding module, used for converting the text image requirement information into a text description feature vector; An image encoding module, used for generating an image feature matrix based on the required information of the text image; A noise prediction module, used for obtaining a predicted image feature matrix based on the text image feature vector, the text description feature vector and the image feature matrix; The image decoding module is used to generate an image matching the required information of the Vincent graph based on the predicted image feature matrix.

2. An image generation method, characterized in that: include: Generate text image feature vector and image feature matrix based on text image requirement information; Converting the text graph requirement information into a text description feature vector; Based on the text image feature vector, the text description feature vector and the image feature matrix, a predicted image feature matrix is ​​obtained; Deconvolution processing is performed on the predicted image feature matrix to generate an image that matches the required information of the Vincent map.

3. The method according to claim 2, characterized in that The step of obtaining a predicted image feature matrix based on the text image feature vector, the text description feature vector and the image feature matrix includes: Based on a preset noise matrix, performing noise processing on the image feature matrix; Perform matrix transformation on the image feature matrix after noise processing to obtain the image feature vector; The image feature vector is used as a query vector input of a first cross-attention module of a text-graph network architecture, and the text image feature vector is used as a key vector input and a value vector input of the first cross-attention module, respectively, and input into the first cross-attention module to output a first predicted noise vector; Using the image feature vector as a query vector input of a second cross-attention module of the text graph network architecture, and using the text description feature vector as a key vector input and a value vector input of the second cross-attention module, respectively inputting the second cross-attention module, and outputting a second predicted noise vector; A predicted image feature matrix is ​​obtained based on the first predicted noise vector and the second predicted noise vector.

4. The method according to claim 3, characterized in that The step of obtaining a predicted image feature matrix based on the first predicted noise vector and the second predicted noise vector comprises: Obtaining a weighted average noise vector based on the first prediction noise vector and the second prediction noise vector; Performing matrix conversion processing on the weighted average noise vector to obtain a predicted image noise matrix; The predicted image noise matrix is ​​subjected to noise reduction processing by using the preset noise matrix to obtain a predicted image feature matrix.

5. The method according to claim 4, characterized in that The step of obtaining a weighted average noise vector based on the first prediction noise vector and the second prediction noise vector comprises: Averaging the first prediction noise vector and the second prediction noise vector to obtain a third prediction noise vector; The third predicted noise vector is used as the query vector input, key vector input and value vector input of the first self-attention module of the Wensheng graph network architecture, input into the first self-attention module, and outputs a weighted average noise vector.

6. The method according to any one of claims 2 to 5, characterized in that The step of generating a text image feature vector based on text image requirement information includes: Obtaining text description information describing text features of the text to be displayed in the text image requirement information; Inputting the text description information into a text image construction module, and outputting a text image matching the text description information; Extracting text image features from the text image to obtain a text image feature vector.

7. The method according to claim 6, characterized in that The step of extracting the text image features from the text image to obtain the text image feature vector includes: Performing convolution processing on the text image to obtain an initial feature vector; Performing matrix conversion processing on the initial feature vector to obtain N visual feature vectors, where N is an integer greater than 1; Using each of the N visual feature vectors as a query vector input, a key vector input, and a value vector input of a second self-attention module of the Vincent graph network architecture, inputting the second self-attention module, and outputting N self-attention noise vectors; Averaging the N self-attention noise vectors to obtain an average feature vector; The average feature vector is flattened to obtain a text image feature vector.

8. The method according to any one of claims 2 to 5, characterized in that Before generating the text image feature vector and the image feature matrix based on the text image requirement information, the method further includes: Get the text and image request information through text input or voice input.

9. The method according to any one of claims 2 to 5, characterized in that Before generating the text image feature vector and the image feature matrix based on the text image requirement information, the method further includes: The training sample is divided into M sub-training samples based on the training parameters, where M is an integer greater than 1; Based on the M sub-training samples, iteratively train the initial model to obtain a first text-generated graph model; Based on Mi sub-training samples, iteratively train the i-th text graph model to obtain the i+1-th text graph model; when i is 1, the i-th text graph model is the first text graph model; when i is an integer greater than 1 and less than M, the i-th text graph model is a text graph model generated after i-1 iterations of training based on the first text graph model; Wherein, in the case where i+1=M, the i+1th Vincent graph model is determined as the Vincent graph training model, and the Vincent graph training model is used to perform image prediction on the Vincent graph requirement information.

10. An image generating device, characterized in that: include: A generation module, used for generating a text image feature vector and an image feature matrix based on text image requirement information; A conversion module, used for converting the text image requirement information into a text description feature vector; A processing module, configured to obtain a predicted image feature matrix based on the text image feature vector, the text description feature vector and the image feature matrix; The generation module is further used to perform deconvolution processing on the predicted image feature matrix to generate an image that matches the required information of the Vincent map.

11. The device according to claim 10, characterized in that The processing module comprises: A processing unit, configured to perform noise addition processing on the image feature matrix based on a preset noise matrix; A matrix conversion unit, used for performing matrix conversion processing on the image feature matrix after the noise addition processing to obtain an image feature vector; A computing unit, configured to use the image feature vector as a query vector input of a first cross-attention module of a text-graph network architecture, and use the text image feature vector as a key vector input and a value vector input of the first cross-attention module, respectively input the first cross-attention module, and output a first predicted noise vector; The computing unit is used to input the image feature vector as a query vector of a second cross-attention module of the text graph network architecture, and input the text description feature vector as a key vector input and a value vector input of the second cross-attention module, respectively, and output a second predicted noise vector; The processing unit is used to obtain a predicted image feature matrix based on the first predicted noise vector and the second predicted noise vector.

12. The device according to claim 11, characterized in that The processing unit is specifically used for: Obtaining a weighted average noise vector based on the first prediction noise vector and the second prediction noise vector; Performing matrix conversion processing on the weighted average noise vector to obtain a predicted image noise matrix; The predicted image noise matrix is ​​subjected to noise reduction processing by using the preset noise matrix to obtain a predicted image feature matrix.

13. The device according to claim 12, characterized in that The processing unit is specifically used for: Averaging the first prediction noise vector and the second prediction noise vector to obtain a third prediction noise vector; The third predicted noise vector is used as the query vector input, key vector input and value vector input of the first self-attention module of the Wensheng graph network architecture, input into the first self-attention module, and outputs a weighted average noise vector.

14. The device according to any one of claims 10 to 13, characterized in that The generating module comprises: An acquisition unit, used for acquiring text description information describing text features of the text to be displayed in the text image requirement information; An output unit, used to input the text description information into a text image construction module, and output a text image matching the text description information; The feature extraction unit is used to extract the text image features in the text image to obtain the text image feature vector.

15. The device according to claim 14, characterized in that The feature extraction unit is specifically used for: Performing convolution processing on the text image to obtain an initial feature vector; Performing matrix conversion processing on the initial feature vector to obtain N visual feature vectors, where N is an integer greater than 1; Using each of the N visual feature vectors as a query vector input, a key vector input, and a value vector input of a second self-attention module of the Vincent graph network architecture, inputting the second self-attention module, and outputting N self-attention noise vectors; Averaging the N self-attention noise vectors to obtain an average feature vector; The average feature vector is flattened to obtain a text image feature vector.

16. The device according to any one of claims 10 to 13, characterized in that The device also includes: The acquisition module is used to acquire the text image requirement information through text input or voice input.

17. The device according to any one of claims 10 to 13, characterized in that The device also includes: A division module, used for dividing the training sample into M sub-training samples based on the training parameters, where M is an integer greater than 1; A training module, used for iteratively training the initial model based on the M sub-training samples to obtain a first text graph model; The training module is further used to iteratively train the i-th text graph model based on Mi sub-training samples to obtain the i+1-th text graph model; when i is 1, the i-th text graph model is the first text graph model; when i is an integer greater than 1 and less than M, the i-th text graph model is a text graph model generated after i-1 iterations of training based on the first text graph model; Wherein, in the case where i+1=M, the i+1th Vincent graph model is determined as the Vincent graph training model, and the Vincent graph training model is used to perform image prediction on the Vincent graph requirement information.

18. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image generation method as described in any one of claims 2 to 9 are implemented.

Images