Artificial intelligence-based text and graph alternation method and device, terminal equipment and medium

Abstract

The invention is suitable for the technical field of artificial intelligence, and provides an artificial intelligence-based text and graph alternation method and device, terminal equipment and a medium, and the method comprises the steps: carrying out the model pre-training of a mask encoder according to a sample pair, and obtaining the sample features of the sample pair according to the mask encoder after the model pre-training; inputting the sample features into a stream model for feature mapping to obtain mapping features, and determining first loss according to the mapping features; performing parameter updating on the stream model, the mask encoder and the picture decoder according to the first loss until the stream model, the mask encoder and the picture decoder converge; and according to the converged stream model, the mask encoder and the picture decoder execute a text-picture alternation task. Based on joint training of the mask encoder, the stream model and the picture decoder, two training tasks of text-to-picture and picture-to-text can be completed at the same time, only one set of model needs to be maintained, the cost of text-to-picture alternation is effectively reduced, and the text-to-picture alternation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based text-image interaction method, device, terminal device and medium. Background Art

[0002] With the increasing development of artificial intelligence, in some application scenarios, it is necessary to automatically generate pictures that match the text information based on the input text, for example, automatically generate advertising pictures based on the user's description text; in other application scenarios, it is necessary to automatically generate description text that matches the picture based on the picture content, for example, automatically generate corresponding product descriptions based on product pictures. Therefore, the problem of text-image interaction (generating pictures by inputting text or generating text by inputting pictures) is receiving more and more attention.

[0003] In the existing text-image intergeneration process, two independent sets of image generation models and text generation models are required to perform image generation tasks and text generation tasks respectively, resulting in a high cost for text-image intergeneration. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a method, apparatus, terminal device and medium for text-image intergeneration based on artificial intelligence to solve the problem of high text-image intergeneration cost in the prior art.

[0005] A first aspect of an embodiment of the present application provides a method for text-image interaction based on artificial intelligence, including:

[0006] Pre-training a mask encoder model according to the sample pair, and obtaining sample features of the sample pair according to the mask encoder after the model pre-training, wherein the sample pair includes a sample text and a sample image;

[0007] Inputting the sample features into the stream model for feature mapping to obtain mapping features, and determining a first loss according to the mapping features;

[0008] updating parameters of the flow model, the mask encoder, and the picture decoder according to the first loss until the flow model, the mask encoder, and the picture decoder converge;

[0009] If a picture generation request is received, the text to be generated in the picture generation request is input into the converged stream model and the converged picture decoder to generate the picture, so as to obtain a generated picture;

[0010] If a text generation request is received, the image to be generated in the text generation request is input into the converged flow model and the converged mask encoder to perform text generation to obtain generated text.

[0011] Furthermore, the mask encoder model is pre-trained according to the sample pairs, including:

[0012] Performing feature encoding on the sample text and the sample image according to the mask encoder to obtain text features and image features, and determining contrast loss and mask mark loss according to the text features;

[0013] Inputting the image features into the picture decoder to decode the picture to obtain a decoded picture, and determining a mean square error loss based on the decoded picture and the sample picture;

[0014] A second loss is determined according to the mean square error loss, the contrast loss and the mask label loss, and parameters of the mask encoder are updated according to the second loss.

[0015] Furthermore, feature encoding is performed on the sample text and the sample image according to the mask encoder to obtain text features and image features, including:

[0016] According to the mask encoder, the sample text and the sample image are respectively segmented and divided to obtain text segmentation and segmentation images;

[0017] Randomly masking the text segmentation, and performing vector encoding on the randomly masked text segmentation to obtain a text vector;

[0018] Feature alignment is performed on the text vector and the segmented image to obtain the text feature and the image feature.

[0019] Further, determining the contrast loss and the mask mark loss according to the text feature includes:

[0020] Input the text features into a pre-trained multi-layer perceptron to perform similarity calculation to obtain feature similarity, and perform loss calculation on the feature similarity according to an information noise contrast estimation loss function to obtain the contrast loss;

[0021] The mask prediction position in the text feature is obtained, and the loss of the mask prediction position is calculated according to the cross entropy loss function to obtain the mask marking loss.

[0022] Furthermore, the sample features are input into the stream model for feature mapping to obtain mapping features, including:

[0023] Acquire a feature segmentation point in the sample feature, and perform feature segmentation on the sample feature according to the feature segmentation point to obtain a first segmentation feature and a second segmentation feature, wherein the feature segmentation point represents a segmentation position between a text vector and an image vector in the sample feature;

[0024] According to the first fully-connected module and the second fully-connected module, the first segmentation feature is fully-connected to obtain a first fully-connected feature and a second fully-connected feature;

[0025] Performing a dot product calculation on the second segmentation feature and the first fully connected feature to obtain a dot product feature, and fusing the dot product feature with the second fully connected feature to obtain a fused feature;

[0026] The first segmentation feature and the fusion feature are concatenated to obtain the mapping feature.

[0027] Further, determining a first loss according to the mapping feature includes:

[0028] Performing vector matching on the mapping feature and a preset Gaussian distribution image to obtain a vector dimension value, and performing probability distribution calculation on the vector dimension value according to a probability density function to obtain a probability density value;

[0029] The difference between the preset value and the probability density value is calculated to obtain a loss value, and the quotient between the loss value and the total number of the vector dimension values ​​is calculated to obtain the first loss.

[0030] Further, inputting the text to be generated in the image generation request into the converged stream model and the converged image decoder to generate an image, to obtain a generated image, includes:

[0031] Sampling a preset Gaussian distribution image to obtain a first sampling vector, and inputting the to-be-generated text and the first sampling vector into the converged stream model for feature mapping to obtain a first target mapping feature;

[0032] The first target mapping feature is input into the converged picture decoder to generate a picture, so as to obtain the generated picture.

[0033] A second aspect of an embodiment of the present application provides a text-image intergeneration device, including:

[0034] A pre-training unit, configured to perform model pre-training on a mask encoder according to a sample pair, and obtain sample features of the sample pair according to the mask encoder after model pre-training, wherein the sample pair includes a sample text and a sample image;

[0035] A feature mapping unit, configured to input the sample feature into a stream model for feature mapping to obtain a mapping feature, and determine a first loss according to the mapping feature;

[0036] a convergence unit, configured to update parameters of the flow model, the mask encoder, and the picture decoder according to the first loss until the flow model, the mask encoder, and the picture decoder converge;

[0037] A picture generating unit, configured to, upon receiving a picture generating request, input the text to be generated in the picture generating request into the converged stream model and the converged picture decoder to generate a picture, so as to obtain a generated picture;

[0038] The text generation unit is used for, if a text generation request is received, inputting the image to be generated in the text generation request into the converged flow model and the converged mask encoder to perform text generation to obtain generated text.

[0039] A third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements each step of the artificial intelligence-based text-image interaction method provided in the first scheme when executing the computer program.

[0040] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the artificial intelligence-based text-image interaction method provided by the first scheme.

[0041] The embodiments of the present application provide an artificial intelligence-based text-image interaction method, apparatus, terminal device and medium. By pre-training the mask encoder model through sample pairs, the text and image can be effectively co-encoded and mapped to a unified representation space, so that the feature expressions of the image and the text are consistent. Based on the joint training of the mask encoder, the flow model and the image decoder, the two training tasks of text-to-image and image-to-text can be completed simultaneously. Only one set of models needs to be maintained, which effectively reduces the cost of text-to-image interaction and improves the efficiency of text-to-image interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0044] Figure 2 is a flow chart of an embodiment of the text-image intergeneration method based on artificial intelligence according to the present application;

[0045] Figure 3 yes Figure 2 Schematic diagram of sample feature acquisition in the embodiment;

[0046] Figure 4 yes Figure 2 Schematic diagram of mapping feature generation in an embodiment;

[0047] Figure 5 is a structural schematic diagram of an embodiment of a text-image intergeneration device according to the present application;

[0048] Figure 6 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0050] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0051] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0052] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0053] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0054] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0055] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0056] It should be noted that the artificial intelligence-based text-image interaction method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the text-image interaction device is generally set in the server / terminal device.

[0057] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0058] Continue to refer Figure 2 , showing a flowchart of an embodiment of the artificial intelligence-based text-image intergeneration method according to the present application. According to different needs, the order of the steps in the flowchart can be changed, and some steps can be omitted. The artificial intelligence-based text-image intergeneration method provided in the embodiment of the present application can be applied to any scenario of generating text based on pictures, or generating pictures based on text, then the artificial intelligence-based text-image intergeneration method can be applied to products in these scenarios, for example, in medical scenarios, the text description of medical images is automatically generated based on medical image pictures; for example, in financial scenarios, insurance product posters are automatically generated based on insurance product descriptions. The artificial intelligence-based text-image intergeneration method comprises the following steps:

[0059] Step S201, pre-training a mask encoder model according to the sample pair, and obtaining sample features of the sample pair according to the mask encoder after the model pre-training;

[0060] In this embodiment, the electronic device (eg Figure 1 The server / terminal device shown in the figure) can receive the text generation request or the image generation request through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0061] Specifically, the sample pair includes a sample text and a sample image, and the sample pair is composed of <sample text, sample image>, where the sample text is used to describe the text of the corresponding image. In this step, the mask encoder is pre-trained using the data set of <sample text, sample image>, and the mask encoder can be a masked multimodal encoder. The masked multimodal encoder (MME) is an encoder structure that performs feature learning and representation through a mask strategy under multimodal input. Through the MME after model pre-training, text and images can be mapped to the same representation space.

[0062] Optionally, pre-train the mask encoder model based on sample pairs, including:

[0063] Performing feature encoding on the sample text and the sample image according to the mask encoder to obtain text features and image features, and determining contrast loss and mask mark loss according to the text features;

[0064] Among them, by using a mask encoder to perform feature encoding on sample text and sample images, the text features of the sample text and the image features of the sample images in the same representation space can be effectively obtained, and the contrast loss (Contrastive Loss) and masked token loss (Masked Token Loss) are calculated based on the text features.

[0065] Inputting the image features into a picture decoder to decode the picture to obtain a decoded picture, and determining a mean square error loss based on the decoded picture and the sample picture;

[0066] Among them, by inputting image features into the image decoder for image decoding, the image features can be effectively restored to the decoded image, and the mean square error loss (Mean-Square Error) is obtained by calculating the difference between the decoded image and the sample image.

[0067] A second loss is determined according to the mean square error loss, the contrast loss and the mask mark loss, and parameters of the mask encoder are updated according to the second loss until the mask encoder satisfies a preset convergence condition, which can be set according to demand; wherein, based on the mean square error loss, the contrast loss and the mask mark loss, the loss of the mask encoder during model pre-training, that is, the second loss, can be effectively determined.

[0068] Furthermore, feature encoding is performed on the sample text and the sample image according to the mask encoder to obtain text features and image features, including:

[0069] According to the mask encoder, the sample text and the sample image are respectively segmented and divided to obtain text segmentation and segmentation images;

[0070] Randomly masking the text segmentation, and performing vector encoding on the randomly masked text segmentation to obtain a text vector;

[0071] Performing feature alignment on the text vector and the segmented image to obtain the text feature and the image feature;

[0072] See also Figure 3 , a text segmenter can be used to segment and vectorize the sample text. A special tag [CLS] is set before the text segmentation, and a separator [SEP] is added between the end of the text segmentation and the first position of the segmented image. The text segmentation includes "table", "on", "put", "a lot", "banana", "and", and "fruit". Since the size of the sample images is inconsistent, the sample images are segmented and uniformly transformed into segmented images of uniform size. The segmented images can be 256*256 size images.

[0073] For text segmentation, a 40% probability can be adopted to cover the original segmentation with a mask, that is, the text segmentation can be replaced with [masked]. For example, "banana" is replaced with [masked], and the vector value of [masked] can be replaced with a random value based on Gaussian distribution.

[0074] In this step, the text vector and the segmented image are input into a layer normalized (LayerNorm) in the mask encoder, and the network model based on the self-attention mechanism (transformer) is pre-placed. This can effectively normalize the inputs of different modalities and align the features between text and image.

[0075] Each feature of the MME output corresponds to an input feature. The number and position of inputs and outputs in MME are aligned. During model pre-training, text vectors, segmented images, special labels, and separators are used as model inputs.

[0076] Furthermore, determining the contrast loss and the mask mark loss according to the text feature includes:

[0077] Input the text features into a pre-trained multi-layer perceptron to perform similarity calculation to obtain feature similarity, and perform loss calculation on the feature similarity according to an information noise contrast estimation loss function to obtain the contrast loss;

[0078] Obtaining a mask prediction position in the text feature, and performing loss calculation on the mask prediction position according to a cross entropy loss function to obtain the mask marking loss;

[0079] Among them, the pre-training process of the mask encoder needs to calculate the contrast loss, mask labeling loss and mean square error loss for the output. The contrast loss is the value of the sim() function in the Information Noise-Contrastive Estimation (InfoNCE) formula, which is calculated by a multilayer perceptron (MLP). The MLP includes two fully connected layers and a sigmoid activation function. The input of the MLP network is the output feature of the corresponding position of [CLS], that is, Figure 3 The first position feature is output in . The first position feature is passed through the MLP network to output feature similarity. The feature similarity is set to the value of the sim() function in the InfoNCE loss function formula. After substituting it into the InfoNCE loss function formula, the contrast loss can be calculated. The loss calculation formula is as follows:

[0080]

[0081] Among them, f(x i ,y i ) is the feature similarity output by MLP when the sample pair <text i, image i> is used as input, that is, the similarity between text and image. The denominator is the cumulative value of feature similarities from 1 to N. In this step, N is the number of sample pairs, E represents the expectation, and f(x i , yj) is the feature similarity output by MLP when the sample pair <text i, picture j> is used as input. The expected loss value of the current number of iterations is calculated. When there are 32 sample pairs, the expected loss value will be calculated 32 times. The expected loss values ​​in the 32 iteration results are accumulated and then divided by 32 to calculate the total expectation and obtain the comparative loss.

[0082] In this step, the mask encoder predicts the word at the [masked] position, and calculates the loss based on the mask prediction position in the text feature and the true position of [masked] to obtain the masked tag loss, where the cross entropy losses of all masked words are added together and divided by the number of masked words to obtain the masked tag loss.

[0083] In this embodiment, after all decoded images are generated, the minimum mean square error is calculated for the segmented images corresponding to the position, and the minimum mean square errors are added and divided by the number of decoded images to obtain the mean square error loss. The mean square error loss, contrast loss and mask label loss are added to obtain the second loss, and the second loss gradient is back-propagated to the parameters in the mask encoder, including the parameters in the decoder. The pre-trained mask encoder can predict the words at the [masked] position, the decoder can generate a small image, and the position corresponding to [CLS] can output the feature similarity of the text and the image.

[0084] Furthermore, the pre-trained mask encoder is used as a feature encoder to perform feature encoding on the text to be generated and the image to be generated, and the sample features (including text and image) of each sample pair can be obtained.

[0085] Step S202, inputting the sample features into the stream model for feature mapping to obtain mapping features, and determining a first loss according to the mapping features;

[0086] Among them, by inputting sample features into the flow model for feature mapping, the mapping features corresponding to the sample features can be effectively obtained, and the model loss of the flow model can be effectively calculated based on the mapping features to obtain the first loss. The flow model is a generative model built on an invertible function. The flow model forms a composite transformation process by stacking multiple reversible transformation layers, thereby mapping the input data to the latent variable space and realizing the inverse mapping from latent variables to data.

[0087] Optionally, the sample features are input into a stream model for feature mapping to obtain mapping features, including:

[0088] Acquire feature segmentation points in the sample features, and perform feature segmentation on the sample features according to the feature segmentation points to obtain a first segmentation feature and a second segmentation feature;

[0089] According to the first fully-connected module and the second fully-connected module, the first segmentation feature is fully-connected to obtain a first fully-connected feature and a second fully-connected feature;

[0090] Performing a dot product calculation on the second segmentation feature and the first fully connected feature to obtain a dot product feature, and fusing the dot product feature with the second fully connected feature to obtain a fused feature;

[0091] Performing feature splicing on the first segmentation feature and the fusion feature to obtain the mapping feature;

[0092] Among them, the feature segmentation point represents the segmentation position between the text vector and the image vector in the sample feature, and the segmentation position is determined according to the positions of the adjacent text vector and the image vector. A first fully connected module and a second fully connected module are provided in the reversible transformation layer in the flow model, and the module structure of the first fully connected module and the second fully connected module can be set according to the requirements. In this step, the parameters of the fully connected layer between the first fully connected module and the second fully connected module are different, and the first fully connected module includes two layers of fully connected layers, and the second fully connected module includes three layers of fully connected layers. The feature linearity between the first fully connected feature and the second fully connected feature after the first fully connected module and the second fully connected module are different. The dot product feature is obtained by performing dot product calculation on the second segmentation feature and the first fully connected feature, and the dot product feature is fused with the second fully connected feature to obtain the fused feature, which effectively improves the linear conversion of the sample feature.

[0093] See also Figure 4 , the process of the flow model is to map a sample feature Z to a mapping feature X. The first d items of the mapping feature X [x1...x d ] can be directly obtained from the first d items of sample feature Z [z1...z d ] is copied. The first d items of sample feature Z [z1...z d ] After the first fully connected module F and the second fully connected module H, the first fully connected feature β and the second fully connected feature γ are obtained. [x d+1 ...x D ]by[z d+1 ...z D ] and β, and then add γ to get the fusion feature [x d+1 ...x D ], and [x1...x d ] and [x d+1 ...x D ] to concatenate the vectors and obtain the mapping feature X.

[0094] Further, determining a first loss according to the mapping feature includes:

[0095] Performing vector matching on the mapping feature and a preset Gaussian distribution image to obtain a vector dimension value, and performing probability distribution calculation on the vector dimension value according to a probability density function to obtain a probability density value;

[0096] Calculate the difference between a preset value and the probability density value to obtain a loss value, and calculate the quotient between the loss value and the total number of vector dimension values ​​to obtain the first loss;

[0097] Among them, the mapping feature is vector matched with the preset Gaussian distribution picture, the value of each dimension of the vector position corresponding to the preset Gaussian distribution picture in the mapping feature is obtained, the vector dimension value y is obtained, and the vector dimension value y is used as the input of the probability density function of the standard Gaussian to calculate the probability density function of the standard Gaussian (normal) distribution:

[0098]

[0099] Among them, the mean is μ, the standard deviation is σ, the mean of the standard Gaussian distribution is 0, and the standard deviation is 1. After obtaining the probability density value p(y), the maximum likelihood estimation algorithm is used to calculate the loss, that is, the probability density value of the vector dimension value on each dimension of the sample feature Z is calculated separately, and the difference after subtracting each probability density value from 1 is used to obtain the loss value. The loss values ​​are added and then divided by the total number of vector dimension values ​​to obtain the first loss.

[0100] In this step, vector matching is performed between the mapping feature and the preset Gaussian distribution image to obtain the vector dimension value on the vector position dimension corresponding to the mapping feature and the preset Gaussian distribution image, and the probability density function is used to perform probability distribution calculation on the vector dimension value. The probability density value corresponding to the mapping feature can be effectively calculated, and the loss value corresponding to the mapping feature can be effectively obtained by calculating the difference between the preset value and the probability density value. The loss value of the flow model, that is, the first loss, is obtained by calculating the quotient between the loss value and the total number of vector dimension values.

[0101] Step S203, updating parameters of the flow model, the mask encoder, and the picture decoder according to the first loss until the flow model, the mask encoder, and the picture decoder converge;

[0102] The first loss is fed back to the flow model, the mask encoder and the picture decoder to synchronously update the parameters in the network until the flow model, the mask encoder and the picture decoder converge.

[0103] Step S204: if a picture generation request is received, inputting the text to be generated in the picture generation request into the converged stream model and the converged picture decoder to generate a picture, thereby obtaining a generated picture;

[0104] Among them, based on the converged flow model and the converged image decoder, the image generation task can be automatically executed, and the image is generated according to the text to be generated in the image generation request to obtain the generated image.

[0105] Optionally, inputting the text to be generated in the image generation request into the converged stream model and the converged image decoder to generate an image, to obtain a generated image, includes:

[0106] Sampling a preset Gaussian distribution image to obtain a first sampling vector, and inputting the to-be-generated text and the first sampling vector into the converged stream model for feature mapping to obtain a first target mapping feature;

[0107] The first target mapping feature is input into the converged picture decoder to generate a picture, so as to obtain the generated picture.

[0108] Step S205, if a text generation request is received, the image to be generated in the text generation request is input into the converged stream model and the converged mask encoder to generate text, so as to obtain generated text;

[0109] Among them, based on the converged flow model and the converged mask encoder, the text generation task can be automatically performed, and text generation is performed according to the to-be-generated image in the text generation request to obtain the generated text.

[0110] Optionally, inputting the image to be generated in the text generation request into the converged stream model and the converged mask encoder to generate text to obtain generated text, including:

[0111] The preset Gaussian distribution picture is sampled to obtain a second sampling vector, and the picture to be generated and the second sampling vector are input into the converged flow model for feature mapping to obtain a second target mapping feature; wherein, the preset Gaussian distribution picture can be set according to demand, for example, the preset Gaussian distribution picture can be set to a standard Gaussian distribution picture.

[0112] The second target mapping feature is input into the converged mask encoder to generate text, so as to obtain the generated text.

[0113] In this embodiment, by pre-training the mask encoder model through sample pairs, the text and image can be effectively co-encoded and mapped to a unified representation space, so that the feature expressions of the image and text are consistent. Based on the joint training of the mask encoder, the flow model and the image decoder, the two training tasks of text-to-image and image-to-text can be completed at the same time. Only one set of models needs to be maintained, which effectively reduces the cost of text-to-image interaction and improves the efficiency of text-to-image interaction. In this embodiment, a masked multimodal encoder is used in combination with a training framework of a volume-variable flow model to co-encode text and images and map them into a unified representation space, so that the feature expressions of the image and text are consistent. At the same time, the sample features generated by the masked multimodal encoder obey the distribution of the unified representation space, and then the sample features are transformed through the Jacobian matrix to allow the flow model to learn the mapping of the real data distribution to the Gaussian distribution. The target mapping feature generation of the flow model only takes one step, which effectively improves the efficiency of text-to-image interaction.

[0114] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0115] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned generated pictures and generated texts, the above-mentioned generated pictures and generated texts can also be stored in a node of a blockchain.

[0116] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0117] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0118] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0120] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0121] Further references Figure 5 , as a response to the above Figure 2 The present application provides an embodiment of a text-image intergeneration device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices. Figure 5 As shown, the text-image intergeneration device 300 described in this embodiment includes: a pre-training unit 301, a feature mapping unit 302, a convergence unit 303, an image generation unit 304 and a text generation unit 305, wherein:

[0122] The pre-training unit 301 is used to perform model pre-training on the mask encoder according to the sample pair, and obtain sample features of the sample pair according to the mask encoder after the model pre-training, wherein the sample pair includes a sample text and a sample image.

[0123] Optionally, the pre-training unit 301 is further used to: perform feature encoding on the sample text and the sample image according to the mask encoder to obtain text features and image features, and determine contrast loss and mask labeling loss according to the text features;

[0124] Inputting the image features into the picture decoder to decode the picture to obtain a decoded picture, and determining a mean square error loss based on the decoded picture and the sample picture;

[0125] A second loss is determined according to the mean square error loss, the contrast loss and the mask label loss, and parameters of the mask encoder are updated according to the second loss.

[0126] Furthermore, the pre-training unit 301 is also used to: perform word segmentation and segmentation on the sample text and the sample image according to the mask encoder to obtain text segmentation and segmentation images;

[0127] Randomly masking the text segmentation, and performing vector encoding on the randomly masked text segmentation to obtain a text vector;

[0128] Feature alignment is performed on the text vector and the segmented image to obtain the text feature and the image feature.

[0129] Furthermore, the pre-training unit 301 is also used to: input the text features into the pre-trained multi-layer perceptron to perform similarity calculation to obtain feature similarity, and perform loss calculation on the feature similarity according to the information noise contrast estimation loss function to obtain the contrast loss;

[0130] The mask prediction position in the text feature is obtained, and the loss of the mask prediction position is calculated according to the cross entropy loss function to obtain the mask marking loss.

[0131] The feature mapping unit 302 is used to input the sample feature into the stream model for feature mapping to obtain mapping features, and determine the first loss according to the mapping features.

[0132] Optionally, the feature mapping unit 302 is further used to: obtain a feature segmentation point in the sample feature, and perform feature segmentation on the sample feature according to the feature segmentation point to obtain a first segmentation feature and a second segmentation feature, wherein the feature segmentation point represents a segmentation position between a text vector and an image vector in the sample feature;

[0133] According to the first fully-connected module and the second fully-connected module, the first segmentation feature is fully-connected to obtain a first fully-connected feature and a second fully-connected feature;

[0134] Performing a dot product calculation on the second segmentation feature and the first fully connected feature to obtain a dot product feature, and fusing the dot product feature with the second fully connected feature to obtain a fused feature;

[0135] The first segmentation feature and the fusion feature are concatenated to obtain the mapping feature.

[0136] Furthermore, the feature mapping unit 302 is further used to: perform vector matching on the mapping feature and a preset Gaussian distribution image to obtain a vector dimension value, and perform probability distribution calculation on the vector dimension value according to a probability density function to obtain a probability density value;

[0137] The difference between the preset value and the probability density value is calculated to obtain a loss value, and the quotient between the loss value and the total number of the vector dimension values ​​is calculated to obtain the first loss.

[0138] The convergence unit 303 is used to update the parameters of the flow model, the mask encoder and the picture decoder according to the first loss until the flow model, the mask encoder and the picture decoder converge.

[0139] The image generation unit 304 is configured to, if an image generation request is received, input the text to be generated in the image generation request into the converged stream model and the converged image decoder to perform image generation to obtain a generated image.

[0140] Optionally, the image generation unit 304 is further used to: sample a preset Gaussian distribution image to obtain a first sampling vector, and input the text to be generated and the first sampling vector into the converged flow model for feature mapping to obtain a first target mapping feature;

[0141] The first target mapping feature is input into the converged picture decoder to generate a picture, so as to obtain the generated picture.

[0142] The text generation unit 305 is used for, if a text generation request is received, inputting the image to be generated in the text generation request into the converged flow model and the converged mask encoder to perform text generation to obtain generated text.

[0143] In this embodiment, the mask encoder model is pre-trained through sample pairs, so that the text and image can be effectively encoded together and mapped to a unified representation space, so that the feature expressions of the image and text are consistent. Based on the joint training of the mask encoder, the flow model and the image decoder, the two training tasks of text-to-image and image-to-text can be completed at the same time. Only one set of models needs to be maintained, which effectively reduces the cost of text-to-image interaction and improves the efficiency of text-to-image interaction.

[0144] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0145] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0146] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0147] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the text-image interaction method based on artificial intelligence, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0148] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the artificial intelligence-based text-image interaction method.

[0149] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0150] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0151] In the embodiments of the present application, by pre-training the mask encoder model through sample pairs, the text and image can be effectively encoded together and mapped to a unified representation space, so that the feature expressions of the image and text are consistent. Based on the joint training of the mask encoder, the flow model and the image decoder, the two training tasks of text-to-image and image-to-text can be completed simultaneously. Only one set of models needs to be maintained, which effectively reduces the cost of text-to-image intergeneration and improves the efficiency of text-to-image intergeneration.

[0152] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned artificial intelligence-based text-image interaction method.

[0153] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0154] In the embodiments of the present application, by pre-training the mask encoder model through sample pairs, the text and image can be effectively encoded together and mapped to a unified representation space, so that the feature expressions of the image and text are consistent. Based on the joint training of the mask encoder, the flow model and the image decoder, the two training tasks of text-to-image and image-to-text can be completed simultaneously. Only one set of models needs to be maintained, which effectively reduces the cost of text-to-image intergeneration and improves the efficiency of text-to-image intergeneration.

[0155] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0156] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A text-image intergeneration method based on artificial intelligence, characterized in that: include: Pre-training a mask encoder model according to the sample pair, and obtaining sample features of the sample pair according to the mask encoder after the model pre-training, wherein the sample pair includes a sample text and a sample image; Inputting the sample features into the stream model for feature mapping to obtain mapping features, and determining a first loss according to the mapping features; updating parameters of the flow model, the mask encoder, and the picture decoder according to the first loss until the flow model, the mask encoder, and the picture decoder converge; If a picture generation request is received, the text to be generated in the picture generation request is input into the converged stream model and the converged picture decoder to generate the picture, so as to obtain a generated picture; If a text generation request is received, the image to be generated in the text generation request is input into the converged flow model and the converged mask encoder to perform text generation to obtain generated text.

2. The method for text-image intergeneration based on artificial intelligence according to claim 1, characterized in that: Pre-train the mask encoder model based on sample pairs, including: Performing feature encoding on the sample text and the sample image according to the mask encoder to obtain text features and image features, and determining contrast loss and mask mark loss according to the text features; Inputting the image features into the picture decoder to decode the picture to obtain a decoded picture, and determining a mean square error loss based on the decoded picture and the sample picture; A second loss is determined according to the mean square error loss, the contrast loss and the mask label loss, and parameters of the mask encoder are updated according to the second loss.

3. The method for text-image intergeneration based on artificial intelligence according to claim 2, characterized in that: The sample text and the sample image are feature encoded according to the mask encoder to obtain text features and image features, including: According to the mask encoder, the sample text and the sample image are respectively segmented and divided to obtain text segmentation and segmentation images; Randomly masking the text segmentation, and performing vector encoding on the randomly masked text segmentation to obtain a text vector; Feature alignment is performed on the text vector and the segmented image to obtain the text feature and the image feature.

4. The method for text-image intergeneration based on artificial intelligence according to claim 2, characterized in that: Determining a contrast loss and a mask marking loss according to the text feature includes: Input the text features into a pre-trained multi-layer perceptron to perform similarity calculation to obtain feature similarity, and perform loss calculation on the feature similarity according to an information noise contrast estimation loss function to obtain the contrast loss; The mask prediction position in the text feature is obtained, and the loss of the mask prediction position is calculated according to the cross entropy loss function to obtain the mask marking loss.

5. The method for text-image intergeneration based on artificial intelligence according to claim 1, characterized in that: The sample features are input into the stream model for feature mapping to obtain mapping features, including: Acquire a feature segmentation point in the sample feature, and perform feature segmentation on the sample feature according to the feature segmentation point to obtain a first segmentation feature and a second segmentation feature, wherein the feature segmentation point represents a segmentation position between a text vector and an image vector in the sample feature; According to the first fully-connected module and the second fully-connected module, the first segmentation feature is fully-connected to obtain a first fully-connected feature and a second fully-connected feature; Performing a dot product calculation on the second segmentation feature and the first fully connected feature to obtain a dot product feature, and fusing the dot product feature with the second fully connected feature to obtain a fused feature; The first segmentation feature and the fusion feature are concatenated to obtain the mapping feature.

6. The method for text-image intergeneration based on artificial intelligence according to claim 1, characterized in that: Determining a first loss according to the mapping feature includes: Performing vector matching on the mapping feature and a preset Gaussian distribution image to obtain a vector dimension value, and performing probability distribution calculation on the vector dimension value according to a probability density function to obtain a probability density value; The difference between the preset value and the probability density value is calculated to obtain a loss value, and the quotient between the loss value and the total number of the vector dimension values ​​is calculated to obtain the first loss.

7. The method for text-image intergeneration based on artificial intelligence according to claim 1, characterized in that: Inputting the text to be generated in the image generation request into the converged stream model and the converged image decoder to generate an image, thereby obtaining a generated image, including: Sampling a preset Gaussian distribution image to obtain a first sampling vector, and inputting the to-be-generated text and the first sampling vector into the converged stream model for feature mapping to obtain a first target mapping feature; The first target mapping feature is input into the converged picture decoder to generate a picture, so as to obtain the generated picture.

8. A text-image intergeneration device, characterized in that: include: A pre-training unit, configured to perform model pre-training on a mask encoder according to a sample pair, and obtain sample features of the sample pair according to the mask encoder after model pre-training, wherein the sample pair includes a sample text and a sample image; A feature mapping unit, configured to input the sample feature into a stream model for feature mapping to obtain a mapping feature, and determine a first loss according to the mapping feature; a convergence unit, configured to update parameters of the flow model, the mask encoder, and the picture decoder according to the first loss until the flow model, the mask encoder, and the picture decoder converge; A picture generating unit, configured to, upon receiving a picture generating request, input the text to be generated in the picture generating request into the converged stream model and the converged picture decoder to generate a picture, so as to obtain a generated picture; The text generation unit is used for, if a text generation request is received, inputting the image to be generated in the text generation request into the converged flow model and the converged mask encoder to perform text generation to obtain generated text.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images