Artificial intelligence-based text-image mutual generation method and device, terminal equipment and medium

Abstract

The application is suitable for the field of artificial intelligence technology, and provides a text-image mutual generation method and device based on artificial intelligence, terminal equipment and medium, the method comprises the following steps: model pre-training is carried out on a mask encoder according to a sample pair, and sample characteristics of the sample pair are obtained according to the mask encoder after model pre-training; the sample characteristics are input into a flow model for characteristic mapping to obtain mapping characteristics, and a first loss is determined according to the mapping characteristics; parameters of the flow model, the mask encoder and a picture decoder are updated according to the first loss until the flow model, the mask encoder and the picture decoder converge; and a text-image mutual generation task is performed according to the converged flow model, mask encoder and picture decoder. Based on the common training of the mask encoder, the flow model and the picture decoder, the two training tasks of text-to-image and image-to-text can be completed at the same time, only one set of model needs to be maintained, the cost of text-image mutual generation is effectively reduced, and the efficiency of text-image mutual generation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a text-image mutual generation method and device based on artificial intelligence, a terminal device and a medium. BACKGROUND

[0002] With the increasing development of artificial intelligence, in some application scenarios, it is necessary to automatically generate a picture that meets the information of the input text, for example, to automatically generate an advertisement picture according to a user's description text; and in some other application scenarios, it is necessary to automatically generate a description text that meets the picture according to the picture content, for example, to automatically generate a corresponding product description according to a product picture, therefore, the problem of text-image mutual generation (generating a picture according to an input text or generating a text according to an input picture) is increasingly valued by people.

[0003] In the existing text-image mutual generation process, two independent sets of picture generation models and text generation models are required to correspondingly perform picture generation tasks and text generation tasks, resulting in a high cost of text-image mutual generation. SUMMARY

[0004] Therefore, the present application provides a text-image mutual generation method and device based on artificial intelligence, a terminal device and a medium to solve the problem of high cost of text-image mutual generation in the prior art.

[0005] A first aspect of the present application provides a text-image mutual generation method based on artificial intelligence, comprising:

[0006] Model pre-training is performed on a mask encoder according to a sample pair, and sample features of the sample pair are obtained according to the mask encoder after model pre-training, the sample pair comprising a sample text and a sample picture;

[0007] The sample features are input into a flow model for feature mapping to obtain mapping features, and a first loss is determined according to the mapping features;

[0008] Parameters of the flow model, the mask encoder and a picture decoder are updated according to the first loss until the flow model, the mask encoder and the picture decoder converge;

[0009] If a picture generation request is received, a text to be generated in the picture generation request is input into the converged flow model and the converged picture decoder for picture generation to obtain a generated picture;

[0010] If a text generation request is received, a picture to be generated in the text generation request is input into the converged flow model and the converged mask encoder for text generation to obtain a generated text.

[0011] Further, the model pre-training of the mask encoder according to the sample comprises:

[0012] According to the mask encoder, the sample text and the sample picture are feature encoded to obtain text features and image features, and the contrast loss and the mask label loss are determined according to the text features.

[0013] The image features are input into the picture decoder for picture decoding to obtain a decoded picture, and the mean square error loss is determined according to the decoded picture and the sample picture.

[0014] The second loss is determined according to the mean square error loss, the contrast loss and the mask label loss, and the parameters of the mask encoder are updated according to the second loss.

[0015] Further, the mask encoder is used to feature encode the sample text and the sample picture to obtain text features and image features, which comprises:

[0016] The sample text and the sample picture are segmented by the mask encoder to obtain text segmentation and segmented pictures.

[0017] The text segmentation is randomly masked, and the text vector is obtained by vector encoding the randomly masked text segmentation.

[0018] The text vector and the segmented picture are feature aligned to obtain the text features and the image features.

[0019] Further, the contrast loss and the mask label loss are determined according to the text features, which comprises:

[0020] The text features are input into the pre-trained multi-layer perception to calculate the feature similarity, and the loss calculation of the feature similarity is performed according to the information noise contrast estimation loss function to obtain the contrast loss.

[0021] The mask prediction position in the text features is obtained, and the loss calculation of the mask prediction position is performed according to the cross-entropy loss function to obtain the mask label loss.

[0022] Further, the sample features are input into the flow model for feature mapping to obtain mapping features, which comprises:

[0023] The feature segmentation points in the sample features are obtained, and the sample features are feature segmented according to the feature segmentation points to obtain first segmentation features and second segmentation features, and the feature segmentation points represent the segmentation positions between the text vectors and the image vectors in the sample features.

[0024] According to the first full connection module and the second full connection module, full connection processing is respectively performed on the first segmentation feature to obtain first full connection features and second full connection features;

[0025] The second segmentation feature is subjected to dot product calculation with the first full connection feature to obtain dot product features, and the dot product features are fused with the second full connection features to obtain fused features;

[0026] The first segmentation feature is subjected to feature splicing with the fused features to obtain the mapping features.

[0027] Further, a first loss is determined according to the mapping features, including:

[0028] The mapping features are subjected to vector matching with a preset Gaussian distribution picture to obtain vector dimension values, and a probability density function is used to perform probability distribution calculation on the vector dimension values to obtain probability density values;

[0029] A difference value between a preset value and the probability density values is calculated to obtain a loss value, and a quotient value between the loss value and a total number of the vector dimension values is calculated to obtain the first loss.

[0030] Further, a text to be generated in the picture generation request is input into the converged flow model and the converged picture decoder to generate a picture, including:

[0031] A preset Gaussian distribution picture is sampled to obtain a first sampling vector, and the text to be generated and the first sampling vector are input into the converged flow model to perform feature mapping to obtain first target mapping features;

[0032] The first target mapping features are input into the converged picture decoder to generate a picture to obtain the generated picture.

[0033] A second aspect of an embodiment of the application provides a text-picture mutual generation device, including:

[0034] A pre-training unit is configured to perform model pre-training on a mask encoder according to a sample pair, and obtain sample features of the sample pair from the mask encoder after model pre-training, the sample pair including a sample text and a sample picture;

[0035] A feature mapping unit is configured to input the sample features into a flow model to perform feature mapping to obtain mapping features, and determine a first loss according to the mapping features;

[0036] A convergence unit is configured to perform parameter updating on the flow model, the mask encoder and a picture decoder according to the first loss until the flow model, the mask encoder and the picture decoder converge.

[0037] a picture generation unit, configured to, if a picture generation request is received, input text to be generated in the picture generation request into the converged flow model and the converged picture decoder to generate a picture, to obtain a generated picture.

[0038] a text generation unit, configured to, if a text generation request is received, input a picture to be generated in the text generation request into the converged flow model and the converged mask encoder to generate text, to obtain generated text.

[0039] A third aspect of the embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements each step of the method for text-image mutual generation based on artificial intelligence provided by the first scheme when executing the computer program.

[0040] A fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and each step of the method for text-image mutual generation based on artificial intelligence provided by the first scheme is implemented when the computer program is executed by a processor.

[0041] The method, device, terminal device and medium for text-image mutual generation based on artificial intelligence provided by the embodiment of the present application can effectively code and map text and image to a unified representation space through model pre-training of the mask encoder by using samples, so that the feature expressions of the picture and the text have consistency, and based on the joint training of the mask encoder, the flow model and the picture decoder, the two training tasks of text-to-image and image-to-text can be completed at the same time, only one set of model needs to be maintained, the cost of text-image mutual generation is effectively reduced, and the efficiency of text-image mutual generation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the schemes in the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

[0044] Figure 2 is a flowchart of one embodiment of the method for text-image mutual generation based on artificial intelligence according to the present application;

[0045] Figure 3 is Figure 2 is a schematic diagram of sample feature acquisition in the embodiment;

[0046] Figure 4 is Figure 2 a schematic diagram of mapping feature generation in an embodiment;

[0047] Figure 5 is a structural schematic diagram of an embodiment of a text-image coexisting device according to the present application;

[0048] Figure 6 is a structural schematic diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the description and the drawings are to be regarded as illustrative in nature and are not intended to limit the application; the terminology used in the description and the claims of the present application and the above description of the drawings includes the terms specifically mentioned above, as well as their derivatives.

[0050] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase that the phrase in the specification appear not necessarily all refer to the same embodiment, or are mutually exclusive or alternative embodiments.

[0051] In order to make the technical personnel in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings below.

[0052] As Figure 1 shown, the system architecture 100 can include a terminal device 101, a network 102 and a server 103, and the terminal device 101 can be a notebook computer 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0053] A user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0054] The terminal device 101 can be various electronic devices with display screens and supporting web browsing, in addition to the notebook computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an electronic book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer and a desktop computer, etc.

[0055] The server 103 can be a server providing various services, such as a background server supporting the pages displayed on the terminal device 101.

[0056] It should be noted that the method for text-image mutual generation based on artificial intelligence provided in the embodiments of the present application is generally executed by a server / terminal device, and correspondingly, the text-image mutual generation apparatus is generally arranged in a server / terminal device.

[0057] It should be understood that Figure 1 The number of terminal devices, networks and servers in

[0058] With reference to Figure 2 , a flowchart of one embodiment of the method for text-image mutual generation based on artificial intelligence according to the present application is shown. The order of the steps in the flowchart can be changed, and some steps can be omitted according to different needs. The method for text-image mutual generation based on artificial intelligence provided in the embodiments of the present application can be applied to any scenario of generating text based on pictures or generating pictures based on text, and then the method for text-image mutual generation based on artificial intelligence can be applied to products in these scenarios, for example, in a medical medical scenario, automatically generating a medical image text description based on a medical image picture; for example, in a financial scenario, automatically generating an insurance product poster based on an insurance product description. The method for text-image mutual generation based on artificial intelligence includes the following steps:

[0059] Step S201, model pre-training is performed on a mask encoder according to a sample pair, and sample features of the sample pair are obtained according to the mask encoder after model pre-training;

[0060] In this embodiment, the electronic device (e.g. Figure 1 The server / terminal device shown in the figure can receive a text generation request or a picture generation request through a wired connection or a wireless connection. It should be noted that the wireless connection can include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection.

[0061] Specifically, the sample pair includes a sample text and a sample picture, and the sample pair is composed of <sample text, sample picture>, wherein the sample text is used to describe the corresponding picture in words. In this step, the mask encoder is pre-trained using the data set of <sample text, sample picture>, and the mask encoder can be a mask multimodal encoder. The mask multimodal encoder (MME) is an encoder structure that learns and represents features through a mask strategy under multimodal input. Through the pre-trained MME, the text and the image can be mapped to the same representation space.

[0062] Optionally, the mask encoder is model pre-trained according to the sample pair, including:

[0063] According to the mask encoder, the sample text and the sample picture are feature encoded to obtain text features and image features, and a contrastive loss and a masked token loss are determined according to the text features.

[0064] Wherein, through the mask encoder, the sample text and the sample picture are feature encoded, which can effectively obtain the text features of the sample text and the image features of the sample picture in the same representation space, and calculate the contrastive loss and the masked token loss according to the text features.

[0065] The image features are input into the picture decoder for picture decoding to obtain a decoded picture, and a mean-square error loss is determined according to the decoded picture and the sample picture.

[0066] Wherein, by inputting the image features into the picture decoder for picture decoding, the image features can be effectively restored to the decoded image, and the mean-square error loss can be obtained by calculating the difference between the decoded picture and the sample picture.

[0067] determine a second loss according to the mean square error loss, the contrastive loss and the mask token loss, and perform parameter updating on the mask encoder according to the second loss until the mask encoder meets a preset convergence condition, which can be set according to requirements; wherein, based on the mean square error loss, the contrastive loss and the mask token loss, the loss of the mask encoder during model pre-training can be effectively determined, that is, the second loss.

[0068] Further, feature encoding is performed on the sample text and the sample picture according to the mask encoder to obtain text features and image features, including:

[0069] According to the mask encoder, text segmentation and segmentation picture are obtained by performing word segmentation and segmentation on the sample text and the sample picture respectively.

[0070] Random mask is performed on the text segmentation, and vector encoding is performed on the text segmentation after random mask to obtain a text vector.

[0071] Feature alignment is performed on the text vector and the segmentation picture to obtain the text features and the image features.

[0072] Please refer to Figure 3 The sample text can be segmented and vector encoded by a text segmentation tokenizer, a special label [CLS] is set before text segmentation, a segment symbol [SEP] is added between the end of the text segmentation and the beginning of the segmentation picture, and the text segmentation includes "table", "on", "put", "many", "banana", "and" and "fruit". Since the sample picture size is inconsistent, the sample picture is segmented to uniformly transform into a segmentation picture with a uniform size, and the segmentation picture can be a 256*256 size picture.

[0073] For text segmentation, the original segmentation can be covered with a mask with a probability of 40%, that is, the text segmentation can be replaced with [masked], for example, "banana" is replaced with [masked], and the vector value of [masked] can be replaced with a random value based on a Gaussian distribution.

[0074] In this step, the text vector and the segmentation picture are input into a network model based on a self-attention mechanism (transformer) in front of a layer normalization (LayerNorm) in the mask encoder, which can effectively normalize the input of different modalities and align the features between the text and the image.

[0075] Each feature output by the MME corresponds to an input feature, and the number and position of the input and output in the MME are aligned. During model pre-training, the text vector, the segmentation picture, the special label and the segment symbol are used as model inputs.

[0076] Further, the contrast loss and the mask label loss are determined according to the text features, and the determining comprises:

[0077] The text features are input into a pre-trained multilayer perceptron for similarity calculation to obtain feature similarity, and loss calculation is performed on the feature similarity according to an information noise contrast estimation loss function to obtain the contrast loss.

[0078] The mask prediction position in the text features is obtained, and loss calculation is performed on the mask prediction position according to a cross-entropy loss function to obtain the mask label loss.

[0079] The pre-training process of the mask encoder needs to calculate the contrast loss, the mask label loss and the mean square error loss on the output, the contrast loss is the value of a sim() function in an information noise contrast estimation loss function (InfoNCE) formula, which is calculated by a multilayer perceptron (MLP), the MLP includes two fully connected layers and a sigmoid activation function, the input of the MLP network is the output feature of the [CLS] corresponding position, that is, Figure 3 the feature of the first position output in the formula (1). After the feature of the first position is output by the MLP network, the feature similarity is output, the feature similarity is set as the value of the sim() function in the InfoNCE loss function formula, and the contrast loss can be calculated after the formula of the InfoNCE loss function is substituted, and the formula of the loss calculation is as follows:

[0080]

[0081] wherein, f(x i , y i ) is the feature similarity output by the MLP when the sample pair <text i, picture i> is input, that is, the similarity of the text and the picture. The denominator is the cumulative value of the feature similarity from 1 to N, in this step, N is the number of sample pairs, E represents expectation, f(x i , yj) is the feature similarity output by the MLP when the sample pair <text i, picture j> is input. The loss expectation value is calculated 32 times, and the loss expectation values in the 32 iterations are accumulated and then divided by 32 to calculate the total expectation to obtain the contrast loss.

[0082] In this step, the mask encoder predicts the words in the [masked] position, calculates the loss based on the masked prediction position in the text feature and the true position of [masked], and obtains the mask label loss. After adding the cross-entropy loss of all masked words and dividing by the number of masked words, the mask label loss is obtained.

[0083] In this embodiment, after all the decoded pictures are generated, the minimum mean square error of the segmented picture corresponding to the position is calculated, and the minimum mean square error is added and divided by the number of decoded pictures to obtain the mean square error loss. The mean square error loss, the contrast loss and the mask label loss are added to obtain the second loss. The second loss gradient is back-propagated to the parameters in the mask encoder, including the parameters in the Decoder. The pre-trained mask encoder can predict the words in the [masked] position, the Decoder can generate small pictures, and the [CLS] corresponding position can output the similarity of the text and picture features.

[0084] Further, the pre-trained mask encoder can be used as a feature encoder encoder to encode the features of the to-be-generated text and the to-be-generated picture, and obtain the sample features (including text and image) of each sample pair.

[0085] In step S202, the sample features are input into the stream model for feature mapping to obtain mapping features, and a first loss is determined based on the mapping features.

[0086] The sample features are input into the stream model for feature mapping to effectively obtain the mapping features corresponding to the sample features, and the model loss of the stream model is effectively calculated based on the mapping features to obtain the first loss. The stream model is a generative model constructed based on an invertible function (Invertible Function). The stream model forms a complex transformation process by stacking multiple reversible transformation layers, thereby mapping the input data to the latent variable space and realizing the inverse mapping from the latent variable to the data.

[0087] Optionally, the sample features are input into the stream model for feature mapping to obtain mapping features, including:

[0088] The feature segmentation points in the sample features are obtained, and the sample features are segmented based on the feature segmentation points to obtain first segmentation features and second segmentation features.

[0089] The first segmentation features are respectively fully connected based on the first full connection module and the second full connection module to obtain first full connection features and second full connection features.

[0090] The second segmentation feature is calculated by point multiplication with the first full connection feature to obtain a point multiplication feature, and the point multiplication feature is fused with the second full connection feature to obtain a fusion feature;

[0091] The first segmentation feature is spliced with the fusion feature to obtain the mapping feature;

[0092] The feature segmentation point represents the segmentation position between the sample feature text vector and the image vector, and the segmentation position is determined according to the positions of adjacent text vectors and image vectors. The reversible transformation layer in the flow model is provided with a first full connection module and a second full connection module. The module structures of the first full connection module and the second full connection module can be set according to requirements. In this step, the parameters of the full connection layers between the first full connection module and the second full connection module are different, the first full connection module includes two layers of full connection layers, and the second full connection module includes three layers of full connection layers. The feature linearities between the first full connection feature and the second full connection feature after full connection processing by the first full connection module and the second full connection module are different. By calculating the second segmentation feature by point multiplication with the first full connection feature, a point multiplication feature is obtained, and the point multiplication feature is fused with the second full connection feature to obtain a fusion feature, effectively improving the linear conversion of the sample feature.

[0093] Please refer to Figure 4 , the process of the flow model is to map a sample feature Z to a mapping feature X. The first d items [x1...x d ] of the mapping feature X can be directly copied from the first d items [z1...z d ] of the sample feature Z. The first d items [z1...z d ] of the sample feature Z are processed through the first full connection module F and the second full connection module H to obtain the first full connection feature β and the second full connection feature γ. The first d items [x1...x d+1 ...x D ] of the mapping feature X are obtained by calculating the point multiplication of [z d+1 ...z D ] and β and adding γ. The first d items [x1...x d+1 ...x D ] of the mapping feature X are obtained by splicing the vectors of [x1...x d ] and [x d+1 ...x D ].

[0094] Further, a first loss is determined according to the mapping feature, comprising:

[0095] The mapping feature is matched with a preset Gaussian distribution picture to obtain a vector dimension value, and a probability density value is calculated by performing a probability distribution calculation on the vector dimension value according to a probability density function.

[0096] a difference value between the preset value and the probability density value is calculated to obtain a loss value, and a quotient value between the loss value and a total number of the vector dimension values is calculated to obtain the first loss;

[0097] wherein the mapping feature is subjected to vector matching with the preset Gaussian distribution picture to obtain a value in each dimension of a vector position corresponding to the preset Gaussian distribution picture in the mapping feature, to obtain a vector dimension value y, the vector dimension value y is taken as an input of a probability density function of a standard Gaussian, and the probability density function of the standard Gaussian (normal) distribution is calculated;

[0098]

[0099] wherein the mean value is μ, the standard deviation is σ, the mean value of the standard Gaussian distribution is 0, and the standard deviation is 1. After the probability density value p(y) is obtained, a loss is calculated using a maximum likelihood estimation algorithm, that is, a probability density value is calculated for each vector dimension value in each dimension of the sample feature Z, a difference value after each probability density value is subtracted by 1 is obtained, the loss value is obtained by adding the loss values and dividing by the total number of the vector dimension values, and the first loss is obtained.

[0100] In this step, the mapping feature is subjected to vector matching with the preset Gaussian distribution picture to obtain a vector dimension value in a vector position dimension corresponding to the mapping feature and the preset Gaussian distribution picture, the vector dimension value is subjected to probability distribution calculation by the probability density function, the probability density value corresponding to the mapping feature can be effectively calculated, the loss value corresponding to the mapping feature can be effectively obtained by calculating a difference value between the preset value and the probability density value, and the loss of the flow model, that is, the first loss, is obtained by calculating a quotient value between the loss value and the total number of the vector dimension values.

[0101] In step S203, the parameters of the flow model, the mask encoder and the picture decoder are updated according to the first loss until the flow model, the mask encoder and the picture decoder converge.

[0102] wherein the first loss is back-propagated to the parameters in the synchronous updating network of the flow model, the mask encoder and the picture decoder until the flow model, the mask encoder and the picture decoder converge.

[0103] In step S204, if a picture generation request is received, the text to be generated in the picture generation request is input into the converged flow model and the converged picture decoder to generate a picture.

[0104] The converged flow model and the converged picture decoder can automatically perform a picture generation task, generate a picture according to the to-be-generated text in the picture generation request, and obtain a generated picture.

[0105] Optionally, the to-be-generated text in the picture generation request is input into the converged flow model and the converged picture decoder for picture generation, and a generated picture is obtained.

[0106] The preset Gaussian distribution picture is sampled to obtain a first sampling vector, and the to-be-generated text and the first sampling vector are input into the converged flow model for feature mapping to obtain a first target mapping feature.

[0107] The first target mapping feature is input into the converged picture decoder for picture generation to obtain the generated picture.

[0108] In step S205, if a text generation request is received, the to-be-generated picture in the text generation request is input into the converged flow model and the converged mask encoder for text generation to obtain generated text.

[0109] The converged flow model and the converged mask encoder can automatically perform a text generation task, generate text according to the to-be-generated picture in the text generation request, and obtain generated text.

[0110] Optionally, the to-be-generated picture in the text generation request is input into the converged flow model and the converged mask encoder for text generation to obtain generated text, including:

[0111] The preset Gaussian distribution picture is sampled to obtain a second sampling vector, and the to-be-generated picture and the second sampling vector are input into the converged flow model for feature mapping to obtain a second target mapping feature. The preset Gaussian distribution picture can be set according to requirements, for example, the preset Gaussian distribution picture can be set as a standard Gaussian distribution picture.

[0112] The second target mapping feature is input into the converged mask encoder for text generation to obtain the generated text.

[0113] In the embodiment, the model pre-training of the sample pair on the mask encoder can effectively code and map the text and the image into a unified representation space, so that the feature expressions of the picture and the text have consistency, and the common training of the mask encoder, the flow model and the picture decoder can complete the two training tasks of text-to-image and image-to-text, only one set of model needs to be maintained, the cost of text-image mutual generation is effectively reduced, and the efficiency of text-image mutual generation is improved. In the embodiment, the training framework of the mask multi-modal encoder combined with the volume variable flow model is used to code and map the text and the image into a unified representation space, so that the feature expressions of the picture and the text have consistency, and the sample features generated by the mask multi-modal encoder conform to the distribution of the unified representation space, and then the sample features are converted through the Jacobian matrix to make the flow model learn the mapping from the real data distribution to the Gaussian distribution, and the target mapping feature generation of the flow model only needs one step, thereby effectively improving the efficiency of text-image mutual generation.

[0114] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0115] It should be emphasized that, in order to further ensure the privacy and security of the generated picture and the generated text, the generated picture and the generated text can also be stored in a node of a blockchain.

[0116] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. Blockchain, in essence, is a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block containing information about a batch of network transactions, used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0117] The embodiment of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0118] The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technologies of artificial intelligence mainly include computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of each method. Among them, the storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0120] It should be understood that, although each step in the flowchart of the accompanying drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0121] Further referring to Figure 5 , as an implementation of the method shown in Figure 2 , the present application provides an embodiment of a text-image mutual generation device. The device embodiment corresponds to the method embodiment shown in Figure 2 , and the device can be applied to various electronic devices. As shown in Figure 5 , the text-image mutual generation device 300 described in the embodiment includes a pre-training unit 301, a feature mapping unit 302, a convergence unit 303, a picture generation unit 304, and a text generation unit 305, wherein:

[0122] The pre-training unit 301 is configured to pre-train a mask encoder according to a sample pair, and obtain sample features of the sample pair according to the mask encoder after model pre-training, wherein the sample pair includes a sample text and a sample picture.

[0123] Optionally, the pre-training unit 301 is further configured to: perform feature encoding on the sample text and the sample picture according to the mask encoder to obtain text features and image features, and determine a contrast loss and a mask label loss according to the text features;

[0124] input the image features into the picture decoder to perform picture decoding to obtain a decoded picture, and determine a mean square error loss according to the decoded picture and the sample picture;

[0125] determine a second loss according to the mean square error loss, the contrast loss and the mask label loss, and perform parameter updating on the mask encoder according to the second loss.

[0126] Further, the pre-training unit 301 is further configured to: perform word segmentation and segmentation on the sample text and the sample picture respectively according to the mask encoder to obtain text segmentation and segmented pictures;

[0127] perform random mask on the text segmentation, and perform vector encoding on the text segmentation after random mask to obtain a text vector;

[0128] perform feature alignment on the text vector and the segmented pictures to obtain the text features and the image features.

[0129] Still further, the pre-training unit 301 is further configured to: input the text features into a pre-trained multi-layer perception to perform similarity calculation to obtain a feature similarity, and perform loss calculation on the feature similarity according to an information noise contrast estimation loss function to obtain the contrast loss;

[0130] obtain a mask prediction position in the text features, and perform loss calculation on the mask prediction position according to a cross-entropy loss function to obtain the mask label loss.

[0131] The feature mapping unit 302 is configured to input the sample features into a flow model to perform feature mapping to obtain mapped features, and determine a first loss according to the mapped features.

[0132] Optionally, the feature mapping unit 302 is further configured to: obtain a feature segmentation point in the sample features, and perform feature segmentation on the sample features according to the feature segmentation point to obtain first segmented features and second segmented features, the feature segmentation point indicating a segmentation position between a text vector and an image vector in the sample features;

[0133] perform full connection processing on the first segmented features respectively according to a first full connection module and a second full connection module to obtain first full connection features and second full connection features;

[0134] perform dot product calculation on the second segmentation feature and the first fully connected feature to obtain a dot product feature, and fuse the dot product feature with the second fully connected feature to obtain a fused feature;

[0135] perform feature splicing on the first segmentation feature and the fused feature to obtain the mapping feature.

[0136] Further, the feature mapping unit 302 is further configured to perform vector matching on the mapping feature and a preset Gaussian distribution picture to obtain a vector dimension value, and perform probability distribution calculation on the vector dimension value according to a probability density function to obtain a probability density value.

[0137] calculate a difference value between a preset value and the probability density value to obtain a loss value, and calculate a quotient value between the loss value and a total number of the vector dimension values to obtain the first loss.

[0138] The convergence unit 303 is configured to perform parameter updating on the flow model, the mask encoder and the picture decoder according to the first loss until the flow model, the mask encoder and the picture decoder converge.

[0139] The picture generation unit 304 is configured to, if a picture generation request is received, input a to-be-generated text in the picture generation request into the converged flow model and the converged picture decoder to perform picture generation to obtain a generated picture.

[0140] Optionally, the picture generation unit 304 is further configured to sample a preset Gaussian distribution picture to obtain a first sampling vector, and input the to-be-generated text and the first sampling vector into the converged flow model to perform feature mapping to obtain a first target mapping feature.

[0141] input the first target mapping feature into the converged picture decoder to perform picture generation to obtain the generated picture.

[0142] The text generation unit 305 is configured to, if a text generation request is received, input a to-be-generated picture in the text generation request into the converged flow model and the converged mask encoder to perform text generation to obtain generated text.

[0143] In this embodiment, the model pre-training of the mask encoder through the sample pair can effectively encode and map the text and the image to a unified representation space, so that the feature expressions of the picture and the text have consistency. Based on the joint training of the mask encoder, the flow model and the picture decoder, the two training tasks of text-to-image and image-to-text can be completed at the same time, only one set of model needs to be maintained, the cost of text-image mutual generation is effectively reduced, and the efficiency of text-image mutual generation is improved.

[0144] To solve the above technical problems, the embodiment of the present application further provides a computer device. For details, please refer to Figure 6 , Figure 6 The basic structure block diagram of the computer device of the embodiment is shown in the figure.

[0145] The computer device 4 comprises a memory 41, a processor 42 and a network interface 43 which are connected to each other through a system bus. It should be noted that only the computer device 4 with the memory 41, the processor 42 and the network interface 43 is shown in the figure, but it should be understood that all the components shown are not required to be implemented, and more or less components can be alternatively implemented. Among them, the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0146] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device and the like.

[0147] The memory 41 at least includes one type of readable storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, for example, a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit and the external storage device of the computer device 4. In this embodiment, the memory 41 is generally used to store an operating system and various application software installed on the computer device 4, for example, computer readable instructions of the text-image co-evolution method based on artificial intelligence, etc. In addition, the memory 41 can also be used to temporarily store various data that has been output or will be output.

[0148] The processor 42 in some embodiments can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer readable instructions or process data stored in the memory 41, for example, the computer readable instructions of the text-image co-evolution method based on artificial intelligence.

[0149] The network interface 43 can include a wireless network interface or a wired network interface, which is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0150] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0151] In the embodiments of the present application, the model pre-training of the mask encoder by the sample pair can effectively code and map the text and the image to a unified representation space, so that the feature expressions of the picture and the text have consistency. Based on the joint training of the mask encoder, the flow model and the picture decoder, the two training tasks of text-to-image and image-to-text can be completed at the same time, only one set of model needs to be maintained, the cost of text-image co-evolution is effectively reduced, and the efficiency of text-image co-evolution is improved.

[0152] The application also provides another implementation, namely providing a computer readable storage medium, the computer readable storage medium stores computer readable instructions, the computer readable instructions can be executed by at least one processor to make the at least one processor execute the steps of the artificial intelligence-based text-image mutual generation method as described above.

[0153] Compared with the prior art, the embodiments of the application have the following beneficial effects:

[0154] In the embodiments of the application, the model pre-training of the mask encoder by the sample pair can effectively code and map the text and the image to a unified representation space, so that the feature expressions of the picture and the text have consistency, and based on the joint training of the mask encoder, the flow model and the picture decoder, the two training tasks of text-to-image and image-to-text can be completed at the same time, only one set of model needs to be maintained, the cost of text-image mutual generation is effectively reduced, and the text-image mutual generation efficiency is improved.

[0155] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods described in various embodiments of the application.

[0156] Obviously, the above-described embodiments are only some of the embodiments of the application, not all the embodiments, and the preferred embodiments of the application are given in the drawings, but do not limit the patent scope of the application. The application can be implemented in many different forms, and on the contrary, the purpose of providing these embodiments is to make the disclosure of the application more thorough and comprehensive. Although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent replacements to some technical features. Any equivalent structure made by using the contents of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the application.

Claims

1. An artificial intelligence-based text-image mutual generation method, characterized in that, The method comprises the following steps: model pre-training is performed on a mask encoder according to a sample pair, and sample features of the sample pair are obtained according to the model pre-trained mask encoder, the sample pair comprising a sample text and a sample picture; the sample features are input into a flow model for feature mapping to obtain mapping features, and a first loss is determined according to the mapping features, wherein the flow model is a generative model constructed based on a reversible function; parameters of the flow model, the mask encoder and a picture decoder are updated according to the first loss until the flow model, the mask encoder and the picture decoder converge; if a picture generation request is received, the text to be generated in the picture generation request is input into the converged flow model and the converged picture decoder for picture generation to obtain a generated picture; if a text generation request is received, the picture to be generated in the text generation request is input into the converged flow model and the converged mask encoder for text generation to obtain a generated text; the sample features are input into a flow model for feature mapping to obtain mapping features, comprising: a feature segmentation point in the sample features is obtained, and the sample features are segmented according to the feature segmentation point to obtain first segmented features and second segmented features, the feature segmentation point representing a segmentation position between a text vector and an image vector in the sample features; first full connection features and second full connection features are obtained by performing full connection processing on the first segmented features according to first full connection modules and second full connection modules, respectively; dot product features are obtained by performing dot product calculation on the second segmented features and the first full connection features, and the dot product features are fused with the second full connection features to obtain fused features; the first segmented features are spliced with the fused features to obtain the mapping features. 2.The AI-based text-image mutual generation method of claim 1, wherein, The model pre-training is performed on the mask encoder according to the sample pair, comprising: text features and image features are obtained by performing feature encoding on the sample text and the sample picture according to the mask encoder, and a contrast loss and a mask mark loss are determined according to the text features; decoded pictures are obtained by inputting the image features into the picture decoder for picture decoding, and a mean square error loss is determined according to the decoded pictures and the sample pictures; a second loss is determined according to the mean square error loss, the contrast loss and the mask mark loss, and parameters of the mask encoder are updated according to the second loss. 3.The AI-based text-image mutual generation method of claim 2, wherein, The text features and the image features are obtained by performing feature encoding on the sample text and the sample picture according to the mask encoder, comprising: text word segmentation and segmented pictures are obtained by performing word segmentation and segmentation on the sample text and the sample picture according to the mask encoder, respectively; random masks are performed on the text word segmentation, and vector encoding is performed on the text word segmentation after the random masks to obtain text vectors; feature alignment is performed on the text vectors and the segmented pictures to obtain the text features and the image features. 4.The AI-based text-image mutual generation method of claim 2, wherein, The contrast loss and the mask mark loss are determined according to the text features, comprising: input the text features into a pre-trained multi-layer perceptron for similarity calculation to obtain feature similarity, and perform loss calculation on the feature similarity according to an information noise contrast estimation loss function to obtain the contrast loss; obtain a mask prediction position in the text features, and perform loss calculation on the mask prediction position according to a cross-entropy loss function to obtain the mask label loss. 5.The AI-based text-image mutual generation method of claim 1, wherein, determining a first loss according to the mapping features, including: performing vector matching on the mapping features and a preset Gaussian distribution picture to obtain a vector dimension value, and performing probability distribution calculation on the vector dimension value according to a probability density function to obtain a probability density value; calculating a difference value between a preset value and the probability density value to obtain a loss value, and calculating a quotient value between the loss value and a total number of the vector dimension values to obtain the first loss. 6.The AI-based text-image mutual generation method of claim 1, wherein, inputting the to-be-generated text in the picture generation request into the converged stream model and the converged picture decoder for picture generation to obtain a generated picture, including: sampling a preset Gaussian distribution picture to obtain a first sampling vector, and inputting the to-be-generated text and the first sampling vector into the converged stream model for feature mapping to obtain first target mapping features; inputting the first target mapping features into the converged picture decoder for picture generation to obtain the generated picture.

7. A text-image intergrowth device, characterized by comprising: The text-picture mutual generation device implements the steps of the method according to any one of claims 1 to 6, and the text-picture mutual generation device includes: a pre-training unit configured to pre-train a mask encoder according to a sample pair, and obtain sample features of the sample pair according to the mask encoder after model pre-training, the sample pair including a sample text and a sample picture; a feature mapping unit configured to input the sample features into a stream model for feature mapping to obtain mapping features, and determine a first loss according to the mapping features; a convergence unit configured to perform parameter updating on the stream model, the mask encoder and a picture decoder according to the first loss until the stream model, the mask encoder and the picture decoder converge; a picture generation unit configured to, if a picture generation request is received, input to-be-generated text in the picture generation request into the converged stream model and the converged picture decoder for picture generation to obtain a generated picture; a text generation unit configured to, if a text generation request is received, input to-be-generated picture in the text generation request into the converged stream model and the converged mask encoder for text generation to obtain generated text.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 6.

Images