Text generation model training based on diffusion model, text generation method and device

Abstract

The embodiment of the present application discloses a text generation model training method and device based on a diffusion model, and a text generation method and device, comprising: obtaining style text associated sample data; the style text associated sample data comprises style text picture sample data and reference text sample data; inputting the style text associated sample data into a feature extraction module of a text generation model to extract first style text features of the style text associated sample data through the feature extraction module; wherein the first style text features comprise first text features and first style picture features; calculating a first feature optimization index data between the first text features and the first style picture features through the text generation model; and training the text generation model through the first feature optimization index data. The technical scheme of the embodiment of the present application can improve the processing efficiency of the text generation model training sample, and further improve the training efficiency of the model and the efficiency of the generated text.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image generation, and in particular to a text generation model training method and device based on a diffusion model, a text generation method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the rapid development of information technology and the rise of mobile Internet, in order to meet the needs of individualization, different artistic style fonts are commonly used in daily documents.

[0003] At present, a text generation model can be used to generate the required text type of a user on demand. Generally, the text generation model can be trained using artificial intelligence algorithms such as deep learning algorithms. After successful model training, the model can be applied to quickly and intelligently generate the required text type of a user on demand.

[0004] The inventors have found that in the prior art, when developers train a recognition model that can accurately recognize fonts with style and patterns, a large number of text or artistic character samples need to be collected. The collection work is not only time-consuming and laborious, but also there is a situation that the collected data samples are not comprehensive and complete. SUMMARY

[0005] Embodiments of the present application provide a text generation model training method and device based on a diffusion model, a text generation method and device, an electronic device, and a storage medium, which can improve the processing efficiency of text generation model training samples, and further improve the training efficiency of the text generation model and the efficiency of generating text using the model.

[0006] According to an aspect of the present application, a text generation model training method based on a diffusion model is provided, comprising:

[0007] Obtaining style text associated sample data; the style text associated sample data includes style text picture sample data and reference text sample data;

[0008] Inputting the style text associated sample data into a feature extraction module of a text generation model to extract first style text features of the style text associated sample data through the feature extraction module; wherein the first style text features include first text features and first style picture features;

[0009] Calculating a first feature optimization index data between the first text features and the first style picture features through the text generation model;

[0010] Training the text generation model through the first feature optimization index data.

[0011] According to another aspect of the present application, a text generation method is provided, comprising:

[0012] obtaining to-be-generated style text associated data; wherein the to-be-generated style text associated data comprises to-be-generated style text picture data and to-be-generated text data;

[0013] inputting the to-be-generated style text associated data into a feature extraction module of a text generation model to extract second style text features of the to-be-generated style text associated data by the feature extraction module; wherein the second style text features comprise second text features and second style picture features;

[0014] generating a second target style text picture according to the second text features and the second style picture features by the text generation model;

[0015] wherein the text generation model is trained by the text generation model training method.

[0016] According to another aspect of the present application, a text generation model training device based on a diffusion model is provided, comprising:

[0017] a style text associated sample data obtaining module for obtaining style text associated sample data; the style text associated sample data comprises style text picture sample data and reference text sample data;

[0018] a first feature extraction module for inputting the style text associated sample data into a feature extraction module of a text generation model to extract first style text features of the style text associated sample data by the feature extraction module; wherein the first style text features comprise first text features and first style picture features;

[0019] a first feature optimization index data calculating module for calculating first feature optimization index data between the first text features and the first style picture features by the text generation model;

[0020] a text generation model training module for training the text generation model by the first feature optimization index data.

[0021] According to another aspect of the present application, a text generation device is provided, comprising:

[0022] a to-be-generated style text associated data obtaining module for obtaining to-be-generated style text associated data; wherein the to-be-generated style text associated data comprises to-be-generated style text picture data and to-be-generated text data;

[0023] a second feature extraction module configured to input the to-be-generated style text-related data into a feature extraction module of the text generation model to extract second style text features of the to-be-generated style text-related data by the feature extraction module; wherein the second style text features comprise second text features and second style picture features;

[0024] a second target style text picture generation module configured to generate a second target style text picture according to the second text features and the second style picture features by the text generation model;

[0025] The text generation model is trained by the text generation model training method provided in the embodiments of the present application.

[0026] According to another aspect of the present application, an electronic device is provided, which comprises:

[0027] at least one processor; and

[0028] a memory connected to the at least one processor in communication; wherein,

[0029] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the text generation model training method based on the diffusion model according to any one of the embodiments of the present application, or implement the text generation method as described in the embodiments of the present application.

[0030] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the text generation model training method based on the diffusion model according to any one of the embodiments of the present application, or implement the text generation method as described in the embodiments of the present application when executed by the processor.

[0031] The technical scheme of the embodiment of the present application firstly acquires style text associated sample data, then inputs the style text associated sample data into a feature extraction module of a text generation model to extract first style text features of the style text associated sample data through the feature extraction module, calculates first feature optimization index data between first text features and first style picture features through the text generation model, and finally trains the text generation model through the first feature optimization index data. After the text generation model training is completed, style text associated data to be generated is acquired, the style text associated data to be generated is input into the feature extraction module of the text generation model to extract second style text features of the style text associated data to be generated through the feature extraction module, and finally a second target style text picture is generated by the text generation model according to the second text features and second style picture features, thereby solving the problem of difficult collection of text generation model training samples, improving the processing efficiency of the text generation model training samples, and further improving the training efficiency of the text generation model and the efficiency of generating text by using the model.

[0032] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0034] Figure 1 is a flowchart of a text generation model training method based on a diffusion model provided by an embodiment of the present application;

[0035] Figure 2 is a flowchart of another text generation model training method based on a diffusion model provided by an embodiment of the present application;

[0036] Figure 3 is a flowchart of another text generation model training method based on a diffusion model provided by an embodiment of the present application;

[0037] Figure 4 is a flowchart of a text generation method provided by an embodiment of the present application;

[0038] Figure 5 is a structural schematic diagram of a text generation model training device based on a diffusion model provided by an embodiment of the present application;

[0039] Figure 6 is a structural schematic diagram of a text generation device provided by an embodiment of the present application.

[0040] Figure 7 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the protection scope of the present application.

[0042] It should be noted that the terms "comprising" and "having" and any variations thereof are intended to cover not exclusively containing, for example, a process, method, system, product or device containing a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0043] Embodiment one

[0044] Figure 1 is a flowchart of a text generation model training method based on a diffusion model provided by an embodiment of the present application. The embodiment can be applicable to the case of artistic style personalized migration of text. The method can be executed by a text generation model training device. The device can be realized by software and / or hardware, and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. The present application does not limit the specific device type of the electronic device. Correspondingly, as shown in Figure 1 the method includes the following operations:

[0045] S110, acquiring style text associated sample data; the style text associated sample data includes style text picture sample data and reference text sample data.

[0046] The style text associated sample data can be text data with a certain style, which is used to train the text generation model. The style text picture sample data can be picture sample data used to express a certain style form. The reference text sample data can be text sample data that needs to be processed in style.

[0047] In the embodiment of the present application, the style-text associated sample data can be obtained from a browser or an open source database.

[0048] In the embodiment of the present application, the style-text associated sample data is input into a feature extraction module of the text generation model to extract first style-text features of the style-text associated sample data by the feature extraction module.

[0049] The first style-text features can include first text features and first style picture features. The first style-text features are obtained by the feature extraction module from the style-text associated sample data and include style-text associated sample data information and text picture style information. The first text features are obtained by the feature extraction module from the reference text sample data and include style-text associated sample data information. The first style picture features are obtained by the feature extraction module from the style-text picture sample data and include text picture style information.

[0050] In the embodiment of the present application, the style-text associated sample data is input into the feature extraction module of the text generation model to obtain the first style-text features of the style-text associated sample data by the feature extraction module.

[0051] Specifically, the first style-text features can be composed of the first text features including the style-text associated sample data information and the first style picture features including the text picture style information.

[0052] In the embodiment of the present application, the first feature optimization index data between the first text features and the first style picture features is calculated by the text generation model.

[0053] The text generation model can be a model for automatically adding an expected text style to the reference text sample data. The first feature optimization index data can be used to evaluate the difference between the predicted value and the true value of the text generation model.

[0054] In the embodiment of the present application, the first text features and the first style picture features are calculated by the text generation model to obtain the first feature optimization index data between the first text features and the first style picture features.

[0055] In the embodiment of the present application, the first feature optimization index data is used to train the text generation model.

[0056] Correspondingly, after the first feature optimization index data is calculated, the first feature optimization index data can be used to optimize the parameter of the text generation model, so as to realize the training of the text generation model.

[0057] The technical scheme of the embodiment of the application first acquires style-text associated sample data, then inputs the style-text associated sample data into a feature extraction module of a text generation model, so as to extract first style-text features of the style-text associated sample data through the feature extraction module, calculate first feature optimization index data between the first text features and the first style picture features through the text generation model, and finally train the text generation model through the first feature optimization index data, thereby solving the problem of difficulty in collecting training samples of the text generation model, improving the processing efficiency of the training samples of the text generation model, and further improving the training efficiency of the text generation model and the efficiency of generating text by using the model.

[0058] Embodiment Two

[0059] Figure 2 is a flowchart of another text generation model training method based on a diffusion model provided by the second embodiment of the application. The present embodiment is based on the above-mentioned embodiment and is embodied in detail. In the present embodiment, a specific optional implementation manner of extracting first style-text features by the feature extraction module and calculating first feature optimization index data by the text generation model is given. Correspondingly, as shown in Figure 2 the method of the present embodiment can include:

[0060] S210, acquiring pre-training picture sample data.

[0061] S220, pre-training the picture feature extraction module according to the pre-training picture sample data.

[0062] S230, acquiring style-text associated sample data; the style-text associated sample data includes style-text picture sample data and reference text sample data.

[0063] The pre-training picture sample data can be picture sample data used to train the picture feature extraction module. The picture feature extraction module can be a functional module used to extract picture feature information.

[0064] In the embodiment of the application, before training the text generation model, the picture feature extraction module included in the text generation model can be pre-trained first. Specifically, pre-training picture sample data can be acquired first, and then the pre-training picture sample data is input into the picture feature extraction module as input data of the picture feature extraction module for learning and training.

[0065] S240, input the style text association sample data to a feature extraction module of the text generation model to extract first style text features of the style text association sample data through the feature extraction module.

[0066] In an optional embodiment of the present application, the feature extraction module can include a picture feature extraction module and a text feature extraction module; the inputting the style text association sample data to the feature extraction module of the text generation model to extract the first style text features of the style text association sample data through the feature extraction module can include: inputting the style text picture sample data to an encoder of the picture feature extraction module to extract picture features of the style text picture sample data through the encoder of the picture feature extraction module; performing feature dimension processing on the picture features of the style text picture sample data to obtain dimension-processed picture features; taking the dimension-processed picture features as the first style picture features.

[0067] The picture feature extraction module can be used to extract features of picture information. The text feature extraction module can be used to extract features of text information. The encoder can be used to extract features of pictures.

[0068] In the embodiment of the present application, the style text picture sample data is taken as input data of the picture feature extraction module and is sent to the encoder of the picture feature extraction module, and the encoder of the picture feature extraction module is used to extract picture features of the style text picture sample data. Then, the picture features of the style text picture sample data are subjected to feature dimension processing to obtain dimension-processed picture features. Finally, the dimension-processed picture features are taken as the first style picture features.

[0069] In a specific example, the picture feature extraction module is taken as an example of an autoencoder. First, the style text picture sample data is taken as input data of the picture feature extraction module autoencoder and is sent to the encoder of the picture feature extraction module autoencoder, and the encoder is used to extract picture features of the style text picture sample data. Then, the extracted picture features are subjected to random sampling to realize feature dimension processing, and dimension-processed picture features are obtained as the first style picture features.

[0070] In an optional embodiment of the present application, the inputting the style text associated sample data into the feature extraction module of the text generation model to extract the first style text feature of the style text associated sample data by the feature extraction module can include: inputting the reference text sample data into the text feature extraction module to extract the text feature of the reference text sample data by the text feature extraction module; and taking the text feature of the reference text sample data as the first text feature.

[0071] The text feature extraction module can be a feature extraction module for extracting text information.

[0072] In the embodiment of the present application, the reference text sample data is first taken as the input data of the text feature extraction module and sent to the text feature extraction module, then the text feature extraction module is used to extract the text feature of the reference text sample data, and finally the text feature of the reference text sample data is taken as the first text feature.

[0073] S250, calculating a first feature optimization index data between the first text feature and the first style picture feature by the text generation model.

[0074] In the embodiment of the present application, the calculating the first feature optimization index data between the first text feature and the first style picture feature by the text generation model can include: sampling the first text feature according to a first set step length to obtain a current sampling time text feature; adding a current time noise to the style text picture sample data at the current sampling time to obtain a current sampling time noise-added picture; inputting the current sampling time text feature and the current sampling time noise-added picture into a semantic segmentation model of the text generation model to obtain a current sampling time noise-added text picture; and calculating a difference between the current sampling time noise-added text picture and a standard noise-added text picture as the first feature optimization index data.

[0075] The first set step length can be a distance for indicating sampling movement. The current sampling time text feature can be a text feature obtained by sampling the first text feature at the current time. The current time noise can be a noise required to be added to the style text picture at the current time. The current sampling time noise-added picture can be a picture obtained by adding noise to the style text picture at the current time. The semantic segmentation model can be a model for generating the current sampling time noise-added text picture. The current sampling time noise-added text picture can be a text picture processed by adding noise at the current sampling time. The standard noise-added text picture can be a text picture processed by standard Gaussian noise at the current sampling time.

[0076] In the embodiment of the present application, first, the first character feature is sampled according to a first set step size to obtain a character feature at a current sampling time, and then random noise at the current time is added to the style character picture sample data at the current sampling time to obtain a noisy picture at the current sampling time. The character feature at the current sampling time and the noisy picture at the current sampling time are input into a semantic segmentation model of a character generation model as input data of the semantic segmentation model, and the semantic segmentation model is used to obtain a noisy character picture at the current sampling time. The first feature optimization index data of the semantic segmentation model is obtained by calculating the difference between the noisy character picture at the current sampling time and a standard noisy character picture.

[0077] In a specific example, the first set step size is 1000 steps. In every 1000 steps, one step is randomly selected as the current sampling time to sample the first character feature to obtain the character feature at the current sampling time.

[0078] Optionally, the random noise at the current time can be added to the style character picture sample data at the current sampling time based on the following formula to obtain the noisy picture at the current sampling time:

[0079]

[0080]

[0081] wherein, X t represents the noisy picture at time t, X t-1 represents the noisy picture at time t-1, q(x) represents the prior distribution of x, β t represents the parameter value associated with the current time t, N(X t ; a, b) represents the Gaussian distribution to which the noisy picture at time t conforms, X1 represents the noisy picture at time 1, and I represents the unit matrix.

[0082] In the embodiment of the present application, the character generation model can be a diffusion model Diffusion Model, and the semantic segmentation model can be a UNET (Unity Networking) in the Diffusion Model. The character feature at the current sampling time and the noisy picture at the current sampling time are input into the semantic segmentation model of the character generation model Diffusion Model as input data of the semantic segmentation model, and the semantic segmentation model is used to obtain the noisy character picture at the current sampling time. The first feature optimization index data of the semantic segmentation model is obtained by calculating the difference between the noisy character picture at the current sampling time and a standard noisy character picture.

[0083] S260, training the character generation model by using the first feature optimization index data.

[0084] In an optional embodiment of the present application, after the first feature optimization index data is used to train the text generation model, the method can further include: obtaining a noise-added text picture output by the text generation model; inputting the noise-added text picture into a decoder of a picture feature extraction module in the feature extraction module to decode the noise-added text picture by the decoder of the picture feature extraction module to obtain a first target style text picture of a set dimension.

[0085] The decoder can be configured to analyze features of a picture. The first target style text picture can be a text picture data with a certain text style obtained by decoding the picture feature extraction module.

[0086] In the embodiment of the present application, the noise-added text picture is obtained by the above steps, and the noise-added text picture is input as input data of the picture feature extraction module in the feature extraction module into the decoder of the picture feature extraction module. The decoder is used to decode the noise-added text picture to obtain a feature map, and the size of the feature map is adjusted to obtain the first target style text picture of 512x512.

[0087] In an optional embodiment of the present application, after the first feature optimization index data is used to train the text generation model, the method can further include: inputting test text sample data into a text feature extraction module in the feature extraction module to extract test text features of the test text sample data by the text feature extraction module; inputting the test text features into an attention module of a semantic segmentation model of the text generation model to obtain attention output test text features; sampling the attention output test text features according to a second set step to obtain sampled test text features; inputting the sampled test text features into a decoder of a picture feature extraction module in the feature extraction module to decode the sampled test text features by the decoder of the picture feature extraction module to obtain a target test style text picture of a set dimension.

[0088] The second set step can be a distance for indicating sampling movement. The test text sample data can be text test data with a certain text style. The test text features can be text features of the test text sample data. The attention output test text features can be text features extracted by the attention module. The sampled test text features can be text features obtained by sampling at a current time. The target test style text picture can be a picture containing certain text picture style information obtained by the feature extraction module.

[0089] In the embodiment of the present application, first, the test text sample data is taken as the input data of the feature extraction module, and is sent to the text feature extraction module in the feature extraction module. The text feature extraction module is used to extract features from the test text sample data to obtain test text features. Then, the test text features are taken as the input data of the text generation model, and are sent to the attention module of the text generation model. The attention module is used to extract features from the test text features to obtain attention output test text features. Further, a second step size is set to 100 steps. In each 100 steps, one step is selected as the current sampling time by reverse sampling to sample the attention output test text features, and test text features at the current sampling time are obtained. The sampled test text features are taken as the input data of the feature extraction module, and are sent to the decoder of the picture feature extraction module in the feature extraction module. The decoder is used to decode the sampled test text features to obtain a feature map, and the size of the feature map is adjusted to obtain a target test style text picture of 512x512.

[0090] Figure 3 is a flowchart of another text generation model training method based on a diffusion model provided in the second embodiment of the present application, as shown in Figure 3 the text data is first sent into the text encoder to obtain text sample features, the picture data with a style pattern is sent into the encoder to obtain picture features, the picture features are sampled to obtain picture sample features, the noise, the text sample features and the picture sample features are taken as the input data of the text generation model, are sent to the text generation model for learning and training, and the output features are decoded by the decoder to obtain the text picture with the style.

[0091] The technical solution of the embodiment of the present application first acquires style text associated sample data and pre-trained picture sample data, pre-trains a picture feature extraction module according to the pre-trained picture sample data. Then, the style text associated sample data is input into the feature extraction module of the text generation model to extract first style text features of the style text associated sample data by the feature extraction module. A first feature optimization index data between the first text features and the first style picture features is calculated by the text generation model, and finally the text generation model is trained by the first feature optimization index data. By extracting the first style text features and the first style picture features, the personalized migration of the text style is realized, and the generation efficiency of the text is improved.

[0092] Embodiment three

[0093] Figure 4is a flowchart of a text generation method provided by Embodiment Three of the present application. The present embodiment can be used in the case of automatically generating text with a certain style using the text generation model trained in the above-mentioned embodiments. The method can be executed by a text generation device, which can be implemented in software and / or hardware and generally integrated in an electronic device, which can be a terminal device or a server device. The present embodiment does not limit the specific type of electronic device. Accordingly, as shown in Figure 4 the method of the present embodiment can include:

[0094] S310, obtaining to-be-generated style text associated data; wherein the to-be-generated style text associated data includes to-be-generated style text picture data and to-be-generated text data.

[0095] The to-be-generated style text associated data can be text data with a certain style of text. The to-be-generated style text picture data can be picture data for expressing a certain style of text. The to-be-generated text data can be text data that needs to be added with a certain style of text.

[0096] In the present embodiment, the to-be-generated style text associated data can be obtained from a browser through copying or from an open source database. In the present embodiment, the method of obtaining the to-be-generated style text associated data is not limited and can be any method that can obtain the to-be-generated style text associated data.

[0097] S320, inputting the to-be-generated style text associated data to a feature extraction module of the text generation model to extract second style text features of the to-be-generated style text associated data through the feature extraction module.

[0098] The second style text features include second text features and second style picture features. The first text features can be obtained by extracting the to-be-generated text data using the feature extraction module and contain features of the to-be-generated style text associated data information. The second style text features can be obtained by extracting the to-be-generated style text picture data using the feature extraction module and contain features of a certain style of text picture.

[0099] In the present embodiment, the to-be-generated style text associated sample data is input to the feature extraction module of the text generation model as input data of the feature extraction module of the text generation model, and the second style text features of the to-be-generated style text associated sample data are obtained by using the feature extraction module.

[0100] S330, generating a second target style text picture according to the second text features and the second style picture features through the text generation model.

[0101] The text generation model is trained by the text generation model training method provided in the embodiments of the present application. The second target style text picture can be a text picture with a certain text style obtained by the text generation model.

[0102] In the embodiments of the present application, the second text feature and the second style picture feature are taken as input data of the text generation model and input into the text generation model for learning and training to obtain the second target style text picture.

[0103] The technical scheme of the embodiments of the present application first acquires style text associated sample data, then inputs the style text associated sample data into a feature extraction module of a text generation model to extract first style text features of the style text associated sample data through the feature extraction module, calculates first feature optimization index data between the first text features and first style picture features through the text generation model, and finally trains the text generation model through the first feature optimization index data. After the text generation model is trained, to-be-generated style text associated data is acquired, the to-be-generated style text associated data is input into the feature extraction module of the text generation model to extract second style text features of the to-be-generated style text associated data through the feature extraction module, and finally a second target style text picture is generated by the text generation model according to the second text features and second style picture features, thereby solving the problem of difficult collection of training samples of the text generation model, realizing personalized migration of text artistic style, and improving font generation efficiency.

[0104] Embodiment Four

[0105] Figure 5 is a structural schematic diagram of a text generation model training device based on a diffusion model provided in Embodiment Four of the present application, as Figure 5 shown, the device comprises a style text associated sample data acquisition module 410, a first feature extraction module 420, a first feature optimization index data calculation module 430, and a text generation model training module 440, wherein:

[0106] The style text associated sample data acquisition module 410 is configured to acquire style text associated sample data; the style text associated sample data comprises style text picture sample data and reference text sample data.

[0107] The first feature extraction module 420 is configured to input the style text associated sample data into a feature extraction module of a text generation model to extract first style text features of the style text associated sample data through the feature extraction module; wherein the first style text features comprise first text features and first style picture features.

[0108] The first feature optimization index data calculation module 430 is configured to calculate first feature optimization index data between the first text feature and the first style picture feature by using the text generation model.

[0109] The text generation model training module 440 is configured to train the text generation model by using the first feature optimization index data.

[0110] The technical scheme of the embodiment of the application first acquires style-text association sample data, then inputs the style-text association sample data into a feature extraction module of a text generation model, extracts a first style-text feature of the style-text association sample data by using the feature extraction module, calculates first feature optimization index data between a first text feature and a first style picture feature by using the text generation model, and finally trains the text generation model by using the first feature optimization index data, thereby solving the problem of difficulty in collecting training samples of the text generation model, improving the processing efficiency of the training samples of the text generation model, and further improving the training efficiency of the text generation model and the efficiency of generating text by using the model.

[0111] Optionally, the feature extraction module comprises a picture feature extraction module and a text feature extraction module; the first feature extraction module 420 is specifically configured to input the style-text picture sample data into an encoder of the picture feature extraction module, extract picture features of the style-text picture sample data by using the encoder of the picture feature extraction module, perform feature dimension processing on the picture features of the style-text picture sample data to obtain dimension-processed picture features, and take the dimension-processed picture features as the first style picture features; and the first feature extraction module 420 is further configured to input the reference text sample data into the text feature extraction module, extract text features of the reference text sample data by using the text feature extraction module, and take the text features of the reference text sample data as the first text features.

[0112] Optionally, the text generation model training apparatus further comprises a picture sample data pre-training module, which is specifically configured to acquire pre-training picture sample data, and pre-train the picture feature extraction module according to the pre-training picture sample data.

[0113] Optionally, the first feature optimization index data calculation module 430 is specifically used for: sampling the first text feature according to a first set step size to obtain the text feature at the current sampling time; adding noise at the current time to the style text image sample data at the current sampling time to obtain a noisy image at the current sampling time; inputting the text feature at the current sampling time and the noisy image at the current sampling time into the semantic segmentation model of the text generation model to obtain a noisy text image at the current sampling time; and calculating the difference between the noisy text image at the current sampling time and the standard noisy text image as the first feature optimization index data.

[0114] Optionally, the text generation model training module 440 is specifically used for: obtaining a noisy text image output by the text generation model; inputting the noisy text image into the decoder of the image feature extraction module in the feature extraction module, so as to decode the noisy text image through the decoder of the image feature extraction module to obtain a first target style text image of a set dimension.

[0115] Optionally, the text generation model training module 440 is further configured to: input test text sample data into the text feature extraction module of the feature extraction module, so as to extract test text features of the test text sample data through the text feature extraction module; input the test text features into the attention module of the semantic segmentation model of the text generation model to obtain attention output test text features; sample the attention output test text features according to a second set step size to obtain sampled test text features; input the sampled test text features into the decoder of the image feature extraction module of the feature extraction module, so as to decode the sampled test text features through the decoder of the image feature extraction module to obtain a target test style text image of a set dimension.

[0116] The text generation model training device described above can execute the text generation model training method based on the diffusion model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the text generation model training method based on the diffusion model provided in any embodiment of the present invention.

[0117] Example 5

[0118] Figure 6 This is a schematic diagram of the structure of a text generation device provided in Embodiment 5 of the present invention, as shown below. Figure 6 As shown, the device includes: a module 510 for acquiring associated data of text in a desired style, a second feature extraction module 520, and a second target style text image generation module 530, wherein:

[0119] The to-be-generated style text association data obtaining module 510 is configured to: obtain to-be-generated style text association data; wherein the to-be-generated style text association data comprises to-be-generated style text picture data and to-be-generated text data.

[0120] The second feature extraction module 520 is configured to: input the to-be-generated style text association data into a feature extraction module of a text generation model, so as to extract second style text features of the to-be-generated style text association data by the feature extraction module; wherein the second style text features comprise second text features and second style picture features.

[0121] The second target style text picture generation module 530 is configured to: generate a second target style text picture according to the second text features and the second style picture features by the text generation model; wherein the text generation model is trained by the text generation model training method provided in the embodiments of the present application.

[0122] The technical scheme of the embodiments of the present application firstly obtains style text association sample data, then inputs the style text association sample data into a feature extraction module of a text generation model, so as to extract first style text features of the style text association sample data by the feature extraction module, calculates a first feature optimization index data between the first text features and the first style picture features by the text generation model, and finally trains the text generation model by the first feature optimization index data. After the text generation model is trained, to-be-generated style text association data is obtained, the to-be-generated style text association data is input into the feature extraction module of the text generation model, so as to extract second style text features of the to-be-generated style text association data by the feature extraction module, and finally generate a second target style text picture according to the second text features and the second style picture features by the text generation model, which solves the problem of difficult collection of training samples of the text generation model, improves the processing efficiency of the training samples of the text generation model, and further improves the training efficiency of the text generation model and the efficiency of generating text by using the model.

[0123] The text generation device described above can execute the text generation method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the present embodiment can be referred to the text generation method provided by any of the embodiments of the present application.

[0124] Embodiment six

[0125] Figure 7A structural diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0126] As shown in Figure 7 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected in communication with the at least one processor 11, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0127] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0128] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the text generation model training method or the text generation method based on the diffusion model.

[0129] In some embodiments, the text generation model training method or the text generation method can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 18. In some embodiments, parts or all of the computer program can be loaded and / or installed onto electronic device 10 via, e.g., ROM 12 and / or communication unit 19. When the computer program is loaded onto RAM 13 and executed by processor 11, one or more steps of the diffusion model based text generation model training method or text generation method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the diffusion model based text generation model training method or text generation method by any other suitable means, e.g., by means of firmware.

[0130] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0131] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0132] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0134] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0135] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

Claims

1. A method for training a text generation model based on a diffusion model, characterized in that, include: Obtain style text association sample data; the style text association sample data includes style text image sample data and baseline text sample data; The style text association sample data is input into the feature extraction module of the text generation model to extract the first style text feature of the style text association sample data; wherein, the text generation model is a diffusion model, and the first style text feature includes a first text feature and a first style image feature; The text generation model is used to calculate the first feature optimization index data between the first text feature and the first style image feature; The text generation model is trained using the first feature optimization index data; The step of calculating the first feature optimization index data between the first text feature and the first style image feature through the text generation model includes: The first text feature is sampled according to a first set step size to obtain the text feature at the current sampling time; noise at the current sampling time is added to the style text image sample data at the current sampling time to obtain a noisy image at the current sampling time; the text feature at the current sampling time and the noisy image at the current sampling time are input into the semantic segmentation model of the text generation model to obtain a noisy text image at the current sampling time; the difference between the noisy text image at the current sampling time and the standard noisy text image is calculated as the first feature optimization index data; The method further includes, after training the text generation model using the first feature optimization index data: Obtain the noisy text image output by the text generation model; input the noisy text image into the decoder of the image feature extraction module in the feature extraction module, so that the noisy text image is decoded by the decoder of the image feature extraction module to obtain a first target style text image of a set dimension.

2. The method according to claim 1, characterized in that, The feature extraction module includes an image feature extraction module and a text feature extraction module; The step of inputting the style text association sample data into the feature extraction module of the text generation model, so as to extract the first style text feature of the style text association sample data through the feature extraction module, includes: The style text image sample data is input into the encoder of the image feature extraction module so that the image features of the style text image sample data can be extracted by the encoder of the image feature extraction module. The image features of the style text image sample data are subjected to feature dimension processing to obtain dimension-processed image features. The image features processed in the aforementioned dimensions are used as the first style image features; The benchmark text sample data is input into the text feature extraction module to extract the text features of the benchmark text sample data. The text features of the benchmark text sample data are used as the first text feature.

3. The method according to claim 2, characterized in that, Before inputting the style text image sample data into the image feature extraction module to extract the image features of the style text image sample data through the image feature extraction module, the method further includes: Obtain pre-trained image sample data; The image feature extraction module is pre-trained based on the pre-trained image sample data.

4. The method according to claim 1, characterized in that, After training the text generation model using the first feature optimization index data, the method further includes: The test text sample data is input into the text feature extraction module in the feature extraction module so that the test text features of the test text sample data can be extracted by the text feature extraction module. The test text features are input into the attention module of the semantic segmentation model of the text generation model to obtain the attention output test text features; The attention output test text features are sampled according to the second set step size to obtain the sampled test text features; The sampled test text features are input into the decoder of the image feature extraction module in the feature extraction module, so that the sampled test text features are decoded by the decoder of the image feature extraction module to obtain a target test style text image of a set dimension.

5. A method for generating text, characterized in that, include: Obtain the text association data of the style to be generated; wherein, the text association data of the style to be generated includes the text image data of the style to be generated and the text data to be generated; The text association data to be generated is input into the feature extraction module of the text generation model, so as to extract the second style text features of the text association data to be generated through the feature extraction module; wherein, the second style text features include second text features and second style image features; The text generation model generates a second target style text image based on the second text features and the second style image features. The text generation model is trained using the text generation model training method described in any one of claims 1-4.

6. A text generation model training device based on a diffusion model, characterized in that, include: The style text association sample data acquisition module is used to acquire style text association sample data; the style text association sample data includes style text image sample data and baseline text sample data; The first feature extraction module is used to input the style text association sample data into the feature extraction module of the text generation model, so as to extract the first style text features of the style text association sample data through the feature extraction module; wherein, the text generation model is a diffusion model, and the first style text features include first text features and first style image features; The first feature optimization index data calculation module is used to calculate the first feature optimization index data between the first text feature and the first style image feature through the text generation model. A text generation model training module is used to train the text generation model using the first feature optimization index data; Specifically, the first feature optimization index data calculation module is used to sample the first text feature according to a first set step size to obtain the text feature at the current sampling time; add noise at the current time to the style text image sample data at the current sampling time to obtain a noisy image at the current sampling time; input the text feature at the current sampling time and the noisy image at the current sampling time into the semantic segmentation model of the text generation model to obtain a noisy text image at the current sampling time; and calculate the difference between the noisy text image at the current sampling time and the standard noisy text image as the first feature optimization index data. The text generation model training module is specifically used to obtain a noisy text image output by the text generation model after training the text generation model with the first feature optimization index data; input the noisy text image to the decoder of the image feature extraction module in the feature extraction module, so that the noisy text image can be decoded by the decoder of the image feature extraction module to obtain a first target style text image of a set dimension.

7. A text generation device, characterized in that, include: The module for acquiring text association data of style to be generated is used to acquire text association data of style to be generated; wherein, the text association data of style to be generated includes text image data of style to be generated and text data to be generated; The second feature extraction module is used to input the text association data of the style to be generated into the feature extraction module of the text generation model, so as to extract the second style text features of the text association data of the style to be generated through the feature extraction module; wherein, the second style text features include second text features and second style image features; The second target style text image generation module is used to generate a second target style text image based on the second text features and the second style image features through the text generation model. The text generation model is trained using the text generation model training method described in any one of claims 1-4.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the text generation model training method based on the diffusion model as described in any one of claims 1-4, or to implement the text generation method as described in claim 5.

9. A computer storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the text generation model training method based on the diffusion model as described in any one of claims 1-4, or the text generation method as described in claim 5.

Images