Image generation method, device, electronic device, medium and program product

Abstract

The present disclosure provides a method for generating an image, which can be applied to the field of artificial intelligence technology and the financial field. The method includes: obtaining a training data set, wherein the training data set includes one-to-one corresponding text data and picture data; extracting text features of the text data and picture features of the picture data; using the text features and picture features to train and obtain a picture generation model; calculating a first probability vector of the text features and a second probability vector of the picture features; fitting the first probability vector and the second probability vector, and updating the picture generation model using the text features and picture features corresponding to the minimum fitting value; inputting the text of the picture to be generated into the updated picture generation model to generate the picture. The picture generation method disclosed in the present disclosure can train a stable and reliable picture generation model by minimizing the fitting of the distribution information of the picture data features and the distribution information of the text data features.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology and the financial field, and specifically to a picture generation method, device, electronic device, medium and program product. Background Art

[0002] When Party A and Party B are communicating about a project or drafting a contract, including in the middle or even later stages of the project, constant changes and modifications may be required. Party A may not describe it clearly enough, which may lead to a misunderstanding between Party B and Party B. Or Party A may not know exactly what it wants and only give a general description, which may lead to frequent changes in project requirements. In addition, Party A may only give a vague framework and fail to give a detailed description of the project details. However, if Party A's requirements can be converted into pictures, they will appear intuitive and detailed.

[0003] There is no applicable image generation method and device in the prior art, so the efficiency of project communication is relatively low. There is an urgent need for an image generation method and device that assists project communication. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a method, device, electronic device, medium and program product for generating pictures to improve project communication efficiency, so as to at least partially solve the above technical problems.

[0005] According to a first aspect of the present disclosure, a method for generating an image is provided, including: obtaining a training data set, wherein the training data set includes one-to-one corresponding text data and picture data; extracting text features of the text data and picture features of the picture data; obtaining an image generation model by training the text features and the picture features; calculating a first probability vector of the text features and a second probability vector of the picture features; fitting the first probability vector and the second probability vector, and updating the image generation model by using the text features and the picture features corresponding to the minimum fitting value; and inputting the text of the image to be generated into the updated image generation model to generate the image.

[0006] According to an embodiment of the present disclosure, extracting text features of text data and image features of image data includes: increasing the number of convolutional layers of the convolutional neural network by adding multi-step residuals, and training to obtain an image feature extraction unit; extracting image features through the image feature extraction unit.

[0007] According to an embodiment of the present disclosure, extracting text features of text data and image features of image data also includes: using the outputs of a current position encoder and multiple previous position decoders as inputs of a current position decoder to train a text feature extraction unit; and extracting text features through the text feature extraction unit.

[0008] According to an embodiment of the present disclosure, fitting the first probability vector and the second probability vector, and updating the image generation model using the text features and picture features corresponding to the minimum fitting value include: calculating the difference between the first probability vector and the second probability vector; and selecting the text features and picture features corresponding to the probability vector whose difference is less than the feature value to update the image generation model.

[0009] According to an embodiment of the present disclosure, selecting text features and picture features corresponding to a probability vector whose difference value is less than a feature value to update the picture generation model includes:

[0010]

[0011] To obtain the probability vector of the difference being less than the eigenvalue, where p i Represents the probability vector distribution of image data, q i Represents the probability vector distribution of text data, i represents the i-th group of text-image data pairs, and N represents the number of text-image data pairs used for model training.

[0012] According to an embodiment of the present disclosure, obtaining a training data set includes: collecting source images and text descriptions corresponding to the source images; and filtering the source images and text descriptions to obtain a training data set.

[0013] According to an embodiment of the present disclosure, filtering source images and text descriptions includes: filtering source images according to any one of image format, image size, image aspect ratio and image content health regulations and their combinations; removing text descriptions as well as personal information and titles that do not comply with text content health regulations.

[0014] According to an embodiment of the present disclosure, filtering the source image and text description further includes: controlling the pixels of each dimension of the source image to be less than 500, and / or the ratio of the dimensions to be no greater than 2.

[0015] According to an embodiment of the present disclosure, the source image and the text description appear in pairs or are deleted.

[0016] A second aspect of the present disclosure provides a picture generation device, including: an acquisition module, used to acquire a training data set, wherein the training data set includes one-to-one corresponding text data and picture data; an extraction module, used to extract text features of the text data and picture features of the picture data; a training module, used to train a picture generation model using the text features and the picture features; a calculation module, used to calculate a first probability vector of the text features and a second probability vector of the picture features; a data distribution fitting module, used to fit the first probability vector and the second probability vector, and update the picture generation model using the text features and the picture features corresponding to the minimum fitting value; and a picture generation module, used to input the text of the picture to be generated into the updated picture generation model to generate the picture.

[0017] According to an embodiment of the present disclosure, the extraction module includes: a first acquisition unit, used to increase the number of convolutional layers of the convolutional neural network by adding multi-step residuals, and train to obtain a picture feature extraction unit; a picture feature extraction unit, used to extract picture features.

[0018] According to an embodiment of the present disclosure, the extraction module also includes: a second acquisition unit, used to use the output of the current position encoder and multiple previous position decoders as the input of the current position decoder to train a text feature extraction unit; and a text feature extraction unit, used to extract text features.

[0019] According to an embodiment of the present disclosure, the image generation device further includes: a collection submodule for collecting source images and text descriptions corresponding to the source images; and a filtering submodule for filtering the source images and text descriptions to obtain a training data set.

[0020] According to an embodiment of the present disclosure, the filtering submodule includes: an image filtering unit, which is used to filter the source image according to any one of the image format, image size, image aspect ratio and image content health regulations and their combinations; a text filtering unit, which is used to eliminate text descriptions, personal information and titles that do not comply with the text content health regulations.

[0021] According to an embodiment of the present disclosure, the filtering submodule further includes: a control unit, configured to control the number of pixels in each dimension of the source image to be less than 500, and / or the ratio of the dimensions to be no greater than 2.

[0022] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned image generation method.

[0023] The fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned image generation method.

[0024] The fifth aspect of the present disclosure also provides a computer program product, including a computer program, which implements the above-mentioned image generation method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0026] Figure 1A diagram schematically illustrates an application scenario of a method, apparatus, device, medium, and program product for generating a picture according to an embodiment of the present disclosure;

[0027] Figure 2 The flowchart of the picture generation method according to the embodiment of the present disclosure is schematically shown;

[0028] Figure 3 A flowchart of a method for obtaining a training data set according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 4 The flowchart of the source data filtering method according to the embodiment of the present disclosure is schematically shown;

[0030] Figure 5 The flowchart of the method for extracting image features according to the embodiment of the present disclosure is schematically shown;

[0031] Figure 6 The structure diagram of the CNN convolutional layer according to the embodiment of the present disclosure is schematically shown;

[0032] Figure 7 The flowchart of the text feature extraction method according to the embodiment of the present disclosure is schematically shown;

[0033] Figure 8 A flowchart of a probability vector fitting method according to an embodiment of the present disclosure is schematically shown;

[0034] Fig. 9 The structure block diagram of the image generating device according to the embodiment of the present disclosure is schematically shown;

[0035] Fig.10 The structure block diagram of the acquisition module according to the embodiment of the present disclosure is schematically shown;

[0036] Fig.11 The structure block diagram of the filtering submodule according to the embodiment of the present disclosure is schematically shown;

[0037] Fig.12 The structure block diagram of the extraction module according to the embodiment of the present disclosure is schematically shown;

[0038] Fig.13 A structural block diagram schematically shows a picture generating device according to another embodiment of the present disclosure; and

[0039] Fig.14 A block diagram of an electronic device suitable for implementing the picture generating method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0040] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0041] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0042] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0043] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0044] The embodiments of the present disclosure provide a method, device, equipment, medium and program product for generating an image, which can be used in the financial field or other fields. It should be noted that the method, device, equipment, medium and program product for generating an image of the present disclosure can be used in the financial field, and can also be used in any field other than the financial field, and the application field of the method, device, equipment, medium and program product of the present disclosure is not limited.

[0045] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0046] Figure 1 The application scenario diagram of the picture generation method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0047] According to the embodiments of the present disclosure, Figure 1As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0048] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0049] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0050] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0051] It should be noted that the image generation method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the image generation device provided in the embodiment of the present disclosure can generally be set in the server 105. The image generation method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image generation device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0052] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0053] The following will be based on Figure 1 The scene described by Figure 2~Figure 8 The picture generating method of the disclosed embodiment is described in detail.

[0054] Figure 2 The flowchart of the picture generating method according to the embodiment of the present disclosure is schematically shown.

[0055] like Figure 2 As shown, an embodiment of the present disclosure provides a method for generating an image, for example, comprising:

[0056] S210, obtaining a training data set, wherein the training data set includes one-to-one corresponding text data and image data.

[0057] Figure 3 A flow chart of a method for obtaining a training data set according to an embodiment of the present disclosure is schematically shown.

[0058] According to the embodiments of the present disclosure, Figure 3 As shown, for example, a training data set is obtained through operations S211 to S212.

[0059] S211, collecting source images and text descriptions corresponding to the source images.

[0060] According to an embodiment of the present disclosure, for example, images and corresponding text descriptions are crawled from business-related web pages, and each image needs to have a corresponding text description. The original images and text descriptions are obtained by crawling related web pages to form a set of image-text data pairs. It is not necessary to collect data again every time, and the previous data can be reused, and the new data can be directly merged with the old data.

[0061] S212, filtering the source images and text descriptions to obtain a training data set.

[0062] Figure 4 The flowchart of the source data filtering method according to the embodiment of the present disclosure is schematically shown.

[0063] According to the embodiments of the present disclosure, Figure 4 As shown, for example, source images and text descriptions are filtered through operations S2121 to S2123.

[0064] S2121, filter the source image according to any one of the image format, image size, image aspect ratio and image content health regulations and their combination.

[0065] S2122, remove text descriptions, personal information and titles that do not meet the text content health regulations.

[0066] According to an embodiment of the present disclosure, in order to improve the quality of source data for training convolutional neural networks, after collecting pictures and corresponding text descriptions, it is necessary to filter the pictures and text descriptions in order to obtain original training data pairs that meet the requirements. For example, filtering is performed based on the encoding format, size, aspect ratio, and whether the content of the picture is healthy. At the same time, the text is filtered to remove unhealthy language, personal information, titles, and other text. It should be noted that pictures and text appear in pairs. When a picture or text is determined to be not in compliance with the requirements, the picture-text data pair will be deleted together.

[0067] S2123, control the pixels of each dimension of the source image to be less than 500, and / or the ratio of the dimensions to be no greater than 2.

[0068] According to an embodiment of the present disclosure, in order to train the convolutional neural network more accurately and quickly, the pixels of each image should not be too large, the pixels of each dimension are, for example, less than 500, and the ratio of the dimensions does not exceed 2.

[0069] S220, extracting text features of the text data and image features of the image data.

[0070] Figure 5 The flowchart of the method for extracting image features according to an embodiment of the present disclosure is schematically shown.

[0071] According to the embodiments of the present disclosure, Figure 5 As shown, for example, image features are extracted through operations S221 to S222.

[0072] S221, increasing the number of convolutional layers of the convolutional neural network by adding multi-step residuals, and training to obtain an image feature extraction unit.

[0073] Figure 6 The figure schematically shows a CNN convolutional layer structure diagram according to an embodiment of the present disclosure.

[0074] According to an embodiment of the present disclosure, for example, a CNN (convolutional neural network) is used to extract image features from the obtained text image data pair. In order to improve the accuracy of the extracted image features, Figure 6 As shown in the figure, for example, by adding multi-step residuals, the CNN network can have more convolutional layers, and the image features can be extracted through a deeper network to alleviate the problem of gradient disappearance. Figure 6It is only a partial schematic diagram of the convolutional layer network structure. In actual situations, the number of network layers is much larger. The present disclosure, for example, uses a 3*3 convolution kernel, and after the convolution, for example, a ReLU activation function is connected, and finally a pooling layer with average pooling is connected, and after the pooling layer, a fully connected layer is connected to output a feature vector of a specified dimension, that is, the number of neurons in the last layer of the fully connected layer represents the dimension of the feature vector. The image feature extraction module used to extract image features needs to be pre-trained. During training, for example, the Adam optimizer is used to update the network parameters, wherein the updated parameters include, for example, the value of the convolution kernel and the parameters of the fully connected layer.

[0075] S222, extracting picture features through a picture feature extraction unit.

[0076] Figure 7 The flowchart of the text feature extraction method according to the embodiment of the present disclosure is schematically shown.

[0077] According to the embodiments of the present disclosure, Figure 7 As shown, for example, text features are extracted through operations S223 to S224.

[0078] S223, using the outputs of the current position encoder and multiple previous position decoders as inputs of the current position decoder, and training to obtain a text feature extraction unit.

[0079] According to an embodiment of the present disclosure, for example, text features of text are extracted from the obtained text-image data pair through BERT (a general language model based on a bidirectional Transformer, which captures long-distance information based on a self-attention mechanism and generates word vectors at the text semantic level). In order to make full use of historical text information, the present disclosure provides a new Transformer structure, for example, including multiple encoders and decoders, and the input of the decoder at the current position is the output of the encoder at the current position plus the output of multiple decoders before the current position, so as to more effectively extract text features. The text feature extraction module for extracting text features is, for example, composed of multiple such Transformers stacked together. The text feature extraction module needs to be trained in advance, and the parameters updated during training are the parameters in each Transformer.

[0080] S224, extracting text features through a text feature extraction unit.

[0081] S230, training a picture generation model using text features and picture features.

[0082] S240, calculating a first probability vector of the text feature and a second probability vector of the image feature.

[0083] According to an embodiment of the present disclosure, for example, two layers of neural networks are connected after the image feature extraction module and the text feature extraction module, respectively, and the two layers of neural networks can splice the feature vectors of the image and the text and convert them into distribution information. After the image and text feature vectors are spliced ​​and converted by the two layers of neural networks, the probability vectors of the image and the text can be calculated, and the values ​​in the vectors are probability values, which represent the distribution information.

[0084] S250, fitting the first probability vector and the second probability vector, and updating the image generation model using the text features and image features corresponding to the minimum fitting value.

[0085] Figure 8 The flowchart of the probability vector fitting method according to the embodiment of the present disclosure is schematically shown.

[0086] According to the embodiments of the present disclosure, Figure 8 As shown, for example, the probability vector is fitted through operations S251 to S252.

[0087] S251, calculating the difference between the first probability vector and the second probability vector.

[0088] S252, select text features and picture features corresponding to the probability vector whose difference is less than the eigenvalue to update the picture generation model.

[0089] According to the embodiments of the present disclosure, the minimum fitting value of the two probability vectors is obtained, that is, the distribution information of the image and text is minimized. If the distribution information of the two is a two-dimensional vector, each vector will have a position in the coordinate system, the vector of the image will have a clustering area, and the vector of the text will also have a clustering area. Let the two clustering areas overlap as much as possible, that is, when the difference between the two probability vectors is less than an eigenvalue, their distribution information is minimized. High-dimensional vectors are comparisons in high-dimensional space, and two-dimensional vectors can be regarded as a miniature of high-dimensional vectors. For example, the objective function

[0090] (1)

[0091] To obtain the probability vector of the difference being less than the eigenvalue, where p i Represents the probability vector distribution of image data, q irepresents the probability vector distribution of text data, i represents the i-th group of text-image data pairs, and N represents the number of text-image data pairs used for model training. Formula (1) can be used to obtain multiple groups of image-text data pairs with the closest probability vector values. These image-text data pairs are used to train the model and update the parameters of the neural network to make the image and text data closely associated. The smaller the distribution difference between the image and text data, the greater their association, and the better the effect of the trained model. For example, the self-attention mechanism is used to model the training model to ensure that each input image character is associated with all input text characters.

[0092] S260: Input the text of the image to be generated into the updated image generation model to generate the image.

[0093] According to an embodiment of the present disclosure, after obtaining an image generation model through minimizing distribution difference training, a piece of text is input into the model, and the model can automatically generate multiple semantically related images according to the description of the text.

[0094] In summary, the present disclosure provides a method for generating an image, which extracts the image feature vector and the text feature vector from the acquired image and text data pair through the image feature extraction module and the text feature extraction module respectively, and minimizes the difference between the two distribution information after converting the two feature vectors into distribution information respectively, so as to train or update a stable and reliable image generation model, and obtain better output results than the model trained based on the method of calculating the similarity of feature data. Through the image generation method and device disclosed in the present disclosure, the customer needs described in text can be converted into images, so as to achieve clearer and more specific needs docking and avoid frequent communication.

[0095] Based on the above-mentioned image generation method, the present disclosure also provides an image generation device. Fig. 9 The device is described in detail.

[0096] Fig. 9 The structural block diagram of the image generating device according to an embodiment of the present disclosure is schematically shown.

[0097] According to the embodiments of the present disclosure, Fig. 9 As shown, the image generating device 600 of this embodiment includes, for example, an acquisition module 610 , an extraction module 620 , a training module 630 , a calculation module 640 , a data distribution fitting module 650 and an image generating module 660 .

[0098] The acquisition module 610 is used to acquire a training data set, wherein the training data set includes text data and image data in one-to-one correspondence. In one embodiment, the acquisition module 610 can be used to perform the operation S210 described above, which will not be described in detail here.

[0099] The extraction module 620 is used to extract text features of text data and image features of image data. In one embodiment, the extraction module 620 can be used to perform the operation S220 described above, which will not be described in detail here.

[0100] The training module 630 is used to train the image generation model using the text features and the image features. In one embodiment, the training module 630 can be used to perform the operation S230 described above, which will not be described in detail here.

[0101] The calculation module 640 is used to calculate the first probability vector of the text feature and the second probability vector of the image feature. In one embodiment, the calculation module 640 can be used to perform the operation S240 described above, which will not be described in detail here.

[0102] The data distribution fitting module 650 is used to fit the first probability vector and the second probability vector, and update the image generation model using the text features and image features corresponding to the minimum fitting value. In one embodiment, the data distribution fitting module 650 can be used to perform the operation S250 described above, which will not be repeated here.

[0103] The picture generation module 660 is used to input the text of the picture to be generated into the updated picture generation model to generate the picture. In one embodiment, the picture generation module 660 can be used to perform the operation S260 described above, which will not be repeated here.

[0104] According to an embodiment of the present disclosure, any multiple modules of the acquisition module 610, the extraction module 620, the training module 630, the calculation module 640, the data distribution fitting module 650 and the image generation module 660 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 610, the extraction module 620, the training module 630, the calculation module 640, the data distribution fitting module 650 and the image generation module 660 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 610, the extraction module 620, the training module 630, the calculation module 640, the data distribution fitting module 650 and the image generation module 660 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is executed.

[0105] Fig.10 The structural block diagram of the acquisition module according to the embodiment of the present disclosure is schematically shown.

[0106] According to the embodiments of the present disclosure, Fig.10 As shown, the acquisition module 610 includes, for example, a collection submodule 611 and a filtering submodule 612 .

[0107] The collection submodule 611 is used to collect source images and text descriptions corresponding to the source images. In one embodiment, the collection submodule 611 can be used to perform the operation S211 described above, which will not be described in detail here.

[0108] The filtering submodule 612 is used to filter the source images and text descriptions to obtain a training data set. In one embodiment, the filtering submodule 612 can be used to perform the operation S212 described above, which will not be described in detail here.

[0109] Fig.11 The structural block diagram of the filtering submodule according to an embodiment of the present disclosure is schematically shown.

[0110] According to the embodiments of the present disclosure, Fig.11 As shown, the filtering submodule 612 includes, for example, a picture filtering unit 6121 , a text filtering unit 6122 and a control unit 6123 .

[0111] The image filtering unit 6121 is used to filter the source image according to any one of the image format, image size, image aspect ratio and image content health regulations and their combination. In one embodiment, the image filtering unit 6121 can be used to perform the operation S2121 described above, which will not be repeated here.

[0112] The text filtering unit 6122 is used to remove text descriptions, personal information and titles that do not meet the text content health regulations. In one embodiment, the text filtering unit 6122 can be used to perform the operation S2122 described above, which will not be repeated here.

[0113] The control unit 6123 is used to control the pixels of each dimension of the source image to be less than 500, and / or the ratio of the dimensions to be no greater than 2. In one embodiment, the control unit 6123 may be used to perform the operation S2123 described above, which will not be described in detail herein.

[0114] Fig.12 The structural block diagram of the extraction module according to the embodiment of the present disclosure is schematically shown.

[0115] According to the embodiments of the present disclosure, Fig.12 As shown, the extraction module 620 includes, for example, a first acquisition unit 621 , a picture feature extraction unit 622 , a second acquisition unit 623 and a text feature extraction unit 624 .

[0116] The first acquisition unit 621 is used to increase the number of convolutional layers of the convolutional neural network by adding multi-step residuals, and train to obtain an image feature extraction unit. In one embodiment, the first acquisition unit 621 can be used to perform the operation S221 described above, which will not be repeated here.

[0117] The picture feature extraction unit 622 is used to extract picture features. In one embodiment, the picture feature extraction unit 622 can be used to perform the operation S222 described above, which will not be described in detail here.

[0118] The second acquisition unit 623 is used to train a text feature extraction unit using the outputs of the current position encoder and multiple previous position decoders as inputs of the current position decoder. In one embodiment, the second acquisition unit 623 can be used to perform the operation S223 described above, which will not be repeated here.

[0119] The text feature extraction unit 624 is used to extract text features. In one embodiment, the text feature extraction unit 624 can be used to perform the operation S224 described above, which will not be described in detail here.

[0120] Fig.13 The structural block diagram of a picture generating device according to another embodiment of the present disclosure is schematically shown.

[0121] According to the embodiments of the present disclosure, Fig.13 As shown, the image generation device includes, for example, a data processing module and a data distribution fitting module. The data processing module includes, for example, an image feature extraction unit and a text feature extraction unit, which are used to extract image features and text features from the input image-text data pair, respectively. After the image features and text features are converted into distribution information in the data distribution fitting module, the difference between the two distributions is minimized, and finally the image generation model is trained. The image generation device can store structured data and model data based on a big data platform.

[0122] The unfinished details in the product example section are similar to those in the method example section. Please refer to the method example section and will not be repeated here.

[0123] Fig.14 A block diagram of an electronic device suitable for implementing the picture generating method according to an embodiment of the present disclosure is schematically shown.

[0124] like Fig.14 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 to a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0125] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0126] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0127] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0128] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0129] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the item recommendation method provided by the embodiment of the present disclosure.

[0130] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 901. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0131] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0132] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0133] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0134] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0136] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for generating an image, comprising: Acquire a training data set, wherein the training data set includes one-to-one corresponding text data and image data; Extracting text features of the text data and image features of the image data; Using the text features and the image features, and adopting a self-attention mechanism to train an image generation model; Calculating a first probability vector of the text feature and a second probability vector of the image feature; Fitting the first probability vector and the second probability vector, and updating the picture generation model using the text feature and the picture feature corresponding to the minimum fitting value; The text of the image to be generated is input into the updated image generation model to generate the image.

2. The image generation method according to claim 1, characterized in that: The extracting of the text features of the text data and the image features of the image data comprises: By adding multi-step residuals, the number of convolutional layers of the convolutional neural network is increased, and the image feature extraction unit is trained; The picture feature is extracted by the picture feature extraction unit.

3. The image generation method according to claim 1, characterized in that: The extracting of the text features of the text data and the image features of the image data further includes: The outputs of the current position encoder and multiple previous position decoders are used as inputs of the current position decoder to train a text feature extraction unit; The text feature is extracted by the text feature extraction unit.

4. The image generation method according to claim 1, characterized in that: The fitting of the first probability vector and the second probability vector, and updating the picture generation model using the text feature and the picture feature corresponding to the minimum fitting value comprises: Calculating a difference between the first probability vector and the second probability vector; The text feature and the picture feature corresponding to the probability vector whose difference value is less than the feature value are selected to update the picture generation model.

5. The image generation method according to claim 4, characterized in that: The selecting the text feature and the picture feature corresponding to the probability vector whose difference value is less than the feature value to update the picture generation model comprises: Through the objective function To obtain the probability vector that the difference is less than the eigenvalue, where p i Represents the probability vector distribution of image data, q i Represents the probability vector distribution of text data, i represents the i-th group of text-image data pairs, and N represents the number of text-image data pairs used for model training.

6. The image generation method according to claim 1, characterized in that: The obtaining of the training data set comprises: Collect source images and text descriptions corresponding to the source images; The source image and the text description are filtered to obtain the training data set.

7. The image generation method according to claim 6, characterized in that: The filtering of the source image and the text description includes: Filter the source images according to any one of image format, image size, image aspect ratio and image content health regulations, or a combination thereof; Remove the text descriptions, personal information and titles that do not meet the text content family regulations.

8. The image generation method according to claim 6, characterized in that: The filtering of the source image and the text description further comprises: The pixels of each dimension of the source image are controlled to be less than 500, and / or the ratio of the dimensions is controlled to be no greater than 2.

9. The image generation method according to claim 6, characterized in that: The source image and the text description appear in pairs or are deleted.

10. A picture generating device, comprising: An acquisition module, used to acquire a training data set, wherein the training data set includes one-to-one corresponding text data and image data; An extraction module, used to extract text features of the text data and image features of the image data; A training module, used to utilize the text features and the image features and adopt a self-attention mechanism to train and obtain an image generation model; A calculation module, used for calculating a first probability vector of the text feature and a second probability vector of the image feature; a data distribution fitting module, configured to fit the first probability vector and the second probability vector, and update the picture generation model using the text feature and the picture feature corresponding to the minimum fitting value; and The picture generation module is used to input the text of the picture to be generated into the updated picture generation model to generate the picture.

11. The image generating device according to claim 10, characterized in that: The extraction module comprises: A first acquisition unit is used to increase the number of convolutional layers of the convolutional neural network by adding multi-step residuals, and train to obtain an image feature extraction unit; The image feature extraction unit is used to extract the image features.

12. The image generating device according to claim 10, characterized in that: The extraction module also includes: A second acquisition unit is used to train a text feature extraction unit by using the outputs of the current position encoder and a plurality of previous position decoders as inputs of the current position decoder; The text feature extraction unit is used to extract the text features.

13. The image generating device according to claim 10, characterized in that: The acquisition module comprises: A collection submodule, used to collect source images and text descriptions corresponding to the source images; The filtering submodule is used to filter the source image and the text description to obtain the training data set.

14. The image generating device according to claim 13, characterized in that: The filtering submodule comprises: An image filtering unit, configured to filter the source image according to any one of image format, image size, image aspect ratio and image content health regulations, and a combination thereof; The text filtering unit is used to remove the text descriptions, personal information and titles that do not comply with the text content health regulations.

15. The image generating device according to claim 13, characterized in that: The filtering submodule also includes: A control unit is used to control the pixels of each dimension of the source image to be less than 500 and / or the ratio of the dimensions to be no greater than 2.

16. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the image generation method according to any one of claims 1 to 9.

17. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the image generation method according to any one of claims 1 to 9.

18. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image generation method according to any one of claims 1 to 9 is implemented.

Images