Image generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image generation method and device, electronic equipment and a storage medium, belongs to the technical field of image processing, and is suitable for the field of financial science and technology. The method comprises the following steps: acquiring a sample description text and a sample image; wherein the sample description text is used for describing a sample image; performing constraint information extraction on the sample description text to obtain constraint information features; performing feature coding on the constraint information features to obtain constraint information codes; performing text coding on the sample description text based on a preset image generation model to obtain a description text code; training a preset image generation model according to the constraint information code, the description text code and the sample image to obtain a target image generation model; and obtaining a target description text, and performing image generation on the target description text based on the target image generation model. According to the embodiment of the invention, the accuracy of the image generated based on the text can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, applicable to the field of financial technology, and in particular to an image generation method and device, an electronic device and a storage medium. Background Art

[0002] Generating images based on text means generating images that match the description by inputting text descriptions. For example, in a financial product marketing campaign, creative redemption posters suitable for the campaign are generated by inputting text descriptions such as product names, points information, and redemption rules. The technology of generating images based on text is usually implemented through network models such as DALL-E2, but because network models such as DALL-E2 may freely interpret text descriptions, the generated images are not accurate enough. Therefore, how to improve the accuracy of images generated based on text has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an image generation method and device, an electronic device and a storage medium, aiming to improve the accuracy of images generated based on text.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides an image generation method, the method comprising:

[0005] Acquire a sample description text and a sample image; wherein the sample description text is used to describe the sample image;

[0006] Extracting constraint information from the sample description text to obtain constraint information features;

[0007] Performing feature encoding on the constraint information feature to obtain a constraint information code;

[0008] Performing text encoding on the sample description text based on a preset image generation model to obtain a description text encoding;

[0009] The preset image generation model is trained according to the constraint information code, the description text code and the sample image to obtain a target image generation model;

[0010] Obtain target description text, and generate an image of the target description text based on the target image generation model.

[0011] In some embodiments, the step of training the preset image generation model according to the constraint information code, the description text code and the sample image to obtain a target image generation model includes:

[0012] Performing image encoding on the sample image based on the preset image generation model to obtain a sample image code;

[0013] Based on the preset image generation model, the description text code and the constraint information code are aggregated and calculated to obtain a constraint description aggregate code;

[0014] The preset image generation model is trained according to the sample image, the constraint description aggregation code and the sample image code to obtain the target image generation model.

[0015] In some embodiments, the step of training the preset image generation model according to the sample image, the constraint description aggregate code, and the sample image code to obtain the target image generation model includes:

[0016] Based on the preset image generation model, a priori feature extraction is performed on the sample image code and the constraint description aggregate code to obtain image text prior features;

[0017] Performing image generation on the image text prior features based on the preset image generation model to obtain a predicted image;

[0018] The preset image generation model is trained based on the predicted image and the sample image to obtain the target image generation model.

[0019] In some embodiments, the step of training the preset image generation model based on the predicted image and the sample image to obtain the target image generation model includes:

[0020] Performing image loss calculation according to the sample image and the predicted image to obtain image loss data;

[0021] Performing semantic loss calculation based on the sample image and the predicted image to obtain semantic loss data;

[0022] The preset image generation model is trained based on the semantic loss data and the image loss data to obtain a target image generation model.

[0023] In some embodiments, the performing semantic loss calculation according to the sample image and the predicted image to obtain semantic loss data includes:

[0024] Performing semantic extraction on the sample image to obtain sample semantic information;

[0025] Performing semantic extraction on the predicted image to obtain predicted semantic information;

[0026] The cosine similarity calculation is performed on the sample semantic information and the predicted semantic information to obtain the semantic loss data.

[0027] In some embodiments, extracting constraint information from the sample description text to obtain constraint information features includes:

[0028] Performing knowledge domain identification on the sample description text to obtain text knowledge domain information;

[0029] Screening a preset regular matching template library according to the text knowledge domain information to obtain a target regular matching template;

[0030] The constraint information of the sample description text is identified according to the target regular matching template to obtain the constraint information feature.

[0031] In some embodiments, the step of identifying constraint information on the sample description text according to the target regular matching template to obtain the constraint information feature includes:

[0032] Performing word segmentation on the sample description text to obtain a text word sequence;

[0033] Performing part-of-speech tagging on the text word sequence to obtain a word part-of-speech sequence;

[0034] Keyword matching is performed on the word part-of-speech sequence according to the target regular matching template to obtain the constraint information feature.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image generating device, the device comprising:

[0036] A data acquisition module, used to acquire a sample description text and a sample image; wherein the sample description text is used to describe the sample image;

[0037] An information extraction module, used to extract constraint information from the sample description text to obtain constraint information features;

[0038] A feature encoding module, used for performing feature encoding on the constraint information feature to obtain constraint information code;

[0039] A text encoding module, used for performing text encoding on the sample description text based on a preset image generation model to obtain a description text encoding;

[0040] A model training module, used for training the preset image generation model according to the constraint information code, the description text code and the sample image to obtain a target image generation model;

[0041] The image generation module is used to obtain target description text and generate an image of the target description text based on the target image generation model.

[0042] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0044] The present application proposes an image generation method and device, an electronic device and a storage medium, which obtain a sample image and a sample description text describing the sample image, then extract constraint information from the sample description text to obtain constraint information features, then feature encode the constraint information features to obtain constraint information codes, and at the same time, perform text encoding on the sample description text based on a preset image generation model to obtain description text codes, and finally train the preset image generation model according to the constraint information codes, description text codes and sample images, so that during the training process, the preset image generation model can not only perceive how to generate sample images from the description text codes, but also enhance the perception ability of the preset image generation model to perceive the constraint information in the sample description text, so as to obtain a target image generation model that can accurately generate images according to the text and the information displayed by the image will not exceed the constraint information; further, the target description text is obtained, and image generation is performed on the target description text based on the target image generation model, so as to achieve accurate text-based image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of an image generation method provided in an embodiment of the present application;

[0046] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0047] Figure 3 yes Figure 2 Flow chart of step S203 in FIG.

[0048] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0049] Figure 5 yes Figure 3 Flow chart of step S403 in FIG.

[0050] Figure 6 yes Figure 5 Flow chart of step S503 in FIG.

[0051] Figure 7 yes Figure 6 Flowchart of step S602 in FIG.

[0052] Figure 8 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] First, some nouns involved in this application are analyzed:

[0058] Image Encoding: In the field of artificial intelligence and computer vision, especially in neural network-based applications, image encoding refers to the process of converting image data into feature representations through neural networks. The main purpose of image encoding is to extract key information from an image and convert it into a more concise and abstract form, which is usually used for subsequent image analysis and processing tasks such as image classification, object recognition, and image retrieval. These features encode the visual content of the image, allowing computers to understand and process image data more efficiently. For example, in facial recognition systems, image encoding extracts individual facial features through neural networks for verification or identification; in self-driving cars, image encoding helps the vehicle understand its surroundings, such as the location of pedestrians, road signs, and other vehicles.

[0059] Text Encoding: In the field of artificial intelligence and natural language processing, especially in neural network-based applications, text encoding refers to the process of converting text data into feature representations through neural networks. The core goal of text encoding is to extract semantic and structural information from text and convert it into a mathematical expression, namely a feature vector, so that the machine can more effectively handle complex language tasks such as text classification, sentiment analysis, and machine translation. For example, in sentiment analysis, text encoding helps the model understand the emotional tendency of user comments; in chatbots, it enables the robot to understand and generate natural conversations.

[0060] Semantic Loss: In the field of artificial intelligence and machine learning, especially in the process of model training, semantic loss is a loss function that evaluates the performance of the model in understanding the semantic level of data. This concept belongs to the intersection of computer science and statistics and is an important branch of artificial intelligence. Semantic loss is mainly used to guide deep learning models, such as convolutional neural networks or recurrent neural networks, to more accurately capture and reproduce the complex semantic structure in the data. In practical applications, such as technical fields such as natural language processing, image recognition or speech recognition, semantic loss helps optimize the model output to ensure its semantic correctness and coherence. For example, in machine translation, semantic loss can ensure that the translation is not only literally correct, but also culturally and contextually consistent; in image description generation, semantic loss is used to ensure that the generated text description accurately reflects the meaning of the image content. By combining traditional loss functions with semantic loss, the performance and accuracy of the model in processing high-level semantic tasks can be significantly improved.

[0061] Generating images based on text means generating images that match the description by inputting text descriptions. For example, in a financial product marketing campaign, creative redemption posters suitable for the campaign are generated by inputting text descriptions such as product names, points information, and redemption rules. The technology of generating images based on text is usually implemented through network models such as DALL-E2, but because network models such as DALL-E2 may freely interpret text descriptions, the generated images are not accurate enough. Therefore, how to improve the accuracy of images generated based on text has become an urgent problem to be solved.

[0062] Based on this, the embodiments of the present application provide an image generation method and device, an electronic device and a storage medium, aiming to improve the accuracy of images generated based on text.

[0063] An image generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the image generation method in the embodiments of the present application is described.

[0064] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0066] The image generation method provided in the embodiment of the present application relates to the field of image processing technology and is applicable to the field of financial technology. The image generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image generation method, etc., but is not limited to the above forms.

[0067] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0068] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0069] Figure 1 is an optional flowchart of the image generation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0070] Step S101, obtaining a sample description text and a sample image; wherein the sample description text is used to describe the sample image;

[0071] Step S102, extracting constraint information from the sample description text to obtain constraint information features;

[0072] Step S103, feature encoding the constraint information feature to obtain the constraint information code;

[0073] Step S104, performing text encoding on the sample description text based on a preset image generation model to obtain a description text encoding;

[0074] Step S105, training a preset image generation model according to the constraint information code, the description text code and the sample image to obtain a target image generation model;

[0075] Step S106, obtaining target description text, and performing image generation on the target description text based on the target image generation model.

[0076] Steps S101 to S106 shown in the embodiment of the present application are as follows: by acquiring a sample image and a sample description text that describes the sample image, then extracting constraint information from the sample description text to obtain constraint information features, then feature encoding the constraint information features to obtain constraint information codes, and at the same time text encoding the sample description text based on a preset image generation model to obtain description text codes, and finally training the preset image generation model based on the constraint information codes, description text codes and sample images, so that during the training process, the preset image generation model can not only perceive how to generate sample images from the description text codes, but also enhance the perception ability of the preset image generation model to perceive the constraint information in the sample description text, so as to obtain a target image generation model that can accurately generate images based on the text and the information displayed by the image will not exceed the constraint information; further, the target description text is acquired, and image generation is performed on the target description text based on the target image generation model, so as to achieve accurate text-based image generation.

[0077] In step S101 of some embodiments, the sample image is a financial product marketing image, such as a promotional poster of a product in a financial product marketing activity, which includes elements such as a specific product name, points information or exchange rules. The sample description text is text information corresponding to the sample image and is used to describe the content, scene and related details of the image, such as: points exchange rules, specifying that points can be exchanged for a movie ticket, and the activity is valid until the end of the year.

[0078] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S203:

[0079] Step S201, performing knowledge domain recognition on the sample description text to obtain text knowledge domain information;

[0080] Step S202, screening a preset regular matching template library according to text knowledge domain information to obtain a target regular matching template;

[0081] Step S203, identifying constraint information of the sample description text according to the target regular matching template to obtain constraint information features.

[0082] In steps S201 to S203 shown in the embodiment of the present application, knowledge domain information of the text is obtained by performing knowledge domain identification on the sample description text, thereby extracting the core information in the text and classifying it, and then screening the preset regular matching template library according to the text knowledge domain information to obtain the target regular matching template, and finally, constraint information of the sample description text is identified according to the target regular matching template to obtain the constraint information feature, so as to further identify the constraint information of the sample description text according to the template corresponding to the knowledge domain of the sample description text, thereby extracting accurate constraint information from the sample description text, providing a constraint information basis for the subsequent generation of accurate images based on the constraint information, so that the generated image will not be freely interpreted and conforms to the content described in the sample description text.

[0083] In step S201 of some embodiments, knowledge domain identification is the process of automatically extracting information related to a specific field from the sample description text. Knowledge domain identification can determine the knowledge field to which the sample description text belongs by a rule-based method, a machine learning-based method, or a knowledge graph-based method. The text knowledge domain information is a label value used to indicate the knowledge field to which the sample description text belongs. For example, the sample description text is "Points can be redeemed for selected products, including electronic products, household items, and redemption coupons. Every 1,000 points can be redeemed for one ticket, and the activity ends on December 31st." Then the knowledge graph-based method determines that the field to which the sample description text belongs is a financial product marketing activity.

[0084] In step S202 of some embodiments, a plurality of regular matching templates are included in the preset regular matching template library, and each regular matching template corresponds to a text knowledge domain information. For example, the text knowledge domain information is "financial product marketing activities", and the regular matching template corresponding to "financial product marketing activities" is obtained in the preset regular matching template library according to "financial product marketing activities", and the target regular matching template is obtained. The target regular matching template is a predefined rule for extracting a specific type of information from the input text, and is composed of one or more regular expressions for matching specific content in the text. For example, the regular expression for matching the date can be \d{4}-\d{2}-\d{2}, which is used to identify the date format such as "XXXX-XX-XX" (year-month-day).

[0085] See also Figure 3 In some embodiments, step S203 may include but is not limited to steps S301 to S303:

[0086] Step S301, segmenting the sample description text into words to obtain a text word sequence;

[0087] Step S302, performing part-of-speech tagging on the text word sequence to obtain a word part-of-speech sequence;

[0088] Step S303: perform keyword matching on the word part-of-speech sequence according to the target regular matching template to obtain constraint information features.

[0089] In steps S301 to S303 shown in the embodiment of the present application, the sample description text is segmented to obtain a text word sequence, and then the text word sequence is tagged with parts of speech to obtain a word part-of-speech sequence. Finally, the word part-of-speech sequence is keyword matched according to the target regular matching template to obtain constraint information features. In this way, the accuracy of information extraction is improved through the combination of part-of-speech tagging and regular matching. At the same time, the regular matching template provides constraints for the model, reduces the model's over-interpretation of the text description, ensures the accuracy of the generated image, and also achieves consistency between the generated image and the information in the text task. The information in the generated image must come from the keyword information in the text.

[0090] In step S301 of some embodiments, the text word sequence refers to a sequential set of words obtained by dividing the sample description text according to words. For example, the sample description text is "By participating in this activity, users can use points to redeem selected products.", and the text word sequence is "["through", "participate", "this", "activity", ",","user", "can", "use", "points", "redeem", "selected", "product"]".

[0091] In step S302 of some embodiments, part-of-speech tagging is to add a label to the text word sequence, indicating the grammatical category of the word in the sentence, and the part-of-speech label generally includes different grammatical categories such as noun, verb, adjective, adverb or pronoun. The word part-of-speech sequence is to add a grammatical category to each word on the basis of the text word sequence, forming a word grammatical category pair, for example ("product", "noun"). In one embodiment, the part-of-speech tagging is based on statistical part-of-speech tagging using a hidden Markov model to perform part-of-speech tagging according to the context.

[0092] In step S303 of some embodiments, keyword matching is to identify and extract keywords that match the target regular matching template from the text. For example, the target regular matching template is "(running|swimming|yoga|cycling|fitness)\s*(daily|weekly|monthly)\s*(at least|many times|several times)\s*(minutes|hours)\s*(improve|enhance|help)\s*(health|physical strength|endurance|flexibility)", and the word part-of-speech sequence is (weekly (time), at least (adverb), three times (quantifier), running (noun), help (verb), enhance (verb), physical strength (noun), endurance (noun), improve (verb), overall (adjective), health status (noun)). Then, based on the regular expression in the target regular matching template, the word part-of-speech sequence is regularly matched to obtain the constraint information feature: "Exercise type: running, exercise frequency: weekly, number of exercises: at least three times, improvement goals: improve physical strength, endurance, and improve health".

[0093] In step S103 of some embodiments, feature encoding refers to converting the constraint information feature into a vector representation, and after feature encoding the constraint information feature, a vector is obtained, that is, the constraint information code. In one embodiment, the constraint information feature is feature encoded by a pre-trained BERT model, and the constraint information feature is input into the BERT model to obtain the constraint information code.

[0094] In step S104 of some embodiments, text encoding refers to converting the sample description text into a vector representation, extracting key information in the sample description text and converting it into a high-dimensional vector to obtain the description text encoding. In one embodiment, the preset image generation model is the DALL-E2 model, and the DALL-E2 model includes a CLIP text encoder. The sample description text is text encoded based on the CLIP text encoder to obtain the description text encoding.

[0095] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S403:

[0096] Step S401, performing image encoding on a sample image based on a preset image generation model to obtain a sample image encoding;

[0097] Step S402, performing aggregation calculation on the description text code and the constraint information code based on a preset image generation model to obtain a constraint description aggregate code;

[0098] Step S403, training a preset image generation model according to the sample image, the constraint description aggregation code and the sample image code to obtain a target image generation model.

[0099] In steps S401 to S403 shown in the embodiment of the present application, the sample image is encoded based on the preset image generation model to obtain the sample image code, and then the description text code and the constraint information code are aggregated based on the preset image generation model to obtain the constraint description aggregate code. Finally, the preset image generation model is trained according to the sample image, the constraint description aggregate code and the sample image code, so as to enhance the preset image generation model's perception ability of the potential relationship between text and image, and obtain a target image generation model that can accurately generate images based on text.

[0100] In step S401 of some embodiments, image coding is a process of converting a sample image into a high-dimensional feature vector by a preset image generation model, thereby converting the image into a more compact and meaningful vector representation, so that the preset image generation model can understand the image content at a higher level, and then use it for image generation. The sample image coding is a vector representation obtained by encoding the sample image by the preset image generation model. In one embodiment, the preset image generation model is a DALL-E2 model, which includes a CLIP image encoder, and the sample image is encoded based on the CLIP image encoder to obtain the sample image encoding.

[0101] In step S402 of some embodiments, the description text code and the constraint information code are aggregated based on a preset image generation model to obtain a constraint description aggregate code, and the aggregate calculation is shown in formula (1):

[0102] Text code=α·text1+text2 (1),

[0103] Among them, Text code is the constraint description aggregation code, α is the constraint condition factor that can be optimized in model training, which is a hyperparameter in the preset image generation model, text1 is the constraint information code, and text2 is the description text code.

[0104] See also Figure 5 In some embodiments, step S403 may include but is not limited to steps S501 to S503:

[0105] Step S501, extracting prior features of sample image codes and constraint description aggregate codes based on a preset image generation model to obtain image text prior features;

[0106] Step S502, performing image generation on the image text prior features based on a preset image generation model to obtain a predicted image;

[0107] Step S501 , training a preset image generation model based on the predicted image and the sample image to obtain a target image generation model.

[0108] In steps S501 to S503 shown in the embodiment of the present application, prior features are extracted from the sample image encoding and the constraint description aggregation encoding based on the preset image generation model to obtain image text prior features, and then image generation is performed on the image text prior features based on the preset image generation model to obtain a predicted image. Finally, the preset image generation model is trained based on the predicted image and the sample image to obtain a target image generation model, thereby enhancing the preset image generation model's perception ability between text and image and between constraint information and image, and obtaining a target image generation model that can accurately generate images.

[0109] In step S501 of some embodiments, prior feature extraction refers to the process of extracting potential correlation features from sample image codes and constraint description aggregate codes. In one embodiment, the preset image generation model is the DALL-E2 model, and DALL-E2 includes a prior module, which generates a vector representation of the sample image code from the constraint description aggregate code to help generate an image associated with the sample description text. The sample image code and the constraint description aggregate code are input into the prior module together to obtain the image text prior features.

[0110] In step S502 of some embodiments, the image text prior features are input into a preset image generation model, and the preset image generation model outputs a predicted image based on the image text prior features. In one embodiment, the preset image generation model is a DALL-E2 model, which includes an image decoder, whose input is the image text prior features and whose output is a predicted image.

[0111] See also Figure 6 In some embodiments, step S503 may include but is not limited to steps S601 to S603:

[0112] Step S601, performing image loss calculation based on the sample image and the predicted image to obtain image loss data;

[0113] Step S602, performing semantic loss calculation based on the sample image and the predicted image to obtain semantic loss data;

[0114] Step S603: training a preset image generation model based on the semantic loss data and the image loss data to obtain a target image generation model.

[0115] In steps S601 to S603 shown in the embodiment of the present application, image loss calculation is performed based on the sample image and the predicted image to obtain image loss data, and semantic loss calculation is performed based on the sample image and the predicted image to obtain semantic loss data. Finally, a preset image generation model is trained based on the semantic loss data and the image loss data to obtain a target image generation model, thereby training the model from both the dimension of image pixels and the dimension of image semantics at the same time, so that the model can be continuously optimized from two dimensions: accuracy and semantic information expressed by the image, and a target image generation model that can correctly express semantics and accurately express images is obtained.

[0116] In step S601 of some embodiments, image loss calculation is performed based on pixel points of the sample image and the predicted image to obtain image loss data. In one embodiment, the image loss calculation is a mean square error calculation, and the mean square error calculation is performed based on the sample image and the predicted image to obtain image loss data.

[0117] See also Figure 7 In some embodiments, step S602 may include but is not limited to steps S701 to S703:

[0118] Step S701, performing semantic extraction on the sample image to obtain sample semantic information;

[0119] Step S702, performing semantic extraction on the predicted image to obtain predicted semantic information;

[0120] Step S703, performing cosine similarity calculation on the sample semantic information and the predicted semantic information to obtain semantic loss data.

[0121] In steps S701 to S703 shown in the embodiment of the present application, semantic extraction is performed on the sample image to obtain sample semantic information, and semantic extraction is performed on the predicted image to obtain predicted semantic information. Finally, cosine similarity calculation is performed on the sample semantic information and the predicted semantic information to obtain semantic loss data, so that the model can well perceive the potential relationship between semantics and images.

[0122] In step S701 of some embodiments, semantic extraction refers to performing semantic recognition on an image to obtain semantic information in the image. For example, if the image includes elements of cartoon characters, the elements of the cartoon characters are extracted and converted into text information through semantic extraction. In one embodiment, text information in an image is extracted using OCR technology and converted into text to obtain sample semantic information.

[0123] In step S702 of some embodiments, the step of performing semantic extraction on the predicted image is similar to the principle of performing semantic extraction on the sample image, which will not be described in detail here.

[0124] In step S703 of some embodiments, the sample semantic information is converted into a sample vector through a pre-trained BERT model, and the predicted semantic information is converted into a predicted vector through a pre-trained BERT model, and then cosine similarity calculation is performed based on the sample vector and the predicted vector to obtain semantic loss data.

[0125] In step S603 of some embodiments, the semantic loss data and the image loss data are first aggregated to obtain aggregated loss data. The aggregated calculation is shown in formula (2):

[0126] L=λ1·L i +λ2·L sms (2)

[0127] Among them, L is the aggregate loss data, L i is the image loss data, L sms is the semantic loss data, λ1 is the balancing factor of the image loss data, which is a learnable hyperparameter in the preset image generation model, and λ2 is the balancing factor of the semantic loss data, which is a learnable hyperparameter in the preset image generation model.

[0128] In step S106 of some embodiments, the target description text is a description text, constraint information is extracted from the target description text to obtain target constraint information, and then feature encoding is performed on the target constraint information to obtain target constraint code. At the same time, the target description text is input into a text encoder in a target image generation model for text encoding to obtain a target text code. Then, the target constraint code and the target text code are input into a priori module in the target image generation model to obtain priori features, and then the priori features are input into an image decoder in the target image generation model to obtain a target image.

[0129] See also Figure 8 The embodiment of the present application further provides an image generating device, which can implement the above-mentioned image generating method, and the device includes:

[0130] The data acquisition module 801 is used to acquire sample description text and sample image; wherein the sample description text is used to describe the sample image;

[0131] An information extraction module 802 is used to extract constraint information from the sample description text to obtain constraint information features;

[0132] The feature encoding module 803 is used to perform feature encoding on the constraint information feature to obtain the constraint information code;

[0133] A text encoding module 804 is used to perform text encoding on the sample description text based on a preset image generation model to obtain a description text encoding;

[0134] The model training module 805 is used to train the preset image generation model according to the constraint information code, the description text code and the sample image to obtain the target image generation model;

[0135] The image generation module 806 is used to obtain the target description text and generate an image of the target description text based on the target image generation model.

[0136] The specific implementation of the image generating device is substantially the same as the specific implementation of the above-mentioned image generating method, and will not be described in detail here.

[0137] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0138] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0139] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0140] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the image generation method of the embodiment of this application;

[0141] Input / output interface 903, used to implement information input and output;

[0142] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0143] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0144] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0145] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned image generation method is implemented.

[0146] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The image generation method, image generation device, electronic device and storage medium provided by the embodiments of the present application obtain a sample image and a sample description text that describes the sample image, then extract constraint information from the sample description text to obtain constraint information features, then feature encode the constraint information features to obtain constraint information codes, and at the same time, perform text encoding on the sample description text based on a preset image generation model to obtain description text codes, and finally train the preset image generation model based on the constraint information codes, description text codes and sample images, so that during the training process, the preset image generation model can not only perceive how to generate sample images from the description text codes, but also enhance the perception ability of the preset image generation model to perceive the constraint information in the sample description text, so as to obtain a target image generation model that can accurately generate images based on the text and the information displayed by the image will not exceed the constraint information; further, the target description text is obtained, and image generation is performed on the target description text based on the target image generation model, so as to achieve accurate text-based image generation.

[0148] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0149] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0150] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0152] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0153] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0158] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire a sample description text and a sample image; wherein the sample description text is used to describe the sample image; Extracting constraint information from the sample description text to obtain constraint information features; Performing feature encoding on the constraint information feature to obtain a constraint information code; Performing text encoding on the sample description text based on a preset image generation model to obtain a description text encoding; The preset image generation model is trained according to the constraint information code, the description text code and the sample image to obtain a target image generation model; Obtain target description text, and generate an image of the target description text based on the target image generation model.

2. The method according to claim 1, characterized in that The step of training the preset image generation model according to the constraint information code, the description text code and the sample image to obtain a target image generation model includes: Performing image encoding on the sample image based on the preset image generation model to obtain a sample image code; Based on the preset image generation model, the description text code and the constraint information code are aggregated and calculated to obtain a constraint description aggregate code; The preset image generation model is trained according to the sample image, the constraint description aggregation code and the sample image code to obtain the target image generation model.

3. The method according to claim 2, characterized in that The step of training the preset image generation model according to the sample image, the constraint description aggregation code and the sample image code to obtain the target image generation model includes: Based on the preset image generation model, a priori feature extraction is performed on the sample image code and the constraint description aggregate code to obtain image text prior features; Performing image generation on the image text prior features based on the preset image generation model to obtain a predicted image; The preset image generation model is trained based on the predicted image and the sample image to obtain the target image generation model.

4. The method according to claim 3, characterized in that The step of training the preset image generation model based on the predicted image and the sample image to obtain the target image generation model includes: Performing image loss calculation according to the sample image and the predicted image to obtain image loss data; Performing semantic loss calculation based on the sample image and the predicted image to obtain semantic loss data; The preset image generation model is trained based on the semantic loss data and the image loss data to obtain a target image generation model.

5. The method according to claim 4, characterized in that The performing semantic loss calculation according to the sample image and the predicted image to obtain semantic loss data includes: Performing semantic extraction on the sample image to obtain sample semantic information; Performing semantic extraction on the predicted image to obtain predicted semantic information; The cosine similarity calculation is performed on the sample semantic information and the predicted semantic information to obtain the semantic loss data.

6. The method according to any one of claims 1 to 5, characterized in that: The step of extracting constraint information from the sample description text to obtain constraint information features includes: Performing knowledge domain identification on the sample description text to obtain text knowledge domain information; Screening a preset regular matching template library according to the text knowledge domain information to obtain a target regular matching template; The constraint information of the sample description text is identified according to the target regular matching template to obtain the constraint information feature.

7. The method according to claim 6, characterized in that The step of identifying constraint information of the sample description text according to the target regular matching template to obtain the constraint information feature includes: Performing word segmentation on the sample description text to obtain a text word sequence; Performing part-of-speech tagging on the text word sequence to obtain a word part-of-speech sequence; Keyword matching is performed on the word part-of-speech sequence according to the target regular matching template to obtain the constraint information feature.

8. An image generating device, characterized in that: The device comprises: A data acquisition module, used to acquire a sample description text and a sample image; wherein the sample description text is used to describe the sample image; An information extraction module, used to extract constraint information from the sample description text to obtain constraint information features; A feature encoding module, used for performing feature encoding on the constraint information feature to obtain constraint information code; A text encoding module, used for performing text encoding on the sample description text based on a preset image generation model to obtain a description text encoding; A model training module, used for training the preset image generation model according to the constraint information code, the description text code and the sample image to obtain a target image generation model; The image generation module is used to obtain target description text and generate an image of the target description text based on the target image generation model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the image generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 7 is implemented.

Images