Image generation method and apparatus, electronic device, and storage medium

Abstract

The application provides an image generation method, device, electronic equipment and storage medium, wherein the method comprises: acquiring a first text; extracting a text semantic feature and a text emotion feature of the first text; and generating a first image corresponding to the first text based on the text semantic feature and the text emotion feature. The method, device, electronic equipment and storage medium provided by the application can make the generated first image exhibit content and emotion information that conforms to the first text, especially in the scenario of generating an expression package, the first text used for interaction can be more in line with the characteristics, and the expression package used for interaction is more creative. Moreover, even if the first text does not include a description of each part of the human body, the first image can be generated, the range of texts used for image generation is greatly widened, the expression package generation service can be provided for the interaction text of daily chat, and the generality, instantaneity and convenience of image generation are ensured.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Current methods for generating emojis mainly involve editing parameters based on existing image or video clips, automatically cropping suitable images, matching corresponding text to images, or training a GAN (Generative Adversarial Network) model by semantically labeling emoji images into multiple human body parts such as head shape, facial features, and upper body to synthesize emoji images.

[0003] However, the generated emojis often don't match the intended use case, lack creativity, and may even pose copyright risks. Furthermore, synthesizing emoji images by training models to segment human body parts has significant limitations, and the generated content is monotonous, leading to inconvenience. For example, when users use emojis in communication, they typically type broad semantic expressions or emotional statements like "The ship of friendship capsizes so easily" or "How embarrassing!" without providing specific descriptions of the human body's state, and the desired generated result may not even be a human image. Summary of the Invention

[0004] This invention provides an image generation method, apparatus, electronic device, and storage medium to solve the problems of existing emojis not matching usage scenarios, lacking creativity, and having poor applicability.

[0005] This invention provides an image generation method, comprising:

[0006] Get the first text;

[0007] Extract the semantic features and sentiment features of the first text;

[0008] Based on the semantic features and emotional features of the text, a first image corresponding to the first text is generated.

[0009] According to the image generation method provided by the present invention, the step of generating a first image corresponding to the first text based on the text semantic features and the text sentiment features further includes:

[0010] Extract the image semantic features and / or image emotion features of the first image;

[0011] Based on the similarity between the text semantic features and the image semantic features, and / or the similarity between the text sentiment features and the image sentiment features, a consistency score for the first image is determined;

[0012] Image recommendations are made based on the consistency score of the first image.

[0013] According to the image generation method provided by the present invention, the extraction of image semantic features and / or image emotion features of the first image includes:

[0014] Image feature extraction is performed on the first image, and the extracted image features are used as the semantic features of the image; and / or,

[0015] Based on the image emotion extraction model, the image emotion features of the first image are extracted. The image emotion extraction model is the part of the facial expression recognition model used to extract facial expression features.

[0016] According to the image generation method provided by the present invention, generating a first image corresponding to the first text based on the text semantic features and the text sentiment features includes:

[0017] Based on the image generation model, image encoding features corresponding to the text semantic features and the text emotion features are determined, and the image encoding features are decoded to obtain the first image;

[0018] The image generation model is trained based on sample text and sample images corresponding to the sample text.

[0019] According to the image generation method provided by the present invention, the step of determining image encoding features corresponding to the text semantic features and the text sentiment features based on an image generation model, and decoding the image encoding features to obtain the first image, includes:

[0020] Receive an editing operation on the second image, cover the area corresponding to the editing operation in the second image, and obtain a third image;

[0021] Based on the image generation model, the text semantic features, the text sentiment features, and the image coding features of the third image are applied to predict the region coding features of the occluded area in the third image, and the region coding features are decoded to obtain the first image.

[0022] According to the image generation method provided by the present invention, the training steps of the image generation model include:

[0023] The sample image is masked to obtain a masked image;

[0024] Using the semantic features and sentiment features of the sample text as samples, and the sample images as labels, the model is trained to obtain a preliminary generation model;

[0025] Using the semantic features and sentiment features of the sample text, as well as the image encoding features of the covered image, as samples and the sample image as a label, the preliminary generation model is fine-tuned to obtain the image generation model.

[0026] According to the image generation method provided by the present invention, the step of extracting the text emotion features includes:

[0027] Based on the text emotion extraction model, the text emotion features of the first text are extracted. The text emotion extraction model and the text emotion classification model constitute a text emotion recognition model. The text emotion classification model is used to classify emotions based on the text emotion features.

[0028] The present invention also provides an image generation apparatus, comprising:

[0029] Text unit, used to obtain the first text;

[0030] The extraction unit is used to extract the text semantic features and text sentiment features of the first text;

[0031] The generation unit is used to generate a first image corresponding to the first text based on the text semantic features and the text sentiment features.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image generation method as described above.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image generation method as described above.

[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image generation method as described above.

[0035] The image generation method, apparatus, electronic device, and storage medium provided by this invention acquire first text; extract semantic and emotional features of the first text; and generate a first image corresponding to the first text based on the semantic and emotional features. This allows the generated first image to display content and emotional information that closely matches the first text, especially in emoji generation scenarios, better aligning with the characteristics of the first text used for interaction, resulting in more creative emojis. Furthermore, even if the first text does not include descriptions of human body parts, a first image can still be generated, greatly expanding the range of texts used for image generation. This provides corresponding emoji generation services for everyday chat interaction text, ensuring the versatility, immediacy, and convenience of image generation. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is one of the flowcharts illustrating the image generation method provided by the present invention;

[0038] Figure 2 This is a flowchart illustrating the image recommendation method provided by the present invention;

[0039] Figure 3 This is a schematic diagram of the training process of the image generation model provided by the present invention;

[0040] Figure 4 This is one of the schematic diagrams of the image editing process provided by the present invention;

[0041] Figure 5 This is the second schematic diagram of the image editing process provided by the present invention;

[0042] Figure 6 This is a schematic diagram of the training process of the image generation model provided by the present invention;

[0043] Figure 7 This is the second flowchart illustrating the image generation method provided by the present invention;

[0044] Figure 8 This is a schematic diagram of the image generation device provided by the present invention;

[0045] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] Current methods for generating emojis primarily involve editing parameters of existing images or video clips. However, the generated emojis often don't match the intended use cases, lack creativity, and may pose copyright risks. Additionally, there are methods that synthesize emojis based on text. These methods involve semantically annotating emoji images by dividing them into multiple parts such as head shape, facial features, and upper body, training a GAN model to synthesize emoji images. However, these solutions require the text to include descriptions of different body parts, which is a significant limitation. Furthermore, the generated images are all human figures, resulting in monotonous content that is unsuitable for generating emojis corresponding to interactive text in everyday chat, leading to inconvenience.

[0048] To address the aforementioned problems, this invention provides an image generation method that generates images based on text semantic and emotional features, making image generation more convenient and resulting in images that better match the semantics and emotion of the text, thus achieving higher image quality. It is understood that the images referred to here can be chat emoticons or images from other applications. Figure 1 This is one of the flowcharts illustrating the image generation method provided by the present invention, such as... Figure 1 As shown, the method includes:

[0049] Step 110, obtain the first text;

[0050] Here, "first text" refers to the text used for image generation. First text can be text entered by the user during the interaction, such as "Here's a little flower for you." First text can also be text used to specifically describe the image, such as dividing the image into multiple parts like head shape, facial features, and upper body with detailed semantic annotations.

[0051] Understandably, when the first text is the text entered during the interaction, performing image generation based on the first text to obtain the first image that can be used as an emoticon can ensure the immediate application of emoticons during the interaction and improve the user experience.

[0052] Step 120: Extract the text semantic features and text sentiment features of the first text;

[0053] Specifically, in interactive scenarios, the initial text typically conveys not only semantic information but also the emotional information the user wishes to express. To make the images generated from the text more vivid and better suited to the user's interactive context, it is necessary to extract not only the semantic features of the initial text but also its emotional features.

[0054] Here, the text semantic features refer to the content information contained in the semantic expression of the first text. The text semantic features of the first text can be extracted by a pre-trained language model, such as BERT (Bidirectional Encoder Representation from Transformers) and its subsequent variant Roberta.

[0055] Furthermore, the text sentiment features of the first text refer to the emotional information contained in the emotional expression of the first text. For example, the text sentiment feature of "Baby is heartbroken" is sadness. Here, the text sentiment features of the first text can be extracted using a pre-trained sentiment feature extraction model. For example, the encoder in a sentiment classification model can be used as a sentiment feature extraction model to achieve text sentiment feature extraction.

[0056] Step 130: Based on the semantic features of the text and the emotional features of the text, generate a first image corresponding to the first text.

[0057] Specifically, after obtaining the semantic and emotional features of the first text, image generation can be performed based on these features. In this process, feature mapping can be performed on the semantic and emotional features of the first text to obtain the corresponding image encoding features. Then, by decoding the image encoding features of the first text, the first image corresponding to the first text can be obtained. Here, the first image refers to an image that matches the first text both semantically and emotionally. Specifically, in an interactive scenario, the first image can be an emoji image corresponding to the first text.

[0058] Understandably, compared to related technologies that only focus on textual semantics when generating images, this embodiment of the invention considers not only the semantic features of the first text but also its emotional features when generating the first image. The application of these emotional features enables the generated first image to display emotional information matching the first text, thus making it more suitable for practical application scenarios.

[0059] The method provided in this invention generates a first image corresponding to the first text based on the text semantic features and text emotional features of the first text. This allows the generated first image to display content and emotional information that is more closely related to the first text. In particular, in the context of emoji generation, it can better match the characteristics of the first text used for interaction, making the emojis used in interaction more creative.

[0060] Furthermore, the generation of the first image relies on the semantic and emotional features of the first text. Even if the first text does not contain descriptions of different parts of the human body, the first image can still be generated, greatly expanding the range of texts used for image generation. This enables the generation of corresponding emoticons for interactive texts generated from daily chats, ensuring the versatility, immediacy, and convenience of image generation.

[0061] To improve the quality of the generated first image and allow users to obtain a more desirable image, based on the above embodiments, this invention proposes steps after obtaining the first image. Figure 2 This is a flowchart illustrating the image recommendation method provided by the present invention, as shown below. Figure 2 As shown, a first image corresponding to the first text is generated, followed by:

[0062] Step 210: Extract the image semantic features and / or image emotion features of the first image;

[0063] Step 220: Determine the consistency score of the first image based on the similarity between the text semantic features and the image semantic features, and / or the similarity between the text sentiment features and the image sentiment features.

[0064] Specifically, the number of first images generated in the above steps corresponding to the first text can be one or multiple. For any given first image, the consistency score of the first image reflects the degree of fit between the first image and the first text, and can be comprehensively evaluated from the following aspects: specifically, the consistency between the textual semantic features of the first text and the image semantic features of the first image, and / or the consistency between the textual emotional features of the first text and the image emotional features of the first image. Here, the image semantic features of the first image refer to the content information of the objects contained in the image, such as "people" or "flowers" in the image. The image semantic features of the first image can be extracted using image feature encoders such as the ViT model or the ResNet model. In addition, the image emotional features of the first image refer to the emotional content information expressed by the first image. For example, the image generated from "Baby is feeling bitter" usually expresses the emotion of sadness. The image emotional features of the first image can be extracted using a facial expression recognition model or an encoder in an emotion recognition model; this embodiment of the invention does not specifically limit this.

[0065] Specifically, when measuring the semantic and / or emotional consistency between the first image and the first text, the consistency score of the first image can be determined based on the similarity between the semantic features of the text and the semantic features of the image, or the similarity between the emotional features of the text and the emotional features of the image, or both of the above. The similarity referred to here can be calculated using similarity algorithms such as cosine similarity and Euclidean distance. A higher similarity indicates a closer match between the first text and the corresponding generated first image. For example, if the first text is "Baby is heartbroken," and its emotional feature is sadness, then if the generated first image contains a sad, tearful expression, the emotional similarity between the first image and the first text is high, meaning the first image better matches the corresponding first text. Conversely, if the generated first image contains a laughing expression, the emotional similarity between the first image and the first text is low, meaning the first image does not closely match the corresponding first text.

[0066] Specifically, regarding semantic similarity, considering that there are some trained text-image matching models in related technologies, such as the CLIP (Contrastive Language-Image Pre-training) model, which can directly calculate the similarity between text semantic features and image semantic features.

[0067] After obtaining the semantic similarity and / or emotional similarity between the first text and the first image, the consistency score of the first image can be determined directly from one of the similarities, or the semantic similarity and emotional similarity between the first text and the first image can be weighted to obtain the consistency score of the first image.

[0068] Furthermore, in interactive scenarios, the first text typically emphasizes emotional expression more, with relatively less direct semantic description. Therefore, when calculating the consistency score of the first image based on a weighted average of semantic and emotional similarity between the first text and the first image, the weighting factors can be adjusted to control the proportion of each. The specific formula for calculating the consistency score is as follows:

[0069] Score=α*f(p i ,p t )+(1-α)*f(e i ,e t )

[0070] Where Score is the final consistency score of the first image, α is the weighting of emotion consistency, f() is the similarity calculation function, and p i ,p t These are the image sentiment features of the first image and the text sentiment features of the first text, respectively. i ,e t These are the image semantic features of the first image and the text semantic features of the first text, respectively.

[0071] Step 230: Based on the consistency score of the first image, perform image recommendation.

[0072] Specifically, based on the consistency score of each first image, the first images can be filtered or sorted, and image recommendations can be made based on the filtering or sorting results. For example, first images with low consistency scores can be filtered out, and the remaining first images can be used for image recommendations. Alternatively, the first images can be sorted according to their consistency scores, and the first images with higher consistency scores can be displayed to the user first. This embodiment of the invention does not impose specific limitations.

[0073] The method provided in this invention calculates the semantic and / or emotional consistency scores of each generated first image relative to the first text, thereby achieving image recommendations that are more faithful to the first text. This helps improve the quality and universality of images recommended to users, thus enhancing the user experience. Furthermore, based on image recommendations, it improves both the user experience and the usage rate of images.

[0074] Based on any of the above embodiments, step 210, extracting the image semantic features and / or image emotion features of the first image, includes:

[0075] Image feature extraction is performed on the first image, and the extracted image features are used as the semantic features of the image; and / or,

[0076] Based on the image emotion extraction model, the image emotion features of the first image are extracted. The image emotion extraction model is the part of the facial expression recognition model used to extract facial expression features.

[0077] Here, considering that the semantic features of an image actually reflect the information of the content of things contained in the first image, and this information is usually also reflected in the image features, the extraction of semantic features of an image can be achieved through an image feature extraction model, such as the ViT model or the ResNet model.

[0078] Furthermore, since the first image typically contains facial expressions with exaggerated facial features, and these expressions generally convey human emotions, it reflects the emotion of the first image. Therefore, image emotion features can be extracted using a facial expression recognition model. Specifically, after detecting a face in the first image, the image's emotional features can be extracted using an expression recognition model. If no face is detected, it is assumed that the first image does not carry a significant emotion, and the image emotion features are assumed to be neutral.

[0079] Based on any of the above embodiments, step 130 includes:

[0080] Based on the image generation model, image encoding features corresponding to the text semantic features and the text emotion features are determined, and the image encoding features are decoded to obtain the first image;

[0081] The image generation model is trained based on sample text and sample images corresponding to the sample text.

[0082] Specifically, in the process of image generation based on an image generation model, image encoding features corresponding to text semantic features and text sentiment features can be mapped first. These image encoding features reflect the informational features of the image to be generated that matches the text semantic features and text sentiment features in terms of its content. Based on this, decoding the image encoding features yields the first image that matches the first text.

[0083] Furthermore, the image generation model can pre-model the relationship between text semantic features, text sentiment features, and image encoding features. In practical applications, the input text semantic features and text sentiment features can be used to gradually decode the image encoding features from the image start identifier (BOI). Subsequently, the image encoding features are decoded and reconstructed to obtain the first image. The image generation model here can be a Transformer model or other model structures; this embodiment of the invention does not specifically limit it.

[0084] Before performing this step, an image generation model needs to be trained. The training of the image generation model can be achieved by the following steps: First, collect sample text and the sample images corresponding to the sample text; then, train the model based on the collected sample text and its corresponding sample images to obtain the image generation model.

[0085] During the training process of the image generation model, you can refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the training process of the image generation model provided by the present invention. Specifically, during the training process of the image generation model, a graph encoder can be applied to encode the sample images to obtain sample encoded features. The graph encoder here can be a VQVAE (Vector Quantized-Variational AutoEncoder) model. Furthermore, a text sentiment feature extraction module can be used to extract the text sentiment features of the sample text, and a text semantic feature extraction module can be used to extract the text semantics of the sample text. Then, based on the sample encoded features obtained from the sample images, and the differences between the image encoded features obtained by mapping the text semantic features and text sentiment features of the sample text, a loss function can be constructed. The training model is then iterated based on the loss function to obtain the image generation model.

[0086] Alternatively, the semantic and sentiment features of the sample text can be mapped to obtain image encoding features. The image encoding features can then be decoded using a graph decoder to obtain a predicted image. Based on the difference between the predicted image and the sample image, a loss function can be constructed. The training model can then be iterated based on the loss function to obtain an image generation model. This embodiment of the invention does not impose specific limitations on this approach.

[0087] Based on the above embodiments, Figure 4 This is one of the schematic diagrams of the image editing process provided by the present invention, such as... Figure 4 As shown, in step 130, determining the image encoding features corresponding to the text semantic features and the text sentiment features based on the image generation model, and decoding the image encoding features to obtain the first image, includes:

[0088] Step 410: Receive an editing operation for the second image, cover the area corresponding to the editing operation in the second image, and obtain a third image;

[0089] The second image here is the image that the user expects to modify locally in conjunction with the first text. The second image can be an image generated earlier based on the first text, or it can be an image uploaded by the user.

[0090] For the second image, the user can select the area that needs to be modified through editing operations. These editing operations can take the form of circle selection, point selection, or sliding selection, allowing the user to select unsatisfactory areas in the second image. Upon detecting and receiving the user's editing operation on the second image, the system can locate the area to be modified based on the position of the editing operation, i.e., the area corresponding to the editing operation, and then mask this area. Once masking is complete, the third image is obtained.

[0091] Step 420: Based on the image generation model, apply the text semantic features, the text sentiment features, and the image coding features of the third image to predict the region coding features of the occluded area in the third image, and decode the region coding features to obtain the first image.

[0092] Specifically, based on the image generation model, the occluded areas in the third image can be regionalized to generate a first image that matches the first text in terms of both semantics and emotion. It can be understood that the first image here inherits the occluded areas from the third image.

[0093] In the generation of the first image, the image generation model, in addition to applying the semantic and sentiment features of the first text, also incorporates the image coding features of the uncovered region of the third image, predicting the image coding features of the covered region in the third image, denoted here as region coding features. It can be understood that these region coding features are obtained by mapping the semantic and sentiment features of the first text with the image coding features of the uncovered region of the third image. After obtaining the region coding features, feature decoding can be used to obtain the region image of the covered region in the third image, and the third image and the region image are iterated to obtain the first image.

[0094] The image generation model here can be obtained by fine-tuning a model that can generate images based on text, i.e., a preliminary generation model. The fine-tuning training steps are as follows: The sample image can be graph-encoded. Based on the sample encoding features obtained from the graph encoding, and the difference between the image encoding obtained by mapping the text semantic features, text sentiment features, and image encoding features of the occluded image from the preliminary generation model, a loss function can be constructed. Then, the parameters of the preliminary generation model are iterated based on the loss function to obtain the image generation model. Alternatively, the text semantic features, text sentiment features, and image encoding features of the occluded image from the preliminary generation model can be mapped to obtain image encoding. The image encoding is then decoded to obtain the predicted image corresponding to the occluded image. Based on the difference between the predicted image and the sample image, a loss function is constructed. Then, the parameters of the preliminary generation model are iterated based on the loss function to obtain the image generation model. This embodiment of the invention does not specifically limit the specific steps.

[0095] The method provided in this embodiment of the invention combines editing operations and first text to achieve image modification of a second image, thereby improving the flexibility of image generation in practical applications and enhancing the user experience.

[0096] Especially when the second image is generated by an image generation method, modifying the second image based on the user's editing operations can enhance the user's sense of participation, thereby making it easier and faster to generate personalized emojis.

[0097] In addition, the second image can also be an image created or selected by the user. Based on this image modification, a personalized first image can be generated, thereby enhancing the fun of image generation.

[0098] Furthermore, Figure 5 This is a second schematic diagram of the image editing process provided by the present invention, as shown below. Figure 5 As shown, the image generation model adds image editing functionality to the initial image generation model, forming an image generation model that includes both image generation and editing. Specifically, in the image editing process, the first text can be input into the text sentiment feature extraction module and the text semantic feature extraction module to obtain the text semantic features and text sentiment features of the first text. Additionally, a graph encoder can be applied to perform graph encoding on the acquired third image to obtain the image encoding features of the third image. Subsequently, the text semantic features and text sentiment features of the first text, as well as the image encoding features of the third image, can be input into the image generation model. The image generation model then predicts the region encoding features of the occluded areas in the third image based on the input features. Finally, a graph decoder can be applied to decode the region encoding features, thereby obtaining the first image corresponding to the first text and the third image.

[0099] Based on any of the above embodiments Figure 6 This is a schematic diagram of the training process of the image generation model provided by the present invention, as shown below. Figure 6 As shown, the training steps of the image generation model include:

[0100] Step 610: Cover the sample image to obtain a covered image;

[0101] Step 620: Using the semantic features and sentiment features of the sample text as samples and the sample images as labels, perform model training to obtain a preliminary generated model;

[0102] Step 630: Using the semantic features and sentiment features of the sample text, as well as the image encoding features of the masked image, as samples and the sample image as labels, the preliminary generation model is fine-tuned to obtain the image generation model.

[0103] Specifically, the training of the image generation model can be divided into two stages, corresponding to steps 620 and 630.

[0104] Step 620 is used to implement basic image generation functionality, that is, to obtain a preliminary generation model with the ability to generate images based on text through training. Specifically, a loss function can be constructed based on the sample encoding features obtained by graph encoding the sample image and the difference between the image encoding obtained by mapping the text semantic features and text sentiment features of the sample text by the initial model. Then, the parameters of the initial model are iterated based on the loss function to obtain the preliminary generation model. Alternatively, the initial model can be used to map the text semantic features and text sentiment features of the sample text to obtain the image encoding, and the image encoding can be decoded to obtain the predicted image. A loss function can be constructed based on the difference between the predicted image and the sample image. Then, the parameters of the initial model are iterated based on the loss function to obtain the preliminary generation model.

[0105] Step 630 is used to fine-tune the initial generation model, thereby obtaining an image generation model that possesses both text-based image generation and image modification capabilities by combining text and images. Specifically, the sample images can be graph-encoded. Based on the sample encoding features obtained from the graph encoding, and the difference between the image encoding obtained by mapping the text semantic features, text sentiment features, and image encoding features of the occluded image from the initial generation model, a loss function can be constructed. Then, the parameters of the initial generation model can be iterated based on the loss function to obtain the image generation model. Alternatively, the text semantic features, text sentiment features, and image encoding features of the occluded image from the initial generation model can be mapped to obtain image encoding. The image encoding can then be decoded to obtain a predicted image. Based on the difference between the predicted image and the sample image, a loss function can be constructed. Then, the parameters of the initial generation model can be iterated based on the loss function to obtain the image generation model.

[0106] The resulting image generation model not only has the function of generating images from text, but also has the function of combining text and images to adjust the images. It can be applied to interactive scenarios where users specify modifications to images, or to interactive tasks where users adjust aspects of previously generated images that they are dissatisfied with, thereby enhancing user engagement in image generation.

[0107] Based on any of the above embodiments, the step of extracting text sentiment features includes:

[0108] Based on the text emotion extraction model, the text emotion features of the first text are extracted. The text emotion extraction model and the text emotion classification model constitute a text emotion recognition model. The text emotion classification model is used to classify emotions based on the text emotion features.

[0109] Specifically, a text emotion recognition model can consist of two parts: a text emotion extraction model and a text emotion classification model. For the text emotion recognition task, the text emotion extraction model acts as an encoder, extracting emotional features from the input text, while the text emotion classification model acts as a classifier, classifying emotions based on the extracted emotional features. The model structure of the text emotion recognition model can be based on common emotion classification models such as BERT or BiLSTM (Bi-directional Long Short-Term Memory), without specific limitations. It can be understood that the text emotion recognition model can be obtained through supervised training based on sample text and its corresponding emotion labels. In this process, the text emotion extraction model within the text emotion recognition model can fully learn the emotional features of the sample text, thus possessing the ability to extract emotional features. In practical applications, emotion labels can use common emotion classification label types, such as happiness, sadness, fear, anger, surprise, and disgust, or user-defined type classifications can be used as needed, such as sarcastic or cute.

[0110] Therefore, in the application of this invention, only the text emotion extraction model in the text emotion recognition model can be applied to extract text emotion features for the first text, thereby obtaining text emotion features. Since the text emotion extraction model participates in the supervised learning task of text emotion recognition during the training phase, the text emotion features extracted based on the text emotion extraction model can accurately and reliably reflect the emotions contained in the first text, thereby generating a first image that is more in line with the user's original intention.

[0111] Based on any of the above embodiments Figure 7 This is a second schematic flowchart of the image generation method provided by the present invention, as shown below. Figure 7 As shown, the method includes:

[0112] The first image can be generated based on an image generation model. Here, the generation of the first image depends on the first text, or on the first text and a third image. Since the third image is not necessary for image generation, it is not required here. Figure 7 It is shown in a dashed box.

[0113] Specifically, first, the first text for which the corresponding image needs to be generated is determined; then, the first text is input into the text emotion feature extraction module and the text semantic feature extraction module respectively, thereby obtaining the text emotion feature and text semantic feature of the first text.

[0114] Next, the text sentiment features and text semantic features of the first text are input into the image generation model, which then uses these features to generate the corresponding first image.

[0115] Alternatively, if a third image exists, the textual sentiment features and semantic features of the first text, as well as the image encoding features of the third image, can be input into the image generation model. The image generation model can then generate the corresponding first image using the textual sentiment features and semantic features of the first text and the image encoding features of the third image.

[0116] The resulting first image may exist in multiple forms.

[0117] After obtaining multiple first images, image emotion feature extraction and image semantic feature extraction modules can be applied to extract image emotion features and image semantic features for each first image, respectively, resulting in image emotion features and image semantic features for each first image. Combining the text emotion features and text semantic features of the first text, the emotion similarity between the text emotion features and image emotion features, and the semantic similarity between the text semantic features and image semantic features, can be calculated for each first image. The consistency score of the first image can be calculated using the semantic similarity and / or emotion similarity. Based on the consistency score of the first images, they can be ranked, thereby completing image recommendation for each first image.

[0118] Based on any of the above embodiments Figure 8 This is a schematic diagram of the image generation device provided by the present invention, as shown below. Figure 8 As shown, the device includes:

[0119] Text unit 810 is used to acquire the first text;

[0120] Extraction unit 820 is used to extract the text semantic features and text sentiment features of the first text;

[0121] The generation unit 830 is used to generate a first image corresponding to the first text based on the text semantic features and the text sentiment features.

[0122] The apparatus provided in this invention generates a first image corresponding to the first text based on the text semantic features and text emotional features of the first text. This allows the generated first image to display content and emotional information that is more closely related to the first text. In particular, in the context of emoji generation, it can better match the characteristics of the first text used for interaction, making the emojis used in interaction more creative.

[0123] Furthermore, the generation of the first image relies on the semantic and emotional features of the first text. Even if the first text does not contain descriptions of different parts of the human body, the first image can still be generated, greatly expanding the range of texts used for image generation. This enables the generation of corresponding emoticons for interactive texts generated from daily chats, ensuring the versatility, immediacy, and convenience of image generation.

[0124] Based on any of the above embodiments, the generation unit further includes a consistency score determination unit, which is used for:

[0125] Extract the image semantic features and / or image emotion features of the first image;

[0126] Based on the similarity between the text semantic features and the image semantic features, and / or the similarity between the text sentiment features and the image sentiment features, a consistency score for the first image is determined;

[0127] Image recommendations are made based on the consistency score of the first image.

[0128] Based on any of the above embodiments, the extraction unit is further configured to:

[0129] Image feature extraction is performed on the first image, and the extracted image features are used as the semantic features of the image; and / or,

[0130] Based on the image emotion extraction model, the image emotion features of the first image are extracted. The image emotion extraction model is the part of the facial expression recognition model used to extract facial expression features.

[0131] Based on any of the above embodiments, the generating unit is used for:

[0132] Based on the image generation model, image encoding features corresponding to the text semantic features and the text emotion features are determined, and the image encoding features are decoded to obtain the first image;

[0133] The image generation model is trained based on sample text and sample images corresponding to the sample text.

[0134] Based on any of the above embodiments, the generating unit is used for:

[0135] Receive an editing operation on the second image, cover the area corresponding to the editing operation in the second image, and obtain a third image;

[0136] Based on the image generation model, the text semantic features, the text sentiment features, and the image coding features of the third image are applied to predict the region coding features of the occluded area in the third image, and the region coding features are decoded to obtain the first image.

[0137] Based on any of the above embodiments, the generating unit is used for:

[0138] The sample image is masked to obtain a masked image;

[0139] Using the semantic features and sentiment features of the sample text as samples, and the sample images as labels, the model is trained to obtain a preliminary generation model;

[0140] Using the semantic features and sentiment features of the sample text, as well as the image encoding features of the covered image, as samples and the sample image as a label, the preliminary generation model is fine-tuned to obtain the image generation model.

[0141] Based on any of the above embodiments, the extraction unit is further configured to:

[0142] Based on the text emotion extraction model, the text emotion features of the first text are extracted. The text emotion extraction model and the text emotion classification model constitute a text emotion recognition model. The text emotion classification model is used to classify emotions based on the text emotion features.

[0143] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute an image generation method, which includes: acquiring first text; extracting semantic features and sentiment features of the first text; and generating a first image corresponding to the first text based on the semantic features and the sentiment features.

[0144] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image generation method provided by the above methods. The method includes: acquiring a first text; extracting text semantic features and text sentiment features of the first text; and generating a first image corresponding to the first text based on the text semantic features and the text sentiment features.

[0146] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an image generation method provided by the methods described above, the method comprising: acquiring a first text; extracting text semantic features and text sentiment features of the first text; and generating a first image corresponding to the first text based on the text semantic features and the text sentiment features.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image generation method, characterized in that, include: Get the first text; Extract the semantic features and sentiment features of the first text; Based on the semantic features and emotional features of the text, a first image corresponding to the first text is generated; The step of generating a first image corresponding to the first text based on the text semantic features and the text sentiment features includes: The system receives an editing operation on the second image, and covers the area corresponding to the editing operation in the second image to obtain a third image; the second image refers to the image that the user expects to modify locally in conjunction with the first text. Based on the image generation model, the text semantic features, the text sentiment features, and the image coding features of the unmasked area of ​​the third image are applied to predict the region coding features of the masked area in the third image, and the region coding features are decoded to obtain the first image.

2. The image generation method according to claim 1, characterized in that, The step of generating a first image corresponding to the first text based on the text semantic features and the text sentiment features, further includes: Extract the image semantic features and / or image emotion features of the first image; Based on the similarity between the text semantic features and the image semantic features, and / or the similarity between the text sentiment features and the image sentiment features, a consistency score for the first image is determined; Image recommendations are made based on the consistency score of the first image.

3. The image generation method according to claim 2, characterized in that, The extraction of image semantic features and / or image emotion features from the first image includes: Image feature extraction is performed on the first image, and the extracted image features are used as the semantic features of the image; and / or, Based on the image emotion extraction model, the image emotion features of the first image are extracted. The image emotion extraction model is the part of the facial expression recognition model used to extract facial expression features.

4. The image generation method according to any one of claims 1 to 3, characterized in that, The image generation model is trained based on sample text and sample images corresponding to the sample text.

5. The image generation method according to claim 4, characterized in that, The training steps of the image generation model include: The sample image is masked to obtain a masked image; Using the semantic features and sentiment features of the sample text as samples, and the sample images as labels, the model is trained to obtain a preliminary generation model; Using the semantic features and sentiment features of the sample text, as well as the image encoding features of the covered image, as samples and the sample image as a label, the preliminary generation model is fine-tuned to obtain the image generation model.

6. The image generation method according to any one of claims 1 to 3, characterized in that, The steps for extracting the emotional features of the text include: Based on the text emotion extraction model, the text emotion features of the first text are extracted. The text emotion extraction model and the text emotion classification model constitute a text emotion recognition model. The text emotion classification model is used to classify emotions based on the text emotion features.

7. An image generation apparatus, characterized in that, include: Text unit, used to obtain the first text; The extraction unit is used to extract the text semantic features and text sentiment features of the first text; The generation unit is configured to generate a first image corresponding to the first text based on the text semantic features and the text sentiment features; The generation unit is specifically used for: The system receives an editing operation on the second image, and covers the area corresponding to the editing operation in the second image to obtain a third image; the second image refers to the image that the user expects to modify locally in conjunction with the first text. Based on the image generation model, the text semantic features, the text sentiment features, and the image coding features of the unmasked area of ​​the third image are applied to predict the region coding features of the masked area in the third image, and the region coding features are decoded to obtain the first image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image generation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image generation method as described in any one of claims 1 to 6.

Images