Image display method and device and electronic equipment

Abstract

The invention discloses an image display method and device and electronic equipment, and belongs to the technical field of electronics. The image display method comprises the following steps: receiving first input information, and generating a first text based on the first input information; and displaying a first image described by the first text based on the first text.

Description

Technical Field

[0001] The present application belongs to the field of electronic technology, and specifically relates to an image display method, device and electronic equipment. Background Art

[0002] Stable Diffusion is a text-to-image model based on deep learning. It can not only perform text-to-image tasks, but also image-to-image tasks, image super-resolution adjustment tasks, and other computer vision tasks.

[0003] In the related technology, in order to improve the ability of the text-generated graph model to understand the text input by the user, electronic devices usually use the T5 large language model to generate large-scale, high-quality training samples for the text-generated graph model, and then train the text-generated graph model based on these training samples to improve the consistency between the images generated by the text-generated graph model and the input text.

[0004] Although the above method can improve the consistency between text and images of the text-generated graph model, the text-generated graph model still has difficulty understanding text with complex semantics and cannot understand the deep meaning of the text, which may result in the final generated image not meeting user needs. Therefore, the above method of generating images using the text-generated graph model has low accuracy. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image display method, device and electronic device, which can improve the accuracy of generating images using a Wensheng graph model.

[0006] In a first aspect, an embodiment of the present application provides an image display method, the method comprising: receiving first input information; generating first text based on the first input information; and displaying a first image described by the first text based on the first text.

[0007] In a second aspect, an embodiment of the present application provides an image display method device, the device comprising a receiving module, a generating module and a display module. The receiving module is used to receive first input information. The generating module is used to generate a first text based on the first input information received by the receiving module. The display module is used to display a first image described by the first text based on the first text generated by the generating module.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the image display method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the image display method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the image display method as described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the image display method as described in the first aspect.

[0012] In the embodiment of the present application, first input information is received; based on the first input information, a first text is generated; based on the first text, a first image described by the first text is displayed. In this way, after the user inputs the first input information, the electronic device recognizes the semantics of the first input information, and adjusts the first input information according to the semantics to obtain a first text that is easier for the text graph model to understand, so that the first image displayed based on the first text is more compatible with the first input information input by the user, the image quality is better, and the accuracy of the generated image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flowcharts of the image display method provided in the embodiment of the present application;

[0014] Figure 2 It is a schematic diagram of the model architecture provided in the embodiment of the present application;

[0015] Figure 3 This is the second flow chart of the image display method provided in the embodiment of the present application;

[0016] Figure 4a It is one of the schematic diagrams of the interactive interface provided in the embodiment of the present application;

[0017] Figure 4b This is the second schematic diagram of the interactive interface provided in the embodiment of the present application;

[0018] Figure 4c This is the third schematic diagram of the interactive interface provided in the embodiment of the present application;

[0019] Figure 5 This is the fourth schematic diagram of the interactive interface provided in the embodiment of the present application;

[0020] Figure 6 This is the third flow chart of the image display method provided in the embodiment of the present application;

[0021] Figure 7 This is the fourth flow chart of the image display method provided in the embodiment of the present application;

[0022] Figure 8a This is the fifth schematic diagram of the interactive interface provided in the embodiment of the present application;

[0023] Figure 8b This is the sixth schematic diagram of the interactive interface provided in the embodiment of the present application;

[0024] Fig. 9 It is one of the schematic diagrams of the image provided in the embodiment of the present application;

[0025] Fig.10 This is the second schematic diagram of the image provided in the embodiment of the present application;

[0026] Fig.11 This is the third schematic diagram of the image provided in the embodiment of the present application;

[0027] Fig.12 This is the fourth schematic diagram of the image provided in the embodiment of the present application;

[0028] Fig.13 is a flow chart of an image display method provided in an embodiment of the present application;

[0029] Fig.14 is a flow chart of an image display method provided in an embodiment of the present application;

[0030] Fig.15 is a schematic structural diagram of an image display device provided in an embodiment of the present application;

[0031] Fig.16 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0032] Fig.17 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0034] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0036] First, the technologies involved in the embodiments of the present application are explained.

[0037] Stable Diffusion: is a deep learning-based text-to-image model that can generate new images with high realism and artistry by learning the underlying rules in a large amount of image data. This model uses a diffusion process to generate images, starting from a random noise image and gradually adding semantic information until it finally generates an image that matches the input text description. It can perform computer vision tasks such as text-to-image tasks, image-to-image tasks, image local modification tasks, image elimination tasks, and image expansion tasks.

[0038] Text-to-image task: The task of generating images based on text.

[0039] Image-to-image task: The task of processing images based on text to generate new images.

[0040] LLM: The full name of LLM is Large Language Model. It is a large language model and an advanced artificial intelligence technology in the field of deep learning. It can understand and generate natural language through training with massive text data. It is widely used in scenarios such as chatbots, text generation and language translation.

[0041] VLM: Visual Language Model, a multimodal artificial intelligence model that can process both visual information such as images and language information such as text. VLM combines computer vision and natural language processing technologies to perform a variety of tasks, such as image description, visual question answering, document understanding, and image caption generation.

[0042] CLIP: The full name of CLIP is Contrastive Language-Image Pre-training. It is a multimodal model of image + language. Its core function is to establish the correlation between images and texts. Through pre-training with a large amount of image and text data, CLIP can understand the relationship between images and texts and realize multimodal learning.

[0043] ViT: The full name of Vision Transformer is Vision Transformer, which is used to handle image recognition tasks and computer vision tasks.

[0044] Prompt: In artificial intelligence (AI) painting, prompt refers to the text description or keywords provided by the user to the AI ​​system to guide the AI ​​to generate images of a specific style or theme. These prompts have a direct impact on the content, style, color, etc. of the AI-generated images.

[0045] The image display method, device and electronic device provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0046] The image display method provided in the embodiment of the present application can be applied to text-generated images or image-generated image scenarios.

[0047] In the text-based image scenario, the user inputs a prompt word in the input box of the interactive interface, and the electronic device generates an image described by the prompt word based on the prompt word through the text-based image model, and displays the image on the interactive interface.

[0048] In the text-generated image scenario, the user enters a prompt word and an image in the input box of the interactive interface. The electronic device processes the image based on the prompt word through the text-generated image model, generates a new image, and displays the new image on the interactive interface.

[0049] To this end, an embodiment of the present application provides an image display method, which receives first input information; generates a first text based on the first input information; and displays a first image described by the first text based on the first text. In this way, after the user inputs the first input information, the electronic device recognizes the semantics of the first input information, and adjusts the first input information according to the semantics to obtain a first text that is easier for the text-based graph model to understand, so that the first image displayed based on the first text is more compatible with the first input information input by the user, the image quality is better, and the accuracy of the generated image is improved.

[0050] The execution subject of the image display method provided in the embodiment of the present application may be an image display device. Exemplarily, the image display device may be an electronic device, or a functional component or functional entity in the electronic device. The image display method provided in the embodiment of the present application will be described exemplarily below by taking the execution subject as an electronic device as an example.

[0051] Figure 1 is a flow chart of an image display method provided in an embodiment of the present application, such as Figure 1 As shown, the image display method provided in the embodiment of the present application may include the following steps 101 to 103.

[0052] Step 101: An electronic device receives first input information.

[0053] In some embodiments of the present application, the first input information may be information input by a user, and the first input information may be used to inform the electronic device of the content of the image required by the user, and may also be used to instruct the electronic device to generate an image based on the first input information.

[0054] In some embodiments of the present application, the first input information may be the second prompt word, or the first input information may include the second prompt word and the second image.

[0055] In some embodiments of the present application, the first input information is a second prompt word, and the first input information can be used to instruct the electronic device to generate an image described by the second prompt word, or can be used to instruct the electronic device to generate an image including the content described by the second prompt word.

[0056] In some embodiments of the present application, the first input information includes a second prompt word and a second image, and the first input information can be used to instruct the electronic device to generate an image including the content described by the second prompt word and the content of the second image.

[0057] In some embodiments of the present application, when a user needs to generate an image, the electronic device can start an image generation application or function and display an interactive interface, and the user can input first input information in the interactive interface.

[0058] Furthermore, the above-mentioned interactive interface may display an interactive control and an input box, and the user may input the first input information in the interactive interface through the input box, or through the interactive control and the input box. Exemplarily, the above-mentioned interactive control may include at least one of a voice input control, an option control, a send control, an image upload control, and the like.

[0059] In some embodiments of the present application, when the user inputs the second prompt word, the user can edit and input the second prompt word in the input box. Alternatively, the user can trigger the interactive control to record the voice, and the electronic device converts the recorded voice into text to input the second prompt word.

[0060] In some embodiments of the present application, when the user inputs the second image, the user can trigger the interactive control to enter the image upload interface. The user captures a new image or selects an existing image and uploads it in the image upload interface to input the second image.

[0061] Example 1, in the Wenshengtu scenario, taking the above first input information as the second prompt word as an example, assuming that the interactive interface includes an interactive control and an input box, and the interactive control is a send control, the user can enter the second prompt word in the input box of the interactive interface and trigger the send control of the interactive interface. In this way, the user enters the second prompt word into the electronic device, and the electronic device receives the second prompt word.

[0062] For example, the interactive interface of the electronic device includes an input box, in which the user enters the second prompt word "a girl wearing a white skirt and long black hair" and clicks a send control to input the second prompt word.

[0063] Example 2, in the image-generating scene, taking the above-mentioned first input information including the second prompt word and the second image as an example, it is assumed that the interactive interface includes an interactive control and an input box, and the interactive control includes an option control and a send control. The user can first trigger the option control of the interactive interface to select the second image, and enter the second prompt word in the input box of the interactive interface, and then trigger the send control of the interactive interface; or the user can first enter the second prompt word in the input box of the interactive interface, and then trigger the option control of the interactive interface to select the second image, and then trigger the send control of the interactive interface. In this way, the user inputs the second image and the second prompt word to the electronic device, and the electronic device receives the second prompt word.

[0064] For example, the interactive interface of the electronic device includes an input box, a send control, and an option control. After the user clicks the option control, the electronic device displays the option interface. After the user clicks the image upload control on the option interface, the electronic device displays the image upload interface. The user selects the second image on the image upload interface. After clicking the confirmation control, the electronic device displays the second image in the input box. Then the user enters the second prompt word "wear a red bow" in the input box and then clicks the send control. Alternatively, the user first enters the second prompt word "wear a red bow" in the input box and then clicks the option control. The electronic device displays the option interface. After the user clicks the image upload control on the option interface, the electronic device displays the image upload interface. The user selects the second image on the image upload interface and clicks the confirmation control. The input box of the electronic device simultaneously displays the second prompt word and the second image, and the user then clicks the send control. In this way, the input of the second prompt word and the second image is realized.

[0065] Step 102: The electronic device generates a first text based on the first input information.

[0066] In some embodiments of the present application, the electronic device may generate the first text directly based on the first input information after receiving the first input information.

[0067] In some other embodiments of the present application, the electronic device may generate a first text based on the first input information after detecting the user input to the send control of the interactive interface and determining that the user input is completed.

[0068] For example, taking the first input information as the second prompt word as an example, assuming that the interactive interface includes an interactive control and an input box, and the interactive control includes a send control, after the user enters the second prompt word "a girl wearing a white skirt and long black hair" in the input box, clicks the send control of the interactive interface, and the electronic device detects the click input of the send control, it generates a first text based on the first input information.

[0069] For example, taking the first input information including the second prompt word and the second image as an example, assuming that the interactive interface includes an interactive control and an input box, and the interactive control includes an option control and a send control, the user enters the second prompt word "wear a red bow" in the input box, selects the second image through the option control, and clicks the send control to send the second prompt word and the second image. The electronic device detects the click input of the send control and generates a first text based on the first input information.

[0070] In some other embodiments of the present application, after receiving the first input information input by the user, if no user operation is detected within a preset time period, indicating that the user input is completed, the electronic device can generate a first text based on the first input information.

[0071] The above-mentioned preset duration can be set or adjusted according to actual needs, and the embodiment of the present application does not limit this. Exemplarily, the value range of the preset duration can be 2 seconds-10 seconds, for example, the preset duration is 5 seconds.

[0072] For example, taking the first input information as the second prompt word as an example, assuming that the interactive interface includes an interactive control and an input box, and the interactive control includes a send control, the user enters the second prompt word "a girl in a white skirt and long black hair" in the input box, and clicks the send control of the interactive interface. If the electronic device does not detect the user's operation within 5 seconds, a first text is generated based on the second prompt word.

[0073] For example, taking the first input information including the second prompt word and the second image as an example, assuming that the interactive interface includes an interactive control and an input box, and the interactive control includes an option control and a send control, the user enters the second prompt word "wear a red bow" in the input box, selects the second image through the option control, and clicks the send control to send the second prompt word and the second image. If the electronic device does not detect the user's operation within 5 seconds, a first text is generated based on the second prompt word.

[0074] In some embodiments of the present application, the electronic device may polish the first input information through a text generation model to obtain the first text.

[0075] It should be noted that the polishing in the embodiments of the present application can be referred to as prompt polishing. In AI painting, prompt polishing refers to optimizing the text instructions input to the AI ​​model in order to generate an image that better meets the expectations.

[0076] In some embodiments of the present application, the above-mentioned text generation model can be a multimodal VLM model, which is used to identify the semantics of the input information and polish the input information according to the semantics, such as expanding the input information by adding detailed descriptions to obtain richer text and output it. The input information can be information in various modalities such as text and / or images, that is, the text generation model can process the input information in various modalities into text output.

[0077] In some embodiments of the present application, in a text image scene, the first input information may be a second prompt word, and the step 102 may be implemented by the following step 1021 .

[0078] Step 1021: The electronic device identifies semantic information of the second prompt word through the large language model in the text generation model, and adjusts the second prompt word based on the semantic information of the second prompt word to obtain the first text.

[0079] In some embodiments of the present application, the above-mentioned large language model can be any deep learning model that can generate natural language text or understand the semantics of natural language text. For example, the large language model can be a variant based on a transformer model (Transformer model), such as a large language model based on the Transformer bidirectional encoder (Bidirectional Encoder Representation from Transformers, BERT) series, a large language model of the Generative Pre-trained Transformer (GPT) series, etc. These models process input text through a multi-layer Transformer structure to generate output text.

[0080] In some embodiments of the present application, taking the large language model as a GPT series model based on the Transformer architecture as an example, the above-mentioned large language model may include an input layer (Input Layer), an embedding layer (Embedding Layer), a positional encoding layer (Positional Encoding Layer), a Transformer encoder layer, an output layer (Output Layer) and a sampling layer (Sampling Layer).

[0081] In some embodiments of the present application, the electronic device inputs the second prompt word into the above-mentioned input layer, and converts the second prompt word into a digital form through the input layer, that is, tokenizes the second prompt word, treats each character or each word in the second prompt word as a segmentation, and then maps each segmentation word to a unique digital identifier (Identity document, ID) to obtain a token ID for each segmentation word. Then, according to the order of each segmentation word in the second prompt word, the token ID of each segmentation word is combined into a sequence of token IDs, and a special start (such as <s> ) and end (such as< / s> )mark.

[0082] In some embodiments of the present application, the electronic device inputs the sequence token IDs output by the input layer into the embedding layer, and maps each token ID in the sequence into a high-dimensional vector space through the embedding layer to obtain an embedding vector (embedding) for each token, and obtains an embedding matrix with a shape of [sequence length, embedding dimension], where each element in the matrix is ​​an embedding vector of a token, and these embedding vectors contain the semantic information of each word segment.

[0083] In some embodiments of the present application, the electronic device inputs the embedding matrix output by the embedding layer into the position encoding layer, and adds position information to the embedding vector of each token through the position encoding layer, so that the large language model can distinguish the same words in different positions. The position encoding can be fixed or learnable, and finally an embedding matrix with position information is obtained.

[0084] In some embodiments of the present application, the electronic device inputs the positional embedding matrix output by the position encoding layer into the Transformer encoder layer, calculates the relationship between each token and other tokens in the positional embedding matrix through the multi-head self-attention mechanism (Multi-Head Attention) in the Transformer encoder layer, and updates the embedding vector of each token, and then sequentially passes through the feed-forward neural network (Feed-Forward Neural Network, FFN), layer normalization (Layer Normalization) and residual connections (Residual Connections) in the Transformer encoder layer to obtain the updated vector representation of each token.

[0085] In some embodiments of the present application, the electronic device inputs the vector representation of each token output by the Transformer encoder layer into the output layer, maps the vector representation of each token to the space of the vocabulary size through the output layer, obtains the score of each token corresponding to each possible output vocabulary, and then obtains a probability distribution matrix with a shape of [sequence length, vocabulary size], in which each element of the matrix represents the probability of generating the corresponding vocabulary at a given position.

[0086] In some embodiments of the present application, the electronic device inputs the probability distribution matrix output by the output layer into the sampling layer, performs sampling according to the probability distribution through the sampling layer, selects the most likely vocabulary or selects vocabulary according to a certain strategy, and then converts each token in the token sequence obtained from the sampling results into a segmentation, and then merges the converted segmentations to obtain the first text.

[0087] In some embodiments of the present application, the large language model learns a large number of language patterns and rules during the training process. In the process of polishing the second prompt word, these learned patterns and rules can be used to generate a more detailed and natural first text. In addition, the large language model can generate corresponding details by understanding the theme, emotion, style, etc. of the second prompt word text, and can also retrieve information related to the second prompt word in its knowledge base to generate more accurate and richer detailed descriptions, and then adjust the second prompt word according to these detailed descriptions to obtain the first text.

[0088] For example, in a Wensheng picture scenario, assuming that the second prompt word input by the user is "Draw a young woman standing on a green lawn", after the electronic device polishes the second prompt word through the Wensheng picture model, the first text obtained is "Draw a young woman, she is standing on the green lawn, wearing a light green long skirt with small white flowers dotted on the skirt, which looks very fresh and natural. She wears white socks and black shoes on her feet, with a straw hat of the same color. She holds a bunch of white flowers in her hand, which looks very sweet and peaceful."

[0089] In some embodiments of the present application, when the second prompt word is a prompt word that is difficult to understand, such as poetry, idioms, allegorical sayings, abstract descriptions, abbreviations, etc., the semantic information of the second prompt word is identified through a large language model, and the semantic information can be understood as a word explanation of the second prompt word. Based on the semantic information, the second prompt word is expanded with the help of the knowledge base of the large language model, that is, descriptions related to the second prompt word are searched from the knowledge base, and these descriptive contents are added on the basis of the second prompt word. In this way, the second prompt word can be converted into a first text that is easier to understand, and the model can generate an image with better quality based on the first text.

[0090] For example, assuming that the second prompt word is "a lone sail in the distance", after the second prompt word is recognized by the large language model, the semantic information of the second prompt word obtained can be "a lone sailboat gradually disappears in the distance", and after the second prompt word is expanded based on the semantic information, the first text obtained can be "an ink painting depicting a lone sail in the distance, a lone sailboat gradually goes away in the vast ocean, with hazy mountains and light clouds in the background. The picture uses simple lines and elegant tones to create a tranquil and distant atmosphere.".

[0091] For example, assuming that the second prompt word is "a waterfall plunges three thousand feet", after the second prompt word is recognized by the large language model, the semantic information of the second prompt word can be "the waterfall falling straight down from the high cliff seems to be several thousand feet high", and after the second prompt word is expanded based on the semantic information, the first text obtained can be "the spectacular waterfall rushes down from the towering cliffs, like a giant silver dragon, with the force of thunder, falling three thousand feet, hitting the rocks and water below, splashing countless crystal water, as if millions of dazzling diamonds are shining in the sun, forming a shocking natural wonder.".

[0092] For example, assuming that the second prompt word is "the formula of life", after the second prompt word is recognized by the large language model, the semantic information of the second prompt word obtained can be "chemical formulas and vibrant plants". After the second prompt word is expanded based on the semantic information, the first text obtained can be "in a laboratory, a scientist holds a test tube with glowing liquid in it, the background is a blackboard full of chemical formulas, and next to it is a growing plant, symbolizing the miracle of life.".

[0093] In some embodiments of the present application, when the second prompt word is a simple or brief description prompt word, the large language model can recognize the semantic information of the second prompt word, and use the knowledge base of the large language model to add detailed descriptions to the second prompt word, such as background content, character clothing, lighting, scene mood, image style, etc., so that the second prompt word is richer, the image drawn by the model is more natural, and the accuracy of image generation is improved.

[0094] For example, assuming that the second prompt word is "draw a rabbit that always fights", after the second prompt word is recognized by the large language model, the semantic information of the second prompt word can be "draw a rabbit, and show through some details that this rabbit can fight", and after the second prompt word is expanded based on the semantic information, the first text obtained can be "draw a rabbit that can fight, wearing red boxing gloves, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats and illuminated by spotlights. Colorful comics.".

[0095] For example, assuming that the second prompt word is "draw a female painter", after the second prompt word is recognized by the large language model, the semantic information of the second prompt word can be "draw a woman, and show that the woman is a painter through the scene". After expanding the second prompt word based on the semantic information, the first text obtained can be "In the beautiful mountains and forests, a gentle female painter, wearing a simple long skirt, holding a palette and a brush, is concentrating on depicting the scenery in front of her."

[0096] In this way, the electronic device understands the user's intention of inputting the second prompt word through the text generation model, identifies the semantic information of the second prompt word, supplements the detailed content according to the semantic information of the second prompt word, adjusts the second prompt word, and obtains a first text with richer content. The user does not need to rack his brains to input rich prompt words, which reduces the difficulty of user input.

[0097] In some embodiments of the present application, in a picture-to-picture scenario, the first input information may include a second prompt word and a second image. The above step 102 may be implemented by the following steps 1022 to 1024.

[0098] Step 1022: The electronic device encodes the second image through the image encoder in the text generation model to obtain image features, and encodes the second prompt word through the text encoder in the text generation model to obtain first text features.

[0099] In some embodiments of the present application, the image encoder may be an encoder or model capable of encoding images, such as CLIPViT. The text encoder may be an encoder or model capable of encoding text, such as CLIPText Encoder or BERT.

[0100] It should be noted that the above image features and the first text features are both represented in the form of vectors.

[0101] In some embodiments of the present application, taking CLIPViT as an example, the image encoder is used to encode the second image through the image encoder, and the implementation process of obtaining image features may include: first, the second image is divided into image blocks (patches) of fixed size through the image encoder, such as dividing the image into 196 image blocks, each of which is 16x16 pixels in size. Then, each image block is linearly projected and mapped to a vector space of fixed dimension, such as linearly projecting the 768 pixel values ​​contained in each small image block to a 768-dimensional embedding space to obtain an embedding vector for each image block. Then, a position code is added to each image block, and the position code is a fixed vector associated with the position of the image block to obtain an embedding vector including the position code for each image block. Then, the embedding vectors including the position code of multiple image blocks are spliced ​​to obtain a sequence, and the sequence is subjected to feature extraction using a multi-head self-attention mechanism to obtain a deep feature representation of each image block, and then the output deep feature representations of all image blocks are averaged and pooled to obtain the image features of the second image.

[0102] In this way, the image features obtained by the image encoder include global features and local features of the second image, and can more accurately represent the second image.

[0103] In some embodiments of the present application, taking the text encoder as BERT as an example, the implementation process of encoding the second prompt word by the text encoder to obtain the first text feature may include: first preprocessing the second prompt word, including converting uppercase to lowercase, removing special characters and numbers, etc. Then use the word segmenter to cut the second prompt word into words or single words, each word or single word is called a segmentation, and then convert each segmentation into an embedded vector of fixed length, which can characterize the semantic information of the segmentation. Then, add a position code for each segmentation, and the position information is independent of the content of the segmentation. Then, add the position code and the embedded vector of each segmentation to obtain the final vector representation, which contains the semantic information and position information of each segmentation, and splice the vector representations of multiple segmentations to obtain the input sequence. Then, through each Transformer encoder in the text encoder, the input sequence is converted into a context vector (context vector), which captures the semantic information of the segmentation in the context. Then, the output of the Transformer encoder is pooled to obtain a fixed-length vector to represent the entire sequence, and the fixed-length vector is the above-mentioned first text feature.

[0104] It should be noted that the process of the above-mentioned image encoder encoding the second image and the text encoder encoding the second prompt word is only an example. In actual use, the electronic device can use any method that can encode an image to obtain image features, and can use any method that can encode text to obtain text features. The embodiments of the present application do not limit this.

[0105] Step 1023: The electronic device converts the image feature into a second text feature through a linear mapping layer in the text generation model.

[0106] Since image features are a numerical representation of image data, they capture the main information and structure in the image. However, this representation is still different from the representation of text features, and image features are difficult to be understood by models that process text. Therefore, electronic devices need to map image features to text features.

[0107] In some embodiments of the present application, the linear mapping layer may also be referred to as a projection layer, which is used to map image features into text features that can be processed by a model for processing text.

[0108] For example, the electronic device can use an encoder-decoder structure to build a cross-modal conversion model that can encode image features into vectors in a latent space and decode them into text features. Alternatively, the electronic device can use multimodal alignment technology to train a model that can learn the correspondence between image features and text features and realize the conversion between them.

[0109] In this way, after the electronic device encodes the image into image features, it maps the image features into second text features so that they can be input into the large language model for understanding and processing.

[0110] Step 1024: The electronic device identifies first semantic information represented by the first text feature and second semantic information represented by the second text feature through the large language model in the text generation model, and adjusts the second semantic information based on the first semantic information and the second semantic information to obtain the first text.

[0111] In some embodiments of the present application, since the first text feature is obtained by encoding the second prompt word, the first text feature can represent the first prompt word, and the second text feature is obtained by converting the image feature obtained by encoding the second image, the second text feature can represent the second image. Therefore, the first semantic information recognized by the electronic device through the large language model can be understood as the semantic information of the first prompt word, and the second semantic information recognized by the large language model can be understood as the image content of the second image.

[0112] In some embodiments of the present application, the electronic device may splice the first text feature and the second text feature and input them into the large language model, the large language model identifies the first semantic information represented by the first text feature and the second semantic information represented by the second text feature, and merges the first semantic information and the second semantic information to obtain the first text. Exemplarily, the electronic device may add the first semantic information to a suitable position in the second semantic information to obtain the first text.

[0113] For example, assuming that the first semantic information is "put on a wristband on him", and the second semantic information is "a youthful male college student, wearing a red No. 24 jersey, is playing basketball on the basketball court. There are many spectators on the sidelines of the basketball court, cheering enthusiastically, and the atmosphere is quite warm." Then the first text obtained by the large language model is "a youthful male college student, wearing a red No. 24 jersey, wearing a wristband on his right hand, is playing basketball on the basketball court.".

[0114] For example, assuming that the first semantic information is "put on a white sun hat for her" and the second semantic information is "a young woman, she is standing on the green grass, wearing a light green long skirt, the skirt is dotted with small white flowers, she looks very fresh and natural.", then the first text obtained by the large language model is "a young woman, she is standing on the green grass, wearing a white sun hat, wearing a light green long skirt, the skirt is dotted with small white flowers, she looks very fresh and natural.".

[0115] For example, see Figure 2 , the electronic device converts the second image X vInput the image encoder in the text generation model, and use the image encoder to encode the second image X v Encode the image feature Z v , and then the image feature Z v Input linear mapping layer, that is, projection layer W, image feature Z v After mapping, it is converted into the second text feature H v Furthermore, the electronic device responds to the second prompt word X q Encode to get the first text feature H q , and then the second text feature H v and the first text feature H q Merge the input into the large language model to get the first text X a .

[0116] In this way, the electronic device recognizes the intention of the second image and the second prompt word input by the user through the text generation model, and merges the image content of the second image with the second prompt word to obtain a first text that is easier for the text-generated graph model to understand. The image generated based on the first text meets the user's needs and has higher quality. In addition, since the text generation model converts the second image into text, that is, the input of the text-generated graph model is text, it can be understood that the text-generated graph model performs the graph-generated graph task based on the first text. Since there is no image in the input, there is no need to fuse the image information and text information in the text-generated graph model, which can avoid the situation where the image information and text information are incorrectly fused when the related technology implements the graph-generated graph task, thereby improving the accuracy and controllability of image generation.

[0117] Step 103: The electronic device displays a first image described by the first text based on the first text.

[0118] In some embodiments of the present application, the electronic device may input the first text into a text-image model, generate a first image described by the first text through the text-image model, and display the first image on the electronic device.

[0119] In some embodiments of the present application, Figure 1 ,like Figure 3 As shown, the above step 103 can be specifically implemented through the following step 1031.

[0120] Step 1031: The electronic device displays a first image and a first text described by the first text based on the first text.

[0121] In some embodiments of the present application, after the electronic device generates a first image based on the first text, it can not only display the first image on the interactive interface, but also display the first text, so that the user can modify or enter supplementary text based on the first text to generate a new image.

[0122] For example, in the Wenshengtu scenario, Figure 4a As shown, an input box, an option control, and a voice input control are displayed below the interactive interface of the electronic device. After the user enters the second prompt word "Draw a fighting rabbit" in the input box of the interactive interface and sends it, the interactive interface of the electronic device displays the second prompt word, and the electronic device polishes the second prompt word through the VLM model to generate a first text, and then inputs the first text into the Wenshengtu model to generate a first image, and displays the first image and the first text "A fighting rabbit, wearing red boxing gloves, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats and illuminated by spotlights. Colorful comics.". In addition, an edit control and a regenerate control are also displayed below the area where the first text is displayed in the interactive interface, so that the user can edit the first text.

[0123] For example, in a graph-to-graph scenario, Figure 5 As shown, an input box, an option control, and a voice input control are displayed below the interactive interface of the electronic device. After the user inputs the second prompt word in the input box of the interactive interface and selects the second image through the option control, and sends the second prompt word and the second image, the interactive interface of the electronic device displays the second prompt word and the second image, and the electronic device generates a first text based on the second prompt word and the second image through the VLM model, and then inputs the first text into the text graph model to generate the first image, and displays the first image and the first text on the interactive interface. In addition, an edit control and a regenerate control are also displayed below the area where the first text is displayed in the interactive interface, so that the user can edit the first text.

[0124] In this way, the interactive interface of the electronic device can not only display the first image, but also display the first text, which is convenient for the user to re-edit the first text or enter new prompt words based on the first text to obtain new text. That is, the user does not need to re-enter the new text, which reduces the user's operations and improves the user's convenience.

[0125] In some embodiments of the present application, Figure 3 ,like Figure 6 As shown, after the above step 1031, the image display method provided in the embodiment of the present application may further include the following steps 104 and 105.

[0126] Step 104: The electronic device receives editing input for the first text.

[0127] Step 105: In response to the editing input, the electronic device displays an image described by the edited first text based on the edited first text.

[0128] In some embodiments of the present application, the above-mentioned editing input may include adding content, deleting content, or modifying content in the first text.

[0129] In some embodiments of the present application, when the interactive interface of the electronic device displays the first image and the first text, the user can directly edit the first text in the text box displaying the first text, and then the electronic device inputs the edited first text into the text-generated graph model to obtain and display the image described by the edited first text.

[0130] Furthermore, the electronic device may simultaneously display the image of the edited first text description and the edited first text on the interactive interface so that the user can continue the operation.

[0131] Exemplarily, the interactive interface of the electronic device may display an editing control and a regeneration control. After the user triggers the editing control, the displayed first text enters an editable state, and then the user can edit the first text. After the user finishes editing, the regeneration control can be triggered. In response to the trigger input of the regeneration control, the electronic device can input the edited first text into a text-generated graph model, generate an image of the edited first text description through the text-generated graph model, and display it.

[0132] For example, in the Wenshengtu scene, Figure 4b As shown, the interactive interface of the electronic device displays an input box, an option control, and a voice input control, and a first text and a first image are displayed in the interactive interface, and an edit control and a regeneration control are displayed below the area displaying the first text, and the edit control is used to prompt the user to edit again. When the user wants to edit the first text, the edit control can be triggered, and the first text is switched to an editable state. Then the user adds the words "and a red hairband" to the first text, and then triggers the regeneration control on the right side of the edit control. The electronic device sends the edited first text to the text graph model without being processed by the VLM model. Then, as Figure 4c As shown, the electronic device generates an image of the edited first text description through a text graph model, and displays the image of the edited first text description and the edited first text "A fighting rabbit, wearing red boxing gloves and a red headband, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats and illuminated by spotlights. Colorful cartoon." In addition, an editing control and a regeneration control can also be displayed below the area displaying the edited first text in the interactive interface, so that the user can continue editing or continuing the conversation based on the edited first text.

[0133] For example, in the image-generating scene, Figure 5As shown, the interactive interface of the electronic device displays an input box, an option control, and a voice input control, and the interactive interface displays a second prompt word "sky plus sun", a second image, a first text "Chinese ink painting style. The center of the picture is a person wearing red clothes, standing on the edge of a cliff. There is a narrow stone path on the edge of the cliff, and the person is gazing into the distance. The background is a vast ocean, the sky intersects with the sea level, and there is a red sunset in the sky." and the first image, and an edit control and a regeneration control are displayed below the area displaying the first text, and the edit control is used to prompt the user to edit again. When the user wants to edit the first text, the edit control can be triggered, and the first text is switched to an editable state, and then the user adds content to the first text, and then triggers the regeneration control on the right side of the edit control, and the electronic device sends the edited first text to the text graph model without being processed by the VLM model. Then, the electronic device generates an image described by the edited first text through the text graph model, and displays the image described by the edited first text and the edited first text on the interactive interface, so that the user can continue editing or continue the conversation based on the edited first text. Furthermore, an editing control and a regeneration control may be displayed below the area displaying the edited first text in the interactive interface, so that the user can edit the edited first text.

[0134] In this way, when the first text and the first image are displayed on the interactive interface, if the user wants to modify the prompt word to modify the image, the first text can be quickly edited in the text box, and the edited text does not need to be polished by the text generation model, but can be directly input into the text-image model to generate a new image, thereby improving the efficiency and accuracy of generating new images.

[0135] In some embodiments of the present application, Figure 3 ,like Figure 7 As shown, after the above step 1031, the image display method provided in the embodiment of the present application may further include the following steps 106 to 108.

[0136] Step 106: The electronic device receives a first prompt word.

[0137] Step 107: The electronic device generates a second text based on the first prompt word and the first text.

[0138] Step 108: The electronic device displays an image described by the second text based on the second text.

[0139] In some embodiments of the present application, the first prompt word may be content added by the user based on the first text.

[0140] In some embodiments of the present application, after receiving the first prompt word, the electronic device can combine the second prompt word with the first text, and when it is determined that the second prompt word has a contextual relationship with the first text, use the first prompt word to polish the first text through a text generation model, or use the first text to polish the first prompt word to obtain the second text.

[0141] In some embodiments of the present application, when the interactive interface of the electronic device displays the first image and the first text, the user can continue to enter the first prompt word in the input box, the electronic device inputs the first prompt word into the text generation model, merges the first prompt word and the first text through the text generation model to obtain the second text, and then inputs the second text into the text-generated graph model, generates an image described by the second text through the text-generated graph model and displays it.

[0142] Furthermore, the electronic device may simultaneously display the image described by the second text and the second text on the interactive interface so that the user can continue the operation.

[0143] Exemplarily, the interactive interface of the electronic device may display an input box and a send control, in which the user may enter a first prompt word in the input box, and then trigger the send control to send the first prompt word to the text generation model, the text generation model identifies the semantic information of the first prompt word and the semantic information of the first text, and based on the semantic information of the first prompt word and the semantic information of the first text, adds the first prompt word to an appropriate position of the first text to obtain a second text, and then inputs the second text into the text-generated graph model, and generates and displays an image described by the second text through the text-generated graph model.

[0144] For example, in the Wenshengtu scene, Figure 8a As shown, an input box, a voice input control and a sending control are displayed below the interactive interface of the electronic device, and the interactive interface displays a second prompt word "Draw a fighting rabbit", a first text "A fighting rabbit, wearing red boxing gloves, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats and illuminated by spotlights. Colorful cartoons." and a first image. When the user wants to modify the first image, the user can enter the first prompt word "Put a red headband on it" in the input box and trigger the sending control to send it to the VLM model. As shown in FIG. Figure 8b As shown, after receiving the first prompt word, the VLM model merges the first prompt word with the first text of the previous round to obtain the second text, and then inputs the second text into the text-image model to obtain the image described by the second text, and displays the image described by the second text and the second text.

[0145] For example, in a graph-generated image scene, an input box, a voice input control, and a send control are displayed below the interactive interface of the electronic device, and a first text and a first image are displayed in the interactive interface. When the user wants to modify the first image, the first prompt word "add a person on the cliff" can be entered into the input box, and the send control can be triggered to send to the VLM model. After receiving the first prompt word, the VLM model merges the first prompt word with the first text of the previous round to obtain the second text, and then inputs the second text into the text-generated image model to obtain the image described by the second text and display it.

[0146] In this way, when the first text and the first image are displayed on the interactive interface, if the user wants to modify the prompt word to modify the image, he can continue to enter the new content in the input box. The electronic device can combine the new content with the first text of the previous round to understand the intention, and then generate a second text input into the text-image model. The user does not need to input a long text, which improves the efficiency of generating new images. In addition, the second text is obtained after being polished by multiple rounds of user input and meets the user's latest needs. Therefore, the image generated based on the second text has higher accuracy.

[0147] In some embodiments of the present application, before using the text generation model, the electronic device may first train the pre-trained model to obtain the text generation model. That is, before the above step 102, the image display method provided in the embodiment of the present application may also include the following steps 201 to 203.

[0148] Step 201: The electronic device obtains at least two sample groups.

[0149] In some embodiments of the present application, each sample group in the at least two sample groups may include at least one input sample and an expected output result corresponding to each input sample.

[0150] In some embodiments of the present application, when a sample group includes at least two input samples, every two adjacent input samples of the at least two input samples are in a containment relationship. When a sample group includes at least two expected output results, every two adjacent expected output results of the at least two expected output results are in a containment relationship.

[0151] In some embodiments of the present application, in a text-to-image scenario, an input sample is a text sample, and the expected output result corresponding to the input sample is a polished text sample. In an image-to-image scenario, an input sample includes a text sample and an image sample, and the expected output result corresponding to the input sample includes a description text of the image sample, a description text based on the image sample, and a polished text obtained from the text sample.

[0152] For example, in a Wensheng picture scenario, if the number of at least one input sample is one, the electronic device can use a reverse deduction method to construct a sample group, based on the high-quality drawing prompt words collected on the Internet as the expected output result, and manually extract the main content as the input sample. The following is an example of a sample group: [{"input sample": "draw a picture of a fighting rabbit", "expected output result": "a fighting rabbit, wearing red boxing gloves, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats and illuminated by spotlights. Color comics."}].

[0153] Exemplarily, in the Wenshengtu scene, if the number of at least one input sample is two, the electronic device can continue to expand a round of input samples and expected output results based on the above input sample and expected output result. The following is an example of a sample group: [{"input sample": "Draw a picture of a rabbit that can fight", "expected output result": "A rabbit that can fight, wearing red boxing gloves, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats, there are spotlights shining. Color cartoon."}, {"input sample": "Give it a red headband", "expected output result": "A rabbit that can fight, wearing red boxing gloves and a red headband, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats, there are spotlights shining. Color cartoon."}].

[0154] Exemplarily, in a graph-generated image scene, if the number of at least one input sample is one, the input sample includes an image and text. The expected output result corresponding to the input sample includes an output description and an output polish, wherein the output description is a description of the input image content, and the output polish is a polished text obtained by modifying the text based on the description of the image content and the text polish. The following is an example of a sample group: [{"input sample": image, "add a red hairband to it", "output description": "A fighting rabbit, wearing red boxing gloves, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats, there are spotlights. Color comics.", "output polish": "A fighting rabbit, wearing red boxing gloves and a hairband, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats, there are spotlights. Color comics."}].

[0155] Exemplarily, in a graph-to-graph scenario, if the number of at least one input sample is two, the electronic device may continue to expand a round of input samples and expected output results based on the above input sample and expected output result. The following is an example of a sample group: [{"input sample": image, "add a red headband to it", "output description": "A fighting rabbit, wearing red boxing gloves, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats, with spotlights shining on it. Color cartoon.", "output polish": "A fighting rabbit, wearing red boxing gloves and a red headband, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats, with spotlights shining on it. Color cartoon."}, {"input sample": "add a pair of wrist guards to it", "output description": "A fighting rabbit, wearing red boxing gloves and a red headband, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats, with spotlights shining on it. Color cartoon.", "output polish": "A fighting rabbit, wearing red boxing gloves, a pair of wrist guards and a red headband, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats, with spotlights shining on it. Color cartoon."}].

[0156] Step 202: The electronic device inputs at least one input sample into the pre-trained model to obtain an actual output result corresponding to each input sample.

[0157] Step 203: The electronic device trains the pre-trained model based on the actual output result corresponding to each input sample and the expected output result corresponding to each input sample to obtain a text generation model.

[0158] In some embodiments of the present application, the above-mentioned pre-trained model may be a pre-trained VLM model, which includes a pre-trained image encoder, a large language model, and a linear mapping layer, wherein the image encoder is used to encode the input image, the large language model is used to polish the input text, and the linear mapping layer can convert image features into text features that can be received by the large language model. This model structure can process images and texts simultaneously to achieve multimodal understanding.

[0159] In some embodiments of the present application, when training the pre-trained model, the electronic device can use a large-scale distributed training framework to facilitate rapid iteration. The training uses the pre-trained VLM model and uses the above sample group for fine-tuning. After the loss function decreases to a stable level or decreases to below a preset threshold, the training is stopped.

[0160] In some embodiments of the present application, the electronic device can select multiple groups of model parameters with smaller loss functions, and determine the model corresponding to each group of model parameters, and use a test set to evaluate the model corresponding to each group of model parameters. The evaluation dimensions include the model's dialogue comprehension accuracy, the aesthetics of the output image, and the consistency of the image and text in the output image. The best model parameters are selected as the model parameters finally used by the text-based image model.

[0161] The image display method provided in the embodiment of the present application uses the VLM model to provide a more powerful image and text understanding capability, and establishes a bridge for interaction between the first input information and the Wensheng graph model. By polishing the information input by the user, automatically converting the first input information into text suitable for the Wensheng graph model to understand, and supplementing the detailed description, not only can the quality of the generated image be improved, but also the difficulty of user input can be reduced. Moreover, for the Wensheng graph mode in which images and text are input at the same time, the VLM model is used to uniformly convert the image and text into polished text and input it into the Wensheng graph model, which has strong interpretability, and the text can be manually adjusted later to enhance the controllability of the painting.

[0162] Next, combine Figures 9 to 12 , the use effect of the image display method provided in the embodiment of the present application is explained.

[0163] Fig. 9 a in the figure is an image generated based on the unpolished prompt word (such as the second prompt word mentioned above) “Lonely sail in the distance”. Fig. 9 The b in the picture is based on the polished prompt (such as the first text above): "An ink painting depicting a lone sail in the distance, with a small boat drifting away in the vast ocean, and the background is hazy mountains, light clouds and migrating geese. The picture uses simple lines and elegant colors to create a quiet and distant atmosphere." The image is generated. Fig. 9 It can be seen that when the Wensheng graph model draws the picture according to the unpolished prompt words, it does not understand the artistic conception of the poem, and the generated image does not conform to the intention expressed by the prompt words. However, the image generated according to the polished prompt words reflects the artistic conception of the poem, conforms to the thoughts and feelings expressed in the poem, and is more accurate.

[0164] Fig.10 a in the figure is an image generated based on the unpolished prompt word (such as the second prompt word mentioned above) "Recipe for Life", Fig.10 The image b in the figure is generated based on the polished prompt (such as the first text above): "In a laboratory, a scientist holds a test tube with glowing liquid in it, the background is a blackboard full of chemical formulas, and there is a growing plant next to it, symbolizing the miracle of life." Fig.10It can be seen that when the Wensheng graph model drew the picture based on the unpolished prompt word, it did not understand the meaning of the abstract prompt word "recipe of life", and the generated image was food, which did not conform to the intention of the prompt word. The image generated based on the polished prompt word better explained the picture of "recipe of life" and was more accurate.

[0165] Fig.11 a in the image is generated based on the unpolished prompt word (such as the second prompt word mentioned above) "a cute cat wallpaper", Fig.11 The b in the image is based on the polished prompt word (such as the first text above): "A wallpaper of a cute cat. The cat is sitting on the windowsill. Outside the window is a starry sky and a bright moon. The cat's eyes are sparkling, as if they are in harmony with the starry sky. The background is a quiet night, and the breeze blows the curtains. The whole picture is full of mystery and fantasy." The image is generated. Fig.11 It can be seen that the image generated by the text-based image model based on the unpolished prompt words has no problem, but the picture is relatively monotonous. The image generated based on the polished prompt words has rich pictures and full details, which is convenient for users to edit again.

[0166] Fig.12 a in the image is generated based on the unpolished prompt word (such as the second prompt word mentioned above) "draw a rabbit that fights", Fig.11 The image b in the figure is generated based on the polished prompt words (such as the first text above): "A rabbit that can fight, wearing boxing gloves, with a firm expression and a flexible body. In a simple boxing ring, surrounded by audience seats." Fig.12 It can be seen that the image generated by the text-based image model based on the unpolished prompt words has no problem, but the picture is relatively monotonous. The image generated based on the polished prompt words has rich pictures and full details, which is convenient for users to edit again.

[0167] Fig.13 301 to 307 are steps of the image display method provided in the embodiment of the present application. Taking the Wenshengtu scene as an example, the method may include the following steps 301 to 307.

[0168] Step 301: an interactive interface of an electronic device displays an input box.

[0169] Step 302: The electronic device receives a second prompt word input by the user in the input box.

[0170] Step 303: In response to the input of the second prompt word, the electronic device inputs the second prompt word into the VLM model to obtain the first text.

[0171] Step 304: The electronic device inputs the first text into Stable Diffusion to generate a first image described by the first text.

[0172] Step 305: The electronic device displays the first image and the first text.

[0173] Step 306: The electronic device receives the user's editing input on the first text, and then inputs the edited first text into Stable Diffusion to generate an image described by the edited first text.

[0174] Step 307: The electronic device receives the first prompt word input by the user in the input box of the interactive interface, and then inputs the first prompt word into the VLM model. The VLM model combines the first text of the previous round with the first prompt word to generate an edited first text, and then inputs the edited first text into Stable Diffusion to generate an image described by the edited first text.

[0175] Step 308: The electronic device displays the edited image of the first text description and the edited first text.

[0176] It should be noted that the specific implementation process of the above steps 301 to 308 can refer to the relevant description of the above embodiment. To avoid repetition, this embodiment will not be repeated here.

[0177] Fig.14 It is a flow chart of the image display method provided in an embodiment of the present application. Taking the image generation scene as an example, the method may include the following steps 401 to 407.

[0178] Step 401: an interactive interface of an electronic device displays an input box.

[0179] Step 402: The electronic device receives a second prompt word input by the user in the input box and a second image uploaded by the user.

[0180] Step 403: In response to the input of the second prompt word and the second image, the electronic device inputs the second prompt word and the second image into a VLM model to obtain a first text.

[0181] Step 404: The electronic device inputs the first text into Stable Diffusion to generate a first image described by the first text.

[0182] Step 405: The electronic device displays the first image and the first text.

[0183] Step 406: The electronic device receives the user's editing input on the first text, and then inputs the edited first text into Stable Diffusion to generate an image described by the edited first text.

[0184] Step 407, the electronic device receives the first prompt word input by the user in the input box of the interactive interface, and then inputs the first prompt word into the VLM model. The VLM model combines the first text of the previous round with the first prompt word to generate an edited first text, and then inputs the edited first text into Stable Diffusion to generate an image described by the edited first text.

[0185] Step 408: The electronic device displays the edited image of the first text description and the edited first text.

[0186] It should be noted that the specific implementation process of the above steps 401 to 408 can refer to the relevant description of the above embodiment. To avoid repetition, this embodiment will not be repeated here.

[0187] It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not impose any restrictions on this.

[0188] The image display method provided in the embodiment of the present application can be executed by an image display device. In the embodiment of the present application, an image display device executing the image display method is taken as an example to illustrate the image display device provided in the embodiment of the present application.

[0189] Fig.15 11 is a schematic diagram of the structure of an image display device provided in an embodiment of the present application. The image display device may include: a receiving module 1101, a generating module 1102 and a display module 1103.

[0190] The receiving module 1101 is used to receive first input information.

[0191] The generating module 1102 is used to generate a first text based on the first input information received by the receiving module 1101 .

[0192] The display module 1103 is configured to display the first image described by the first text based on the first text generated by the generating module 1102 .

[0193] In some embodiments of the present application, the display module 1103 is specifically configured to display the first image and the first text based on the first text generated by the generation module 1102 .

[0194] In some embodiments of the present application, the receiving module 1101 is further used to receive editing input of the first text. The display module 1103 is further used to respond to the editing input received by the receiving module 1101 and display an image described by the edited first text based on the edited first text.

[0195] In some embodiments of the present application, the receiving module 1101 is further used to receive a first prompt word. The generating module 1102 is used to generate a second text based on the first text and the first prompt word received by the receiving module 1101. The display module 1103 is further used to display an image described by the second text based on the second text generated by the generating module 1102.

[0196] In some embodiments of the present application, the first input information is a second prompt word. The generation module 1102 is specifically configured to identify the semantic information of the second prompt word through a large language model in the text generation model, and adjust the second prompt word based on the semantic information of the second prompt word to obtain the first text.

[0197] In some embodiments of the present application, the first input information includes a second prompt word and a second image. The generation module 1102 is specifically used to encode the second image through an image encoder in a text generation model to obtain image features, and encode the second prompt word through a text encoder in a text generation model to obtain first text features; and, through a linear mapping layer in the text generation model, convert the image features into second text features; and, through a large language model in the text generation model, identify first semantic information represented by the first text feature and second semantic information represented by the second text feature, and adjust the second semantic information based on the first semantic information and the second semantic information to obtain the first text.

[0198] In the image display device provided by the embodiment of the present application, after the user inputs the first input information, the electronic device recognizes the semantics of the first input information and adjusts the first input information according to the semantics to obtain a first text that is easier for the text-graph model to understand, so that the image displayed based on the first text is more adaptable to the first input information input by the user, the image quality is better, and the accuracy of the generated image is improved.

[0199] The image display device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device, an augmented reality / virtual reality device, a robot, a wearable device, a super mobile personal computer, a netbook or a personal digital assistant, etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0200] The image display device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0201] The image display device provided in the embodiment of the present application can implement each process implemented by each embodiment of the above-mentioned image display method, and will not be described again here to avoid repetition.

[0202] Alternatively, if Fig.16 As shown, an embodiment of the present application further provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores programs or instructions that can be executed on the processor 901, and when the program or instructions are executed by the processor 901, the various steps of the above-mentioned image display method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0203] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0204] Fig.17 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0205] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0206] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.17 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0207] The user input unit 1007 is used to receive first input information. The processor 1010 is used to generate a first text based on the first input information received by the user input unit 1007. The display unit 1006 is used to display a first image described by the first text based on the first text generated by the processor 1010.

[0208] In some embodiments of the present application, the display unit 1006 is specifically configured to display a first image and the first text based on the first text generated by the processor 1010 .

[0209] In some embodiments of the present application, the user input unit 1007 is further used to receive editing input of the first text. The display unit 1006 is further used to display an image described by the edited first text based on the edited first text in response to the editing input received by the user input unit 1007.

[0210] In some embodiments of the present application, the user input unit 1007 is further configured to receive a first prompt word. The processor 1010 is configured to generate a second text based on the first text and the first prompt word received by the user input unit 1007. The display unit 1006 is further configured to display an image described by the second text based on the second text generated by the processor 1010.

[0211] In some embodiments of the present application, the first input information is a first prompt word. The processor 1010 is specifically configured to identify semantic information of a second prompt word through a large language model in a text generation model, and adjust the second prompt word based on the semantic information of the second prompt word to obtain the first text.

[0212] In some embodiments of the present application, the first input information includes a second prompt word and a second image. The processor 1010 is specifically used to encode the second image through an image encoder in a text generation model to obtain image features, and encode the second prompt word through a text encoder in a text generation model to obtain first text features; and, through a linear mapping layer in the text generation model, convert the image features into second text features; and, through a large language model in the text generation model, identify first semantic information represented by the first text feature and second semantic information represented by the second text feature, and adjust the second semantic information based on the first semantic information and the second semantic information to obtain the first text.

[0213] The electronic device provided in the embodiment of the present application, after the user inputs the first input information, the electronic device recognizes the semantics of the first input information, and adjusts the first input information according to the semantics to obtain a first text that is easier for the text graph model to understand, so that the first image displayed based on the first text is more adaptable to the first input information input by the user, the image quality is better, and the accuracy of the generated image is improved.

[0214] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0215] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0216] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0217] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image display embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0218] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0219] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image display embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0220] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0221] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image display embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0222] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0223] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0224] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image display method, characterized in that: The method comprises: receiving first input information; Based on the first input information, generate a first text; Based on the first text, a first image described by the first text is displayed.

2. The method according to claim 1, characterized in that The step of displaying a first image described by the first text based on the first text includes: Based on the first text, the first image and the first text are displayed.

3. The method according to claim 2, characterized in that After displaying the first image and the first text based on the first text, the method further includes: receiving editing input of the first text; In response to the editing input, an image described by the edited first text is displayed based on the edited first text.

4. The method according to claim 2, characterized in that: After displaying the first image and the first text based on the first text, the method further includes: receiving a first prompt word; Generate a second text based on the first prompt word and the first text; Based on the second text, an image described by the second text is displayed.

5. The method according to any one of claims 1 to 4, characterized in that: The first input information is a second prompt word; The step of generating a first text based on the first input information includes: The semantic information of the second prompt word is identified through a large language model in the text generation model, and the second prompt word is adjusted based on the semantic information of the second prompt word to obtain the first text.

6. The method according to any one of claims 1 to 4, characterized in that: The first input information includes a second prompt word and a second image; The step of generating a first text based on the first input information includes: encoding the second image by using an image encoder in a text generation model to obtain image features, and encoding the second prompt word by using a text encoder in the text generation model to obtain first text features; Converting the image feature into a second text feature through a linear mapping layer in the text generation model; The first semantic information represented by the first text feature and the second semantic information represented by the second text feature are identified through the large language model in the text generation model, and the second semantic information is adjusted based on the first semantic information and the second semantic information to obtain the first text.

7. An image display device, characterized in that: The device comprises a receiving module, a generating module and a display module; The receiving module is used to receive first input information; The generating module is configured to generate a first text based on the first input information received by the receiving module; The display module is used to display the first image described by the first text based on the first text generated by the generation module.

8. The device according to claim 7, characterized in that The display module is specifically configured to display the first image and the first text based on the first text generated by the generation module.

9. The device according to claim 8, characterized in that The receiving module is further used to receive editing input of the first text; The display module is further configured to display an image described by the edited first text based on the edited first text in response to the editing input received by the receiving module.

10. The device according to claim 8, characterized in that The receiving module is further used to receive a first prompt word; The generating module is used to generate a second text based on the first text and the first prompt word received by the receiving module; The display module is further used to display the image described by the second text based on the second text generated by the generation module.

11. The device according to claim 9, characterized in that The first input information is a second prompt word; The generation module is specifically used to identify the semantic information of the second prompt word through the large language model in the text generation model, and adjust the second prompt word based on the semantic information of the second prompt word to obtain the first text.

12. The device according to claim 9, characterized in that The first input information includes a second prompt word and a second image; The generating module is specifically used for: encoding the second image by using an image encoder in a text generation model to obtain image features, and encoding the second prompt word by using a text encoder in the text generation model to obtain first text features; Converting the image feature into a second text feature through a linear mapping layer in the text generation model; The first semantic information represented by the first text feature and the second semantic information represented by the second text feature are identified through the large language model in the text generation model, and the second semantic information is adjusted based on the first semantic information and the second semantic information to obtain the first text.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image display method according to any one of claims 1 to 6 are implemented.

14. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the image display method according to any one of claims 1 to 6 are implemented.

Images