Method and terminal device for image generation

Abstract

The application provides a method for generating an image, which comprises: obtaining description information corresponding to an image to be generated, and then generating an image corresponding to the description information according to the description information. The method for generating an image and the terminal device provided by the application are suitable for generating a corresponding image according to obtained image description information.

Description

[0001] This application is a divisional application of the invention patent application with application number 201810132928.3, application date of February 9, 2018, and invention title "Method and Terminal Device for Image Generation". Technical Field

[0002] This invention relates to the field of image processing technology, and more specifically, to a method and terminal device for image generation. Background Technology

[0003] Images are vivid and intuitive forms of expression, and people often use them to convey their ideas. Examples include various charts in business conference reports, design sketches in product design, and navigation maps drawn for route navigation.

[0004] In existing technologies, users usually need to have rich drawing experience when drawing images. For users with insufficient drawing experience, it is difficult to obtain images that accurately reflect their own ideas.

[0005] Therefore, how to obtain images that accurately express the user's ideas is a problem that existing technologies urgently need to solve. Summary of the Invention

[0006] To overcome or at least partially solve the aforementioned technical problems, the following technical solutions are proposed: According to one aspect, embodiments of the present invention provide a method for image generation, comprising: Obtain the descriptive information corresponding to the image to be generated; Based on the description information, an image corresponding to the description information is generated.

[0007] According to another aspect, embodiments of the present invention also provide a terminal device, comprising: Processor; and The memory is configured to store machine-readable instructions that, when executed by the processor, cause the processor to perform methods for image generation.

[0008] According to yet another aspect, an embodiment of the present invention also provides an image generation apparatus, characterized in that it comprises: The acquisition module is used to acquire descriptive information corresponding to the image to be generated; The generation module is used to generate an image corresponding to the description information obtained by the acquisition module.

[0009] This invention provides a method and terminal device for image generation. Compared with the prior art, this invention obtains descriptive information corresponding to the image to be generated, and then generates an image corresponding to the descriptive information based on the descriptive information. That is, this invention can directly generate an intuitive image corresponding to the descriptive information based on the image's descriptive information. Therefore, even if users do not have rich drawing experience, they can obtain an image that accurately expresses their own ideas by inputting descriptive information, which greatly improves the user experience.

[0010] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0011] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a system block diagram of Embodiment 1 of the present invention; Figure 2 This is a flowchart of an image generation method according to an embodiment of the present invention; Figure 3 This is a flowchart of the image generation method in Example 1; Figure 4 This is a schematic diagram illustrating the fusion of multiple GAN models. Figure 5 This is a schematic diagram illustrating the application of reinforcement learning to form a user-related image generation strategy and accelerate the creation process in an embodiment of the present invention; Figure 6 This is a flowchart of the weight adapter module in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the process of adjusting the weight adapter in an embodiment of the present invention; Figure 8 This is a system block diagram of Embodiment 2 of the present invention; Figure 9 This is a flowchart of the image generation method in Example 2; Figure 10 This is a schematic diagram illustrating the artistic positioning of a user based on an image in an embodiment of the present invention; Figure 11 This is a schematic diagram of the image decomposition model in an embodiment of the present invention; Figure 12 This is a schematic diagram illustrating the decomposition of an image based on layout in an embodiment of the present invention; Figure 13 This is a schematic diagram illustrating the artistic positioning of a user's social group in an embodiment of the present invention; Figure 14 This is a schematic diagram of an image generated based on user attribute information in an embodiment of the present invention; Figure 15 (a) is a schematic diagram of the diagonal composition method in an embodiment of the present invention; Figure 15 (b) is a schematic diagram of the nine-square grid composition method in an embodiment of the present invention; Figure 15 (c) is a schematic diagram of the centered composition method in an embodiment of the present invention; Figure 15 (d) is a schematic diagram of the proportional distribution pattern in an embodiment of the present invention; Figure 16 This is a schematic diagram illustrating the image generation method based on a nine-grid layout in an embodiment of the present invention; Figure 17 This is a schematic diagram illustrating the generation of images based on different coloring methods in an embodiment of the present invention; Figure 18 This is a schematic diagram illustrating the generation of images based on different image styles in an embodiment of the present invention; Figure 19 This is a schematic diagram illustrating image generation based on image content in an embodiment of the present invention; Figure 20 This is a schematic diagram of the image evaluation system in an embodiment of the present invention; Figure 21 This is a flowchart of the image generation method in Example 3; Figure 22 This is a schematic diagram illustrating the generation of an image composed of multiple layers based on image description information in an embodiment of the present invention; Figure 23 This is a schematic diagram illustrating the process of grouping elements into different layers in an embodiment of the present invention; Figure 24 This is a flowchart of the image generation method in Example 3; Figure 25(a) is a schematic diagram of drawing auxiliary information in AR navigation according to an embodiment of the present invention; Figure 25(b) is a schematic diagram of generating various descriptive charts in a business meeting report according to an embodiment of the present invention; Figure 25(c) is a schematic diagram of generating or adjusting design drafts in product design according to an embodiment of the present invention; Figure 25(d) is a schematic diagram of generating a painting that conforms to the user's description according to an embodiment of the present invention; Figure 26 This is a schematic diagram of an image generation device according to an embodiment of the present invention; Figure 27 This is a schematic diagram of the structure of the terminal device in an embodiment of the present invention; Figure 28 This is a block diagram of the computing system of the terminal device in an embodiment of the present invention. Detailed Implementation

[0012] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0013] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0014] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0015] Those skilled in the art will understand that the terms "terminal" and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices having a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptop and / or handheld computers or other devices that have and / or include a radio frequency receiver. As used herein, "terminal" or "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Terminal" or "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0016] Images are a natural form of describing works of art and objective things. Text is the most basic way of conveying information, but it is sometimes less intuitive than images. In many cases, people want to transform text into more intuitive images. For example, in business meeting reports, the content of the meeting can be directly converted into various descriptive charts; in product design, product design descriptions can be directly converted into design sketches; in augmented reality (AR) navigation, navigation maps can be directly drawn based on location or route-related descriptions; and user descriptions can be directly converted into artwork.

[0017] Therefore, how to accurately transform users' relevant descriptions into images that reflect their own thoughts has become an urgent problem to be solved.

[0018] To address the technical problems in the prior art, embodiments of the present invention provide an image generation method, as detailed below: This invention proposes an image generation method, which mainly includes how to generate high-quality images based on user-input description information and how to shorten the interactive iteration process required for users to obtain satisfactory images.

[0019] So, how to generate high-quality images based on user-input descriptions includes: First, when a user wants to create an image, they can describe the content of the ideal image using text or voice. For example, they can describe the objects and their attributes, the layout, the color tone, and the texture of the image. Combining the user's input text or voice description with user attribute information, an image that conforms to the user's semantics (i.e., an image that reflects the user's own ideas) can be generated in real time. This allows users to quickly obtain the image they want in the most natural and convenient way. Even users without drawing skills can create paintings with their own unique style.

[0020] Users can also add text or voice descriptions to the images generated by the system. The system will adjust the images based on the user's descriptions. The system can also provide users with some guidance information (also known as prompts) for the images to be generated, such as artistic guidance on layout, color, etc., so that the images obtained by users have more artistic value or personal characteristics.

[0021] When generating images, the system can create multi-layered images, dividing them into multiple layers based on layout, objects, and other information. This facilitates subsequent adjustments by the system and allows users to easily import the images into professional drawing software for modification.

[0022] Furthermore, how to shorten the interactive iteration process required for users to obtain satisfactory images includes: During the iterative process of user interaction, the system model can be improved based on user feedback on the generated images, enabling the system to generate images that meet user needs with less user interaction. The system can also obtain personal information (such as personal information obtained through user-input descriptions) and establish user profiles. Based on the image evaluation model, it can artistically position users and their social groups to generate images that are more in line with users' personal preferences and have higher popularity, thereby shortening the user's interaction iteration process.

[0023] In this embodiment of the invention, the system framework diagram is as follows: Figure 1As shown, by combining a Generative Adversarial Network (GAN) model and a knowledge base, the system generates high-quality images from the user's input description, evaluates the images, and adjusts them based on the user's supplementary descriptions, ultimately producing an image that satisfies the user.

[0024] In this embodiment of the invention, the generation and evaluation of images are mainly carried out from aspects such as layout, color, style, and content to ensure that high-quality images that conform to the user's description are generated. The GAN model library contains sub-GAN models in these four aspects, which control the generation rules of these aspects respectively. The system's knowledge base includes user attribute information (such as user profiles, social group artistic positioning, etc.) and conventional image features (such as common sense information).

[0025] In this embodiment of the invention, users can input voice, text, or images as descriptive information. Users can input voice through a voice assistant application, text through handwriting or keyboard input, or text or images through copy and paste. After receiving the descriptive information input by the user, the system can determine the corresponding text. If the user's input descriptive information contains voice and / or images, the system can convert the input voice into text through voice recognition and the input image into text through image recognition.

[0026] Users can initiate the text-to-image generation process by clicking buttons, using voice commands, etc. The text corresponding to the descriptive information is processed by a natural language understanding module, transforming it into semantics that the GAN model can understand, and then used by the text-to-image generator to generate the corresponding image. The descriptive text can include descriptions of the image's content and characteristics (including but not limited to style, color, layout, and content). For example, in this embodiment of the invention, a multi-GAN model fusion technique is used to decompose and combine the image from four aspects: layout, color, painting style, and content, and incorporates content from a knowledge base to generate a high-quality image from the text.

[0027] After generating a high-quality image, it needs to be evaluated using the knowledge base and GAN model library. Following the evaluation, if the user requires additional descriptions, they can input speech, text, or images as supplementary information. The corresponding text is then processed by a natural language understanding module to convert it into semantics understandable by the GAN model. A text-to-image optimizer then adjusts the generated image to produce a final image that satisfies the user. For example, in this embodiment, the adjustment process also employs multi-GAN model fusion technology, decomposing and fusing the image from four aspects: layout, color, painting style, and content. This is combined with the knowledge base content and the supplementary text to adjust the generated image, ultimately producing a final image that satisfies the user.

[0028] The knowledge base includes user profiles (such as user personas), social group positioning (such as collective intelligence), and common sense information.

[0029] In addition, an image evaluation system model was established to analyze the generated images and provide intelligent optimization and suggestions. The generated images can contain multiple layers of information, making it easy for users to import them into professional drawing software for modification.

[0030] This invention provides a method for image generation, such as... Figure 2 As shown, it includes: Step 201: Obtain the description information corresponding to the image to be generated.

[0031] Specifically, the descriptive information includes at least one of the following: text descriptive information, voice descriptive information, and image descriptive information.

[0032] Step 202: Generate an image corresponding to the description information.

[0033] This invention provides an image generation method. Compared with the prior art, this invention obtains descriptive information corresponding to the image to be generated, and then generates an image corresponding to the descriptive information based on the descriptive information. That is, this invention can directly generate an intuitive image corresponding to the descriptive information based on the image's descriptive information. Therefore, even if users do not have rich drawing experience, they can obtain an image that accurately expresses their own ideas by inputting descriptive information, which greatly improves the user experience.

[0034] Specifically, step 202 includes step 2021a (not labeled in the figure) and step 2021b (not labeled in the figure), wherein, Step 2021a: Based on the obtained description information, determine the image data corresponding to at least two image features respectively.

[0035] Specifically, step 2021a includes step 2021a1 (not labeled in the figure), wherein, Step 2021a1: Based on the obtained descriptive information and at least one of the following information, determine the image data corresponding to at least two image features respectively: User attribute information; general image features corresponding to the description information; user's environment information; user feedback information on the generated image.

[0036] Specifically, determining the image data corresponding to at least two image features includes: step b1 (not labeled in the figure) and step c1 (not labeled in the figure), wherein, Step b1: Determine the weight information corresponding to each image feature.

[0037] Step c1: Based on the obtained description information and the weight information corresponding to each image feature, determine the image data corresponding to at least two image features respectively.

[0038] Specifically, determining the image data corresponding to at least two image features includes: step d1 (not labeled in the figure), wherein, Step d1: For each image feature, generate image data corresponding to at least two specified image features using the corresponding image generation model.

[0039] Furthermore, image generation models include GAN models.

[0040] Step 2021b: Fuse the determined image data to obtain an image corresponding to the description information.

[0041] Furthermore, the weight information corresponding to each image feature is adjusted based on at least one of the following: Description information corresponding to the image to be generated; user attribute information; common image features corresponding to the description information; user's environment information; user feedback information on the generated image.

[0042] Further, step 202 includes step 2022 (not labeled in the figure), wherein, Step 2022: Based on the obtained description information and at least one of the following, generate an image corresponding to the description information: User attribute information; general image features corresponding to the description information; user's environment information; user feedback information on the generated image.

[0043] Furthermore, the method also includes: determining the matching degree between the acquired descriptive information and at least one of the following: user attribute information, conventional image features corresponding to the descriptive information, user's environment information, and user feedback information on the generated image; when the determined matching degree is less than a preset threshold, generating prompt information and / or image adjustment suggestions.

[0044] The image features include at least one of the following: Image color features; image style features; image layout features; image content features.

[0045] Further, step 202 includes step 2023a (not labeled in the figure) and step 2023b (not labeled in the figure), wherein, Step 2023a: Extract image elements related to the image to be generated from the description information, and / or the location information corresponding to each image element.

[0046] The positional information corresponding to each image element includes at least one of the following: the relative positional relationship between the image elements; and the depth information corresponding to each image element.

[0047] Step 2023b: Based on the image elements related to the image to be generated, and / or the position information corresponding to each image element, generate an image corresponding to the description information.

[0048] Furthermore, step 202 specifically includes step 2024 (not labeled in the figure), wherein, Step 2024: Generate an image consisting of multiple layers based on the description information.

[0049] Furthermore, when generating an image composed of multiple layers, auxiliary information is generated to describe the relationships between layer elements.

[0050] Furthermore, when image adjustment information is received, the generated image is adjusted based on the received image adjustment information and auxiliary information used to describe the relationship between layer elements to obtain the adjusted image; The auxiliary information used to describe the relationships between layer elements includes at least one of the following: The layer information for each element; The relative positions of the elements; The area occupied by each element; Depth information of each element in the image.

[0051] Furthermore, the method also includes: obtaining supplementary descriptive information; adjusting the generated image based on the obtained supplementary descriptive information to obtain an adjusted image.

[0052] Specifically, step 202 includes step 2025a (not labeled in the figure) and step 2025b (not labeled in the figure), wherein... Step 2025a: Obtain the multimedia information corresponding to the description information.

[0053] Step 2025b: Add driving assistance information corresponding to the description information to the multimedia information corresponding to the description information, and generate an image containing the driving assistance information.

[0054] The following sections will explain in detail the image generation scheme provided by the embodiments of the present invention through different application scenarios and specific application examples.

[0055] The following details the image generation methods for different application scenarios, as shown in Examples 1 to 3, including: Application Scenario 1, Application Scenario 2, and Application Scenario 3, corresponding to Examples 1, 2, and 3 respectively. Application Scenario 1 generates an image corresponding to the descriptive information by fusing multiple image data; Application Scenario 2 generates a personalized image corresponding to the descriptive information; Application Scenario 3 generates an image composed of multiple layers. Any one of Application Scenario 1, Application Scenario 2, or Application Scenario 3 can achieve image generation independently, or at least two scenarios can be combined to achieve image generation simultaneously. See below for details: Implementation Example 1 mainly includes: generating high-quality images by combining multiple GAN models. GAN stands for Generative Adversarial Network, a primary technique for generating images from text, and mainly consists of the following three parts: 1. Generating target images using a combination of multiple GAN models; 2. Set the weights for each GAN model in the GAN model library; 3. Improve the model by modifying the weights of each GAN through reinforcement learning.

[0056] Example 2 mainly includes: context-based intelligent optimization and recommendation, and specifically includes at least the following two aspects: 1. Contextual information refers to the knowledge base established in the system, including three aspects: user profile information, artistic positioning (personal artistic positioning and the artistic positioning of the user's social group), and common sense; among them, 1) User profile information serves as the contextual constraint for the system to generate or modify images, and is updated in real time based on user input, including information such as nationality, race, gender, and age; 2) Establish an image decomposition model from each artistic level, conduct artistic evaluation of images that satisfy users, obtain users' personal artistic positioning information, and generate images that better suit personal preferences; 3) Artistic positioning of users and their social groups, and real-time updates based on the user's active social networks to generate images that are more popular in the user's social groups.

[0057] 2. Intelligent optimization and suggestions 1) When processing the text description input by the user, the system automatically uses the context as additional information to constrain the generated image, thus shortening the interactive iteration process required to achieve the user's satisfactory result; 2) The GAN model library provides guidance and suggestions to users based on image evaluation results, so as to achieve images that are more in line with the user's personal style, more popular with their social groups, and more artistic.

[0058] Example 3 mainly includes: generating a multi-layered image, which includes at least the following three aspects; wherein, 1. Multiple layers can be generated according to user needs; 2. Generate additional information, such as depth information, to assist in placing elements on different layers; 3. Each layer can be adjusted independently or in combination to generate the final image.

[0059] In this embodiment of the invention, image generation involves text input via voice recognition, user handwriting or keyboard input, or copy-pasting. The user initiates the text-to-image generation process by clicking a button or issuing a voice command. The input text is converted into semantics understandable by the natural language understanding module and the image generation model (GAN model). Therefore, the user-input description must include a description of the image content and characteristics (including but not limited to style, color, layout, and content). A multi-GAN model fusion technique is employed to decompose and combine the image from four aspects: layout, color, painting type, and content. Combined with contextual information from a knowledge base, high-quality images are generated from the text description. An image evaluation system model is established to analyze the image, perform intelligent optimization and suggestions, and the generated image can contain multiple layers of information, facilitating user import and modification into professional drawing software.

[0060] Example 1 The first embodiment of this invention details the application scenario, and the image generation method flow is as follows: Figure 3 As shown: Step 301: Obtain the description information corresponding to the image to be generated.

[0061] Specifically, the descriptive information includes at least one of the following: text descriptive information, voice descriptive information, and image descriptive information.

[0062] Users can input voice, text, or images as descriptive information. Users can input voice through a voice assistant application, text via handwriting or keyboard, or by copying and pasting. After receiving the user's input, the system can determine the corresponding text. If the user's input includes voice and / or images, the system can convert the input voice into text using voice recognition and the input image into text using image recognition.

[0063] The description information entered by the user needs to include a description of the image content and characteristics of the image to be generated (including but not limited to style, color, layout, content, etc.).

[0064] Step 302: Based on the obtained description information, determine the image data corresponding to at least two image features respectively.

[0065] Specifically, the image features include at least one of the following: image color features; image style features; image layout features; and image content features.

[0066] Image layout features refer to the positional and size relationships between various objects in an image. Common compositional methods in images include diagonal lines, the rule of thirds, centering, and proportional distribution. Layout greatly influences the expressiveness of a painting, thus affecting the quality of the image.

[0067] Image color features correspond to the coloring methods in the painting process. Coloring methods are important features in the generated image. Different coloring methods will bring great visual differences. Coloring-related information includes hue, saturation, and brightness. Users usually have their own preferred coloring styles.

[0068] Style characteristics include, but are not limited to, watercolor, oil painting, comics, sketches, drawings, traditional Chinese paintings, and simple line drawings.

[0069] Image content features refer to the common combinations of image content in an image. For example, the ocean can be paired with sailboats, and mountains can be paired with plants. Specifically, they include, but are not limited to, people, seascapes, street scenes, animals, plants, buildings, mountains, rivers, etc.

[0070] Specifically, step 302 includes: step 3021 (not shown in the figure): Step 3021: Based on the obtained descriptive information and at least one of the following information, determine the image data corresponding to at least two image features respectively: User attribute information; The descriptive information corresponds to the conventional image features; User's environment information; User feedback on the generated images.

[0071] The system can create a knowledge base to store user attribute information and common image features. The user attribute information can include: user profiles, social group artistic positioning, etc.

[0072] The user profile includes personal information extracted from user-input descriptions or obtained through other means (such as through designated social media platforms). This personal information includes the user's age, gender, occupation, and other details. The user profile also includes an artistic positioning for the user, allowing them to retain their personal style in their creations. This positioning is achieved by determining the layout, colors, painting style, and content of images preferred by the user based on their input descriptions or an image evaluation system. Social group artistic positioning involves the system performing artistic analysis on the user's long-term active social networks and artistically positioning their social circles, making the user's images more popular within their social networks.

[0073] Artistic positioning refers to establishing an image evaluation system that assigns a user's personal artistic preference based on the evaluation results, thereby providing the user's personal artistic positioning. This invention proposes a novel image evaluation system that decomposes images from multiple artistic levels, obtaining decomposition models of the image at each artistic level. These artistic levels include, but are not limited to, layout, color, painting style, and content.

[0074] Common image features, also known as common sense information, are the regular image features that may be used in the process of generating an image. Examples include common combinations of image content features, common image layouts, common image colors, and common image drawing styles. Further examples include common combinations of image content features, such as a sailboat on the sea in an image; common image layouts, such as some images being centered; common image colors, such as the common color of glazed tiles being golden yellow; and common image drawing styles, such as the common drawing style used to depict characters in anime being comics.

[0075] For example, in embodiments of the present invention, image features include: image layout features, image color features, image style features, and image content features.

[0076] In this embodiment of the invention, based on the acquired descriptive information and user attribute information, image data corresponding to at least two image features are determined. Since user attribute information includes user profiles, social group art positioning, etc., based on the acquired descriptive information and user profiles and / or social group art positioning, image data corresponding to at least two of the following image features (image layout features, image color features, image style features, and image content features) are determined. For example, if the acquired descriptive information is "drawing a courtyard", and the user profile records that the user is Chinese, then the image data corresponding to the image content feature can be determined to be a courtyard house, and the image data corresponding to the image style feature is a traditional Chinese painting style. If the acquired descriptive information is "drawing a courtyard", and the image style feature in the social group art positioning is an oil painting, and the user profile records that the user is Chinese, then the image data corresponding to the image content feature can be determined to be a courtyard house, and the image data corresponding to the image style feature is an oil painting.

[0077] In this embodiment of the invention, based on the acquired description information and the conventional image features corresponding to the description information, image data corresponding to at least two image features are determined. For example, if the acquired description information is "drawing a beach", the conventional image content feature corresponding to the description information is "beach plus sailboat", and the conventional image style feature corresponding to the description information is "oil painting", then the image data corresponding to the content feature of the image can be determined to be "beach + sailboat", and the image data corresponding to the style feature of the image is "oil painting".

[0078] In this embodiment of the invention, the user's environment information refers to the user's current environment, such as objects present in the user's current environment and the colors of those objects. Further, based on the acquired description information and the user's environment information, image data corresponding to at least two image features are determined. For example, if the acquired user-input description information is "sunset, sea," and the terminal detects that the user's current environment is a beach with sailboats in the sea, and the terminal can detect the colors of the sea, beach, sunset, and sailboats in the user's current environment (e.g., the sea is blue, the sunset is reddish-yellow, the sailboats are white, and the beach is yellow), then based on the acquired description information and the user's environment information, the image data corresponding to the image content feature is determined to be "sea + sunset + beach + sailboat," and the image data corresponding to the image color feature is, for example, the sea is blue, the sunset is reddish-yellow, the sailboats are white, and the beach is yellow.

[0079] In this embodiment of the invention, based on the acquired descriptive information and the user's feedback information on the generated image, image data corresponding to at least two image features are determined. The user's feedback information on the generated image may include the user's evaluation information and the user's supplementary descriptive information. Based on the user's evaluation information on the generated image, the user's preferred image content features, image style features, image layout features, and image color features can be determined. Therefore, based on the acquired descriptive information and the aforementioned preferred image features, image data corresponding to at least two image features are determined. For example, based on the user's evaluation information on the generated image, it is determined that the user's preferred image style is a comic book style and the user's preferred image layout is a 3x3 grid layout. If the acquired descriptive information is "draw a house", then based on the user's feedback information on the generated image, the image style feature data corresponding to the image data is a comic book style house, and the image layout feature data corresponding to the image data is a 3x3 grid layout house.

[0080] Specifically, in step 302 or step 3021, "determining the image data corresponding to at least two image features" includes: step A (not labeled in the figure), wherein, Step A: For each image feature, generate image data corresponding to at least two specified image features using the corresponding image generation model.

[0081] Image generation models include GAN models.

[0082] In this embodiment of the invention, the system can create multiple GAN models. Each GAN model corresponds to at least one image feature.

[0083] GAN stands for Generative Adversarial Network, a primary technique for generating images from text. Combining multiple GAN models generates high-quality images.

[0084] Specifically, the image features include at least one of the following: image color features; image style features; image layout features; and image content features.

[0085] Among them, the GAN model corresponding to the image color features (which can be referred to as color GAN) is used to supervise the generation of images with different coloring styles; the GAN corresponding to the image style features (which can be referred to as style GAN) is used to supervise the generation of images with different styles; the GAN corresponding to the image layout features (which can be referred to as layout GAN) is used to supervise the generation of images with different layout styles; and the GAN corresponding to the image content (which can be referred to as content GAN) is used to supervise the generation of images with different content.

[0086] This invention proposes a method for fusing multiple GAN models to generate high-quality images based on user descriptions. This invention proposes that multiple GAN models can be established starting from each artistic element such as layout, color, drawing style, and content. Each GAN model focuses on the supervision task of a single aesthetic factor. The image features generated by these GAN models are then fused together to take into account multiple artistic elements and maximize the quality of the painting.

[0087] Specifically, step 302 or step 3021, "determining the image data corresponding to at least two image features respectively," includes: step B (not labeled in the figure) and step C (not labeled in the figure), wherein, Step B: Determine the weight information corresponding to each image feature.

[0088] Step C: Based on the obtained description information and the weight information corresponding to each image feature, determine the image data corresponding to at least two image features respectively.

[0089] In this embodiment of the invention, the system can set the weights corresponding to each image feature based on the description information input by the user and the knowledge base, so that the generated images are different and closer to the user's requirements.

[0090] For example, some users' creations lean towards a cartoon style (image style feature) rather than focusing on color and layout features, while others prioritize drawing specific content (image content feature) rather than a particular style (image style feature). These needs can be met by adjusting the weights of image features corresponding to various artistic elements for specific users. A complete set of weighted data can determine a default user-related image generation strategy. Applying this strategy allows users to obtain output results that are closer to their creative habits and style.

[0091] Furthermore, the weight information corresponding to each image feature is adjusted based on at least one of the following: Descriptive information corresponding to the image to be generated; User attribute information; The descriptive information corresponds to the conventional image features; User's environment information; User feedback on the generated images.

[0092] In this embodiment of the invention, the weight information corresponding to each image is adjusted based on the description information corresponding to the image to be generated. For example, if the description information corresponding to the image to be generated is "drawing a Chinese courtyard house in the style of traditional Chinese painting", that is, the user is more concerned about the content and style of the image, then the weight information corresponding to each image feature is adjusted to increase the weights corresponding to the image content feature and the image style feature respectively.

[0093] In this embodiment of the invention, user attribute information includes user profile and social group artistic positioning. For example, in the user's social group artistic positioning, the social group to which the user belongs tends to favor a nine-grid layout and has no particular requirements for the content features, color features, and style features of the image. In this case, the weight information corresponding to each image feature is adjusted to increase the weight information corresponding to the image layout feature.

[0094] In this embodiment of the invention, based on the conventional image features corresponding to the description information, the weight information corresponding to each image feature is adjusted. For example, if the description information includes "beach", and the conventional image content feature corresponding to "beach" is "beach + sailboat", and the conventional image style feature corresponding to "beach" is comic style, then the weight information corresponding to each image feature is adjusted by adding the weight information corresponding to the image content feature and the image style feature respectively.

[0095] In this embodiment of the invention, the weight information corresponding to each image feature is adjusted based on the user's environment information. For example, if the description information includes "sea + beach", and the detected user environment is "sea with sailboats", then the weight information corresponding to each image feature is adjusted by increasing the weight information corresponding to the image content feature.

[0096] In this embodiment of the invention, based on user feedback on the generated image, the weight information corresponding to each image feature is adjusted. The user's evaluation information on the generated image includes: the user's evaluation information on the generated image. For example, if the user gives a high evaluation to an image with a diagonal layout and warm colors, then the weight information corresponding to each image feature is adjusted by increasing the weight information corresponding to the image layout feature and the image color feature.

[0097] In this embodiment of the invention, the system includes a reinforcement learning neural network that records descriptive information corresponding to the image to be generated, user attribute information, conventional image features corresponding to the descriptive information, user environment information, and / or user feedback information on the generated image. A weight adapter module modifies the weight information of each image feature using the learned user-related image generation strategy, thereby generating an image that more closely resembles the specific user's creative habits. In this embodiment, since each image feature corresponds to a GAN model, adjusting the weight information corresponding to each image feature is equivalent to adjusting the weight information corresponding to each GAN model.

[0098] For example, users typically need to generate high-quality images with multiple artistic elements. Through repeated interactions and feedback with the system, and ultimately, a satisfactory image output, a user-specific image generation strategy can be formed. This strategy serves as the user's default strategy. During subsequent creative processes, the system automatically applies this default strategy to generate the output image. This allows users to obtain satisfactory results with minimal input, accelerating the creative process. Moreover, the output reflects the user's habits and style, thus more closely aligning with the user's requirements.

[0099] Step 303: Fuse the determined image data to obtain an image corresponding to the description information.

[0100] In this embodiment of the invention, fusing the determined image data means fusing the image data corresponding to each GAN model (which can be called GAN fusion) to obtain an image corresponding to the description information, such as... Figure 4 As shown.

[0101] After adjusting the weight information corresponding to each image feature, the image data corresponding to each image feature is determined by the adjusted weight information of each image. The image data are then fused to generate an image corresponding to the descriptive information.

[0102] For example, if a user's multiple inputs sequentially contain keywords such as "sunset, beach," "oil painting, sailboat, distant mountains," and "higher saturation, lower brightness," the user will ultimately receive a satisfactory output artwork, such as... Figure 5 The output image shown; in this process, through reinforcement learning, the system will adjust the weight information corresponding to each image feature through a weight adapter to meet the user's needs. The next time the system receives the user's input "sunset, beach", it will automatically use "sunset, beach", "oil painting, sailboat, distant mountains", "higher saturation, lower brightness" as keywords to directly obtain the same result. Figure 5The output image has similar quality, taking into account the image's color features, style features, layout features, and content features. In the process of generating the image, the user does not need to repeatedly input keywords such as "sunset, beach", "oil painting, sailboat, distant mountains", "higher saturation, lower brightness" during the image generation process.

[0103] In this embodiment of the invention, the system includes multiple GAN models, a weight adapter, and an evaluator. The weight adapter combines multiple GAN models, and each GAN model includes a generator and a discriminator. The GAN discriminator dynamically evaluates the generated images and adjusts the weights of different GAN models when user needs are met or user feedback is received.

[0104] In this embodiment of the invention, the user-input description information is transformed into information that the computer system can understand through natural language understanding, and then other modules are invoked through GAN fusion to generate an image that matches the description information. In this embodiment, each GAN model's generator generates the image output in the current iteration under the drive of the weight adapter, and each GAN model's discriminator dynamically evaluates the image generated by the generator of that GAN model, adjusting the weights of different GANs when user needs are met and user feedback is received. Both work together to improve their respective functions during training. For example, the color-adjusting GAN generator adjusts the colors to generate an image according to the user's request, and the corresponding discriminator judges whether the quality of the newly generated image is close to the real image; the weight adapter adjusts the GAN fusion through output parameters to meet the user's description information and the information in the knowledge base; the evaluator evaluates various aspects (e.g., style, layout, color, content) of the currently generated image based on the user's description information, the definition and configuration of the weight recognizer, and the output of the GAN discriminator. Furthermore, the evaluator can also combine information from the knowledge base during the evaluation of the currently generated image. In this embodiment of the invention, if the evaluation result exceeds the preset score threshold of the multi-GAN fusion model with the current weight setting, then this result is output; otherwise, the next iteration is performed until the requirement is met. Figure 6 As shown. This process not only ensures the combination of multiple elements, but also guarantees the quality of the image.

[0105] In this embodiment of the invention, the system also includes an application reinforcement learning network that can record description information corresponding to the image to be generated, user attribute information, conventional image features corresponding to the description information, user environment information, and user feedback information on the generated image. After obtaining a better image, the weight adapter module will learn the optimal image generation strategy, and the repeated reinforcement learning process can gradually optimize the system performance.

[0106] The reinforcement neural network inputs each user state and feedback into a reinforcement learning model. This reinforcement learning model learns the weight parameters of the multi-GAN for a specific user. Through repeated learning, the reinforcement learning model generates a set of weight parameter configuration strategies for that specific user to adjust the weight information corresponding to each image feature. The images generated according to this configuration strategy can better meet the user's requirements and are closer to the user's creative habits, thus improving the efficiency of creation. Figure 7 This involves continuously refining the weight adapter through reinforcement learning iterations. Specifically, a reinforcement learning neural network is set up as the agent evaluation model; the weight adapter output state at time t is input. Rewards for feedback from user interactions The agent evaluation model is based on the state at time t. and feedback rewards This will generate enhanced actions. , The weight adapter is instructed to work with the text-to-image generator and optimizer to produce the output state at time t+1. and feedback rewards By repeating the above process and obtaining the best results after several user interactions, the weight adapter module will learn the optimal image generation strategy. This iterative reinforcement learning process gradually optimizes the performance of the weight adapter.

[0107] After the image is generated through steps 301-303, if adjustments and additions to the generated image are needed, adjustments can be made based on the supplementary description information input by the user. See steps 304 (not labeled in the figure) and 305 (not labeled in the figure) for details. Step 304: Obtain supplementary description information.

[0108] In the embodiments of the present invention, the supplementary descriptive information may be supplementary descriptive information corresponding to image color features, supplementary descriptive information corresponding to image style features, supplementary descriptive information corresponding to image layout features, or supplementary descriptive information corresponding to image content. No limitation is made in the embodiments of the present invention.

[0109] Step 305: Adjust the generated image based on the obtained supplementary description information to obtain the adjusted image.

[0110] For example, when a user generates an image for the first time using this system, the input description information includes: "sunset" + "sea surface" + "oil painting type". The system generates an oil painting of a sunset over the sea based on the user's description information. The user finds the image to be rather monotonous and lacks a sense of dynamism, so they add the description "add a sailboat". The system then generates a scene of a lone sailboat in the distance on the sea under the sunset as the adjusted image.

[0111] This invention proposes that if a general GAN ​​model is used to generate images from user descriptions, it may be difficult to simultaneously account for the influence of multiple artistic elements on the image, resulting in low-quality images. By applying multi-GAN fusion technology, multiple artistic elements can be considered during image generation, leading to high-quality image output. Each user's image creation has its own characteristics and preferences for each artistic element. By using reinforcement learning on the feedback during the user-system interaction creation process, image generation strategies tailored to individual user characteristics and habits can be configured to meet personalized user needs.

[0112] Example 2 Embodiment 2 of this invention details the image generation method flow under application scenario 2: Different users often describe the same scene differently, or users may have the same description but different desired image effects. Existing technologies do not take into account individual differences among users and can only generate images that the system considers to correspond to the description information according to uniform rules. This requires users to describe their image information for the generated image as detailed as possible each time they create an image in order to finally obtain a satisfactory image.

[0113] To address the aforementioned technical problems, this invention proposes an image generation method applicable to, for example... Figure 8 The system shown automatically records user profiles (e.g., user profiles obtained through input text), community collective wisdom (e.g., social group art positioning), and common sense information (e.g., geographical and cultural common sense) in a knowledge base. The information output from the knowledge base is used as additional constraint information, applied to the text-to-image generator / text-to-image optimizer, and combined with image evaluation results (e.g., ratings) for intelligent optimization, providing appropriate suggestions. Figure 8In this model, the text-to-image generator and optimizer rely on a GAN model library. The text-to-image generator, while processing user input text, considers user profiles in the knowledge base (e.g., ethnicity, race, age, color preferences, style) and combines the collective wisdom of the user's matched community (e.g., art styles relevant to the user's preferences) and common sense information (e.g., racial and regional characteristics of objects). After initially generating an image, the text-to-image optimizer, combining evaluations and subsequent user input text, similarly utilizes the constraints of the GAN model library and the knowledge base input to further refine the image.

[0114] For example, when a user generates an image using this system, the input description includes: "sunset" + "sea" + "oil painting type". The system generates an oil painting of a sunset over the sea based on the user's description. The user finds the image somewhat monotonous and lacks dynamic beauty, so they add the description "add a sailboat". The system then generates an image of a lone sailboat silhouetted against the sunset over the sea, as the adjusted image. The terminal device can optimize its knowledge base based on the user's input description and supplementary description. When the user subsequently inputs "sunset" + "sea", the terminal device can automatically add the sailboat based on the optimized knowledge base, resulting in a high-quality output.

[0115] For a detailed flowchart of the specific image generation method in this embodiment of the invention, please refer to [link / reference]. Figure 9 ,in, Step 901: Obtain the description information corresponding to the image to be generated.

[0116] The description information includes at least one of the following: text description information, voice description information, and image description information.

[0117] For details, please refer to step 301, which will not be repeated here.

[0118] Step 902: Based on the acquired description information and at least one of the following, generate an image corresponding to the description information: User attribute information; The descriptive information corresponds to the conventional image features; User's environment information; User feedback on the generated images.

[0119] In this embodiment of the invention, the knowledge base established in the system includes user attribute information and conventional image features (such as common sense information). The user attribute information includes user profiles and social group artistic positioning.

[0120] The user profile includes personal information extracted from the user's description information, as well as an artistic positioning of the user (i.e., the user's preferred image color features, image style features, image layout features, and image content features), allowing the user to retain their personal style in their creations. In this embodiment of the invention, the artistic positioning of the user refers to establishing an image evaluation system, and based on the image evaluation results, determining the user's personal artistic preferences, thereby providing the user's personal artistic positioning. This embodiment of the invention proposes a new image evaluation system that decomposes images from multiple artistic levels, obtaining decomposition models of the image at each artistic level, including but not limited to layout, color, style, and content.

[0121] Among them, social group art positioning is a system that performs art analysis on the social networks in which users have been active for a long time, and performs art positioning on their social groups (that is, the image color characteristics, image style characteristics, image layout characteristics, and image content characteristics that the user's social group likes), so that the images created by the user are more popular in their social groups.

[0122] In this embodiment of the invention, an image corresponding to the description information is generated based on the obtained description information and user attribute information. For example, if the obtained description information is "drawing a courtyard", the user attribute information records that the user is Chinese, and the social group art positioning is that the image style preferred by the user's social group is "traditional Chinese painting style", then based on the obtained description information and user attribute information, an image with the image content feature of "courtyard" and the image style feature of "traditional Chinese painting style" is directly generated.

[0123] In this embodiment of the invention, based on the obtained description information and conventional image features, an image corresponding to the description information is directly generated. For example, if the obtained description information is "drawing a beach, in an oil painting style" and the conventional image content features are "beach + sailboat", then the image directly generated corresponding to the description information is an image containing "containing a beach and a sailboat, in an oil painting style".

[0124] In this embodiment of the invention, based on the acquired description information and the user's environment information, an image corresponding to the description information is generated. For example, if the acquired description information is "drawing a beach", and the terminal detects that the user's environment information is "the user is currently on a beach, and there are coconut trees on the beach", then the generated image corresponding to the description information includes "beach + coconut trees".

[0125] In this embodiment of the invention, based on the acquired descriptive information and the user's feedback on the generated image, an image corresponding to the descriptive information is generated. That is, based on the user's feedback on the previously generated image, the user's preference for image features (image color features, image content features, image layout features, and image style features) is obtained. When the user's descriptive information for the image is obtained again, the acquired descriptive information and the user's feedback on the generated image are directly used to generate an image corresponding to the descriptive information. For example, if the user gives a high rating to an image with a diagonal layout and warm colors, then when the user inputs descriptive information for the image again, an image with a diagonal layout and warm colors, corresponding to the descriptive information, is generated based on the user's input descriptive information.

[0126] In step 902, before generating an image corresponding to the description information based on the acquired description information and user attribute information, user attribute information is created.

[0127] Specifically, creating user attribute information includes creating user profile information, an artistic positioning of the user, and a knowledge base corresponding to the artistic positioning of the user's social group, as detailed in (1), (2), and (3) below: 1) Create user profile information: When a user generates an image using the system for the first time, the system will create a default profile for the user. The user's profile records personal information, including but not limited to nationality, race, age, and gender.

[0128] Each time the system receives a description from the user, it extracts the corresponding personal information based on the parsed semantics. If the information is being obtained for the first time, it is recorded. If the information already exists and the two inputs are different, the system updates the established personal profile information based on the latest information.

[0129] Users can also manually create or modify their profiles, add personal information, and create personalized tags to improve their profile information.

[0130] 2) Create a personalized artistic positioning for each user. When a user obtains a satisfactory image after multiple interactions with the system, the system automatically evaluates the final image, performs artistic localization based on a GAN model library, and records the evaluation in the system's knowledge base. Figure 10As shown, after generating a high-quality image, it needs to be evaluated using the knowledge base and GAN model library. Following the evaluation, if the user needs to add a description, they input this information. The corresponding text is then processed by the natural language understanding module, converting it into semantics that the GAN model can understand. A text-to-image optimizer then adjusts the generated image to produce a final image that satisfies the user. When the user creates again, the system generates an image closer to their personal style based on their artistic vision, thus reducing the need for further user interaction and iteration.

[0131] The system determines the artistic levels of image decomposition based on the types of sub-GAN models in the GAN model library, including but not limited to layout, color, style, and content. The system evaluates each satisfactory image obtained by the user based on the GAN model, decomposing it from several artistic levels to obtain the image decomposition model at each artistic level, such as... Figure 11 As shown, a sub-GAN model in the GAN model library decomposes an image based on n mainstream types in its corresponding art level, obtaining the image's proportion in each of these n types as m1, m2, ..., m. n This refers to the decomposition model of an image at this artistic level, allowing the system to derive the user's personal preferences based on this decomposition model.

[0132] Taking the decomposition model of an image in terms of layout as an example, the system decomposes the layout of the image, obtaining the main layout types existing in the image as: M1, M2, M3, …, and the corresponding proportions as m1, m2, m3, …, thus completing the user's positioning of the layout at an artistic level. Figure 12 As shown, the system decomposes the composition of the image, and the layout type of the image is a combination of diagonal and nine-square grid.

[0133] When a user generates an image again using the system, if no layout description is provided during generation or modification, the generated image will prioritize maintaining the same artistic positioning as the user's intended layout. This ensures that the user's image largely retains their own style. If the user wants to change the image's layout, they can supplement the description with text to modify the system-generated image. The system updates the user's artistic positioning in terms of layout based on the layout information input each time, allowing the user to retain their stylistic preferences in future creations.

[0134] 3) Creating an artistic positioning for the user's social groups The user's social group includes the user group of the social application the user belongs to, etc. This is not limited in this embodiment of the invention.

[0135] The social groups users frequent often reflect their preferences to some extent. By artistically positioning the social groups users are active in, and making the images created by users more in line with the artistic positioning of their social groups, the images created by users will be more popular.

[0136] The system obtains the user's social groups and the artistic positioning of members within those groups. For example... Figure 13 As shown, the system can obtain the artistic positioning E of each user j (j=1,2,…,m) in social group i (i=1,2,…,n). j,i :

[0137] Where K1={k 1k},K2={k 2k},K4={k 4k}(k=1,…,p), which correspond to the decomposition results of the four art decomposition levels, where p is the number of type components that can be decomposed at each art level.

[0138] The system is based on E j,i The artistic positioning of social group i can be obtained. :

[0139] The system determines the artistic positioning of user j based on the social groups they belong to. The average artistic positioning of the user across all their social groups is used to obtain the artistic positioning E of the user's social group. j :

[0140] Where N is the number of social groups the user belongs to.

[0141] Further, step 902 can be: generating an image corresponding to the description information based on the acquired description information and the created user attribute information.

[0142] After receiving the description information input by the user, the system automatically uses the user attribute information created in the system, the regular image features corresponding to the description information, the user's environment information, and the user's feedback information on the generated image as constraints to assist the image generation / modification module in completing image optimization.

[0143] Furthermore, when users input descriptive information corresponding to the image to be generated, they may not input the layout, color, style, and content of the image to be generated completely. However, user attribute information generally does not change much. Therefore, the system can combine the existing user attribute information with the descriptive information input by the user to generate an image that matches the user's personal attributes.

[0144] The following steps first combine descriptive information with user profile information to generate an image corresponding to the descriptive information. For example, if a user wants to draw an image representing a traditional Chinese courtyard house, the user might input the descriptive information "a courtyard." In this case, based on the user's profile information that the user is Chinese and likes traditional Chinese architecture, the system will directly generate an image of a courtyard house when generating the image corresponding to the descriptive information. Figure 14 As shown.

[0145] Furthermore, descriptive information can be combined with the user's personal artistic style and / or the artistic style of the user's group to generate an image corresponding to the descriptive information. The following sections will introduce this from four aspects: image layout, image color, image style, and image content. 1) Image layout Commonly used layout methods in images include: diagonal ( Figure 15 a) Nine-square grid ( Figure 15 b) Centered ( Figure 15 c) Geometric distribution ( Figure 15 d) etc. In a diagonal layout, the image is distributed along the diagonal; in a rule-of-thirds layout, the image subject is represented at the intersection of the grid; in a centered layout... Figure 1 Generally used for drawing task settings, with the subject in the very center of the image; in equal proportions, the image is clearly divided, mostly used for drawing scenes, and sometimes for drawing tasks, usually using a third division.

[0146] In this embodiment of the invention, the layout GAN model in the image generation / optimization module of the system generates the layout of the image based on the characteristics of commonly used composition methods. The GAN model in the image evaluation system that decomposes the image layout also decomposes the image layout based on the layout method, thereby extracting the user's preference for the layout.

[0147] like Figure 16 As shown, if the user does not specify any layout requirements in the input description information, but the system's personal attribute information records that the user's personal artistic style is a nine-grid layout or that the nine-grid layout is popular in the user's social group, then the system will default to using the nine-grid layout when optimizing the image.

[0148] If the user's preference for several layout methods is almost equal in the personal attribute information recorded in the system, or if the user's commonly used layout method is unsuitable according to the user's description, the system will recommend a composition method based on the user's current description of the scene. For example, if there is a main character in the scene, and the user has not described a specific layout method, but the user's recorded attribute information does not show a particular preferred layout method, or if centered composition is more popular in their social group, the system will automatically center the composition during optimization to make the image more reasonable. If the system records that the user prefers a rule of thirds composition, the system will provide suggestions during optimization, suggesting that centered composition is more suitable.

[0149] 2) Image color Coloring is an important step in image generation, such as... Figure 17 As shown, different coloring methods can lead to significant visual differences. Users usually have their own preferred coloring styles. This invention proposes that the system records the user's commonly used color information such as hue, saturation, and transparency, which serves as the basis for the color GAN model in the image generation / optimization module to generate the image color system. The GAN model in the image evaluation system, which decomposes the image color information, also decomposes the color positioning of the image based on the hue, saturation, and brightness of the image, thereby extracting the user's color preferences.

[0150] This invention proposes that image color information is also related to some of the user's personal information, such as age and gender. The system intelligently formulates color schemes based on the recorded personal profile information. In addition, the user's artistic positioning in terms of color within their social group, as found in their attribute information, will also serve as a basis for the system to provide suggestions to the user.

[0151] 3) Image style Image styles include, but are not limited to, watercolor, oil painting, comics, sketches, drawings, traditional Chinese paintings, and simple line drawings. In this embodiment of the invention, the system records the painting styles frequently used by the user, serving as the basis for the image style GAN model in the image generation / optimization module to generate image painting types. The GAN model in the image evaluation system, which decomposes the painting type information of an image, also decomposes the image's painting style based on its classification, thereby extracting the user's preferred image styles, such as... Figure 18 As shown, the system generates images corresponding to the user's preferred image style.

[0152] 4) Image content This invention proposes that the system records users' common combinations of image content, which serve as the basis for the image content generated by the painting type GAN model in the image generation / optimization module. The GAN model in the image evaluation system, which decomposes image content information, also decomposes the image's location based on its constituent objects, thereby extracting the user's preferences regarding image content composition. For example... Figure 19 As shown, if a user describes a scene of "sunset over the sea," and the system analyzes the user's environment and deems the scene too monotonous, adding a sailboat in the direction of the sunset would better highlight the vastness of the scene and add a dynamic aesthetic to the image, then the system will intelligently add the "sailboat" element during optimization. Alternatively, if the system's personal attribute information records that the user frequently uses a sailboat to enhance the aesthetics of a sunset over the sea, then the system will automatically add a sailboat when generating an image the next time it receives a description of a similar scene, even if the user did not specify a sailboat in the description, or generate a suggestion message: "Add a sailboat to the user's description."

[0153] In this embodiment of the invention, when generating an image corresponding to the description information based on at least one of the following: user attribute information, conventional image features corresponding to the description information, user environment information, and user feedback information on the generated image, as well as the obtained description information, the user's personal profile information does not need to be re-entered, nor does it require excessive description of personal style preferences. Only the content of the image to be generated and some other special requirements need to be described. This simplifies the user's input significantly, greatly shortens the interactive iteration process, and allows the user to modify the image according to the system's suggestions to obtain an image that is more in line with their personal style, more popular, and more artistic.

[0154] When generating an image corresponding to the description information using Example 2, the GAN model from Example 1 can also be used. For details, please refer to Example 1, which will not be repeated here.

[0155] Since the generated image may not match the user attribute information, the regular image features corresponding to the description information, the characteristics of the user's environment, and the user's feedback on the generated image when generating the image based on the description information, steps 903 and 904 are executed. Step 903: Determine the degree of matching between the acquired description information and at least one of the following: user attribute information, regular image features corresponding to the description information, user's environment information, and user feedback information on the generated image.

[0156] Step 904: When the determined matching degree is less than the preset threshold, generate a prompt message and / or image adjustment suggestions.

[0157] In this embodiment of the invention, when the system generates an image corresponding to the description information, it automatically inputs the recorded user personal attribute information, conventional image features, user environment information, and user feedback information on the generated image as additional information into the image generation / modification module, and combines the image evaluation results to perform intelligent optimization and generate prompt information and / or image adjustment suggestions.

[0158] The following describes steps 903 and 904 in detail, using user attribute information as an example. The determination of the matching degree between the acquired description information and at least one of the following—user attribute information, the corresponding conventional image features, the user's environment information, and the user's feedback on the generated image—is based on the evaluation results provided by the image evaluation system. The system decomposes the image corresponding to the user description information to obtain the image's artistic positioning. It then comprehensively evaluates the decomposition results of each feature of the image based on the user attribute information to arrive at a conclusion, determining whether to generate prompt information and / or image adjustment suggestions.

[0159] like Figure 20 As shown, the image evaluation system analyzes the layout features, color features, image style features, and image content features of an image by using various sub-GANs from a multi-GAN model library, and establishes a decomposition model M. i And obtain the proportion m of each type of component in each decomposition model. ij， Where i (i=1,2,…,4) is the decomposition level, j (j=1,2,…,n) are the various types of components decomposed at level i. The system obtains the user's personal artistic positioning based on the artistic positioning of the user's social group, and thus sets the scoring coefficient K = {k ij (i=1,2,…,4; j=1,2,…,n)}, so that the user's score matches the artistic positioning of their social group, and the scoring coefficient K is stored in the knowledge base for the system to score users in the future.

[0160] The system-generated images are scored based on the user's rating coefficient K recorded in the knowledge base, thus determining the degree to which the generated images match the user's personal artistic positioning.

[0161] When the score is low, meaning the generated image doesn't quite match the user's personal artistic style, the system will prompt the user and provide suggestions for improvement. Similarly, the system scores the image based on the artistic style of the user's social group recorded in the knowledge base. When the score is low, meaning the generated image doesn't quite match the artistic style of the user's social group and may not be popular, the system will prompt the user and provide suggestions for improvement.

[0162] After generating the image corresponding to the description information in step 902, the user may need to adjust the image color features, image style features, image layout features, and image content features in the generated image. This can be done based on the supplementary description information input by the user, as detailed in steps 905 (not labeled in the figure) and 906 (not labeled in the figure). Step 905: Obtain supplementary description information.

[0163] Step 906: Adjust the generated image based on the obtained supplementary description information to obtain the adjusted image.

[0164] For specific implementation details, please refer to steps 304 and 305, which will not be repeated here.

[0165] Example 3 Embodiment 3 of this invention details the image generation method flow under application scenario 3: In existing image generation techniques, the generation system takes user-generated textual descriptions, audio descriptions, and / or image descriptions as input to generate a single-layer image that aggregates all input elements. Modifying any element within this single-layer image often affects other parts of the image.

[0166] For example, in drawing or image processing, if a user needs to modify a single element or several elements in an already generated image, they often need to process the corresponding elements as independent layers, and then reassemble the multiple layers into the final output image after the modifications. Alternatively, modifying elements in a single-layer image using automatic processing methods after generating an existing image often requires complex processing. For instance, moving an object requires detecting and segmenting the image before moving it, and filling the background after moving the object, which is quite complex.

[0167] To address the technical problems in existing technologies, it is necessary to analyze the descriptive information input by the user and generate an image composed of multiple layers, such as... Figure 21 As shown below: Step 2101: Obtain the description information corresponding to the image to be generated.

[0168] The description information includes at least one of the following: text description information, voice description information, and image description information.

[0169] In this embodiment of the invention, the descriptive information may be descriptive information input by the user through text, descriptive information input through voice, or an image.

[0170] Step 2102: Extract image elements related to the image to be generated from the description information, and / or the location information corresponding to each image element.

[0171] The positional information for each image element includes: the relative positional relationship between the image elements; and the depth information for each image element. Relative position determines the coordinates of the element on a plane, while depth information determines the three-dimensional positional relationship of the elements in the image.

[0172] For example, if the user inputs a photo and the following voice description, "Generate an image containing the family in this photo, with a castle and forest in the background and an airplane flying in the air," the elements extracted from this description are people, castle (building), airplane, and forest, as well as the planar position and depth information of each element, to create an image with a three-dimensional effect, with people in the foreground and the castle, forest, and airplane in the background.

[0173] Step 2103: Based on the image elements related to the image to be generated, and / or the position information corresponding to each image element, generate an image composed of multiple layers.

[0174] For example, based on the extracted elements from this description information, such as characters, castles, airplanes, and forests, along with the location information corresponding to each element, a generator is generated as follows: Figure 22 The image shown. In Figure 22 The generated image consists of multiple layers: Layer 0 (a layout map describing the area occupied by each element), Layer 1 (containing backgrounds such as forests), Layer 2 (containing buildings such as castles), Layer 3 (people), and Layer 4 (airplanes).

[0175] It should be noted that when generating an image composed of multiple layers, steps D (not labeled in the figure) and E (not labeled in the figure) can also be used. Step D: Obtain the descriptive information corresponding to the image to be generated.

[0176] Step E: Generate an image consisting of multiple layers based on the description information.

[0177] In this embodiment of the invention, when generating an image composed of multiple layers, language processing is used to extract different image elements from the natural language input and the position information corresponding to each image element. Combined with auxiliary input, and according to the user-defined granularity or the image generation model of user habits, the extracted image elements are generated into the corresponding layers.

[0178] For example, such as Figure 23As shown, the user input includes a photograph and the following voice description: "Generate an image containing the family in this photograph, with a castle and forest in the background, and an airplane flying in the sky." Based on this description, the extracted elements are people, a castle, an airplane, and a forest. Using natural language understanding, the system generates the positional information for each element, setting the people as the foreground and the castle, forest, and airplane as the background. Based on the relationships between the elements, one or more of these elements are placed on corresponding layers, generating an image composed of multiple layers. This allows users to more easily perform professional editing of the generated multi-layered images, or to submit further requests through other automatic processing functions provided by the system. The system can then optimize and modify individual layers or multiple layers, thereby improving the user experience.

[0179] It should be noted that, through steps 2101, 2102, and 2103, based on the image elements related to the image to be generated, and / or the position information corresponding to each image element, a single-layer image can also be generated.

[0180] When generating an image composed of multiple layers in the above manner, step 2104 can also be performed, wherein step 2104 generates auxiliary information to describe the relationship between layer elements.

[0181] The auxiliary information used to describe the relationships between layer elements includes at least one of the following: The layer information for each element; The relative positions of the elements; The area occupied by each element; Depth information of each element in the image.

[0182] The layer information for each element refers to the layers in which each element in an image is located. For example, if the input description is a photo and the following voice description, "Generate an image containing the family in this photo, with a castle and forest in the background and an airplane flying in the air," the elements extracted based on this description are people, castle, airplane, and forest. The forest is located in layer 1, the castle in layer 2, the people in layer 3, and the airplane in layer 4.

[0183] The relative positional relationship of each element refers to the relative positions between elements in an image. This can be the relative positions between elements on the same layer. For example, an airplane can be used as the background of a person.

[0184] The area occupied by each element includes both the position and size of each element. For example, in the layout diagram, the airplane occupies area 1, the forest occupies area 2, the character occupies area 3, and the castle occupies area 4.

[0185] The depth information of each element in the image determines the stereoscopic perspective effect of combining different elements.

[0186] When generating a single-layer image or an image composed of multiple layers using Example 3, at least one of the following can be combined: GAN model, user attribute information, conventional image features corresponding to the description information, user environment information, and user feedback information regarding the generated image. For details, please refer to Example 1 and Example 2, which will not be repeated here.

[0187] After generating the image, it may be necessary to adjust a certain element in the image. That is, when adjustment information for the generated image is detected, step 2105 can be executed, wherein... Step 2105: When image adjustment information is received, the generated image is adjusted based on the received image adjustment information and auxiliary information used to describe the relationship between layer elements to obtain the adjusted image.

[0188] In this embodiment of the invention, adjusting the generated image includes, but is not limited to, adjusting lighting, composition, and element size. In this embodiment, if a user needs to adjust elements in a generated multi-layered image, they can do so in the independent layer containing the element, or by combining adjustments with other layers, thus generating an adjusted image.

[0189] After generating the image, the user may need to adjust the image's color features, style features, layout features, and content features. This can be done based on the user's supplementary description information, as detailed in steps 2106 and 2107. Step 2106: Obtain supplementary description information.

[0190] Step 2107: Adjust the generated image based on the obtained supplementary description information to obtain the adjusted image.

[0191] For specific implementation details, please refer to steps 305 and 306, which will not be repeated here.

[0192] Example 4 This embodiment specifically describes an example of generating corresponding images based on description information and applying them to different fields. Steps 2401 and 2402 are described using the field of vehicle navigation as an example. Figure 24 As shown, where, Step 2401: Obtain the description information corresponding to the image to be generated.

[0193] The description information includes at least one of the following: text description information, voice description information, and image description information.

[0194] Step 2402: Obtain the multimedia information corresponding to the description information.

[0195] In this embodiment of the invention, corresponding keyword information is obtained from the acquired description information, and multimedia information corresponding to the description information is determined based on the keyword information.

[0196] For example, as shown in Figure 25(a), user A wants to drive to the location of user B. User B notifies user A of their current location via a call or instant messaging application. User A's terminal device automatically generates driving assistance information (such as location indication, route indication, direction indication, etc.) based on user B's location description. Specifically, user A's terminal device can obtain the location description information input by user B through a call or instant messaging application. For example, user A's terminal device obtains user B's location description information as "I am at the east foot of a Chinese-style bell and drum tower, north of a three-way intersection" through a call application. User A's terminal device extracts relevant keyword information from user B's location description information. The extracted keyword information can be "Chinese-style bell and drum tower," "east foot," "three-way intersection," and "north." The extracted keyword information can include, but is not limited to, building descriptions, location descriptions, direction descriptions, and environmental descriptions. User A's terminal device matches the multimedia information corresponding to the description information from the multimedia information collected in real time by the AR navigation application, such as matching an image containing "Chinese Bell and Drum Tower" and "Three-way intersection", as shown in Figure 25(a).

[0197] Step 2403: Add driving assistance information corresponding to the description information to the multimedia information corresponding to the description information, and generate an image containing the driving assistance information.

[0198] The driving assistance information in this embodiment of the invention may include at least one of location indication, route indication, and direction indication.

[0199] For example, as shown in Figure 25(a), based on the extracted keyword information "Chinese Bell and Drum Tower", "East Foot", "Three-way Intersection", and "North", user A's terminal device determines the current location of user B in the multimedia information corresponding to the description information. The location indication of user B is added as driving assistance information to the multimedia information determined in step 2802, and an image containing driving assistance information is generated, that is, the location of user B is displayed in real time through AR.

[0200] Alternatively, based on the extracted keyword information "Chinese Bell and Drum Tower", "East Foot", "Three-way Intersection", and "North", User A's terminal device determines the current location of User B and further determines the direction and / or route to User B's current location. The direction and / or route to User B's current location are then added as driving assistance information to the multimedia information determined in step 2402 to generate an image containing driving assistance information.

[0201] Furthermore, generating images corresponding to the descriptive information can also be applied to other fields. For example, based on the input descriptive information, the meeting content in a business meeting report can be directly converted into various descriptive charts, such as pie charts, bar charts, and scatter plots, as shown in Figure 25(b); for example, based on the input descriptive information related to product design, product design drafts can be generated or adjusted, as shown in Figure 25(c); and paintings can be generated based on the user's input descriptive information. For example, if the user's input descriptive information is "A family is boating on a calm sea, and seagulls are singing. Using an American cartoon style, add sandcastles on the beach," a corresponding American cartoon-style painting can be generated, as shown in Figure 25(d).

[0202] After generating an image containing driving assistance information, the user may need to adjust the image's color features, style features, layout features, and content features. This can be done based on supplementary descriptive information input by the user, as detailed in steps 2404 (not labeled in the figure) and 2405 (not labeled in the figure). Step 2404: Obtain supplementary description information.

[0203] Step 2405: Adjust the generated image based on the obtained supplementary description information to obtain the adjusted image.

[0204] This invention provides an image generation apparatus, such as... Figure 26 As shown, the device includes an acquisition module 2601 and a generation module 2602, wherein, The acquisition module 2601 is used to acquire descriptive information corresponding to the image to be generated; The generation module 2602 is used to generate an image corresponding to the description information obtained by the acquisition module 2601.

[0205] This invention provides an image generation apparatus. Compared with the prior art, this invention obtains descriptive information corresponding to the image to be generated, and then generates an image corresponding to the descriptive information based on the descriptive information. That is, this invention can directly generate an intuitive image corresponding to the descriptive information based on the image's descriptive information. Therefore, even if users do not have rich drawing experience, they can obtain an image that accurately expresses their own ideas by inputting descriptive information, which greatly improves the user experience.

[0206] The image generation apparatus provided in this embodiment of the invention is applicable to the above method embodiments, and will not be described again here.

[0207] Based on the above image generation method, embodiments of the present invention also provide a corresponding terminal device, such as... Figure 27 As shown, it includes: a processor 2701; and a memory 2702 configured to store machine-readable instructions that, when executed by the processor, cause the processor to perform the method described above.

[0208] Figure 28 A block diagram of a computing system that can be used to implement a second terminal device according to an embodiment of this disclosure is illustrated. Figure 28 As shown, the computing system 2800 includes a processor 2810, a computer-readable storage medium 2820, an output interface 2830, and an input interface 2840. This computing system 2800 can perform the operations described above. Figure 2 , Figure 3 , Figure 9 , Figure 21 as well as Figure 24 The method described is used to generate an image corresponding to the description information.

[0209] Specifically, processor 2810 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 2810 may also include onboard memory for caching purposes. Processor 2810 may be used for executing reference... Figure 2 , Figure 3 , Figure 9 , Figure 21 as well as Figure 24 The described method flow consists of a single processing unit or multiple processing units representing different actions.

[0210] Computer-readable storage medium 2820 can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, readable storage media can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, apparatuses, or propagation media. Specific examples of readable storage media include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); memories such as random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0211] Computer-readable storage medium 2820 may include computer program 2821, which may include code / computer-executable instructions that, when executed by processor 2810, cause processor 2810 to perform, for example, the above-described combination. Figure 2 , Figure 3 , Figure 9 , Figure 21 as well as Figure 24 The described method flow and any variations thereof. Computer program 2821 can be configured to have computer program code, for example, including computer program modules. For example, in an example embodiment, the code in computer program 2821 may include one or more program modules, such as 2821A, module 2821B, ... It should be noted that the division and number of modules are not fixed. Those skilled in the art can use appropriate program modules or combinations of program modules according to the actual situation. When these combinations of program modules are executed by processor 2810, the processor 2810 can perform, for example, the above-described combination... Figure 2 , Figure 3 , Figure 9 , Figure 21 as well as Figure 24 The described method and any variations thereof.

[0212] According to embodiments of this disclosure, processor 2810 can use output interface 2830 and input interface 2840 to perform the above-described combination. Figure 2 , Figure 3 , Figure 9 , Figure 21 as well as Figure 24 The described method and any variations thereof.

[0213] This invention provides a terminal device. Compared with the prior art, this invention obtains descriptive information corresponding to the image to be generated, and then generates an image corresponding to the descriptive information based on the descriptive information. That is, this invention can directly generate an intuitive image corresponding to the descriptive information based on the image's descriptive information. Therefore, even if users do not have rich drawing experience, they can obtain an image that accurately expresses their own ideas by inputting descriptive information, which greatly improves the user experience.

[0214] The terminal device provided in this embodiment of the invention is applicable to the above method embodiments, and will not be described again here.

[0215] Those skilled in the art will understand that this invention includes devices for performing one or more of the operations described in this application. These devices may be specifically designed and manufactured for the desired purpose, or may include known devices found in general-purpose computers. These devices have computer programs stored therein that can be selectively activated or reconfigured. Such computer programs may be stored in a device (e.g., a computer)-readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, including but not limited to any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. That is, a readable medium includes any medium by which a device (e.g., a computer) stores or transmits information in a readable form.

[0216] Those skilled in the art will understand that each block in these structural diagrams and / or block diagrams and / or flow diagrams, as well as combinations of blocks in these structural diagrams and / or block diagrams and / or flow diagrams, can be implemented using computer program instructions. Those skilled in the art will also understand that these computer program instructions can be provided to a processor of a general-purpose computer, a specialized computer, or other programmable data processing method for implementation, thereby enabling the processor of the computer or other programmable data processing method to execute the schemes specified in the blocks or plurality of blocks of the structural diagrams and / or block diagrams and / or flow diagrams disclosed herein.

[0217] Those skilled in the art will understand that the steps, measures, and schemes in the various operations, methods, and processes discussed in this invention can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this invention can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and schemes in the prior art that are similar to those disclosed in this invention can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0218] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method executed by a terminal device, characterized in that, include: Obtain the descriptive information corresponding to the image to be generated; Based on the obtained description information, an image generation model corresponding to each of the multiple image elements is used to generate image data corresponding to each image element. The multiple image elements include at least two of image color, image style, image layout, and image content. By fusing the image data corresponding to the various image elements, an image corresponding to the acquired descriptive information is obtained.

2. The method according to claim 1, characterized in that, The generation of image data corresponding to each image feature includes: Based on the obtained descriptive information and at least one of the following information, image data corresponding to each image element is generated using the image generation model corresponding to each of the multiple image elements: User attribute information; The descriptive information corresponds to the conventional image features; User's environment information; User feedback on the generated images.

3. The method according to claim 1, characterized in that, The generation of image data corresponding to each image feature includes: Determine the weight information of the image generation model corresponding to each image element; Based on the obtained descriptive information and the weight information of the image generation model corresponding to each image element, the image generation model corresponding to each image element is used to generate image data corresponding to each image element.

4. The method according to claim 3, characterized in that, The method further includes: Adjust the weight information of the image generation model for each image feature based on at least one of the following: Descriptive information corresponding to the image to be generated; User attribute information; The descriptive information corresponds to the conventional image features; User's environment information; User feedback on the generated images.

5. The method according to claim 1, characterized in that, The image generation model includes a Generative Adversarial Network (GAN) model.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Determine the degree of matching between the acquired descriptive information and at least one of the following: user attribute information, conventional image features corresponding to the descriptive information, user's environment information, and user feedback information on the generated image; When the determined matching degree is less than a preset threshold, a prompt message and / or image adjustment suggestions are generated.

7. The method according to claim 1, characterized in that, The generation of image data corresponding to each image feature includes: Extract image elements related to the image to be generated from the description information, and / or the position information corresponding to each image element; Based on the image elements related to the image to be generated, and / or the location information corresponding to each image element, an image generation model corresponding to each image element is used to generate image data corresponding to each image element.

8. The method according to claim 7, characterized in that, The positional information corresponding to each image element includes at least one of the following: the relative positional relationship between the image elements; and the depth information corresponding to each image element.

9. The method according to claim 1, characterized in that, The image corresponding to the acquired descriptive information is an image composed of multiple layers.

10. The method according to claim 9, characterized in that, The method further includes: Generate auxiliary information to describe the relationships between layer elements; When image adjustment information is received, the generated image is adjusted based on the received image adjustment information and auxiliary information used to describe the relationship between layer elements, resulting in an adjusted image. The auxiliary information used to describe the relationships between layer elements includes at least one of the following: The layer information for each element; The relative positions of the elements; The area occupied by each element; Depth information of each element in the image.

11. The method according to claim 1, characterized in that, The method further includes: Get additional description information; Based on the obtained supplementary descriptive information, the generated image is adjusted to obtain the adjusted image.

12. The method according to claim 11, characterized in that, The step of adjusting the generated image based on the acquired supplementary description information to obtain the adjusted image includes: Based on the obtained supplementary description information, the image generation models corresponding to the various image elements are used to adjust the image data of the corresponding image elements in the generated image to obtain the adjusted image.

13. The method according to claim 1, characterized in that, Generate an image corresponding to the description information based on the description information, including: Retrieve the multimedia information corresponding to the description information; Add driving assistance information corresponding to the description information to the multimedia information corresponding to the description information to generate an image containing the driving assistance information.

14. The method according to any one of claims 1-13, characterized in that, The descriptive information includes at least one of the following: text descriptive information, voice descriptive information, and image descriptive information.

15. A terminal device, comprising: processor; as well as A memory configured to store machine-readable instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-14.

Images