Method and system for generating image conception through characters

Abstract

The invention provides a method for generating an image concept through characters. The method comprises the steps that semantic information related to image generation in the characters is analyzed and extracted; mapping the semantic information to a visual feature space to generate a preliminary visual feature vector diagram; generating a preliminary concept of the image according to the preliminary visual feature vector diagram and the semantic information; and optimizing the preliminary conception of the image through reinforcement learning and user feedback to obtain a final image conception. The text content is accurately converted into the image features through the multi-mode learning technology, and the accuracy problem of complex semantic conversion is solved. A three-layer architecture is generated by adopting character analysis, semantic mapping and conception, so that the generation process is clear and controllable, and scene design can be refined. The image conception is continuously optimized through reinforcement learning and user feedback, and the high matching degree of the generated image conception and text description is ensured.

Description

Technical Field

[0001] The present invention relates to the field of AI visual image generation, and in particular to a method and system for generating image concepts using text. Background Art

[0002] In recent years, AI's ability to generate images based on text has become increasingly mature, especially the Stable Diffusion model, which has greatly improved the speed and quality of image generation. However, in the AI ​​ability of text-generated images, the prompt (input prompt word) has a great impact on the quality of the generated image (prompt engineering is a formal search for prompts to retrieve the desired result from the language model, where the desired result depends on the final task and the end user), especially some decorative prompt words can greatly improve the aesthetics of the generated result; but most user-entered prompt descriptions are insufficient. And the text-generated image model relies on large-scale text-image pair training data. However, these models have difficulty accurately understanding the context of the text and image requirements when generating complex scenes. Summary of the invention

[0003] In view of the problems existing in the prior art, a method and system for generating image conception from text is provided, which can accurately form a conception map according to the text description.

[0004] The first aspect of the present invention provides a method for generating image conception from text, comprising:

[0005] Parse and extract semantic information from text related to image generation;

[0006] Mapping semantic information into visual feature space to generate a preliminary visual feature vector map;

[0007] Generating preliminary ideas for images based on preliminary visual feature vector graphs and semantic information;

[0008] Reinforcement learning based on artificial feedback optimizes the initial conception of the image to obtain the final image conception.

[0009] As a preferred solution, the parsing and extracting semantic information related to image generation in the text specifically includes:

[0010] Encode the input text into word vectors and sentence vectors;

[0011] Identify the core elements of word and sentence vectors, including entities, actions, and emotions;

[0012] Semantically annotate each element and output the object and its attribute information, i.e. semantic information;

[0013] As a preferred solution, mapping the semantic information to the visual feature space specifically includes:

[0014] Convert semantic information into image feature vectors through a multimodal Transformer model;

[0015] Generate visual feature tags for each object and background based on the results of multimodal learning;

[0016] Generate a preliminary visual feature vector map for image composition.

[0017] As a preferred solution, the preliminary concept of generating an image based on the preliminary visual feature vector graph and semantic information specifically includes:

[0018] Using the preliminary visual feature vector graph, a preliminary scene framework is formed;

[0019] The spatial position of the object in the scene frame is adjusted according to the semantic information of the text to form a preliminary image concept that conforms to the description.

[0020] As a preferred solution, the reinforcement learning based on artificial feedback optimizes the preliminary conception of the image to obtain the final image conception, which specifically includes:

[0021] Get feedback from users or models, get a score on the matching between the generated image concept and the input text, and determine whether optimization and adjustment are needed;

[0022] The weights are fine-tuned during the reinforcement learning process to obtain the final image concept.

[0023] The second aspect of the present invention provides a system for generating image concepts from text, comprising:

[0024] A text content parsing module is used to extract semantic information related to image generation from input text;

[0025] Multimodal semantic mapping module, used to map semantic information into visual feature space and generate preliminary visual feature vector graph;

[0026] An image concept generation module, used to generate a preliminary concept of an image based on a preliminary visual feature vector map and semantic information;

[0027] The feedback optimization module optimizes the image generation effect based on reinforcement learning of artificial feedback and outputs the final image concept.

[0028] As a preferred solution, the specific working process of the text content parsing module includes:

[0029] Use the pre-trained NLP model to encode the input text, and combine keyword extraction, entity recognition and sentiment analysis technology to identify the entities, actions and emotional core elements in the text;

[0030] Semantically annotate each element, generate lexical semantic structure, and output object and its attribute information, namely semantic information.

[0031] As a preferred solution, the specific working process of the multimodal semantic mapping module includes:

[0032] Convert semantic information into image feature vectors through a multimodal Transformer model or interactive attention mechanism;

[0033] Generate visual feature tags for each object and background based on the results of multimodal learning;

[0034] Generate a preliminary visual feature vector map for image composition.

[0035] As a preferred solution, the specific working process of the image concept generation module includes:

[0036] Input the preliminary visual feature vector map into the Generative Adversarial Network (GAN) or diffusion model to generate a preliminary scene frame;

[0037] The semantic information of the text description and the convolutional neural network are used to adjust the spatial position of the object to form an image concept that conforms to the description.

[0038] As a preferred solution, the specific working process of the feedback optimization module includes:

[0039] Get feedback from users or models, get a score on the matching between the generated image concept and the input text, and determine whether optimization and adjustment are needed;

[0040] The weights are fine-tuned during the reinforcement learning process to obtain the final image concept.

[0041] Compared with the prior art, the beneficial effects of adopting the above technical solution are:

[0042] (1) Multimodal semantic mapping: Through multimodal learning technology, text content is accurately converted into image features, solving the accuracy problem of complex semantic conversion.

[0043] (2) Modular concept generation process: The three-layer architecture of text analysis, semantic mapping, and concept generation makes the generation process clear and controllable, and helps to refine the scenario design.

[0044] (3) Feedback optimization mechanism: The text content parsing module and the multimodal semantic mapping module are continuously optimized through reinforcement learning and user feedback to ensure that the generated image concept is highly matched with the text description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The present invention provides a flowchart of a method for generating image concepts from text.

[0046] Figure 2 Schematic diagram of the system proposed by the present invention that proposes the concept of generating images from text DETAILED DESCRIPTION

[0047] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar modules or modules with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be construed as limitations on the present application. On the contrary, the embodiments of the present application include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0048] In order to accurately interpret the semantics of objects, scenes, and emotions in text descriptions, it provides a basic concept for image generation. Figure 1 , the embodiment of the present invention proposes a method for generating image conception from text, which can accurately form a conception map according to the text description. The specific scheme is as follows:

[0049] Step 1: Parse and extract semantic information related to image generation in the text.

[0050] In this step, the input text content is parsed to extract key visual elements and contextual information, including objects, background, emotions, light and shadow, etc., which are used for the subsequent adjustment of the preliminary concept of the graphic.

[0051] In one embodiment, a pre-trained NLP model (such as BERT, GPT) is used to encode text to form word vectors and sentence vectors; then, key word extraction, entity recognition (NER) and sentiment analysis are combined to identify core elements such as entities, actions, and emotions in the text, such as "deer in the forest in the morning light"; finally, each element is semantically annotated to generate a lexical semantic structure, and information about the object and its attributes (position, scale, color, light and shadow, etc.), that is, semantic information, is output.

[0052] Step 2: Map the semantic information to the visual feature space to generate a preliminary visual feature vector map.

[0053] In this step, the multimodal relationship between text and image is established mainly by mapping text semantic information into visual feature space.

[0054] In this embodiment, a multimodal Transformer (such as CLIP) or an interactive attention mechanism is used to match the text content with the image generation features and generate a preliminary visual feature vector map. In one embodiment, the semantic information is converted into an image feature vector through a multimodal Transformer model; based on the results of multimodal learning, visual feature tags are generated for each object and background, including size, position, color, etc.; finally, a preliminary visual feature vector map for composition is formed.

[0055] Step 3: Generate a preliminary concept of the image based on the preliminary visual feature vector map and semantic information.

[0056] In this step, a preliminary conception of the image is generated mainly based on multimodal semantic mapping and semantic information, including scene layout, object position, viewing angle, etc.

[0057] In one embodiment, a preliminary visual feature vector graph is received through a generative adversarial network (GAN) or a diffusion model to form a preliminary scene framework, and the spatial position of the object is adjusted according to the semantic information of the text description in combination with the Transformer encoding method to form a preliminary concept of the image that conforms to the description. Taking GAN as an example, GAN can generate realistic images through adversarial training of the generator and the discriminator. The specific steps are as follows:

[0058] Step 3.1, encode the text through Transformer;

[0059] Step 3.2, the obtained code is passed through the multi-layer CNN network to generate an image, which is compared with the discriminator and fed back for back propagation;

[0060] Step 3.3: Repeat step 3.2 to make the generated image gradually close to the real image, and form a preliminary concept of the image that conforms to the description.

[0061] Step 4: Optimize the initial conception of the image based on reinforcement learning with artificial feedback to obtain the final image conception.

[0062] In order to ensure the accuracy and consistency of image concept generation, the embodiment of the present invention also proposes a reinforcement learning based on human feedback (RLHF) mechanism. By collecting user feedback, the matching score between the generated image concept and the input text is obtained, and it is determined whether optimization and adjustment are needed. If no adjustment is needed, it can be directly output; if adjustment is needed, the weight is fine-tuned in the reinforcement learning process to ensure that the generated image concept is more consistent with the text description.

[0063] Reinforcement learning with human feedback (RLHF) refers to incorporating human feedback into the model training process so that multimodal large models (MLLMs) can better understand human intentions and generate results that are more in line with human expectations. Multimodal representation learning aims to map data of different modalities into a common semantic space to facilitate cross-modal comparison and reasoning. Common methods include joint embedding, cross-modal attention mechanism, and graph neural networks.

[0064] The training process of RLHF can be divided into three stages: supervised fine-tuning (SFT), reward modeling (RM), and reinforcement learning (RL). Supervised fine-tuning (SFT): Use labeled data to conduct supervised training on the model to improve the performance of the model on specific tasks; Reward modeling (RM): Convert human feedback into reward signals to guide the reinforcement learning process; Reinforcement learning (RL): Use reinforcement learning algorithms (e.g., PPO, SAC) to fine-tune the model so that it can generate outputs that are more in line with human expectations.

[0065] Specifically, in the process of collecting and converting artificial feedback, artificial feedback can be explicit (for example, scoring the model output) or implicit (for example, user clicks or dwell time). Artificial feedback is converted into reward signals to guide the reinforcement learning process.

[0066] During model training and evaluation, select an appropriate multimodal learning model architecture, such as CLIP, ALIGN, Florence, etc. Use the training set to pre-train the model, learn the joint representation of multimodal data, and use the validation set to evaluate the model and adjust hyperparameters. Use the test set to evaluate the model, calculate indicators such as BLEU score and ROUGE score, and evaluate.

[0067] The following is a detailed description of the optimization process of the initial conception of the image:

[0068] Step 4.1. Initialize the model and parameters: Select a pre-trained multimodal model, such as LLM (Large Language Models), and initialize the learnable transformation matrix and bias.

[0069] Step 4.2: Build a reward model: Train a reward model that can evaluate the quality of the model output based on human feedback and return a reward value. This reward value will be used to guide the reinforcement learning algorithm to update the model weights.

[0070] Step 4.3, Generate and Evaluate: Pass the input data (such as text prompts) to the multimodal model to generate output. The output is then sent to the reward model together with the original input to obtain the reward value.

[0071] Step 4.4, Reinforcement Learning Algorithm: Use reinforcement learning algorithms (such as PPO, SAC) to update model weights. The algorithm will optimize the model based on the reward value to generate higher quality output.

[0072] Step 4.5, weight fine-tuning and optimization: During the fine-tuning process, techniques such as LoRA (Low-Rank Adaptation) can be used to update only specific parts of the model (such as the weights of the cross-attention module) while keeping other parts (such as the backbone initialization weights) unchanged.

[0073] Step 4.6, multi-task learning weights: For situations involving multiple loss functions (such as losses of different modalities), adaptive multi-task learning weights can be used to balance the importance of different tasks.

[0074] Step 4.7, Dynamic Weight Adjustment: Dynamically adjust the weights of non-important targets based on the changes in the relationship between targets. This can be achieved by increasing or decreasing the weights to optimize the global goal.

[0075] Step 4.8, Experiment and Evaluation: Evaluate the model performance on multiple cross-modal task domain benchmarks, such as image text retrieval, question answering (QA), video text retrieval, and video QA.

[0076] Step 4.9, Ablation Experiments and Visual Analysis: Conduct ablation experiments to understand the impact of different components on model performance, and perform visual analysis to better understand the model behavior.

[0077] Please refer to Figure 2 The embodiment of the present invention also proposes a system for generating image concept from text, which mainly includes a text content analysis module, a multimodal semantic mapping module, an image concept generation module and a feedback optimization module. Specifically,

[0078] The text content analysis module is used to extract semantic information related to image generation from the input text. In this module, the input text is encoded using a pre-trained NLP model, and the entity, action, and emotional core elements in the text are identified by combining keyword extraction, entity recognition, and sentiment analysis techniques; each element is semantically annotated to generate a lexical semantic structure, and the object and its attribute information, i.e., semantic information, is output.

[0079] The multimodal semantic mapping module is used to map semantic information to the visual feature space and generate a preliminary visual feature vector map. In this module, semantic information is converted into an image feature vector through a multimodal Transformer model or an interactive attention mechanism; visual feature tags are generated for each object and background based on the results of multimodal learning; and a preliminary visual feature vector map for composition is generated.

[0080] The image concept generation module is used to generate a preliminary concept of the image based on the preliminary visual feature vector map and semantic information. In this module, the preliminary visual feature vector map is input into the generative adversarial network (GAN) or diffusion model to generate a preliminary scene framework; the spatial position of the object is adjusted using the semantic information of the text description and the convolutional neural network to form an image concept that meets the description.

[0081] The feedback optimization module is used to optimize the image generation effect using reinforcement learning and user feedback, and output the final image concept. In this module, by collecting feedback from users or models, the matching score between the generated image concept and the input text is obtained, and it is determined whether optimization adjustment is needed. If no adjustment is needed, it can be directly output; if adjustment is needed, reinforcement learning is used to fine-tune the weights of the text content analysis and multimodal mapping modules to ensure that the concept of the generated image is more consistent with the text description.

[0082] The present invention proposes a method and system for generating image concepts based on input text. The core semantics of the text are extracted through natural language processing technology, and the text content is mapped to the image feature space in combination with a multimodal learning method to generate a preliminary concept map, which provides references in terms of composition, object positioning, scene layout, etc., and provides input for subsequent image generation modules.

[0083] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flowchart.

[0084] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0085] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0087] As another aspect, the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes a method for generating an image concept from text as described in the above embodiment.

[0088] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements a method for generating image conception from text described in the above embodiment.

[0089] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0090] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0091] For those skilled in the art, the specific meanings of the above terms in the present invention can be understood in specific situations; the drawings in the embodiments are used to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0092] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for generating image conception from text, characterized in that: include: Parse and extract semantic information from text related to image generation; Mapping semantic information into visual feature space to generate a preliminary visual feature vector map; Generating preliminary ideas for images based on preliminary visual feature vector graphs and semantic information; Reinforcement learning based on artificial feedback optimizes the initial conception of the image to obtain the final image conception.

2. The method of generating image concept from text according to claim 1, characterized in that: The parsing and extracting of semantic information related to image generation from the text specifically includes: Encode the input text into word vectors and sentence vectors; Identify the core elements of word and sentence vectors, including entities, actions, and emotions; Semantically annotate each element and output the object and its attribute information, i.e., semantic information.

3. The method of generating image concept from text according to claim 1, characterized in that: Mapping the semantic information to the visual feature space specifically includes: Convert semantic information into image feature vectors through a multimodal Transformer model; Generate visual feature tags for each object and background based on the results of multimodal learning; Generate a preliminary visual feature vector map for image composition.

4. The method of generating image concept from text according to claim 1, characterized in that: The preliminary concept of generating an image based on the preliminary visual feature vector graph and semantic information specifically includes: Using the preliminary visual feature vector graph, a preliminary scene framework is formed; The spatial position of the object in the scene frame is adjusted according to the semantic information of the text to form a preliminary image concept that conforms to the description.

5. The method of generating image concept from text according to claim 1, characterized in that: The reinforcement learning based on artificial feedback optimizes the preliminary conception of the image to obtain the final image conception, which specifically includes: Get user feedback, get the matching score between the generated image concept and the input text, and determine whether optimization and adjustment are needed; The weights are fine-tuned during the reinforcement learning process to obtain the final image concept.

6. A system for generating image concepts from text, characterized in that: include: A text content parsing module is used to extract semantic information related to image generation from input text; Multimodal semantic mapping module, used to map semantic information into visual feature space and generate preliminary visual feature vector graph; An image concept generation module, used to generate a preliminary concept of an image based on a preliminary visual feature vector map and semantic information; The feedback optimization module optimizes the image generation effect based on reinforcement learning of artificial feedback and outputs the final image concept.

7. The system for generating image concept from text according to claim 6, characterized in that: The specific working process of the text content parsing module includes: Use the pre-trained NLP model to encode the input text, and combine keyword extraction, entity recognition and sentiment analysis technology to identify the entities, actions and emotional core elements in the text; Semantically annotate each element, generate lexical semantic structure, and output object and its attribute information, namely semantic information.

8. The system for generating image concept from text according to claim 6, characterized in that: The specific working process of the multimodal semantic mapping module includes: Convert semantic information into image feature vectors through a multimodal Transformer model or interactive attention mechanism; Generate visual feature tags for each object and background based on the results of multimodal learning; Generate a preliminary visual feature vector map for image composition.

9. The system for generating image concept from text according to claim 6, characterized in that: The specific working process of the image concept generation module includes: Input the preliminary visual feature vector map into the Generative Adversarial Network (GAN) or diffusion model to generate a preliminary scene frame; The semantic information of the text description and the convolutional neural network are used to adjust the spatial position of the object to form an image concept that conforms to the description.

10. The system for generating image concept from text according to claim 6, characterized in that: The specific working process of the feedback optimization module includes: Get feedback from users or models, get a score on the matching between the generated image concept and the input text, and determine whether optimization and adjustment are needed; The weights are fine-tuned during the reinforcement learning process to obtain the final image concept.

Images