Image generation method, apparatus, device, and storage medium
By receiving user prompts, extracting features, and generating sampling time step sequences in a neural network model, the problem of poor image generation quality in existing technologies is solved, achieving the effect of improving image quality and reducing device pressure under the same computational cost.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNITED NETWORK COMM GRP CO LTD
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-02
AI Technical Summary
In existing technologies, the method of generating images through user prompts and fixed time step sequences cannot adaptively allocate computing resources according to the characteristics of the image generation task, resulting in poor image quality under the same computing cost.
By receiving target prompts from the user, features are extracted and input into a trained neural network model for trajectory prediction, generating a sampled time step sequence, which replaces the fixed time step sequence to generate the target image.
At the same computational cost, the quality of image generation is improved, and the computational overhead is reduced through a lightweight neural network model, thus reducing the operating pressure on electronic devices.
Smart Images
Figure CN122134871A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to an image generation method, apparatus, device, and storage medium. Background Technology
[0002] The user's image generation process refers to a process guided by user prompts, starting from pure noise, and gradually denoising to generate a clear image consistent with the semantics through multiple calculations. Each step of the process is controlled by the prompts to ensure content matching, ultimately converting text into a high-quality image.
[0003] In the prior art, traditional image generation methods generate images corresponding to user prompts by using user prompts and a preset fixed time step sequence, guided by the text description of the user prompts, and performing multi-step calculations of the fixed time step sequence.
[0004] However, in existing technologies, the method of generating images through user prompts and fixed time step sequences cannot adaptively allocate computing resources according to the characteristics of the image generation task, resulting in poor image quality under the same computing cost. Summary of the Invention
[0005] This application provides an image generation method, apparatus, device, and storage medium to solve the problem that the method of generating images through user prompts and fixed time step sequences cannot adaptively allocate computing resources according to the characteristics of the image generation task, resulting in poor image quality under the same computing cost.
[0006] In a first aspect, this application provides an image generation method applied to an electronic device, comprising:
[0007] Receive target prompt words sent by the user terminal, and extract features from the target prompt words to obtain target prompt word features;
[0008] The target prompt word features are input into a trained neural network model for trajectory prediction to obtain the target sampling trajectory, wherein the trained neural network model is trained by a pre-built connected network model;
[0009] Determine the corresponding offline map based on the target sampling trajectory;
[0010] Trajectory search is performed on the offline map to generate a sampling time step sequence;
[0011] Generate the target image corresponding to the target prompt word based on the sampling time step sequence;
[0012] Output the target image corresponding to the target prompt word to the user terminal.
[0013] In one possible design, the construction process of the neural network model includes: designing the network hierarchy of the neural network model to be constructed to obtain a multidimensional input layer, a preset number of multidimensional hidden layers, and a multidimensional output layer; configuring activation functions for each multidimensional hidden layer to complete the configuration of each multidimensional hidden layer; after completing the configuration of each multidimensional hidden layer, sequentially establishing connections between the multidimensional input layer, each multidimensional hidden layer, and the multidimensional output layer through a fully connected method to obtain the neural network model; performing structural verification on the neural network model to obtain verification results; and when the verification result is successful, training and configuring the loss function and optimizer for the neural network model to complete the construction of the neural network model.
[0014] In one possible design, the training process of the connection network model includes: acquiring multiple historically generated prompt word image pairs; extracting multiple corresponding prompt words and multiple images from the multiple prompt word image pairs; determining the corresponding sampling trajectory of each image through a preset trajectory processing framework; performing feature extraction processing on the multiple prompt word image pairs and the multiple prompt words to obtain corresponding prompt word features; and training the neural network model based on each sampling trajectory and each prompt word feature to obtain a trained neural network model.
[0015] In one possible design, the step of performing feature extraction processing on the plurality of prompt word images and the plurality of prompt words to obtain corresponding prompt word features includes: determining corresponding user style features based on a plurality of historically generated prompt word images; performing feature extraction processing on each prompt word using a preset feature extraction tool to obtain corresponding initial prompt word features; and performing cross-modal attention fusion of each initial prompt word feature with the user style features to obtain corresponding prompt word features.
[0016] In one possible design, determining the corresponding sampling trajectory of each image through a preset trajectory processing framework includes: determining the expected noise value of any image at each time step in a preset time step sequence under the preset trajectory processing framework; performing denoising estimation and reconstruction error processing on each expected noise value and corresponding time step through a diffusion model to obtain reconstruction error results; integrating the reconstruction error results corresponding to each time step to obtain the sampling trajectory corresponding to the image; and determining the sampling trajectory corresponding to the remaining images among the plurality of images.
[0017] In one possible design, training the neural network model based on each sampling trajectory and each prompt word feature to obtain a trained neural network model includes: integrating each prompt word feature and corresponding sampling trajectory into corresponding data pairs, and integrating each data pair into a dataset, wherein each sampling trajectory corresponds one-to-one with each prompt word feature; determining a portion of the data pairs in the dataset as a training set according to a preset partitioning ratio; and training the neural network model based on the training set using forward propagation and backpropagation algorithms to obtain a trained neural network model.
[0018] In one possible design, training the neural network model using the training set according to the forward propagation and back propagation algorithms to obtain a trained neural network model includes: inputting the prompt word features of any data pair in the training set into the neural network model and performing prediction processing through the forward propagation algorithm to obtain a predicted sampling trajectory; measuring the error between the sampling trajectory in the data pair and the predicted sampling trajectory using the loss function; updating the model parameters of the neural network model according to the error using the back propagation algorithm and optimizer; and after the neural network model has been updated, iteratively updating the neural network model repeatedly based on the remaining data pairs in the training set until the model converges to obtain a trained neural network model.
[0019] Secondly, this application provides an image generation apparatus for use in electronic devices, comprising:
[0020] The first extraction module is used to receive target prompt words sent by the user terminal and extract features from the target prompt words to obtain target prompt word features;
[0021] The input module is used to input the target prompt word features into a trained neural network model for trajectory prediction processing to obtain the target sampling trajectory, wherein the trained neural network model is trained by a pre-built connected network model;
[0022] The first determining module is used to determine the corresponding offline map based on the target sampling trajectory;
[0023] The search module is used to perform trajectory search on the offline map to generate a sampling time step sequence;
[0024] The generation module is used to generate the target image corresponding to the target prompt word based on the sampling time step sequence;
[0025] The output module is used to output the target image corresponding to the target prompt word to the user terminal.
[0026] Thirdly, this application provides an electronic device, including: at least one processor and a memory;
[0027] The memory stores computer-executed instructions;
[0028] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the image generation method as described in the first aspect and various possible designs of the first aspect.
[0029] Fourthly, this application provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the image generation method described in the first aspect and various possible designs of the first aspect.
[0030] The image generation method, apparatus, device, and storage medium provided in this application receive target prompt words sent by a user terminal and extract features from the target prompt words to obtain target prompt word features; input the target prompt word features into a trained neural network model for trajectory prediction processing to obtain a target sampling trajectory, wherein the trained neural network model is trained by a pre-constructed connected network model; determine the corresponding offline map based on the target sampling trajectory; perform trajectory search on the offline map to generate a sampling time step sequence; generate a target image corresponding to the target prompt word based on the sampling time step sequence; and output the target image corresponding to the target prompt word to the user terminal. By performing multiple data processing on the target prompt words until a sampling time step sequence that meets the requirements is obtained, it is used to replace the fixed time step sequence. The target image is generated based on the sampling time step sequence, thereby improving the image generation quality under the same computational cost. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 This is a schematic diagram illustrating an application scenario of the image generation method provided in the embodiments of this application;
[0033] Figure 2 A flowchart illustrating the image generation method provided in this application embodiment. Figure 1 ;
[0034] Figure 3 A flowchart illustrating the image generation method provided in this application embodiment. Figure 2 ;
[0035] Figure 4This is a schematic diagram of the structure of the image generation apparatus provided in the embodiments of this application;
[0036] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0038] The user's image generation process refers to a multi-step calculation process guided by user prompts, starting from pure noise and proceeding along a progressively denoised path to generate a clear image consistent with the semantics. Each step of the process is controlled by the prompts to ensure content matching, ultimately converting text into a high-quality image. In existing technologies, traditional image generation methods use user prompts and a preset fixed-time-step sequence, guided by the text description of the prompts, to generate the image corresponding to the prompts through multi-step calculations within that fixed-time-step sequence. However, this existing method of generating images through the collaboration of user prompts and fixed-time-step sequences cannot adaptively allocate computational resources according to the characteristics of the image generation task, resulting in poor image quality at the same computational cost.
[0039] To address the aforementioned technical problems, this application proposes the following technical concept: Considering the target prompt words sent by the user terminal, the inventors extract the target prompt word features based on the target prompt words, use a trained neural network model to predict the trajectory of the target prompt word features to obtain the target sampling trajectory, determine the corresponding offline map based on the target sampling trajectory, perform trajectory search on the offline map to generate a sampling time step sequence, and generate the target image corresponding to the target prompt word based on the sampling time step sequence, thereby improving the image generation quality under the same computational cost.
[0040] Figure 1 This is a schematic diagram illustrating an application scenario of the image generation method provided in the embodiments of this application.
[0041] like Figure 1 As shown, the scenario includes: user terminal 101 and electronic device 102.
[0042] Among them, the user terminal 101 can be a display screen or a personal computer or other terminal.
[0043] Electronic device 102 can be a standalone device or a cluster of multiple devices.
[0044] Electronic device 102 receives target prompt words sent by user terminal 101 and extracts features from the target prompt words to obtain target prompt word features; it inputs the target prompt word features into a trained neural network model for trajectory prediction processing to obtain the target sampling trajectory, wherein the trained neural network model is trained by a pre-constructed connected network model; it determines the corresponding offline map based on the target sampling trajectory; it performs trajectory search on the offline map to generate a sampling time step sequence; it generates the target image corresponding to the target prompt word based on the sampling time step sequence; and it outputs the target image corresponding to the target prompt word to user terminal 101. A detailed embodiment is described below.
[0045] Figure 2 A flowchart illustrating the image generation method provided in this application embodiment. Figure 1 The execution entity in this embodiment can be Figure 1 The electronic devices shown in the illustrated embodiments are not specifically limited in this embodiment. Figure 2 As shown, the method includes:
[0046] S201: Receive the target prompt word sent by the user terminal, and extract features from the target prompt word to obtain the target prompt word features.
[0047] In this embodiment, the target prompt word is text, video, or audio, etc.
[0048] For example, the target cue word is: A woman wearing a black T-shirt with a logo, holding a black marker, smiling at the camera. Behind her is the glass wall of a data center server room, on which is clearly written in handwriting: "Noteing the problem of local error accumulation, we innovatively propose to use DAG and path optimization to improve the consistency of accelerated generation."
[0049] For example, the target prompt word is ; Target prompt word features are .
[0050] S202: Input the target prompt word features into the trained neural network model for trajectory prediction processing to obtain the target sampling trajectory, wherein the trained neural network model is trained by a pre-built connected network model.
[0051] In this embodiment, the target sampling trajectory is the predicted optimal sampling trajectory DNA vector.
[0052] For example, the neural network model is .
[0053] For example, the predicted optimal sampling trajectory DNA vector is .
[0054] In this embodiment, the process of constructing the neural network model in S202 includes steps a~e:
[0055] Step a: Design the network hierarchy of the neural network model to be constructed to obtain a multi-dimensional input layer, a predetermined number of multi-dimensional hidden layers, and a multi-dimensional output layer.
[0056] For example, the multidimensional input layer is an L-dimensional input layer.
[0057] In this embodiment, the preset quantity can be any one of 2, 3 or 5, or other quantities.
[0058] For example, the preset number of multidimensional hidden layers is two 2048-dimensional hidden layers.
[0059] For example, the multidimensional output layer is a T-dimensional linearly activated output layer.
[0060] Step b: Configure the activation function for each multidimensional hidden layer to complete the configuration of each multidimensional hidden layer.
[0061] In this embodiment, the activation function can be a ReLU activation function or other activation functions.
[0062] Step c: After configuring each multidimensional hidden layer, the multidimensional input layer, each multidimensional hidden layer, and the multidimensional output layer are connected sequentially using a fully connected method to obtain the neural network model.
[0063] In this embodiment, the neural network model is a lightweight neural network.
[0064] Lightweight neural networks include three-layer MLP models, small variants of Transformer, or convolutional neural networks.
[0065] For example, the lightweight neural network is a three-layer MLP model.
[0066] Among them, MLP (Multi-Layer Perceptron) is a feedforward artificial neural network model, which consists of an input layer, a hidden layer and an output layer.
[0067] Step d: Perform structural verification on the neural network model to obtain the verification results.
[0068] Step e: When the verification result is successful, train and configure the loss function and optimizer of the neural network model to complete the construction of the neural network model.
[0069] The optimizer is the Adam optimizer.
[0070] S203: Determine the corresponding offline map based on the target sampling trajectory.
[0071] Specifically, the corresponding offline map is determined based on the target sampling trajectory and with reference to the preset trajectory processing framework.
[0072] The preset trajectory processing framework can be the CoTj framework or other processing frameworks.
[0073] The CoTj framework is a novel processing architecture for determining sampling trajectories, determining offline graphs, and searching for offline graph trajectories.
[0074] For details regarding the CoTj framework used to determine the sampling trajectory, please refer to [link / reference]. Figure 3 Corresponding implementation examples.
[0075] In addition, the CoTj framework is also used to define the corresponding authentication process and specifications, covering the process of structured modeling and feature extraction of image data.
[0076] In this embodiment, the offline graph is a graph-structured data network where nodes represent intermediate states in the latent space (such as feature representations of noisy images) and edges represent possible state transition paths.
[0077] The offline graph is used to determine an optimal denoising route in advance, rather than letting the model blindly calculate step by step.
[0078] S204: Perform trajectory search on the offline map to generate a sampling time step sequence.
[0079] Specifically, the CoTj framework is used to perform trajectory search on the offline graph to generate a sampling time step sequence.
[0080] Specifically, step S204 includes steps a~c:
[0081] Step a: Using the start and end points of the offline graph as constraints, perform a graph search algorithm in the offline graph to obtain a continuous path sequence.
[0082] The search process can incorporate a visual-language joint inference mechanism, where a large language model performs logical inference on the semantic labels corresponding to continuous path sequences to filter out path branches that do not conform to the proposed diagram.
[0083] Among them, the continuous path sequence is composed of graph nodes in sequence, representing potential feasible trajectories.
[0084] Step b: Discretize the continuous path sequence obtained by the search in time and space, and extract the spatial location points on the path at fixed time intervals to form an equidistant sampling sequence.
[0085] In this embodiment, the fixed time interval needs to be determined according to requirements.
[0086] Step c: Introduce temporal context embedding, use the skip-gram model to encode the timestamps of the sampling points, so that the equidistant sampling sequence retains the temporal features of motion, which is convenient for subsequent behavior prediction or control, and finally outputs a sampling time step sequence.
[0087] S205: Generate the target image corresponding to the target prompt word based on the sampling time step sequence.
[0088] Specifically, a diffusion model is used to perform denoising operations sequentially according to the sampling time step sequence in order to generate the target image corresponding to the target prompt word.
[0089] S206: Output the target image corresponding to the target prompt word to the user terminal.
[0090] In summary, the image generation method provided in this embodiment receives target prompt words sent by the user terminal and extracts features from the target prompt words to obtain target prompt word features; inputs the target prompt word features into a trained neural network model for trajectory prediction processing to obtain the target sampling trajectory, wherein the trained neural network model is trained by a pre-constructed connected network model; determines the corresponding offline map based on the target sampling trajectory; performs trajectory search on the offline map to generate a sampling time step sequence; generates the target image corresponding to the target prompt word based on the sampling time step sequence; and outputs the target image corresponding to the target prompt word to the user terminal. By performing multiple data processing on the target prompt words until a sampling time step sequence that meets the requirements is obtained, it can replace the fixed time step sequence and generate the target image based on the sampling time step sequence, thereby improving the image generation quality under the same computational cost.
[0091] Furthermore, the image generation method provided in this embodiment designs the network hierarchy of the neural network model to be constructed to obtain a multidimensional input layer, a preset number of multidimensional hidden layers, and a multidimensional output layer; configures the activation function for each multidimensional hidden layer to complete the configuration of each multidimensional hidden layer; after completing the configuration of each multidimensional hidden layer, the multidimensional input layer, each multidimensional hidden layer, and the multidimensional output layer are sequentially connected through a fully connected method to obtain the neural network model; the neural network model is structurally verified to obtain the verification result; when the verification result is successful, the neural network model is trained and configured with a loss function and optimizer to complete the construction of the neural network model. By constructing a lightweight neural network model, the computational overhead during inference is ensured to be extremely small (approximately 2 milliseconds), thereby reducing the operating pressure on electronic devices.
[0092] Figure 3A flowchart illustrating the image generation method provided in this application embodiment. Figure 2 In the embodiments of this application, in Figure 2 Based on the provided embodiments, a detailed explanation of the specific implementation method for the training process of the neural network model in S202 is given. For example... Figure 3 As shown, the method includes:
[0093] S301: Obtain multiple pairs of prompt word images generated in the past.
[0094] In this embodiment, the number of multiple prompt word image pairs can be any number of 3000 pairs, 5000 pairs, or 8000 pairs, or other numbers.
[0095] For example, multiple cue word-image pairs are 6000 pairs of cue word-image pairs.
[0096] For example, the set corresponding to the prompt word-image pairs is: .
[0097] Each prompt word corresponds to: .
[0098] The prompts describe a specific scene; the images are high-quality renderings of that scene.
[0099] S302: Extract the corresponding multiple prompt words and multiple images from multiple prompt word image pairs.
[0100] S303: Determine the corresponding sampling trajectory for each image through a preset trajectory processing framework.
[0101] Specifically, step S303 includes steps a~d:
[0102] Step a: Under the preset trajectory processing framework, determine the expected noise value of any image at each time step in the preset time step sequence.
[0103] For example, the default trajectory processing framework is the CoTj framework.
[0104] For example, any image is The preset time step sequence is T=1000; each time step is t.
[0105] For example, the expected noise value is .
[0106] Step b: Using a diffusion model, perform denoising estimation and reconstruction error processing on each expected noise value and its corresponding time step to obtain the reconstruction error result.
[0107] In this embodiment, the diffusion model is an advanced generative artificial intelligence technology. Its core idea is to generate high-quality data through two processes: adding noise and removing noise. It is particularly outstanding in the field of image generation.
[0108] For example, the expected noise value is determined using a diffusion model. Denoising estimation and reconstruction error processing are performed at the corresponding time step t to obtain the reconstruction error result. .
[0109] in, It is a time step The upper bound of the reconstruction error.
[0110] Step c: Integrate the reconstruction error results corresponding to each time step to obtain the sampling trajectory corresponding to the image.
[0111] In this embodiment, the sampling trajectory is the DNA vector of the optimal sampling trajectory.
[0112] For example, the length of the DNA vector is 1000.
[0113] For example, the reconstruction error results at each time step are integrated to obtain the DNA vector of the optimal sampling trajectory corresponding to the image:
[0114]
[0115] Where, vector This determines the structure and edge weights of the corresponding image.
[0116] Furthermore, the DNA vector of each optimal sampling trajectory is .
[0117] Furthermore, the DNA vectors of the optimal trajectories corresponding to different prompt words exhibit a high degree of structural similarity, i.e., similarity ∈ [0.6,1], although the specific values are different. This indicates that the macroscopic topological structure of the optimal trajectory is highly regular and learnable.
[0118] Furthermore, the optimal trajectory DNA vector is not directly strongly correlated with high-level features such as the semantic complexity of the cue words. This means that the optimal trajectory cannot be derived from the cue words through simple rules. Therefore, the following learning process is needed to capture the complex mapping relationship.
[0119] Step d: Determine the sampling trajectory corresponding to each of the remaining images in the multiple images.
[0120] S304: Perform feature extraction processing on multiple prompt word image pairs and multiple prompt words to obtain the corresponding prompt word features.
[0121] Specifically, step S304 includes steps a to c:
[0122] Step a: Determine the corresponding user style features based on multiple historically generated prompt word images.
[0123] In this embodiment, user style features include color distribution features, brushstroke type features, and other style features.
[0124] Step b: Use a preset feature extraction tool to extract features from each prompt word to obtain the corresponding initial prompt word features.
[0125] In this embodiment, the preset feature extraction tool is the text encoder, BERT encoder, or T5 encoder of the current base model, etc.
[0126] For example, the current base model is stable diffusion 3.5.
[0127] In this embodiment, the initial prompt word feature can be a text-encoded feature or a CLIP feature, and its vector length is L.
[0128] CLIP features are joint embedding representations that map text and images to the same semantic space.
[0129] Step c: Perform cross-modal attention fusion between the initial prompt word features and user style features to obtain the corresponding prompt word features.
[0130] For example, each prompt word feature corresponds to: .
[0131] S305: Train the neural network model based on the features of each sampling trajectory and each prompt word to obtain a trained neural network model.
[0132] Specifically, step S305 includes steps a~c:
[0133] Step a: Integrate the features of each prompt word and the corresponding sampling trajectory into corresponding data pairs, and integrate the data pairs into a dataset, wherein each sampling trajectory corresponds one-to-one with each prompt word feature.
[0134] Step b: According to the preset partitioning ratio, select a portion of the data pairs in the dataset as the training set.
[0135] In this embodiment, the preset division ratio can be any of 70%, 80%, or 90%, or other ratios.
[0136] In addition, the remaining data pairs in the dataset are used as the test set to test the trained neural network model.
[0137] Step c: Train the neural network model using the training set according to the forward propagation algorithm and the back propagation algorithm to obtain a trained neural network model.
[0138] Specifically, step c includes steps c1 to c4:
[0139] Step c1: Input the prompt word features of any data pair in the training set into the neural network model and perform prediction processing through the forward propagation algorithm to obtain the predicted sampling trajectory.
[0140] The forward propagation algorithm is used to instruct the input layer, hidden layer, and output layer in the neural network model to process relevant data in sequence.
[0141] For example, the prompt word features are Neural network model The predicted sampling trajectory is .
[0142] Step c2: Measure the error between the sampled trajectory and the predicted sampled trajectory in the data pair using a loss function.
[0143] The loss function is the mean squared error loss function.
[0144] For example, the loss function is:
[0145]
[0146] In the formula, θ represents the optimal sampling trajectory DNA vector corresponding to the sampling trajectory; θ represents the model parameters.
[0147] Step c3: Update the model parameters of the neural network model based on the error using the backpropagation algorithm and optimizer.
[0148] The backpropagation algorithm is used to instruct the output layer, hidden layer, and input layer in a neural network model to process relevant data in reverse.
[0149] For example, the optimizer is the Adam optimizer.
[0150] Specifically, the gradient of the error is calculated using the backpropagation algorithm; and the model parameters θ of the neural network model are updated along the gradient descent direction using the optimizer.
[0151] Step c4: After the neural network model has been updated, iteratively update the neural network model based on the remaining data in the training set until the model converges to obtain a trained neural network model.
[0152] In summary, the image generation method provided in this embodiment acquires multiple historically generated prompt word image pairs; extracts corresponding prompt words and images from the multiple prompt word image pairs; determines the corresponding sampling trajectory of each image through a preset trajectory processing framework; performs feature extraction processing on the multiple prompt word image pairs and multiple prompt words to obtain the corresponding prompt word features; and trains a neural network model based on each sampling trajectory and each prompt word feature to obtain a trained neural network model. By training the neural network model using multiple historically generated prompt word image pairs to obtain a trained neural network model, the accuracy of the trained neural network model in predicting sampling trajectories is improved.
[0153] In addition, the image generation method provided in this embodiment automatically generates sampling trajectories for a large amount of data by determining the preset trajectory processing framework as the CoTj framework, without the need for manual annotation, thus achieving efficient construction of large-scale training data.
[0154] Furthermore, the image generation method provided in this embodiment demonstrates that the optimal trajectory DNA vectors corresponding to different prompt words exhibit high structural similarity, enabling lightweight networks, i.e., well-trained neural network models, to learn general mapping patterns effectively and possess strong generalization capabilities.
[0155] In addition, the image generation method provided in this embodiment determines the corresponding user style features based on multiple historically generated prompt word images; performs feature extraction processing on each prompt word using a preset feature extraction tool to obtain the corresponding initial prompt word features; and performs cross-modal attention fusion between each initial prompt word feature and the user style features to obtain the corresponding prompt word features, so that a sampling trajectory that is more in line with user preferences can be generated based on the prompt word features.
[0156] Figure 4 This is a schematic diagram of the structure of the image generation apparatus provided in an embodiment of this application. Figure 4 As shown, the image generation device includes: a first extraction module 401, an input module 402, a first determination module 403, a search module 404, a generation module 405, and an output module 406.
[0157] The first extraction module 401 is used to receive the target prompt word sent by the user terminal and extract features from the target prompt word to obtain the target prompt word features;
[0158] The input module 402 is used to input the target prompt word features into the trained neural network model for trajectory prediction processing to obtain the target sampling trajectory, wherein the trained neural network model is trained by a pre-built connected network model;
[0159] The first determining module 403 is used to determine the corresponding offline map based on the target sampling trajectory;
[0160] Search module 404 is used to perform trajectory search on the offline map to generate a sampling time step sequence;
[0161] The generation module 405 is used to generate the target image corresponding to the target prompt word based on the sampling time step sequence;
[0162] The output module 406 is used to output the target image corresponding to the target prompt word to the user terminal.
[0163] In one possible implementation, the device further includes:
[0164] The design module is used to design the network hierarchy of the neural network model to be built, so as to obtain a multi-dimensional input layer, a preset number of multi-dimensional hidden layers and a multi-dimensional output layer.
[0165] The first configuration module is used to configure the activation function for each multidimensional hidden layer to complete the configuration of each multidimensional hidden layer.
[0166] The connection module is used to establish connections between the multidimensional input layer, each multidimensional hidden layer, and the multidimensional output layer in a fully connected manner after the configuration of each multidimensional hidden layer is completed, so as to obtain the neural network model.
[0167] The verification module is used to perform structural verification on the neural network model in order to obtain the verification results;
[0168] The second configuration module is used to configure the loss function and optimizer for the neural network model when the verification result is successful, so as to complete the construction of the neural network model.
[0169] In one possible implementation, the device further includes:
[0170] The acquisition module is used to acquire multiple image pairs of prompt words generated in the past;
[0171] The second extraction module is used to extract multiple prompt words and multiple images from multiple prompt word image pairs;
[0172] The second determining module is used to determine the corresponding sampling trajectory of each image through a preset trajectory processing framework;
[0173] The third extraction module is used to perform feature extraction processing on multiple prompt word image pairs and multiple prompt words to obtain the corresponding prompt word features;
[0174] The training module is used to train the neural network model based on the features of each sampling trajectory and each prompt word to obtain a trained neural network model.
[0175] In one possible implementation, the third extraction module specifically includes:
[0176] The determination unit is used to determine the corresponding user style features based on multiple historically generated prompt word images;
[0177] The extraction unit is used to perform feature extraction processing on each prompt word using a preset feature extraction tool to obtain the corresponding initial prompt word features;
[0178] The fusion unit is used to perform cross-modal attention fusion of the initial prompt word features and user style features to obtain the corresponding prompt word features.
[0179] In one possible implementation, the second determining module specifically includes:
[0180] The first determining unit is used to determine the expected noise value of any image at each time step in a preset time step sequence within a preset trajectory processing framework.
[0181] The processing unit is used to perform denoising estimation and reconstruction error processing on each expected noise value and corresponding time step through a diffusion model to obtain the reconstruction error result;
[0182] The integration unit is used to integrate the reconstruction error results corresponding to each time step to obtain the sampling trajectory corresponding to the image;
[0183] The second determining unit is used to determine the sampling trajectory corresponding to each of the remaining images in the multiple images.
[0184] In one possible implementation, the training module specifically includes:
[0185] The integration unit is used to integrate the features of each prompt word and the corresponding sampling trajectory into corresponding data pairs, and to integrate the data pairs into a dataset, wherein each sampling trajectory corresponds one-to-one with each prompt word feature;
[0186] The unit of determination is used to determine a subset of data pairs in the dataset as the training set according to a preset partitioning ratio.
[0187] The training unit is used to train the neural network model according to the forward propagation algorithm and the back propagation algorithm based on the training set to obtain a trained neural network model.
[0188] In one possible implementation, the training unit specifically includes:
[0189] The input unit is used to input the prompt word features of any data pair in the training set into the neural network model for prediction processing through the forward propagation algorithm to obtain the predicted sampling trajectory;
[0190] The measurement unit is used to measure the error between the sampled trajectory and the predicted sampled trajectory in the data pair using a loss function;
[0191] The first update unit is used to update the model parameters of the neural network model according to the error through the backpropagation algorithm and optimizer;
[0192] The second update unit is used to iteratively update the neural network model based on the remaining data in the training set after the neural network model has been updated, until the model converges, so as to obtain a trained neural network model.
[0193] The apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.
[0194] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device of this embodiment includes: a processor 501 and a memory 502; the memory stores computer execution instructions; at least one processor executes the computer execution instructions stored in the memory, causing at least one processor to execute the image generation method described above.
[0195] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.
[0196] When the memory 502 is set up independently, the electronic device also includes a bus 503 for connecting the memory 502 and the processor 501.
[0197] This application also provides a computer storage medium storing computer execution instructions. When the processor executes the computer execution instructions, the image generation method described above is implemented.
[0198] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the image generation method described above.
[0199] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.
[0200] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.
[0201] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.
[0202] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.
[0203] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.
[0204] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.
[0205] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0206] The aforementioned storage media can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage media can be any available medium accessible to general-purpose or special-purpose computers.
[0207] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.
[0208] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. An image generation method, characterized in that, Applied to electronic devices, including: Receive target prompt words sent by the user terminal, and extract features from the target prompt words to obtain target prompt word features; The target prompt word features are input into a trained neural network model for trajectory prediction to obtain the target sampling trajectory, wherein the trained neural network model is trained by a pre-built connected network model; Determine the corresponding offline map based on the target sampling trajectory; Trajectory search is performed on the offline map to generate a sampling time step sequence; Generate the target image corresponding to the target prompt word based on the sampling time step sequence; Output the target image corresponding to the target prompt word to the user terminal.
2. The method according to claim 1, characterized in that, The process of constructing the neural network model includes: The neural network model to be constructed is designed with network layers to obtain a multidimensional input layer, a predetermined number of multidimensional hidden layers, and a multidimensional output layer; Configure the activation function for each multidimensional hidden layer to complete the configuration of each multidimensional hidden layer; After configuring each multidimensional hidden layer, the multidimensional input layer, each multidimensional hidden layer, and the multidimensional output layer are sequentially connected using a fully connected method to obtain a neural network model. The neural network model is structurally verified to obtain the verification results; When the verification result is successful, the loss function and optimizer are trained and configured on the neural network model to complete the construction of the neural network model.
3. The method according to claim 2, characterized in that, The training process of the connection network model includes: Obtain multiple image pairs of previously generated prompt words; Extract the corresponding multiple prompt words and multiple images from the multiple prompt word image pairs; By using a pre-defined trajectory processing framework, the corresponding sampling trajectory for each image is determined; Feature extraction processing is performed on the multiple prompt word image pairs and the multiple prompt words to obtain the corresponding prompt word features; The neural network model is trained based on the features of each sampling trajectory and each prompt word to obtain a trained neural network model.
4. The method according to claim 3, characterized in that, The step of performing feature extraction processing based on the multiple prompt word image pairs and the multiple prompt words to obtain corresponding prompt word features includes: Determine the corresponding user style characteristics based on multiple historically generated prompt word images; By using a preset feature extraction tool, feature extraction processing is performed on each prompt word to obtain the corresponding initial prompt word features; The initial prompt word features are fused with the user style features through cross-modal attention to obtain the corresponding prompt word features.
5. The method according to claim 3, characterized in that, The process of determining the corresponding sampling trajectory for each image through a preset trajectory processing framework includes: Within the preset trajectory processing framework, the expected noise value of any image at each time step in the preset time step sequence is determined; By using a diffusion model, denoising estimation and reconstruction error processing are performed on each expected noise value and its corresponding time step to obtain the reconstruction error result; The reconstruction error results at each time step are integrated to obtain the sampling trajectory corresponding to the image. Determine the sampling trajectory corresponding to each of the remaining images among the plurality of images.
6. The method according to any one of claims 3 to 5, characterized in that, The step of training the neural network model based on each sampling trajectory and each prompt word feature to obtain a trained neural network model includes: Each prompt word feature and its corresponding sampling trajectory are integrated into a corresponding data pair, and each data pair is integrated into a dataset, wherein each sampling trajectory corresponds one-to-one with each prompt word feature; According to a preset division ratio, a portion of the data pairs in the dataset are determined as the training set; The neural network model is trained using the forward propagation algorithm and the back propagation algorithm based on the training set to obtain a trained neural network model.
7. The method according to claim 6, characterized in that, The step of training the neural network model using the training set according to the forward propagation algorithm and the back propagation algorithm to obtain a trained neural network model includes: The prompt word features of any data pair in the training set are input into the neural network model and processed by the forward propagation algorithm to obtain the predicted sampling trajectory. The loss function is used to measure the error between the sampled trajectory in the data pair and the predicted sampled trajectory. The model parameters of the neural network model are updated based on the error using the backpropagation algorithm and optimizer. After the neural network model is updated, iterative updates are performed on the remaining data in the training set until the model converges to obtain a trained neural network model.
8. An image generation apparatus, characterized in that, Applied to electronic devices, including: The first extraction module is used to receive target prompt words sent by the user terminal and extract features from the target prompt words to obtain target prompt word features; The input module is used to input the target prompt word features into a trained neural network model for trajectory prediction processing to obtain the target sampling trajectory, wherein the trained neural network model is trained by a pre-built connected network model; The first determining module is used to determine the corresponding offline map based on the target sampling trajectory; The search module is used to perform trajectory search on the offline map to generate a sampling time step sequence; The generation module is used to generate the target image corresponding to the target prompt word based on the sampling time step sequence; The output module is used to output the target image corresponding to the target prompt word to the user terminal.
9. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the image generation method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, The computer storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, it implements the image generation method as described in any one of claims 1 to 7.