Image generation model training method, system and device

Abstract

The invention discloses a model training method for image generation, which is used for improving the accuracy of a model generated image. The method comprises the following steps: acquiring initial data; obtaining semantic information from the initial data and analyzing the semantic information to obtain semantic features; analyzing human body actions in the initial image corresponding to the semantic information to obtain posture features; obtaining interaction features interacting with the human body action in the initial image; obtaining visual features of the initial image through an independent convolutional network; performing multi-modal information fusion based on the semantic features in combination with the posture features, the interaction features and the visual features to generate a target feature vector; and when all the initial data generate target feature vectors, training a diffusion model through the target feature vectors to obtain a target diffusion model.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, system and apparatus for training an image generation model. Background Technology

[0002] In the field of image generation, deep learning-driven models have made significant progress, with methods based on diffusion models and generative adversarial networks (GANs) capable of generating highly realistic images. However, these models still face many challenges when generating images for specific scenarios.

[0003] From the perspective of semantic accuracy of generated images, existing models struggle to understand and generate images related to complex semantics. For example, when the input contains text prompts with multiple objects and complex relationships, the generated images may fail to accurately reflect the semantic relationships in the text, resulting in issues such as incorrect object positions and illogical actions. This problem is even more pronounced when generating images related to people.

[0004] The datasets upon which existing image generation models rely for training have limitations. Image annotations in these datasets may not be precise enough, especially for human interaction scenes, where annotations may only cover the main objects and actions, ignoring subtle semantic information. Furthermore, the data distribution is uneven, with a large number of images for common scenes but a scarcity of images for some special or rare scenes. This leads to models overlearning common scenes during training, resulting in insufficient ability to generate images for special scenes and a lack of diversity in the generated images. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a model training method for image generation, which improves the accuracy of image generation by the model.

[0006] The technical solution provided in this application is described below: The first aspect of this application provides a method for training an image generation model, including: Acquire initial data, which includes several semantic information and an initial image corresponding to the semantic information; Semantic information is obtained from the initial data and analyzed to obtain semantic features; The human body movements in the initial image corresponding to the semantic information are analyzed to obtain posture features; Obtain the interaction features in the initial image that interact with the human body's actions; Visual features of the initial image are obtained through independent convolutional networks; Based on the semantic features, combined with the posture features, the interaction features, and the visual features, multimodal information fusion is performed to generate a target feature vector; When all the initial data generate target feature vectors, the diffusion model is trained using the target feature vectors to obtain the target diffusion model.

[0007] Optionally, the step of obtaining semantic information from the initial data and analyzing the semantic information to obtain semantic features includes: Obtain the semantic information to be analyzed from the initial data; The semantic information is structurally parsed using natural language processing tools to obtain the parsing results; The parsing results are input into a pre-trained language model to obtain semantic features.

[0008] Optionally, the step of analyzing human actions in the initial image corresponding to the semantic information to obtain pose features includes: Obtain the initial image corresponding to the semantic information; The pose detector identifies the human joint positions and confidence scores in the initial image, and obtains a set of joint attention values. A pose attention map is constructed based on the set of joint attention values ​​to obtain the key pose regions; The key pose regions are transformed into pose features through thresholding and downsampling.

[0009] Optionally, obtaining the interaction features in the initial image that interact with the human body's actions includes: Use a segmentation tool to extract the outlines of people and objects in the initial image; Calculate the distance matrix between the figure outline and the object outline, and identify the key points of the interaction boundary between the figure outline and the object outline using the distance matrix; An interaction attention map is generated based on the key points of the interaction boundary, and interaction features are generated through threshold processing.

[0010] Optionally, obtaining the visual features of the initial image through an independent convolutional network includes: The initial image is convolved using an independent convolutional network to obtain low-level and high-level features. Pooling is performed on the low-level features and the high-level features respectively to obtain salient features; The initial image with the aforementioned salient features is flattened, and visual features are generated through a fully connected layer.

[0011] Optionally, the step of fusing multimodal information based on the semantic features, the pose features, the interaction features, and the visual features to generate a target feature vector includes: Unify the basic format of the semantic features, the posture features, the interaction features, and the visual features; By concatenating features, the semantic features, the posture features, the interaction features, and the visual features are fused into an initial feature vector; The weights of all features contained in the comprehensive feature vector are dynamically allocated through an attention mechanism to obtain an intermediate feature vector; The intermediate feature vectors are weighted and fused to obtain the target feature vector.

[0012] Optionally, when all the initial data generates target feature vectors, training the diffusion model using the target feature vectors to obtain the target diffusion model includes: Construct the index relationship between the target feature vector and the initial image; The index relationship is input into the diffusion model; The initial image obtained from the index relationship is transformed into a noisy image through forward diffusion; The training image is obtained by performing reverse denoising on the noisy image using the target feature vector; The loss value of the training image and the noise is calculated using a loss function; The diffusion model is optimized based on the loss value until it converges or reaches a preset threshold. The training of the diffusion model is then complete, and the target diffusion model is output.

[0013] A second aspect of this application provides an image generation model training system, comprising: The first acquisition unit is used to acquire initial data, which includes several semantic information and an initial image corresponding to the semantic information. The second acquisition unit is used to acquire semantic information from the initial data and analyze the semantic information to obtain semantic features; The analysis unit is used to analyze human actions in the initial image corresponding to the semantic information to obtain posture features; The third acquisition unit is used to acquire the interaction features in the initial image that interact with the human body's actions; The fourth acquisition unit is used to acquire the visual features of the initial image through an independent convolutional network; The fusion unit is used to perform multimodal information fusion based on the semantic features, the posture features, the interaction features, and the visual features to generate a target feature vector; The training unit is used to train the diffusion model using the target feature vectors when all the initial data generate target feature vectors, thereby obtaining the target diffusion model.

[0014] Optionally, the second acquisition unit is specifically used for: Obtain the semantic information to be analyzed from the initial data; The semantic information is structurally parsed using natural language processing tools to obtain the parsing results; The parsing results are input into a pre-trained language model to obtain semantic features.

[0015] Optionally, the analysis unit is specifically used for: Obtain the initial image corresponding to the semantic information; The pose detector identifies the human joint positions and confidence scores in the initial image, and obtains a set of joint attention values. A pose attention map is constructed based on the set of joint attention values ​​to obtain the key pose regions; The key pose regions are transformed into pose features through thresholding and downsampling.

[0016] Optionally, the third acquisition unit is specifically used for: Use a segmentation tool to extract the outlines of people and objects in the initial image; Calculate the distance matrix between the figure outline and the object outline, and identify the key points of the interaction boundary between the figure outline and the object outline using the distance matrix; An interaction attention map is generated based on the key points of the interaction boundary, and interaction features are generated through threshold processing.

[0017] Optionally, the fourth acquisition unit is specifically used for: The initial image is convolved using an independent convolutional network to obtain low-level and high-level features. Pooling is performed on the low-level features and the high-level features respectively to obtain salient features; The initial image with the aforementioned salient features is flattened, and visual features are generated through a fully connected layer.

[0018] Optionally, the fusion unit is specifically used for: Unify the basic format of the semantic features, the posture features, the interaction features, and the visual features; By concatenating features, the semantic features, the posture features, the interaction features, and the visual features are fused into an initial feature vector; The weights of all features contained in the comprehensive feature vector are dynamically allocated through an attention mechanism to obtain an intermediate feature vector; The intermediate feature vectors are weighted and fused to obtain the target feature vector.

[0019] Optionally, the training unit is specifically used for: Construct the index relationship between the target feature vector and the initial image; The index relationship is input into the diffusion model; The initial image obtained from the index relationship is transformed into a noisy image through forward diffusion; The training image is obtained by performing reverse denoising on the noisy image using the target feature vector; The loss value of the training image and the noise is calculated using a loss function; The diffusion model is optimized based on the loss value until it converges or reaches a preset threshold. The training of the diffusion model is then complete, and the target diffusion model is output.

[0020] A third aspect of this application provides an image generation model training apparatus, the apparatus comprising: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to execute the first aspect and any one of the optional methods in the first aspect.

[0021] A fourth aspect of this application provides a computer-readable storage medium on which a program is stored, which, when executed on a computer, performs the methods of the first aspect and any one of the first aspects.

[0022] As can be seen from the above technical solutions, this application has the following advantages: This application achieves multimodal information fusion by combining the semantic features with the pose features, interaction features, and visual features. The fused target feature vector is used as the training data for the diffusion model. The diffusion model for image generation is trained using multiple feature vectors, thereby improving the quality, accuracy, and diversity of images generated by the diffusion model. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic flowchart of an embodiment of the image generation model training method in this application; Figure 2a This is a schematic flowchart of an embodiment of the first stage of the image generation model training method in this application; Figure 2b This is a schematic flowchart of an embodiment of the second stage of the image generation model training method in this application; Figure 2c This is a schematic flowchart of an embodiment of the third stage of the image generation model training method in this application; Figure 3 This is a schematic diagram of an embodiment of the image generation model training system in this application; Figure 4 This is a schematic diagram of an embodiment of the image generation model training device in this application. Detailed Implementation

[0025] It should be noted that the image generation model training method provided in this application can be applied to terminals, systems, and servers. For example, a terminal can be a smartphone, computer, tablet, smart TV, smartwatch, portable computer, or a desktop computer, etc. For ease of explanation, this application uses a terminal as the execution subject for illustration.

[0026] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0027] Please see Figure 1 This application first provides an embodiment of an image generation model training method, which includes: S101. Obtain initial data, wherein the initial data includes several semantic information and an initial image corresponding to the semantic information; The initial data is used to generate relevant data for synthesizing multimodal fusion information. In this embodiment, the initial data includes semantic information and the initial image corresponding to the semantic information.

[0028] Semantic information can be indexed by naming the image files of the initial images. This initial data mainly consists of images, and the semantic information is used to describe the specific content of the images.

[0029] The image data in the initial data of this embodiment includes interactive actions of people and other things.

[0030] Specifically, the initial dataset is generated through manual annotation or automatic collection. It contains text descriptions of human interaction scenes and corresponding images. Semantic information is provided in natural language text form, describing the relationships between people, actions, objects, and scenes. The initial images are high-resolution images (e.g., RGB format), containing people and their interaction scenes with objects, corresponding to the semantic information content. This initial data is organized in pairs (text-image pairs) and stored in a database or file for easy subsequent reading, either individually or in batches.

[0031] S102. Obtain semantic information from the initial data and analyze the semantic information to obtain semantic features; In practice, semantic information processing and image processing need to correspond. When semantic information is bound to the initial image, the terminal can still determine the initial image data corresponding to the current semantic information based on its index relationship when processing data. Therefore, it can perform batch processing or processing one by one when processing semantic information or the initial image later.

[0032] Specifically, the terminal performs natural language processing on the text, analyzes its semantic content, extracts key elements from the description, such as people, actions, objects and their relationships, and transforms the text content into fixed-dimensional feature vectors to represent semantic information. Using language processing techniques, the text is mapped to a high-dimensional semantic space to generate semantic feature vectors; therefore, the actual representation of semantic features is in vector form.

[0033] S103. Analyze the human body movements in the initial image corresponding to the semantic information to obtain posture features; Images corresponding to the text descriptions are read from the initial dataset. The human movements in the images are analyzed to identify the spatial locations of key human body parts, such as the head, limbs, and torso. Motion analysis techniques are used to detect the joints or key points of the human body and generate feature representations of the movement patterns. The feature representations are processed to highlight key movement areas and form posture features that reflect the human body's movement state. In practice, the final posture features are represented in the form of a posture mask.

[0034] S104. Obtain the interaction features in the initial image that interact with the human body's actions; The process involves analyzing the interaction areas between people and objects in the initial image, focusing on contact or proximity areas related to actions, identifying the spatial relationship between key parts of the person (such as hands) and objects (such as tools or objects), extracting features from the interaction areas, generating feature representations that represent the interaction relationships, highlighting the contact or associated areas between people and objects, processing the feature representations to form interaction features that reflect the interaction patterns between actions and objects, and obtaining the content as interaction information, which is the interaction feature. The interaction feature is represented by an interaction boundary mask.

[0035] S105. Obtain the visual features of the initial image through an independent convolutional network; The initial image is input into a separate convolutional neural network (CNN). Through multiple convolutional operations, the image's appearance information is extracted, including low-level features such as edges, texture, and color, as well as high-level features such as object shape and structure. The extracted features are then subjected to dimensionality reduction processing to generate fixed-dimensional feature vectors that represent the image's visual information; this visual information constitutes the visual features of the initial image.

[0036] S106. Based on the semantic features, the posture features, the interaction features, and the visual features, multimodal information fusion is performed to generate a target feature vector; Semantic features, pose features, interaction features, and visual features are adjusted to a unified representation format to ensure consistency in dimensions or data structure. The adjusted features are then combined into a comprehensive feature representation that includes multimodal information from text and images. Using fusion techniques, the contribution of each feature is dynamically adjusted to generate a single target feature vector that integrates all modal information.

[0037] Specifically, the semantic features, posture features, interaction features, and visual features actually obtained are not data of the same dimension. Therefore, before fusion, it is necessary to unify the expression dimension and data format of the above features to meet the needs of multimodal information fusion and thus obtain the target feature vector.

[0038] S107. When all the initial data generate target feature vectors, a diffusion model is trained using the target feature vectors to obtain the target diffusion model.

[0039] For each pair of target feature vectors and the initial image, a pairing relationship is established to form a training set. Using a latent space-based diffusion model, with the target feature vectors as conditions, the model is trained to generate images consistent with the initial images. The diffusion model training process includes at least the following: Noise is progressively added to the initial image to generate images with varying noise levels. Guided by the target feature vector, the content of the original image is recovered from the noisy images. The model parameters are optimized to make the generated images closely resemble the initial images and conform to the semantic and visual requirements of the target feature vector. Training continues until the model performance stabilizes (e.g., the quality of the generated images reaches expectations), at which point training is complete.

[0040] The conditions for completing training are that the diffusion model converges or a sufficient number of data iterations are performed. Once these conditions are met, the terminal will output the diffusion model with the current parameters as the target diffusion model.

[0041] This application achieves multimodal information fusion by combining the semantic features with the pose features, interaction features, and visual features. The fused target feature vector is used as the training data for the diffusion model. The diffusion model for image generation is trained using multiple feature vectors, thereby improving the quality, accuracy, and diversity of images generated by the diffusion model.

[0042] Please see Figure 2a , Figure 2b and Figure 2c This application provides another embodiment of an image generation model training method, which includes: S201. Obtain initial data, wherein the initial data includes several semantic information and an initial image corresponding to the semantic information; Step S201 in this embodiment is similar to step S101 in the previous embodiment, and will not be described in detail here.

[0043] S202. Obtain the semantic information to be analyzed from the initial data; Specifically, the terminal extracts a semantic information T from the initial dataset. i Let i represent the current text-image data as the i-th element in the initial data, and T i This corresponds to the text portion of the i-th text-image pair. The text descriptions are read sequentially to ensure that each piece of semantic information is processed in order, preparing for subsequent analysis. The text is stored in natural language and includes relationships between people, actions, objects, and scenes.

[0044] S203. Perform structural parsing on the semantic information using natural language processing tools to obtain the parsing results; Specifically, structural parsing involves using natural language processing tools to perform lexical, syntactic, and semantic analysis on semantic information T. Lexical analysis decomposes semantic information T into a set of words or phrases W = {w1, w2, ... w...}. n}; syntactic analysis determines the grammatical relationships between words and constructs a syntax tree structure G; semantic analysis mines the semantic connections between words and obtains a set of semantic relationships S. For example, for the text prompt "a person is watering flowers in the garden", lexical analysis yields W={person, in, garden, watering flowers}, syntactic analysis determines that "person" is the executor of the action "watering flowers", "in the garden" is an adverbial of place, and semantic analysis clarifies that "watering flowers" is an interactive action between a person and flowers in a garden scene.

[0045] S204. Input the parsing results into the pre-trained language model to obtain semantic features.

[0046] After completing the structural parsing, the terminal obtains the parsing result corresponding to the semantic information, enabling the terminal to process the text prompt using the BERT language model. The semantic information T is input into model L, and the model, based on its trained language knowledge and semantic representations, outputs a semantic feature vector V. text Mathematically, this is represented as:

[0047] Semantic feature vectors contain information about the entities in the text and the semantic relationships between them. Each dimension of the semantic feature vector corresponds to the strength or weight of a specific semantic feature.

[0048] S205. Obtain the initial image corresponding to the semantic information; Extract semantic information T from the initial dataset. i Paired initial image I i The system reads the image data from the initial image and ensures it matches the corresponding text description. The image is in high-resolution RGB format and contains a person and their interaction with objects.

[0049] S206. Identify the human joint positions and confidence scores in the initial image using a posture detector to obtain a set of joint attention values; Using a pose detector to detect the pose of people in an image and obtain the position of human joints. and the corresponding confidence score ,in, This refers to the number of visible joints. The pose attention map is calculated based on joint positions and confidence scores. .

[0050] Specifically, in the context of posture attention maps Previously, it was necessary to determine the locations of all human joints and their confidence scores, and then, based on the confidence scores corresponding to joint i... Calculate the attention value G of the i-th joint at the initial image coordinates (x, y). i (x, y), calculated as follows:

[0051] Where (x, y) is the current calculated coordinate position. The position of the i-th joint It is the standard deviation of the Gaussian distribution.

[0052] The attention values ​​of all joints in the initial image are calculated one by one at all coordinates, and the set of the calculation results is the joint attention value set.

[0053] S207. Construct a posture attention map based on the set of joint attention values ​​to obtain the key posture regions; After obtaining the set of joint attention values, this set is identified as image attention. A pose attention map is then constructed based on this set. Specifically, the joint confidence score is essentially the same parameter as the attention of that joint in the image; therefore, the pose attention map... The calculation method is as follows:

[0054] Specifically, the joint attention values ​​of all visible joints are summed, and the result is reflected in the initial image to obtain a pose attention map. .

[0055] S208. Through threshold processing and downsampling, the key pose region is converted into pose features.

[0056] After acquiring the pose attention map, the terminal applies a threshold to it, generating a binary mask and retaining regions with attention values ​​higher than the threshold. The binary mask is then downsampled to reduce resolution and computational complexity. Finally, the processed mask is converted into pose features, representing key action regions.

[0057] pose attention map Convert to pose mask The process is as follows:

[0058] in, To target the pose attention map threshold, It is a downsampling operation, making Matches the size of the image feature map.

[0059] pose mask Low-pose quality regions are highlighted and used as a representation of pose features for subsequent guided image generation. In this application, pose features are represented using pose masks. This indicates that the pose mask is a binary feature map.

[0060] S209. Use a segmentation tool to extract the outlines of people and objects in the initial image; For the collected images containing human interaction By using segmentation tools, information about the body parts of people and the outlines of objects in an image can be obtained.

[0061] Specifically, segmentation tools (such as Mask R-CNN) are used to analyze the image, identify, and extract the contours of people and objects. The segmentation tool generates pixel-level masks for the person and object regions, marking the contour boundaries. The output is a set of person and object contours.

[0062] S210. Calculate the distance matrix between the figure outline and the object outline, and identify the key points of the interaction boundary between the figure outline and the object outline through the distance matrix. The distance D between the character's joint coordinates and the object's outline points is calculated as follows:

[0063] in, It refers to the number of joints in a character. It describes the number of fixed points on an object. It is a distance matrix.

[0064] S211. Generate an interaction attention map based on the interaction boundary key points, and generate interaction features through threshold processing.

[0065] Similar to pose attention maps, interaction attention maps select the points with the smallest distance between T (constant) pairs as interaction boundary keypoints. Score of key interaction points For this point in the distance matrix The minimum distance in.

[0066] Calculate the interaction boundary attention map based on the interaction key point scores. Interaction Boundary Attention Graph The calculation method is as follows:

[0067] in, It is the position of the i-th interaction boundary key point. The score is the score of the interactive key point, (x,y) is the current calculated coordinate position, and T is one of the points with the smallest distance from each other, so the value is 2T.

[0068] Determine the interaction boundary attention graph Then, the interaction boundary attention map is processed in the same way. Convert to interactive mask The formula is as follows:

[0069] in, It is a threshold for attention at the interaction boundary.

[0070] Interactive boundary mask The interactive boundary region is highlighted and used as an interactive feature to guide image generation.

[0071] S212. Perform convolution operations on the initial image using an independent convolutional network to obtain low-level features and high-level features; The initial image is fed into a separate convolutional neural network (CNN). Through multiple layers of convolution operations, low-level features (such as edges and textures) and high-level features (such as object shape and structure) are extracted. Low-level features are generated by shallow convolutions, and high-level features are generated by deep convolutions.

[0072] Specifically, a visual feature extractor is constructed: a multi-layer convolutional neural network (CNN) is built as the visual feature extractor, denoted as F. This network consists of multiple convolutional layers, pooling layers, and fully connected layers.

[0073] Convolution operations extract low-level features: from the preprocessed image... enter Convolutional layers extract low-level features such as edges and textures from an image through convolution operations, as shown in the following formula:

[0074] in, It is Each convolutional kernel is located at... The convolution result at the point, It is the pixel value at the corresponding position in the input image. It is The weights of each convolutional kernel, It's a bias. and is the size of the convolution kernel.

[0075] S213. Pool the low-level features and the high-level features respectively to obtain salient features; Dimensionality reduction: Pooling layers (max pooling) are used to reduce the dimensionality of the feature map, thus reducing computational cost. The formula is as follows:

[0076] in, These are the feature map values ​​after pooling. is the size of the pooling window, k is the convolution kernel, i and j are the coordinates of the convolution kernel, and m and n are the convolution iteration indices.

[0077] S214. Flatten the initial image with the aforementioned salient features and generate visual features through a fully connected layer.

[0078] The fully connected layer outputs a visual feature vector: Finally, the feature map is converted into a one-dimensional vector through the fully connected layer, resulting in the visual feature vector V of the image. image , the formula is as follows:

[0079] This describes the process of acquiring visual features from the initial image I using a visual feature extractor.

[0080] S215. Unify the basic format of the semantic features, the posture features, the interaction features, and the visual features; As can be seen from the steps above, among semantic features, posture features, interaction features, and visual features, visual features and semantic features are one-dimensional features, while posture features and interaction features are two-dimensional features in the form of masks. Therefore, it is necessary to perform dimensional transformation on posture features and interaction features.

[0081] Specifically, regarding pose masks and interaction boundary mask Dimension transformation is performed using fully connected layers. This is achieved through a learnable weight matrix. and bias vector Convert the pose mask to:

[0082] Similarly, using the weight matrix and bias vector This transforms the interaction boundary mask into:

[0083] Ensure pose mask and interaction boundary mask It has the same dimensions as other feature vectors.

[0084] S216. By feature concatenation, the semantic features, the posture features, the interaction features, and the visual features are fused into an initial feature vector; By employing an attention mechanism, weights for different modal features are dynamically allocated, enabling the model to focus on key features based on input information. First, all feature vectors are concatenated into a composite feature vector, i.e., the initial feature vector V. concat :

[0085] S217. The weights of all features contained in the comprehensive feature vector are dynamically allocated through an attention mechanism to obtain an intermediate feature vector; An attention mechanism is used to assign weights to each part (semantic, pose, interaction, and visual) of the initial feature vector, highlighting important features. Attention scores are calculated using a neural network, and the weights are dynamically adjusted based on feature relevance. These weights are then applied to the initial feature vector to generate a weighted intermediate feature vector.

[0086] Attention weights are calculated using a multilayer perceptron (MLP). First, through Perform a nonlinear transformation, and then... Obtain the normalized attention weights.

[0087] in , This is the weight matrix. b h , As the bias vector, after determining the attention weight of each feature through the above calculation, the intermediate feature vector is obtained by redeploying according to the attention weight.

[0088] S218. The intermediate feature vectors are weighted and fused to obtain the target feature vector.

[0089] The intermediate feature vectors are weighted and fused to integrate the attention-weighted features. A fully connected layer or linear transformation is used to map the intermediate feature vectors to the target dimension, generating the target feature vector. Specifically, this is based on the calculated attention weights. The target feature vector is generated by weighted summation of each feature vector. The calculation formula is:

[0090] in , , , , Attention weight vector The Each element.

[0091] Through this fusion process, It can fully integrate multimodal information, providing strong support for image generation models.

[0092] S219. Construct the index relationship between the target feature vector and the initial image; A pairing index is created for each pair of target feature vectors and their corresponding initial images to ensure that the training data is organized in an orderly manner. The index relationship records the correspondence between feature vectors and images and is stored as key-value pairs or database entries.

[0093] Specifically, the target feature vector With the corresponding initial image I i Perform pairings to build a training dataset. ,in The number of training samples is .

[0094] S220. Input the index relationship into the diffusion model; Specifically, the diffusion model uses the Stable Diffusion model as the basic architecture for image generation. In the initialization phase, the model parameters are reasonably initialized according to the structural characteristics and training requirements of the model to ensure that the model has a good learning starting point in the early stage of training. The data used for training this model is the index relationship established by the target feature vector and the initial image. This data with index relationship is the training data.

[0095] S221. The initial image obtained from the index relationship is converted into a noisy image through forward diffusion; A forward diffusion process is applied to progressively add Gaussian noise to the latent space representation. Noise images are generated at multiple time steps t according to a predefined noise schedule.

[0096] In each iteration, for the input target image Noise is gradually added through a forward diffusion process, transforming it into a noise distribution. The forward diffusion process is based on the principles of the diffusion model and proceeds according to the following formula:

[0097] in , , It is the noise addition rate, which is typically a value over time. A changing function that controls the rate at which noise is added. From the standard normal distribution The noise vector obtained from sampling. With time steps... The increase of the initial image I i Gradually drowned out by noise, and finally in time step At that time, the image completely becomes a noisy image. .

[0098] S222. Perform reverse denoising on the noisy image using the target feature vector to obtain a training image; A denoising network (U-Net) using a diffusion model is used to target feature vectors. Given the condition, predict the noise components in a noisy image.

[0099] Based on the predicted noise, reverse denoising is performed to generate a clearer image.

[0100] target feature vector and noisy images The image is input into the Stable Diffusion model, which attempts to recover the target image from the noisy image through a reverse denoising process.

[0101] The core of the model is a denoising neural network. It is based on the noisy image Predicting noise at time step t In Stable Diffusion, the target feature vector is... This conditional information is input into the denoising neural network, enabling the model to combine multimodal information for denoising.

[0102] Starting from time step T, reverse denoising is performed according to the formula:

[0103] Where z is from the standard normal distribution noise obtained from mid-sample

[0104] As the time step gradually decreases, the model continuously removes noise, and the image gradually becomes clearer, eventually reaching a clearer image at time step [number missing]. The generated image is obtained at that time. .

[0105] S223. Calculate the loss value of the training image and the noise using a loss function; A loss function is used to measure the difference between the image generated by the model and the target image. In StableDiffusion, a commonly used loss function is the mean squared error (MSE) loss function, with the following formula:

[0106] in It's real noise. It is noise in the model prediction.

[0107] This loss function measures the accuracy of the model's noise prediction by calculating the mean square error between the actual noise and the predicted noise, thereby guiding the model's training.

[0108] S224. Optimize the diffusion model based on the loss value until the diffusion model converges or reaches a preset threshold, then complete the training of the diffusion model and output the target diffusion model.

[0109] Repeat the forward diffusion process, reverse denoising process, loss function calculation, and parameter update steps described above for multiple iterations of training until the model converges or reaches the preset number of training rounds. Finally, utilize the model's inference capabilities to generate high-quality images.

[0110] The image generation model training method in the embodiments of this application has been described in detail above. The image generation model training system and apparatus will be described in detail below.

[0111] Please see Figure 3 This application provides an embodiment of an image generation model training system, which includes: The first acquisition unit 301 is used to acquire initial data, the initial data including several semantic information and an initial image corresponding to the semantic information; The second acquisition unit 302 is used to acquire semantic information from the initial data and analyze the semantic information to obtain semantic features; Analysis unit 303 is used to analyze human actions in the initial image corresponding to the semantic information to obtain posture features; The third acquisition unit 304 is used to acquire the interaction features in the initial image that interact with the human body's actions; The fourth acquisition unit 305 is used to acquire the visual features of the initial image through an independent convolutional network; The fusion unit 306 is used to perform multimodal information fusion based on the semantic features, the posture features, the interaction features, and the visual features to generate a target feature vector; Training unit 307 is used to train a diffusion model using the target feature vectors when all the initial data generate target feature vectors, thereby obtaining a target diffusion model.

[0112] Optionally, the second acquisition unit 302 is specifically used for: Obtain the semantic information to be analyzed from the initial data; The semantic information is structurally parsed using natural language processing tools to obtain the parsing results; The parsing results are input into a pre-trained language model to obtain semantic features.

[0113] Optionally, the analysis unit 303 is specifically used for: Obtain the initial image corresponding to the semantic information; The pose detector identifies the human joint positions and confidence scores in the initial image, and obtains a set of joint attention values. A pose attention map is constructed based on the set of joint attention values ​​to obtain the key pose regions; The key pose regions are transformed into pose features through thresholding and downsampling.

[0114] Optionally, the third acquisition unit 304 is specifically used for: Use a segmentation tool to extract the outlines of people and objects in the initial image; Calculate the distance matrix between the figure outline and the object outline, and identify the key points of the interaction boundary between the figure outline and the object outline using the distance matrix; An interaction attention map is generated based on the key points of the interaction boundary, and interaction features are generated through threshold processing.

[0115] Optionally, the fourth acquisition unit 305 is specifically used for: The initial image is convolved using an independent convolutional network to obtain low-level and high-level features. Pooling is performed on the low-level features and the high-level features respectively to obtain salient features; The initial image with the aforementioned salient features is flattened, and visual features are generated through a fully connected layer.

[0116] Optionally, the fusion unit 306 is specifically used for: Unify the basic format of the semantic features, the posture features, the interaction features, and the visual features; By concatenating features, the semantic features, the posture features, the interaction features, and the visual features are fused into an initial feature vector; The weights of all features contained in the comprehensive feature vector are dynamically allocated through an attention mechanism to obtain an intermediate feature vector; The intermediate feature vectors are weighted and fused to obtain the target feature vector.

[0117] Optionally, the training unit 307 is specifically used for: Construct the index relationship between the target feature vector and the initial image; The index relationship is input into the diffusion model; The initial image obtained from the index relationship is transformed into a noisy image through forward diffusion; The training image is obtained by performing reverse denoising on the noisy image using the target feature vector; The loss value of the training image and the noise is calculated using a loss function; The diffusion model is optimized based on the loss value until it converges or reaches a preset threshold. The training of the diffusion model is then complete, and the target diffusion model is output.

[0118] In this embodiment, the functions of each unit are the same as those described above. Figures 1 to 2c The steps in the illustrated embodiments are the same and will not be repeated here.

[0119] Please see Figure 4 This application provides another embodiment of an image generation model training apparatus, including: Processor 401, memory 402, input / output unit 403, bus 404; The processor 401 is connected to the memory 402, the input / output unit 403 and the bus 404; The processor 401 specifically executes... Figures 1 to 2c The specific operations corresponding to the steps in the method will not be elaborated here.

[0120] This application also relates to a computer-readable storage medium on which a program is stored, characterized in that, when the program is run on a computer, it causes the computer to perform any of the methods described above.

[0121] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for training an image generation model, characterized in that, The method includes: Acquire initial data, which includes several semantic information and an initial image corresponding to the semantic information; Semantic information is obtained from the initial data and analyzed to obtain semantic features; The human body movements in the initial image corresponding to the semantic information are analyzed to obtain posture features; Obtain the interaction features in the initial image that interact with the human body's actions; Visual features of the initial image are obtained through independent convolutional networks; Based on the semantic features, combined with the posture features, the interaction features, and the visual features, multimodal information fusion is performed to generate a target feature vector; When all the initial data generate target feature vectors, the diffusion model is trained using the target feature vectors to obtain the target diffusion model.

2. The method according to claim 1, characterized in that, The step of obtaining semantic information from the initial data and analyzing the semantic information to obtain semantic features includes: Obtain the semantic information to be analyzed from the initial data; The semantic information is structurally parsed using natural language processing tools to obtain the parsing results; The parsing results are input into a pre-trained language model to obtain semantic features.

3. The method according to claim 1, characterized in that, The step of analyzing human movements in the initial image corresponding to the semantic information to obtain posture features includes: Obtain the initial image corresponding to the semantic information; The pose detector identifies the human joint positions and confidence scores in the initial image, and obtains a set of joint attention values. A pose attention map is constructed based on the set of joint attention values ​​to obtain the key pose regions; The key pose regions are transformed into pose features through thresholding and downsampling.

4. The method according to claim 1, characterized in that, The step of obtaining the interaction features in the initial image that interact with the human body's actions includes: Use a segmentation tool to extract the outlines of people and objects in the initial image; Calculate the distance matrix between the figure outline and the object outline, and identify the key points of the interaction boundary between the figure outline and the object outline using the distance matrix; An interaction attention map is generated based on the key points of the interaction boundary, and interaction features are generated through threshold processing.

5. The method according to claim 1, characterized in that, The step of obtaining the visual features of the initial image through an independent convolutional network includes: The initial image is convolved using an independent convolutional network to obtain low-level and high-level features. Pooling is performed on the low-level features and the high-level features respectively to obtain salient features; The initial image with the aforementioned salient features is flattened, and visual features are generated through a fully connected layer.

6. The method according to any one of claims 1 to 5, characterized in that, The step of fusing multimodal information based on the semantic features, pose features, interaction features, and visual features to generate a target feature vector includes: Unify the basic format of the semantic features, the posture features, the interaction features, and the visual features; By concatenating features, the semantic features, the posture features, the interaction features, and the visual features are fused into an initial feature vector; The weights of all features contained in the comprehensive feature vector are dynamically allocated through an attention mechanism to obtain an intermediate feature vector; The intermediate feature vectors are weighted and fused to obtain the target feature vector.

7. The method according to any one of claims 1 to 5, characterized in that, When all the initial data generates a target feature vector, the diffusion model is trained using the target feature vector to obtain the target diffusion model, which includes: Construct the index relationship between the target feature vector and the initial image; The index relationship is input into the diffusion model; The initial image obtained from the index relationship is transformed into a noisy image through forward diffusion; The training image is obtained by performing reverse denoising on the noisy image using the target feature vector; The loss value of the training image and the noise is calculated using a loss function; The diffusion model is optimized based on the loss value until it converges or reaches a preset threshold. The training of the diffusion model is then complete, and the target diffusion model is output.

8. A model training system for image generation, characterized in that, The system includes: The first acquisition unit is used to acquire initial data, which includes several semantic information and an initial image corresponding to the semantic information. The second acquisition unit is used to acquire semantic information from the initial data and analyze the semantic information to obtain semantic features; The analysis unit is used to analyze human actions in the initial image corresponding to the semantic information to obtain posture features; The third acquisition unit is used to acquire the interaction features in the initial image that interact with the human body's actions; The fourth acquisition unit is used to acquire the visual features of the initial image through an independent convolutional network; The fusion unit is used to perform multimodal information fusion based on the semantic features, the posture features, the interaction features, and the visual features to generate a target feature vector; The training unit is used to train the diffusion model using the target feature vectors when all the initial data generate target feature vectors, thereby obtaining the target diffusion model.

9. A model training device for image generation, characterized in that, The device includes: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, the program performing the method as claimed in any one of claims 1 to 7 when executed on a computer.

Images