An artificial intelligence image generation method and apparatus

Abstract

This application discloses an artificial intelligence image generation method, comprising: extracting text features from global text description information and text description information of the category to which a local target belongs, which are used to characterize the desired feature information of the artificial intelligence image to be generated, to obtain global text feature information and local text feature information respectively; the text description information of the category to which the local target belongs is used to characterize the desired category of the local target; using a trained first neural network, performing global attention learning on a layout image, which is used to characterize the desired layout of the artificial intelligence image to be generated, and the global text feature information to obtain global features; using a trained second neural network, performing local attention learning on the layout image, the local mask image of the local target, and the local text feature information to obtain local features; and fusing the global features and local features to obtain the artificial intelligence image. This application improves the ability to generate local details of artificial intelligence images.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular, to an artificial intelligence image generation method. Background Technology

[0002] In the field of industrial artificial intelligence, AI models require a large amount of rich and specific sample data for training, making the convenient expansion of sample data a key step.

[0003] Artificial intelligence-generated content technology, which uses various machine learning methods to learn the components of objects from data and then generates content, provides a possible way to obtain sample data.

[0004] Currently, AI image generation is usually guided by natural language descriptions used to describe the overall image. However, the AI ​​images generated by this method lack sufficient detail in local images, making it difficult to apply to sample data augmentation in the field of industrial AI. Summary of the Invention

[0005] This invention provides an artificial intelligence image generation method to improve the precision of local images in artificial intelligence images.

[0006] The first aspect of this invention provides an artificial intelligence image generation method, the method comprising:

[0007] Text feature extraction is performed on the global text description information and the text description information of the category to which at least one local target belongs, which are used to characterize the expected feature information of the AI ​​image to be generated. This yields global text feature information and local text feature information, respectively. The text description information of the category to which the local target belongs is used to characterize the expected category of the local target in the AI ​​image to be generated.

[0008] Using the trained first neural network, global attention is learned on the layout image and global text feature information to obtain global features. The layout image is used to represent the desired layout of the AI ​​image to be generated.

[0009] Using the trained second neural network, local attention learning is performed on the layout image, at least one frame of local mask image of the local target, and local text feature information to obtain local features. The local mask image is used to characterize the desired image position of the local target in the AI ​​image to be generated.

[0010] By fusing global and local features, an artificial intelligence image is obtained. This artificial intelligence image has the same layout as the desired features and layout image, and has local targets at the locations of the desired image that are consistent with the desired category.

[0011] As one possible implementation, the method further includes:

[0012] Feature extraction is performed on the layout image to obtain image features, which are then used as global image features.

[0013] As one possible implementation, the global attention learning is performed in the following manner:

[0014] Based on the first UNet neural network, attention learning is performed on global image features and global text features. The first downsampling layer performs global cross-attention calculation based on global image features and global text features. Each of the remaining sampling layers performs global cross-attention calculation based on the fused image features of the previous sampling layer adjacent to the sampling layer. The fused image features of each downsampling layer are input across layers to the upsampling layer that is symmetrical to the downsampling layer.

[0015] As one possible implementation, the local attention learning is performed in the following manner:

[0016] Based on the second UNet neural network, attention learning is performed on global image features, local mask images, and local text features. The first downsampling layer performs local cross-attention calculation based on global image features, local mask images, and local text feature information. Each of the remaining sampling layers performs local cross-attention calculation based on the fused image features of the previous sampling layer adjacent to the sampling layer. The fused image features of each downsampling layer are input across layers to the upsampling layer symmetrical to the downsampling layer.

[0017] in,

[0018] The first UNet neural network has the same neural network structure as the second UNet neural network.

[0019] As one possible implementation, the fused image features are obtained in the following manner:

[0020] For each sampling layer,

[0021] The global cross-attention result calculated by the sampling layer in the first UNet neural network is accumulated with the local cross-attention result calculated by the sampling layer in the second UNet neural network to obtain the fused image features represented by the accumulated cross-attention result.

[0022] As one possible implementation, the fusion of global and local features includes:

[0023] The global cross-attention result, which represents global features, output from the last sampled layer of the first UNet neural network, is accumulated with the local cross-attention result, which represents local features, output from the last sampled layer of the second UNet neural network, to obtain the artificial intelligence image represented by the accumulated attention result.

[0024] As one possible implementation, the global cross-attention calculation is performed as follows:

[0025] In any sampling layer,

[0026] According to the query vector mapping function, the image features input to this sampling layer are mapped to query vectors to obtain the query vector for this sampling layer.

[0027] By applying the key vector mapping function, key vectors are mapped onto the global text features to obtain the key vectors for this sampling layer.

[0028] The global text features are mapped using a value vector mapping function to obtain the value vector for that sampling layer.

[0029] Based on the query vector, key vector, and value vector of this sampling layer, global cross-attention is calculated according to the attention learning calculation formula to obtain the global features of this sampling layer.

[0030] As one possible implementation, the local cross-attention calculation is performed as follows:

[0031] In any sampling layer,

[0032] The local image features of the sampling layer are obtained by multiplying the image features input to the sampling layer with the pixel values ​​of the corresponding pixels in the local mask image.

[0033] According to the query vector mapping function, query vector mapping is performed on the local image features to obtain the query vector of the local image features of this sampling layer.

[0034] By applying the key vector mapping function, key vectors are mapped onto local text features to obtain the key vectors for that sampling layer.

[0035] By applying the value vector mapping function, value vectors are mapped onto local text features to obtain the value vector of that sampling layer.

[0036] Based on the query vector, key vector, and value vector of this sampling layer, local cross-attention calculation is performed according to the attention learning calculation formula to obtain the local features of this sampling layer.

[0037] As one possible implementation, the first UNet neural network and the second UNet neural network are trained in the following manner:

[0038] For any global image sample

[0039] Textual feature information is extracted from the text description information of the global image sample and the text description information of the category to which at least one local target in the global image sample belongs, respectively, to obtain global text sample information and local text sample information.

[0040] The global image sample, the layout image sample of the global image sample, and the global text sample information are input into the first UNet neural network to perform global attention learning and obtain global feature samples. The layout image sample includes images of the local targets in the global image sample.

[0041] The global image sample, the layout image sample, at least one frame of local mask image sample of the local target, and the local text sample are input into the second UNet neural network to perform local attention learning and obtain local feature samples, wherein each category corresponds to a local mask image sample.

[0042] The global feature samples output by the first UNet neural network and the local feature samples output by the second UNet neural network are fused to obtain artificial intelligence image samples and noisy global image samples.

[0043] Based on the loss function value between the noise in the AI ​​image samples and the noise in the noisy global image samples, the network parameters of the first UNet neural network and the second UNet neural network are adjusted.

[0044] Repeat the process until the desired result is achieved.

[0045] As one possible implementation, the step of inputting the global image sample, the layout image sample of the global image sample, and the global text sample information into the first UNet neural network includes:

[0046] Feature extraction is performed on the global image sample and its layout image sample to obtain the global image feature sample.

[0047] Global image feature samples and global text sample information are input into the first UNet neural network;

[0048] The step of inputting the global image sample, the layout image sample, at least one frame of local mask image sample of the local target, and the local text sample into the second UNet neural network includes:

[0049] The global image feature samples, at least one frame of local mask image samples, and local text samples are input into the second UNet neural network.

[0050] As one possible implementation, the global image samples are image samples from an autonomous driving training set, and these image samples correspond to label samples with three-dimensional spatial coordinates of local targets used to characterize driving constraints.

[0051] The layout image sample is obtained in the following manner:

[0052] For any image sample in the autonomous driving training set,

[0053] Based on the camera parameters used to acquire the image sample, the three-dimensional spatial coordinates of each label sample corresponding to the image sample are mapped to the two-dimensional image coordinates in the two-dimensional image plane to obtain the layout image sample.

[0054] The text description information of the global image samples is obtained in the following manner:

[0055] Image samples from the autonomous driving training set are input into a multimodal large language model to obtain textual description information including at least one of the following: weather, time of day, driving scenario, and signage.

[0056] As one possible implementation, the local mask image is obtained in the following manner:

[0057] For any category to which a local target in the layout image belongs,

[0058] Based on the image position of each local target included in the category in the layout image, the pixel value at the image position is set as the first pixel value, and the pixel values ​​at the other image positions are set as the second pixel value, thus obtaining the local mask image of the category, wherein the first pixel value and the second pixel value are not equal.

[0059] A second aspect of the present invention provides an artificial intelligence image generation apparatus, the apparatus comprising:

[0060] The text feature extraction module is used to extract text features from the global text description information and the text description information of the category to which at least one local target belongs, which are used to characterize the expected feature information of the AI ​​image to be generated. This yields global text feature information and local text feature information, where the text description information of the category to which the local target belongs is used to characterize the expected category of the local target in the AI ​​image to be generated.

[0061] The global attention learning module utilizes the trained first neural network to perform global attention learning on the layout image and global text feature information to obtain global features. The layout image is used to represent the desired layout of the AI ​​image to be generated.

[0062] The local attention learning module is used to perform local attention learning on the layout image, at least one frame of local mask image of the local target, and local text feature information using a trained second neural network to obtain local features. The local mask image is used to represent the desired image position of the local target in the AI ​​image to be generated.

[0063] The fusion module is used to fuse global and local features to obtain an artificial intelligence image. This artificial intelligence image has the same layout and desired features as the layout image and has local targets at the desired image location that are consistent with the desired category.

[0064] The artificial intelligence image generation method provided in this application not only uses global text description information and layout images as guiding labels, but also uses local target category text description information and local mask images as guiding labels. Furthermore, it fuses the global features obtained from global attention learning with the local features obtained from local attention learning layer by layer, thereby improving the ability to generate local details of artificial intelligence images. This method is beneficial for generating small-sized local targets and improves the fineness of local images in artificial intelligence images. Attached Figure Description

[0065] Figure 1 This is a schematic flowchart of an artificial intelligence image generation method according to an embodiment of this application.

[0066] Figure 2 This is a schematic diagram of a diffusion model according to an embodiment of this application.

[0067] Figure 3 This is a schematic diagram of the generation of a layout image of a frame of global image sample data in an embodiment of this application.

[0068] Figure 4 This is a schematic diagram illustrating the generation of tag data in an embodiment of this application.

[0069] Figure 5 This is a schematic diagram of the Unet network in the training diffusion model of an embodiment of this application.

[0070] Figure 6 This is a schematic diagram illustrating how the Unet network, i.e., the diffusion model, is used to generate driving images based on inference in this embodiment.

[0071] Figure 7 This is a schematic diagram of an artificial intelligence image generation device according to an embodiment of this application.

[0072] Figure 8 This is another schematic diagram of the artificial intelligence image generation device according to an embodiment of this application. Detailed Implementation

[0073] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0074] This application provides an artificial intelligence image generation method. It learns global features by global attention learning of the text features of the layout image and the global text description information used to characterize the desired features of the artificial intelligence image to be generated. It learns local features by local attention learning of the layout image, the local mask image of the local target in the layout image, and the text features of the text description information of the category to which the local target belongs. The global features and local features are then fused to generate an artificial intelligence image.

[0075] See Figure 1 As shown, Figure 1 This is a schematic flowchart of an artificial intelligence image generation method according to an embodiment of this application. The method includes:

[0076] Step 101: Extract text features from the global text description information used to characterize the desired feature information of the AI ​​image to be generated and the text description information used to characterize the category to which at least one local target in the AI ​​image to be generated belongs, to obtain global text feature information and local text feature information, respectively. The text description information of the category to which the local target belongs is used to characterize the desired category of the local target.

[0077] As an example,

[0078] Textual description information includes natural language description information, which can be obtained through external input or generated through a multimodal large language model. This application does not impose any restrictions on this.

[0079] There may be one or more local targets, and this application does not limit this; the category to which the local targets belong may be one or more categories. For example, different local targets may have different categories, or all local targets may have the same category, and this application does not limit this.

[0080] As an example, text feature extraction can be achieved using a text encoding model with a neural network structure.

[0081] Step 102: Using the trained first neural network, perform global attention learning on the layout image and global text feature information to obtain global features. The layout image is used to represent the desired layout of the AI ​​image to be generated.

[0082] As an example, the layout image includes annotation information or semantic segmentation information of the region of interest, such as bounding boxes.

[0083] As an example, the first neural network can be a network used for feature extraction and feature fusion in a diffusion model, a deep generative model that demonstrates powerful capabilities in generating, editing, and reconstructing content such as images, videos, and 3D scenes. Alternatively, it can be a network used for feature extraction and feature fusion in a conditional diffusion model, where additional conditional information, such as text descriptions or sketches, is added during the generation process to guide the diffusion model in generating images or videos of a specific style. This application does not impose any limitations on this approach.

[0084] As an example, feature extraction is performed on the layout image, for instance, using an image coding model to extract features and obtain image features as global image features, which helps improve inference efficiency and reduce computation.

[0085] As an example, global attention learning is performed in the following way:

[0086] Based on the first UNet neural network, attention learning is performed on global image features and global text features. The first downsampling layer performs global cross-attention calculation based on global image features and global text features. Each of the remaining sampling layers performs global cross-attention calculation based on the fused image features of the previous sampling layer adjacent to it. The fused image features of each downsampling layer are input across layers to the upsampling layer that is symmetrical to the downsampling layer.

[0087] As an example, global cross-attention is calculated as follows:

[0088] In any sampling layer,

[0089] According to the query vector mapping function, the image features input to this sampling layer are mapped to query vectors to obtain the query vector for this sampling layer.

[0090] By applying the key vector mapping function, key vectors are mapped onto the global text features to obtain the key vectors of the global text features for this sampling layer.

[0091] The global text features are mapped using a value vector mapping function to obtain the value vector for that sampling layer.

[0092] Based on the query vector, key vector, and value vector of this sampling layer, global cross-attention is calculated according to the attention learning calculation formula to obtain the global features of this sampling layer.

[0093] Step 103: Using the trained second neural network, perform local attention learning on the layout image, at least one frame of local mask image of the local target, and local text feature information to obtain local features. The local mask image is used to represent the desired image location of the local target in the AI ​​image to be generated.

[0094] As an example, the second neural network can be a network used for feature extraction and feature fusion in a diffusion model, or it can be a network used for feature extraction and feature fusion in a conditional diffusion model. This application does not impose any limitations on this.

[0095] The network structures of the first neural network and the second neural network can be the same or different, and the embodiments of this application do not impose any restrictions on this.

[0096] As an example, each category corresponds to a local mask image, which helps improve inference efficiency.

[0097] The local attention learning is performed in the following manner:

[0098] Based on the second UNet neural network, attention learning is performed on global image features, local mask images, and local text features. The first downsampling layer performs local cross-attention calculation based on global image features, local mask images, and local text feature information. Each of the remaining sampling layers performs local cross-attention calculation based on the fused image features of the previous sampling layer adjacent to the sampling layer. The fused image features of each downsampling layer are input across layers to the upsampling layer symmetrical to the downsampling layer.

[0099] The first UNet neural network has the same neural network structure as the second UNet neural network, which is beneficial for feature fusion.

[0100] The local attention cross-calculation is performed as follows:

[0101] In any sampling layer,

[0102] The local image features of the sampling layer are obtained by multiplying the image features input to the sampling layer with the pixel values ​​of the corresponding pixels in the local mask image.

[0103] By applying the query vector mapping function, query vectors are mapped to local image features to obtain the query vector for that sampling layer.

[0104] By applying the key vector mapping function, key vectors are mapped onto local text features to obtain the key vectors for that sampling layer.

[0105] By applying the value vector mapping function, value vectors are mapped onto local text features to obtain the value vector of that sampling layer.

[0106] Based on the query vector, key vector, and value vector of this sampling layer, local cross-attention calculation is performed according to the attention learning calculation formula to obtain the local features of this sampling layer.

[0107] The fused image features are obtained as follows:

[0108] For each sampling layer,

[0109] The global cross-attention result calculated by the sampling layer in the first UNet neural network is accumulated with the local cross-attention result calculated by the sampling layer in the second UNet neural network to obtain the fused image features represented by the accumulated cross-attention result.

[0110] Steps 102 and 103 have no strict order and can be executed in parallel.

[0111] Step 104: The global features and local features are fused to obtain an artificial intelligence image. This artificial intelligence image has the same layout and desired features as the layout image, and has a local target at the desired image location that is consistent with the desired category.

[0112] As an example, the global cross-attention result, which is used to represent global features, output from the last sampled layer of the first UNet neural network, and the local cross-attention result, which is used to represent local features, are accumulated to obtain the artificial intelligence image represented by the accumulated attention result.

[0113] This application provides an artificial intelligence image generation method that uses layout images, global text description information, local target mask images, and local target categories as guides. This allows the method to learn global feature information while also learning local features of local targets. By fusing the learned global and local features, the method not only improves the precision of local details in artificial intelligence images but also achieves diversity in artificial intelligence images.

[0114] To facilitate understanding of the embodiments of this application, the following description uses the generation of autonomous driving data as an example. It should be understood that this application is not limited to autonomous driving applications, and is applicable to other applications as well.

[0115] See Figure 2 As shown, Figure 2 This is a schematic diagram of a diffusion model according to an embodiment of this application, wherein the first neural network and the second neural network are neural networks used for feature extraction and feature fusion in the diffusion model.

[0116] The diffusion model in this embodiment includes: a first neural network for global attention learning, a second neural network for local attention learning, an image encoder for mapping the image space to a high-dimensional implicit feature space, and a text encoder for encoding global text description information and local category description information into the same space as the image features output by the image encoder.

[0117] in,

[0118] The global image features output by the image encoder are input into the first and second neural networks. The image encoder avoids directly inputting the image as a control condition into the neural network, which would significantly increase the computational load and training or inference time. This helps reduce the computational scale of the neural network. As an example, the image encoder can be a variational autoencoder (VAE) model.

[0119] The text encoder inputs its global text features, generated based on global text description information, into a first neural network, and its local text features, generated based on local category description information, into a second neural network. As an example, the text encoder can be a Contrastive Language-Image Pre-training (CLIP) model.

[0120] The first neural network and the second neural network have the same network structure. The second neural network also takes a local mask image as input. The global features output by the first neural network are fused with the local features output by the second neural network.

[0121] As an example, the first neural network is a first Unet network, and the second neural network is a second Unet network. In the Unet network, downsampling layers are used for feature extraction, and upsampling layers are used for feature fusion. In this embodiment, global image features are input to the first downsampling layer in the first Unet network and the first downsampling layer in the second Unet network. Global text features are input to the first downsampling layer in the first Unet network, and local text features and local mask images are input to the first downsampling layer in the second Unet network. The global attention learning result of the first downsampling layer in the first Unet network is fused with the local attention learning result of the first downsampling layer in the second Unet network, for example, by accumulating them. The fused attention learning result is then input to the next sampling layer adjacent to the first downsampling layer, and so on. That is, in the first Unet network, each sampling layer other than the first sampling layer performs global cross-attention calculation based on the fused attention learning result of the previous sampling layer adjacent to that sampling layer. Similarly, in the second Unet network, each sampling layer other than the first sampling layer performs local cross-attention calculation based on the fused attention learning result of the previous sampling layer adjacent to that sampling layer.

[0122] Furthermore, the fused image features of each downsampling layer are input across layers to an upsampling layer that is symmetrical to that downsampling layer. The local attention learning result of each sampling layer represents the local features of that sampling layer, the global attention learning result of each sampling layer represents the global features of that sampling layer, and the fused attention learning result of each sampling layer represents the fused image features of that sampling layer.

[0123] To train the diffusion model, data-label pairs consisting of global image sample data and their label information from the existing autonomous driving sample dataset are used to generate sample label data for training the diffusion model. The global image sample data and its label information correspond one-to-one, thus forming data-label pairs.

[0124] As an example, the label information includes the three-dimensional (3D) spatial coordinates of local targets in the global image samples used to characterize driving constraints. Local targets include, but are not limited to, obstacles and traffic signs, such as lane lines.

[0125] Based on the camera intrinsic and extrinsic parameters corresponding to the global image sample data, the spatial coordinate information of the local target is converted into a two-dimensional image on the image plane through coordinate mapping. This two-dimensional image includes the layout image corresponding to the local target and the image coordinate information of the layout image. This two-dimensional image is used as a layout image sample to characterize the layout information of the local target.

[0126] See Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the generation of a layout image from a frame of global image sample data. The frame of global image sample data and its label information form a data-label pair. Through coordinate mapping, the labels are mapped to two-dimensional image data to obtain the layout image sample, which may include lane lines, vehicles, etc.

[0127] For each category to which a local target belongs, the local targets of that category in the layout image sample are masked to obtain a local mask image sample corresponding to that category. For example, the image region where each obstacle (such as a car) of the same category is located in the same image sample is filled with the first pixel value, such as 1, and other regions are filled with the second pixel value, such as 0, to obtain a local mask image sample of the car type obstacle. In this way, if there are n categories of local targets in the image sample, there are n frames of local mask image samples, that is, each category corresponds to one frame of local mask image sample.

[0128] For each category of local targets in the global image sample data, generate a text description for that category, thus obtaining the text description information for that category.

[0129] Global image sample data is input into a multimodal large model inference. Through multimodal large model inference, text descriptions of the global image sample data are generated. The text descriptions include weather, time of day, driving scenario (urban roads, highways, mountains, rural areas, tunnels, etc.), and descriptions of indicators such as signs. The text description information of the global image sample data, local masked image samples of various categories, text descriptions of the categories to which local targets belong, and layout image samples are used as label data.

[0130] As an example, the multimodal large model can be the QWEN2-VL model, LLaVa, etc., and this embodiment does not limit it.

[0131] The above tag generation can be found in [link to relevant documentation]. Figure 4 As shown.

[0132] See Figure 5 As shown, Figure 5 This is a schematic diagram of the Unet network used in training the diffusion model according to an embodiment of this application. In the diagram, the blue boxes in the Unet network represent sampling layers in the first Unet network, the yellow boxes represent sampling layers in the second Unet network, the + symbol indicates that the outputs of the corresponding sampling layers are fused, and the cross-layer connections between symmetrical sampling layers in the same Unet network are not shown; each sampling layer performs cross-attention calculation.

[0133] For any global image sample

[0134] This global image sample I x and the global image sample I x The layout image sample is input to the image encoder. After mapping by the image encoder, image features are obtained as global image feature samples. In the image encoder, the input global image sample and layout image sample are concatenated by channel before feature extraction. The text description C of this global image sample is... global C. Textual description of the category to which the local target belongs local The inputs are fed into a text encoder, and after encoding by the text encoder model, global text feature sample information and σ(C) are obtained respectively. global Local text feature sample information σ(C) local ),

[0135] in,

[0136] Global text feature sample information σ(C) global The local text feature sample information σ(C) is input into the first Unet network. local ), and local masked image samples are input into the second Unet network;

[0137] All global image feature samples are input into the first Unet network and the second Unet network.

[0138] The first Unet network performs global attention learning on the input global text feature samples and the input global image feature samples. The global attention learning process is as follows:

[0139] In each sampling layer of the first Unet network,

[0140] The image feature samples ε input to the sampling layer are obtained by querying the vector mapping function Q. I l The query vector is mapped to obtain the query vector of the sampling layer. The image feature sample input by the first sampling layer is the global image feature sample, and the image features input by the other sampling layers are the fused image feature samples. The fused image feature sample is the fusion of the global feature sample output by the previous sampling layer adjacent to the sampling layer and the local feature sample output by the sampling layer corresponding to the previous sampling layer in the second Unet network. For example, the global features and local features at each pixel are accumulated.

[0141] The global text feature sample information σ(C) is obtained through the key vector mapping function K. global Key vector mapping is performed to obtain the global text feature sample information σ(C) of this sampling layer. global The key vector of )

[0142] The global text feature sample information σ(C) is obtained through the value vector mapping function V. global Value vector mapping is performed to obtain the global text feature sample information σ(C) of this sampling layer. global The value vector of )

[0143] Based on the query vector, key vector, and value vector of this sampling layer, global cross-attention is calculated to obtain the global feature of this sampling layer, which is represented by weights. The mathematical expression is as follows:

[0144]

[0145] Among them, f global Let T represent the global feature sample, T represent the transpose matrix, softmax represent the normalization function, and Q(ε( I l )) represents the query vector mapping function Q on the input image feature samples ε ( I l The query vector mapping performed by K(σ(C)) is the query vector; global )) represents the query vector mapping function K on global text feature sample information σ(C) globalThe key vector mapping performed by ) is the key vector; V(σ(C) global )) represents the value vector mapping function V on global text feature sample information σ(C) global The value vector mapping performed by ) is the value vector.

[0146] The second Unet network performs local attention learning on the input local text feature samples, local mask image samples, and input global image feature samples. The local attention learning process is as follows:

[0147] In each sampling layer of the second Unet network,

[0148] The local masked image sample is compared with the image feature sample ε input to the sampling layer. I l The pixel values ​​at corresponding pixel points are multiplied to obtain the local image feature samples of that sampling layer. The image feature samples input to the first sampling layer are global image feature samples, while the image feature samples input to the remaining sampling layers are fused image feature samples. These fused image feature samples are the fusion of the local feature samples output by the previous sampling layer adjacent to the current sampling layer and the global feature samples output by the sampling layer corresponding to the previous sampling layer in the first Unet network. In other words, they are the fused image feature samples input to the sampling layer corresponding to the previous sampling layer in the first Unet network.

[0149] The query vector for this sampling layer is obtained by mapping query vectors to local image feature samples using the query vector mapping function Q.

[0150] Local text feature sample information σ(C) is obtained through the key vector mapping function K. local Perform key vector mapping to obtain the key vector of this sampling layer.

[0151] The local text feature sample information σ(C) is obtained through the value vector mapping function V. local Perform value vector mapping to obtain the value vector of the sampling layer.

[0152] Based on the query vector, key vector, and value vector of this sampling layer, local cross-attention is calculated to obtain the local feature samples of this sampling layer, which can be expressed mathematically as follows:

[0153]

[0154] Among them, f local Let T represent the local feature sample, T represent the transpose matrix, and softmax represent the normalization function.

[0155] Q(ε) I l )· Im K(σ(C)) represents the query vector mapping performed by the query vector mapping function Q on local image feature samples, i.e., the query vector; local )) represents the query vector mapping function K on the local text feature sample information σ(C) local The key vector mapping performed by ) is the key vector; V(σ(C) local )) represents the value vector mapping function V on local text feature sample information σ(C) local The value vector mapping performed on d is the value vector. k The dimension of the query vector.

[0156] In the last sampling layer of the Unet network, the global feature samples output by the last sampling layer of the first Unet network are fused with the local feature samples output by the last sampling layer of the second Unet network to obtain artificial intelligence image samples and noisy global image samples. The artificial intelligence image samples and noisy global image samples are distinguished by channels.

[0157] Calculate the loss function value between the noise in the noisy global image sample and the noise in the AI ​​image sample, and adjust the model parameters of the Unet network based on the loss function value.

[0158] The training is repeated until the loss function value reaches its minimum, resulting in the trained Unet network, i.e., the diffusion model.

[0159] As an example, the loss function value can be calculated as follows:

[0160]

[0161] Where L is the value of the loss function. The noise in the global image samples is added to conform to a standard Gaussian distribution. This refers to noise in the AI ​​image samples, specifically the noise fitted by the Unet network.

[0162] This represents the 2nd normal form.

[0163] See Figure 6 As shown, Figure 6 This is a schematic diagram illustrating how the Unet network, i.e., the diffusion model, is used to generate driving images based on inference in this embodiment. In this embodiment, layout image samples of global image samples in the autonomous driving dataset can be used as the desired layout image to generate driving images through inference, thereby expanding the autonomous driving dataset.

[0164] Specifically, any layout image is input to the image encoder, global text description information and text description information of the category to which the local target belongs are input to the text encoder, and the local mask image of the local target is input to the second Unet network. Here, the layout image is used to represent the desired layout of the image to be generated, the global text description information is used to represent the desired feature information of the image to be generated, the text description information of the category to which the local target belongs is used to represent the desired category to which the local target belongs in the image to be generated, and the location of the mask region in the local mask image of the local target is used to represent the desired image location region of the local target in the image to be generated.

[0165] The image features output by the image encoder are input as global image features into the first Unet network and the second Unet network. The global text features output by the text encoder are input into the first Unet network, and the local text features output by the text encoder are input into the second Unet network. After inference by the first Unet network and the second Unet network, an artificial intelligence image is generated. This image has the desired layout and desired features, and there is a local target in the desired image location region that is consistent with the desired category.

[0166] It should be understood that, in this embodiment, the local mask image can be input to the second Unet network in parallel across multiple frames. When performing local attention calculation at the same sampling layer, the global image features are multiplied by each local mask image to obtain the local image features of each local mask image. Then, the query vector of each local image feature is calculated. Based on the query vector, key vector, and value vector of each local image feature, local cross-attention calculation is performed to obtain the local features of each local image feature.

[0167] The method for generating driving images in this embodiment can generate a wide variety of desired driving images, such as driving images with different weather conditions, time periods, driving scenarios, and traffic signs. It is particularly effective for generating small target obstacles such as pedestrians and non-motorized vehicles.

[0168] See Figure 7 As shown, Figure 7 This is a schematic diagram of an artificial intelligence image generation apparatus according to an embodiment of this application. The apparatus includes:

[0169] The text feature extraction module is used to extract text features from the global text description information used to characterize the desired feature information of the AI ​​image to be generated, and the text description information of the category to which at least one local target in the layout image belongs, to obtain global text feature information and local text feature information, respectively. The text description information of the category to which the local target belongs is used to characterize the desired category of the local target in the AI ​​image to be generated.

[0170] The global attention learning module utilizes the trained first neural network to perform global attention learning on the layout image and global text feature information to obtain global features. The layout image is used to represent the desired layout of the AI ​​image to be generated.

[0171] The local attention learning module is used to perform local attention learning on the layout image, at least one frame of local mask image of the local target, and local text feature information using a trained second neural network to obtain local features. The local mask image is used to represent the desired image position of the local target in the AI ​​image to be generated.

[0172] The fusion module is used to fuse global and local features to obtain an artificial intelligence image. This artificial intelligence image has the same layout and desired features as the layout image and has local targets at the desired image location that are consistent with the desired category.

[0173] See Figure 8 As shown, Figure 8 This is a schematic diagram of an artificial intelligence image generation apparatus according to an embodiment of this application. The apparatus includes a memory and a processor. The memory stores a computer program, and the processor is configured to execute the computer program to perform the steps of the artificial intelligence image generation method according to an embodiment of this application.

[0174] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0175] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0176] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the artificial intelligence image generation method of this application.

[0177] For the device / network-side equipment / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0178] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0179] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An artificial intelligence image generation method, characterized in that, The method includes: Text feature extraction is performed on the global text description information and the text description information of the category to which at least one local target belongs, which are used to characterize the expected feature information of the AI ​​image to be generated. This yields global text feature information and local text feature information, respectively. The text description information of the category to which the local target belongs is used to characterize the expected category of the local target in the AI ​​image to be generated. Using the trained first neural network, global attention is learned on the layout image and global text feature information to obtain global features. The layout image is used to represent the desired layout of the AI ​​image to be generated. Using the trained second neural network, local attention learning is performed on the layout image, at least one frame of local mask image of the local target, and local text feature information to obtain local features. The local mask image is used to characterize the desired image position of the local target in the AI ​​image to be generated. By fusing global and local features, an artificial intelligence image is obtained. This artificial intelligence image has the same layout as the desired features and layout image, and has local targets at the locations of the desired image that are consistent with the desired category.

2. The artificial intelligence image generation method according to claim 1, characterized in that, The method further includes: Feature extraction is performed on the layout image to obtain image features, which are then used as global image features. The global attention learning is performed in the following manner: Based on the first UNet neural network, attention learning is performed on global image features and global text features. The first downsampling layer performs global cross-attention calculation based on global image features and global text features. Each of the remaining sampling layers performs global cross-attention calculation based on the fused image features of the previous sampling layer adjacent to the sampling layer. The fused image features of each downsampling layer are input across layers to the upsampling layer that is symmetrical to the downsampling layer. The local attention learning is performed in the following manner: Based on the second UNet neural network, attention learning is performed on global image features, local mask images, and local text features. The first downsampling layer performs local cross-attention calculation based on global image features, local mask images, and local text feature information. Each of the remaining sampling layers performs local cross-attention calculation based on the fused image features of the previous sampling layer adjacent to the sampling layer. The fused image features of each downsampling layer are input across layers to the upsampling layer symmetrical to the downsampling layer. in, The first UNet neural network has the same neural network structure as the second UNet neural network.

3. The artificial intelligence image generation method according to claim 2, characterized in that, The fused image features are obtained in the following manner: For each sampling layer, The global cross-attention result calculated by the sampling layer in the first UNet neural network is accumulated with the local cross-attention result calculated by the sampling layer in the second UNet neural network to obtain the fused image features represented by the accumulated cross-attention result. The fusion of global and local features includes: The global cross-attention result, which represents global features, output from the last sampled layer of the first UNet neural network, is accumulated with the local cross-attention result, which represents local features, output from the last sampled layer of the second UNet neural network, to obtain the artificial intelligence image represented by the accumulated attention result.

4. The artificial intelligence image generation method according to claim 2, characterized in that, The global cross-attention calculation is performed as follows: In any sampling layer, According to the query vector mapping function, the image features input to this sampling layer are mapped to query vectors to obtain the query vector for this sampling layer. By applying the key vector mapping function, key vectors are mapped onto the global text features to obtain the key vectors for this sampling layer. The global text features are mapped using a value vector mapping function to obtain the value vector for that sampling layer. Based on the query vector, key vector, and value vector of this sampling layer, global cross-attention is calculated according to the attention learning calculation formula to obtain the global features of this sampling layer.

5. The artificial intelligence image generation method according to claim 2, characterized in that, The local cross-attention calculation is performed as follows: In any sampling layer, The local image features of the sampling layer are obtained by multiplying the image features input to the sampling layer with the pixel values ​​of the corresponding pixels in the local mask image. According to the query vector mapping function, query vector mapping is performed on the local image features to obtain the query vector of the local image features of this sampling layer. By applying the key vector mapping function, key vectors are mapped onto local text features to obtain the key vectors for that sampling layer. By applying the value vector mapping function, value vectors are mapped onto local text features to obtain the value vector of that sampling layer. Based on the query vector, key vector, and value vector of this sampling layer, local cross-attention calculation is performed according to the attention learning calculation formula to obtain the local features of this sampling layer.

6. The artificial intelligence image generation method according to claim 2, characterized in that, The first UNet neural network and the second UNet neural network are trained in the following manner: For any global image sample Textual feature information is extracted from the text description information of the global image sample and the text description information of the category to which at least one local target in the global image sample belongs, respectively, to obtain global text sample information and local text sample information. The global image sample, the layout image sample of the global image sample, and the global text sample information are input into the first UNet neural network to perform global attention learning and obtain global feature samples. The layout image sample includes images of the local targets in the global image sample. The global image sample, the layout image sample, at least one frame of local mask image sample of the local target, and the local text sample are input into the second UNet neural network to perform local attention learning and obtain local feature samples, wherein each category corresponds to a local mask image sample. The global feature samples output by the first UNet neural network and the local feature samples output by the second UNet neural network are fused to obtain artificial intelligence image samples and noisy global image samples. Based on the loss function value between the noise in the AI ​​image samples and the noise in the noisy global image samples, the network parameters of the first UNet neural network and the second UNet neural network are adjusted. Repeat the process until the desired result is achieved.

7. The artificial intelligence image generation method according to claim 6, characterized in that, The step of inputting the global image sample, the layout image sample of the global image sample, and the global text sample information into the first UNet neural network includes: Feature extraction is performed on global image samples and their layout image samples to obtain global image feature samples. Global image feature samples and global text sample information are input into the first UNet neural network; The step of inputting the global image sample, the layout image sample, at least one frame of local mask image sample of the local target, and the local text sample into the second UNet neural network includes: The global image feature samples, at least one frame of local mask image samples, and local text samples are input into the second UNet neural network.

8. The artificial intelligence image generation method according to claim 6 or 7, characterized in that, The global image samples are image samples from the autonomous driving training set. Each image sample corresponds to a label sample with the three-dimensional spatial coordinates of local targets used to characterize driving constraints. The layout image sample is obtained in the following manner: For any image sample in the autonomous driving training set, Based on the camera parameters used to acquire the image sample, the three-dimensional spatial coordinates of each label sample corresponding to the image sample are mapped to the two-dimensional image coordinates in the two-dimensional image plane to obtain the layout image sample. The text description information of the global image samples is obtained in the following manner: Image samples from the autonomous driving training set are input into a multimodal large language model to obtain textual description information including at least one of the following: weather, time of day, driving scenario, and signage.

9. The artificial intelligence image generation method according to any one of claims 1 to 7, characterized in that, The local mask image is obtained in the following manner: For any category to which a local target in the layout image belongs, Based on the image position of each local target included in the category in the layout image, the pixel value at the image position is set as the first pixel value, and the pixel values ​​at the other image positions are set as the second pixel value, thus obtaining the local mask image of the category, wherein the first pixel value and the second pixel value are not equal.

10. An artificial intelligence image generation device, characterized in that, The device includes: The text feature extraction module is used to extract text features from the global text description information and the text description information of the category to which at least one local target belongs, which are used to characterize the expected feature information of the AI ​​image to be generated. This yields global text feature information and local text feature information, where the text description information of the category to which the local target belongs is used to characterize the expected category of the local target in the AI ​​image to be generated. The global attention learning module utilizes the trained first neural network to perform global attention learning on the layout image and global text feature information to obtain global features. The layout image is used to represent the desired layout of the AI ​​image to be generated. The local attention learning module is used to perform local attention learning on the layout image, at least one frame of local mask image of the local target, and local text feature information using a trained second neural network to obtain local features. The local mask image is used to represent the desired image position of the local target in the AI ​​image to be generated. The fusion module is used to fuse global and local features to obtain an artificial intelligence image. This artificial intelligence image has the same layout and desired features as the layout image and has local targets at the desired image location that are consistent with the desired category.

Images