Method, electronic device and computer program product for generating image

Through the spatially perceived multimodal model framework, the target image is generated using description text and position prior information, which solves the problem of image generation in complex environments, realizes image sample generation in fields such as autonomous driving, and improves the accuracy of image generation and diversity.

CN119068080APending Publication Date: 2024-12-03DELL PROD LP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310646119.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-01
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing technology is difficult to generate image samples of real-world scenarios in complex and changeable environments, especially in the fields of autonomous driving and medical care. Deep learning models require a robust and accurate understanding of the three-dimensional world, and existing image generation methods cannot effectively simulate extremes. Scenario changes under conditions.

Method used

By obtaining the description text describing the target image and determining the position prior information, a spatially perceived multimodal model framework is used, including the position prior module, text encoding module, segmentation diagram encoding module, multimodal fusion module, grid deformation module and The image decoding module generates a target image, controls the direction and position of the object in the image, and realizes image generation under extreme conditions.

Benefits of technology

It realizes the generation of diverse and accurate image samples without manual collection, which can be used for rare data simulation and the reliability improvement of autonomous vehicles, enhances the accuracy and diversity of image generation, and is suitable for autonomous driving. and other applications that require complex scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068080A_ABST
    Figure CN119068080A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a method for generating an image, electronic equipment and a computer program product. The method includes acquiring a description text for describing image content of a target image, determining position priori information, and generating the target image based on the description text and the position priori information. According to the method disclosed by the embodiment of the invention, the target image which can be used for rare data simulation under a rare condition can be generated through the input description text. Furthermore, the method allows position-aware editing and operation, and the direction and position of the object generated in the target image can be controlled based on the position priori information, so that the image can be generated efficiently and diverse. Moreover, the method provided by the invention is based on the object type in each subdivided image block, so that the position perception is more accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method for generating an image, an electronic device, and a computer program product. Background Art

[0002] In order to provide AI models with unlimited training samples for scene simulation, various technologies are currently used to automatically create images under many different conditions without the need for manual collection and annotation of data. For example, when training AI models in the fields of security, autonomous driving, and medical care, in order to improve their effectiveness, it is necessary to provide them with a large number of training samples through image generation technology.

[0003] For example, for autonomous driving, it fully utilizes the achievements in computer vision and deep learning to simulate the same driving behavior as humans. More specifically, to build a fully autonomous vehicle, it is important to have a powerful and effective model to understand the three-dimensional world. However, these challenges are caused by the complex and changing environment around the world. For the same street, the scene changes every year, every season, every day, and every minute. The scenes captured by the camera are huge, and it is a big challenge for deep learning models to have a robust and accurate estimate. Summary of the invention

[0004] According to an exemplary embodiment of the present disclosure, a technical solution for generating an image is provided, which can control the direction and position of an object generated in a target image based on position prior information, and can generate images efficiently and diversely.

[0005] In a first aspect of the present disclosure, a method is provided, which may include obtaining a description text for describing the image content of a target image, determining position prior information, the position prior information indicating the type of object in each of a plurality of image blocks of the target image, the position prior information being used to locate the position of the object in the image content in the target image, and generating the target image based on the description text and the position prior information. The method provided in the first aspect can be implemented to control the direction and position of the object generated in the target image based on the position prior information, and can generate images efficiently and diversely.

[0006] In the second aspect of the present disclosure, an electronic device is provided. The electronic device includes: a processor, and a memory coupled to the processor, the memory having instructions stored therein, and the instructions, when executed by the electronic device, cause the electronic device to perform operations, including obtaining a description text for describing the image content of a target image, determining position prior information, the position prior information indicating the type of object in each of a plurality of image blocks of the target image, the position prior information being used to locate the position of the object in the image content in the target image, and generating the target image based on the description text and the position prior information. The electronic device provided by the second aspect can control the direction and position of the object generated in the target image based on the position prior information, and can generate images efficiently and diversely.

[0007] In a third aspect of the present disclosure, there is provided a computer program product, which is tangibly stored on a computer-readable medium and comprises computer-executable instructions which, when executed, cause a computer to perform the method according to the first aspect of the present disclosure.

[0008] From the above description, it can be seen that according to the solutions of the various embodiments of the present disclosure, the direction and position of the object generated in the target image can be controlled based on the position prior information, and the image can be generated effectively and diversely. It should be understood that the invention summary is provided to introduce the selection of concepts in a simplified form, which will be further described in the specific embodiments below. The invention summary is not intended to identify the key features or main features of the present disclosure, nor is it intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing a model framework of a spatial perception multimodal model according to some embodiments of the present disclosure is shown;

[0011] Figure 2 Some embodiments of the present disclosure are shown Figure 1 An example architecture diagram of the model framework in ;

[0012] Figure 3 A schematic diagram showing a form of a position vector according to some embodiments of the present disclosure;

[0013] Figure 4 A schematic diagram showing a target image generated after a location description word is input in a description text according to some embodiments of the present disclosure;

[0014] Figure 5 A schematic diagram showing a target image according to some embodiments of the present disclosure is shown;

[0015] Figure 6 Another schematic diagram of a target image according to an embodiment of the present disclosure is shown;

[0016] Figure 7 A schematic diagram showing an example model that can be combined with a spatial perception multimodal model to generate a target image according to an embodiment of the present disclosure;

[0017] Figure 8 A flowchart of a method for training a spatial perception multimodal model according to an embodiment of the present disclosure is shown;

[0018] Fig. 9 A flowchart showing a method for generating an image according to an embodiment of the present disclosure; and

[0019] Fig.10 A schematic structural diagram of a device that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0022] When training an artificial intelligence model, it is very difficult to have a synthetic scene that is close to the real world. Most work focuses on scene synthesis guided by semantic segmentation. The idea is to have a pixel-level classifier that can identify the boundaries of objects on the road, for example, for autonomous driving scenarios. For medical scenarios, it can identify organs and / or tissues inside and / or outside the human body, but its disadvantage is that it cannot simulate changes caused by physical factors. It should be noted that the scene application of image generation in this specification is only for illustrative purposes and should not be understood as a limitation of the present disclosure. For example, when the autonomous driving scenario is used as an example, the present disclosure should not be understood as being limited to the autonomous driving scenario.

[0023] The present disclosure proposes a method for generating an image, an electronic device, and a computer program product. The method includes obtaining a description text for describing the image content of a target image, determining position prior information, the position prior information indicating the type of object in each of a plurality of image blocks of the target image, the position prior information being used to locate the position of the object in the image content in the target image, and generating the target image based on the description text and the position prior information. The method proposed in the present disclosure can generate a target image under rare circumstances (such as extreme weather) by inputting a description text, so the target image can be used for rare data simulation, for example, it can be used to make autonomous vehicles more reliable under real conditions. In some embodiments of the present disclosure, the method proposed in the present disclosure allows location-aware editing and operation, and can control the direction and position of the object generated in the target image based on the position prior information, so that images can be generated efficiently and diversely. And the method proposed in the present disclosure is based on the type of object in each subdivided image block, so the location perception is more accurate.

[0024] Figure 1A schematic diagram of an example model framework 100 of a spatially aware multimodal model according to some embodiments of the present disclosure is shown. The method proposed in the present disclosure can be implemented based on the model framework 100. In some embodiments, the model framework 100 includes a position prior module 102, a text encoding module 104, a segmentation map encoding module 106, a multimodal fusion module 108, a grid deformation module 110, and an image decoding module 112. The position prior module 102, the text encoding module 104, and the segmentation map encoding module 106 are respectively used to encode input information, and the specific input information of each module is described below. The multimodal fusion module 108 is used to perform multimodal fusion on the outputs of the text encoding module 104 and the segmentation map encoding module 106 to align the outputs of the text encoding module 104 and the segmentation map encoding module 106, that is, map them to the same feature space (e.g., the feature space of the target image), and the grid deformation module 110 is used to sample and deform the output of the position prior module 102 to obtain a deformed grid, which is used to offset the multimodal fusion feature to control the position of the object. The image decoding module 112 is used to decode the offset multimodal fusion features to generate a target image. Figure 1 The model framework 100 is shown in FIG. 1 , but in some embodiments, some modules may selectively not work based on different inputs. For example, when the segmentation map encoding module 106 is not input, the segmentation map encoding module 106 may not work. It is understandable that at this time, the multimodal fusion module 108 only maps the text encoding module 104 to the above feature space. Figure 2 The functions of the modules in the model framework 100 are described in detail. Figure 2 is an example architecture diagram of the model framework 100.

[0025] First, a description text 114 for describing the image content of the target image may be input to the text encoding module 104. For example, if the desired scene is: a winter street in an image that is entirely white, with vehicles on the road and pine trees on the roadside, the description text 114 “Winter street, a white scene, a car on the road, pine trees on the roadsides” may be input. After receiving the description text 114, the text encoding module 104 may extract text features of the description text 114 and input them to the multimodal fusion module 108. The multimodal fusion module 108 maps the text features to the feature space of the target image for subsequent use. Figure 2For example, the text encoding module 202 may include multiple layers (not all of which are shown) for encoding the description text 204, and may also process the encoded features through a series of one-dimensional convolutional layers in the multiple layers to obtain one-dimensional text features, and input the text features into the multimodal fusion module 206. After the multimodal fusion module 206 performs feature space mapping and size adjustment on the text features, the adjusted features are input into the image decoding module 112. For example, after performing feature space mapping, since the obtained features are one-dimensional features, the multimodal fusion module 206 may convert the mapped features into two-dimensional features (for example, converting 1024×1 one-dimensional features into 32×32 two-dimensional features) for subsequent image generation.

[0026] In order to locate the direction and position of the object in the image content generated in the target image, the position prior information, or a part of the position prior information, can also be input to the position prior module 102. The input position prior information can at least indicate the type of object in each of the multiple image blocks of the target image, and the object type can correspond to the object type of the object in the image content. For example, in the above example, the position prior information can include which image blocks / blocks of the multiple image blocks the street, road, vehicle, and pine tree exist in respectively.

[0027] In some examples, the position prior information may include a position vector 116, thereby representing the type of object in each of the plurality of image blocks of the target image in the form of a vector. In this case, the position vector 116 may be input into the position prior module 102. Figure 3 , Figure 3 A form of position vector is shown. For the target image, it can be pre-specified that it is divided into N image blocks 302, 304... (not all are numbered). Assuming that there are a first number of objects in the image content, including vehicles, trees, roads...grass, then for each object, a corresponding one-dimensional vector 306, 308... (hereinafter referred to as a first vector for the convenience of explanation) can be constructed first. Each constructed first vector is labeled with the data recorded therein to indicate which object the first vector corresponds to. For example, the first number of objects can be numbered starting from 1 in the order of vehicles, trees, roads...grass, and the data "1" is marked at the position of the corresponding object number in each first vector, such as Figure 3 As shown in , since the vehicle is numbered 1, the first position of the first vector 306 is marked with data "1", and the rest are marked with data "0". Since the tree is numbered 2, the second position of the first vector 308 is marked with data "1", and the rest are marked with data "0", and so on. It can be understood that the constructed first vector has a first number of , and the value of the first size of the first vector is equal to the first number. If Figure 3There are 16 kinds of objects in total, so the first dimension of the first vector is also 16, that is, the first vector is a one-dimensional vector of 16×1.

[0028] Then, for each object, a corresponding one-dimensional vector 310, 312, ... (hereinafter referred to as a second vector for convenience of explanation) of a second size is constructed. Each constructed second vector is used to mark in which image block(s) the corresponding object type appears through the data recorded therein. For example, each image block may be numbered starting from 1 in a sequence from top to bottom and from left to right, and in each second vector, data "1" is marked at the position of the image block number including the corresponding object type, and data "0" is marked at the rest, such as Figure 3 As shown in , if the vehicle is expected to appear in the 6th, 7th, and 8th image blocks of the target image, "1" can be marked at the 6th, 7th, and 8th positions of the second vector 310, and "0" can be marked at the rest. It can be understood that the value of the second size is equal to the total value of the number of image blocks. For example, for Figure 3 Since there are N image blocks in total, the second size is also N, that is, the second vector is a one-dimensional vector of N×1. It can be understood that the above-mentioned size, quantity, annotation data, etc. should not be understood as a limitation to the present disclosure.

[0029] After obtaining the first vector (assuming N) and the second vector (assuming 16), they can be combined into a vector of N×(N+16). For example, for each object, its corresponding second vector can be spliced ​​after the first vector to obtain a third vector of (N+16)×1. It can be understood that the third vector can be used to indicate whether a specific type of object exists in each of the multiple image blocks. Then, the third vector of each object is spliced ​​to obtain an N×(N+16) position vector. Operations such as combining and labeling position vectors can be performed by a person who wants to generate a target image. For example, a person who wants to generate a target image can generate a position vector with one click after making information related to the position vector in a table.

[0030] The position prior module 102 performs convolution processing on the input position vector 116 to obtain a deformed grid, and inputs the deformed grid into the grid deformation module 110, and the grid deformation module 110 determines how to control the direction and position of the object in the target image. Figure 2 The two-dimensional convolutional network in the position prior module 208 performs convolution processing on the position vector 210 to obtain a deformed grid, and then inputs the deformed grid into the grid deformation module 212. The deformed grid is upsampled by the grid deformation module 212 to determine feature offset information 214 corresponding to the position vector for controlling the direction and position of the object in the target image. In some examples, upsampling can be performed using formula (1).

[0031] in where G is the bicubic interpolation operator which is implemented by using the deformed mesh u i and v j The nearest 8 pixels are defined to calculate the center pixel, u i refers to the offset on the x-axis, v j refers to the displacement on the y-axis, F d Refers to the calculated feature offset information, f d Refers to the value of the pixel in the feature offset information.

[0032] Based on the feature offset information, the features of the input image decoding module 112 can be spatially sampled to offset the features and control the direction and position of the object in the target image. For example, when the feature offset information is obtained by formula (1), since each pixel in the feature offset information records the offset value (x-axis and y-axis), each pixel in the feature can be offset based on the offset value.

[0033] In some examples, the image decoding module 112 gradually samples the offset features through the deconvolution layer to obtain the final generated image. Figure 2 In the example, the image decoding module 208 can perform two upsamplings on the features. In this case, the deformed grid can be upsampled three times at different scales to obtain feature offset information 2141, 2142, and 2143 of different sizes. The sizes of the feature offset information 2141, 2142, and 2143 are respectively the same as the sizes of the features processed by the deconvolution layers 2161, 2162, and 2163, and the features can be offset at the same scale. Through multiple offsets, it is possible to avoid inaccurate control and offset of objects with a small image area when the features only include rough image information when not upsampled.

[0034] In some embodiments, when the description text and the position vector are input, the segmentation map 118 can also be input to the segmentation map encoding module 106 at the same time to describe the position of the object in a visual way, and use cross-modal fusion to combine all different signals as output. At this time, the position prior information also includes the information in the segmentation map 118. For example, a person who hopes to generate a target image can obtain an initial image that includes at least a portion of the image content by means of shooting, making, etc., and then process the initial image to obtain a segmentation map. Then, the segmentation map is input to the segmentation map encoding module 106, and the image segmentation features in the segmentation map are extracted by the segmentation map encoding module 106. The extracted image segmentation features are spatial features that represent the position of the above-mentioned at least a portion of the image content in the target image. In some examples, it can be achieved by Figure 2The segmentation map encoding module 218 shown in the figure extracts two-dimensional image segmentation features. The segmentation map encoding module 218 may include at least a plurality of convolutional layers 2181 , 2182 , and 2183 , and has an asymmetric structure with the image decoding module 208 .

[0035] After obtaining the image segmentation features, the segmentation map encoding module 106 inputs the image segmentation features to the multimodal fusion module 108. The multimodal fusion module 108 maps the text features and the image segmentation features to the same feature space for multimodal fusion to generate multimodal fusion features 220. In some examples, the image segmentation features can be obtained by Figure 2 The multimodal fusion module 206 shown in FIG performs multimodal fusion. The multimodal fusion module 206 processes the text features and the image segmentation features based on the self-attention mechanism. In order to align the text features and the image segmentation features, the average pooling operator can be used to compress the text features and the image segmentation features into a one-dimensional vector, and then regularize it to obtain F T and F I , where F T refers to the processed text features, F I Refers to the processed image segmentation features. Through this method, images and texts can be constrained to be in the same distribution for feature fusion.

[0036] Then, the weight W is determined through the self-attention mechanism. For details, please refer to formula (2) and formula (3).

[0037] F IT =conv(concateF I ,F T ) Formula (2)

[0038]

[0039] Among them, concate refers to the concatenation operation, conv refers to the convolution operation, Q FIT , K FIT and V FIT They refer to the query, key, and value of the concatenated features, respectively, and S refers to the softmax function. I and F T , as shown in formula (4).

[0040] F con =W·F I +(1-W)·F T Formula (4)

[0041] Where W is a confidence score between 0 and 1, which is used to represent the importance between text features and image segmentation features, and F conIt is the output feature, which can be converted into a two-dimensional feature (for example, a 1024×1 one-dimensional feature is converted into a 32×32 two-dimensional feature) for subsequent image generation. It can be understood that if only a segmentation map or a description text is input to the model, the W of the input information is 1.

[0042] Since images are visual aids that describe the location and texture of scene objects, and text is a high-level abstraction that contains human knowledge of traffic scenes, combining these two types of information can encourage the model to find the collinearity between text and images. By combining different signals, the generation of target images can be better supervised, and the collinearity between text and images can enhance feature representation for better image generation.

[0043] In some embodiments, in order to improve the accuracy of image generation and avoid feature generalization during feature fusion, a short connection can be constructed between the segmentation map encoding module 218 and the image decoding module 208 to share image segmentation features between the segmentation map encoding module 218 and the image decoding module 208, such as Figure 2 As shown, short connections can be constructed between 2181 and 2163, 2182 and 2162, and 2183 and 2161 of the same size.

[0044] In addition to supervising the location of objects in the target image through position vectors and segmentation maps, position descriptors can also be input into the description text. Figure 4 , Figure 4 The figure shows the generated target image after the position description word is entered in the description text. Figure 1 The example model framework of the spatial perception multimodal model in the text generates a target image. When the text encoding module 104 extracts the text features, it also extracts the features of the position descriptor, and thus can control the position of the object in the target image. It can be seen that by adding position descriptors to the description text, the position of the object in the target image can be further effectively supervised. Figure 5 , Figure 5 : The target images 508 and 510 generated by the method proposed in the present invention after simultaneously inputting description texts 502 and 504, a segmentation map 506 and a position vector (not shown), as well as the image 512 generated by the traditional method are shown. It can be seen that the method proposed in the present invention can use the description text as a condition to generate more specific scene results, while the traditional method can only generate one scene. Figure 6 Another example of the target image generated by the present disclosure is shown. It can be seen that through the segmentation map, description text and position vector (not shown), the position of the object in the target image can be accurately controlled to obtain the desired target image.

[0045] refer to Figure 7 , Figure 7Some examples that can be used with Figure 1 The example model combined with the example model framework of the spatial perception multimodal model in the example model is used to efficiently generate a target image. In some embodiments, Figure 7 The parameters in the example model in the spatial perception multimodal model can be used as parameters of certain modules. For example, the parameters used to decode and generate images in the example model can be used as some parameters in the image decoding module 112 / 216. Figure 7 The example model shown in is used to extract latent features that summarize the key attributes of the target dataset, and then latent optimization is used to extract new data from the latent features. According to the latent representation, the methods can be divided into explicit generative models and implicit generative models. For explicit generative models, there are three main categories: 1. Variational Autoencoder (VAE); 2. Flow; 3. Diffusion model. For implicit generative models, the most popular one is Generative Adversarial Network (GAN).

[0046] GAN uses a generator and a discriminator to compete with each other to generate fake samples. The discriminator works as a classifier to supervise the data generated by the generator. The competition stops when the discriminator gives a 50% judgment on whether the fake sample is real or fake. One of the representative methods is pix2pixHD. It consists of a two-stage GAN and learns image generation from coarse to fine.

[0047] VAE is well known for its dimensionality reduction. VAE learns a multivariate Gaussian representation of an image. The learned mean u and variance σ are used to sample the latent vector. The overall optimization minimizes the KL divergence between the posteriors of the real data and the generated data. VQ-VAE and VQ-VAE-2 are two representative methods that use VAE to learn data distribution to generate high-quality images.

[0048] On the other hand, Flow learns the probability density function of real data. Through a series of transformations, the variables are repeatedly replaced with new variables according to the variable change theorem, and finally the distribution of the final target variable is obtained. The disadvantage of Flow is that it has a large amount of calculations, which makes it easy to degenerate. Glow is one of the effective methods that can handle large image compression and generation. The idea is to use a reversible 11 convolutional layer to simplify the model.

[0049] Diffusion models have recently attracted attention due to their flexible and effective optimization. Diffusion models extend the idea of ​​the reparameterization trick in one-dimensional latent space to the two-dimensional feature domain. To reduce the complexity, a Markov chain can be established to generate the forward (backward) diffusion process of samples by slowly adding (removing) noise. The whole process iteratively approximates the conditional probability of the real sample given the noise.

[0050] The spatial perception multimodal model provided by the present disclosure or combined Figure 7The spatial perception multimodal model following the example model in can customize the position of the object in the target image through the position prior information, and at the same time, by inputting the description text, freely control the image content of the target image. On the one hand, it can accept text prompts from humans to generate conditional images, thereby controlling the direction of data generation to enhance and improve the model. Taking the autonomous driving scene as an example, in the real world, due to frequent road construction and different traffic conditions, the driving scenes vary greatly, so the spatial editing provided by the present disclosure is a huge advantage for autonomous driving scene generation. On the other hand, target images including rare image content can be automatically generated, increasing the number of target images that can be generated, thereby avoiding the memory and computing power limitations of most edge devices, and efficiently training the autonomous driving system to establish a channel for continuous learning and updating. For example, the spatial perception multimodal model provided by the present disclosure can be connected to the autonomous driving system.

[0051] The present disclosure also provides a method for training a spatial perception multimodal model. Figure 8 , Figure 8 A flow chart of the method is shown in . At box 800, a sample data set is obtained, and the sample data set includes sample description text, position prior information corresponding to the description text, and a real image. In some embodiments, the sample data set may also include a sample segmentation map. At box 802, based on the sample description text and the position prior information, a sample target image is output through a predetermined image generation model. The predetermined image generation model may refer to a spatial perception multimodal model that has not been trained or to a spatial perception multimodal model in training. The specific processing flow can refer to the above. In some embodiments, the sample segmentation map can also be simultaneously input into a predetermined image generation model to output a sample target image. The specific processing flow can refer to the above.

[0052] At box 804, a predetermined image generation model is trained based on the similarity labels between the real image and the sample target image, the similarity labels between the sample description text and the sample target image, the sample dataset (e.g., the real image and the sample description text therein), and the sample target image through contrast loss, adversarial loss, and average error loss.

[0053] To train a predetermined image generation model, contrastive loss can be used to align image and text features, adversarial loss can be used to supervise the visual quality of the generated sample target images, and L1 loss can be used to calculate the pixel difference between the generated sample target images and the real images.

[0054] The formula of contrast loss can refer to formula (5):

[0055]

[0056] Where Lcons is the contrast loss, F(T) is the feature of the sample description text, and F(I' r ) is the image feature of the sample target image, F(I r ) is the image feature of the real image, cos refers to the calculation of cosine similarity, T is the transpose, P IT and P II They refer to the similarity labels between the sample description text and the sample target image, and the similarity labels between the sample description text and the sample target image, respectively. They are all values ​​between 0 and 1, which are used to indicate the degree of similarity between the sample description text and the sample target image, and between the sample description text and the sample target image. S indicates the softmax function, and the other parameters are predetermined parameters. Formula (4) encourages the generated sample target image to be consistent with the real image and the sample description text.

[0057] For adversarial loss L adv , a discriminator is used to identify whether the generated sample target image is visually similar to the real image. The discriminator is pre-trained on an open source dataset. The adversarial loss can encourage the generation of sample target images that are close to real-world scenes (e.g., traffic scenes). For the L1 loss L mae , which calculates the mean absolute error between the sample target image and the true image to minimize the distortion.

[0058] During training, L can be calculated cons , L adv and L mae The weighted sum of is taken as the total loss, that is, formula (6).

[0059] L=αL cons +βL adv +ωL mae Formula (6)

[0060] The weights are weighting factors used to balance loss terms of the same numerical scale.

[0061] In some embodiments, the spatially aware multimodal model can also be trained using the structural similarity (SSIM) loss. The SSIM loss calculates the pixel-level difference between the sample target image and the separated image, i.e., L ssim , which focuses on the edges of objects and ensures that the sample target image has an ideal scene structure. At this point, L can be calculated cons , L adv , L ssim and L mae The weighted sum of is taken as the total loss, that is, formula (7).

[0062] L=αL cons +βL adv +γL ssim+ωL mae Formula (7)

[0063] The weights are weighting factors used to balance loss terms of the same numerical scale.

[0064] At block 806, a trained image generation model is determined. Through training, parameters in the model may be adjusted. For example, during training Figure 2 When the spatial perception multimodal model is constructed, the parameters in the text encoding module 202, the multimodal fusion module 206, the position prior module 208, the image encoding module 208, the segmentation map encoding module 218 and other parts can be continuously adjusted until a trained image generation model that meets the termination conditions is obtained.

[0065] The present disclosure also provides a method for generating an image. Fig. 9 , Fig. 9 The flowchart of the method is shown in FIG. At block 900, a description text is obtained for describing the image content of the target image. The description text may be input by a user or may be automatically generated by a machine, and the present disclosure is not limited thereto. The image content included in the description text may include information such as objects in the target image, the overall image style, season, weather, etc. In addition, in some embodiments, the image content described in the description text may include not only objects that should be in the target image, but also location supervision words for the objects, such as reference words. Figure 4 .

[0066] At block 902, position prior information is determined, the position prior information including the type of object in each of a plurality of image blocks of a target image, for locating the position of an object in the image content in the target image. In some embodiments, the position prior information may include a position vector, and an exemplary description of the position vector may refer to the above and Figure 3 In some embodiments, the size of the position vector may be related to the total number of image blocks and the number of object types of the plurality of image blocks. In other embodiments, the position prior information may also include image segmentation features in the segmentation map, so that the position of the object in the target image may be further supervised by the spatial features in the segmentation map, and the image generation may be supervised by the colinearity of the text and image features, thereby further improving the accuracy of the image generation. The segmentation map may be obtained by processing an image obtained by shooting, making, etc.

[0067] At block 904, a target image is generated based on the description text and the position prior information. In some embodiments, the position prior information only includes the position vector. At this time, the text features of the description text can be extracted and mapped to a specific feature space. At the same time, the position vector is also processed by grid deformation and the like to obtain feature offset information, and based on the feature offset information, the mapped text features are feature offset to obtain features for generating the target image. For example, the position vector can be input into the position prior module 102, the description text can be input into the text encoding module 104, and based on Figure 1 The example frame in is processed to generate a target image. It can be understood that the segmentation map encoding module 106 does not work at this time.

[0068] In some embodiments, feature offset information of multiple different sizes may be obtained, and multiple feature offsets may be performed based on the feature offset information of multiple different sizes, and the multiple feature offsets respectively correspond to the features used to generate the target image after multiple upsampling. Figure 2 The mesh deformation module 212 and the image decoding module 216 are processed.

[0069] In some embodiments, the location prior information also includes image segmentation features. In this case, after the image segmentation features and text features are multimodally fused, feature offset is performed on the multimodal fusion features based on feature offset information to obtain features used to generate the target image. For example, the segmentation map can be input into Figure 1 The segmentation map encoding module 106 in the image segmentation feature extraction module is used to extract the image segmentation feature, and the multimodal feature fusion module 108 is used to perform multimodal feature fusion on the image segmentation feature and the text feature, and then the multimodal fusion feature is subjected to feature shifting. Similarly, feature shifting information of multiple different sizes can be obtained, and multiple feature shifting can be performed based on the feature shifting information of multiple different sizes. For example, Figure 2 The mesh deformation module 212 and the image decoding module 216 in the image decoding module are processed. In some embodiments, the image segmentation features can be further fused with the multimodal fusion features to avoid feature generalization in the multimodal feature fusion process. For example, Figure 1 and Figure 2 The short connection in , shares the image segmentation features between the segmentation map encoding module 106 / 218 and the image decoding module 112 / 216.

[0070] According to the various embodiments of the present disclosure described above and their combined embodiments, the efficiency of configuring the functional state can be improved. Figures 1 to 9 The implementation method shown in the figure is only for reference. Figures 1 to 9The architecture or process illustrated in the figure may have other different forms, and may also include more or fewer functional modules and / or units. These modules and / or units may be partially or fully implemented as hardware modules, software modules, firmware modules or any combination thereof, and the embodiments of the present disclosure are not limited to this.

[0071] It is to be understood that the specific names and / or protocols of the various components in the system described herein are only to help those skilled in the art to better understand the concept of the present disclosure, and are not intended to limit the scope of the present disclosure in any way. Moreover, in other embodiments, more or better components may be included, and alternative components with the same or similar functions may also be included.

[0072] Fig.10 1 shows a schematic structural diagram of an example device 1000 that can be used to implement some embodiments of the present disclosure. The device 1000 can be implemented as a server or a PC, etc. The embodiments of the present disclosure do not limit the specific implementation type of the device 1000. Fig.10 As shown, the device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1002 or computer program instructions loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0073] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0074] The processing unit 1001 can execute the various methods and / or processes described above, for example Figure 8 or Fig. 9 For example, in some embodiments, Figure 8 or Fig. 9The method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by CPU 1001, the above-described Figure 8 or Fig. 9 Alternatively, in other embodiments, the CPU 1001 may be configured to execute in any other appropriate manner (eg, by means of firmware, etc.). Figure 8 or Fig. 9 method.

[0075] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0076] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0077] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0078] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0079] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, and conventional procedural programming languages. The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server.

[0080] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer or other programmable data processing device, thereby producing a machine so that when these instructions are executed by the processing unit of a computer or other programmable data processing device, a device for realizing the functions / actions specified in one or more boxes in the flow chart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions make the computer, programmable data processing device and / or other equipment work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for realizing various aspects of the functions / actions specified in one or more boxes in the flow chart and / or block diagram. Computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device, so that a series of operating steps are performed on the computer, other programmable data processing device or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing device, or other device realize the functions / actions specified in one or more boxes in the flow chart and / or block diagram.

[0081] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0082] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0083] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0084] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0085] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a target image, include: Get the description text used to describe the image content of the target image; Determining position a priori information, the position a priori information indicating a type of an object in each of a plurality of image blocks of the target image, the position a priori information being used to locate a position of an object in the image content in the target image; as well as The target image is generated based on the description text and the location prior information. 2 . The method according to claim 1 , wherein the position prior information comprises at least a position vector, and the position vector is used to indicate the object type in each of the plurality of image blocks.

3. The method according to claim 2, wherein the position vector is a combination of a first number of one-dimensional vectors of a first size and a first number of one-dimensional vectors of a second size corresponding to the first number of one-dimensional vectors of the first size, the value of the first size is equal to the first number, the first number is the number of object types, the value of the second size is equal to the total number of image blocks of the multiple image blocks, and each one-dimensional vector of the first size is combined with the corresponding one-dimensional vector of the second size to indicate whether a specific type of object exists in each of the multiple image blocks.

4. The method according to claim 2, wherein determining the location prior information further comprises: include: Acquire a segmentation map, wherein the segmentation map includes at least a portion of the image content; Based on the segmentation map, the position prior information is determined to include image segmentation features in the segmentation map, where the image segmentation features are used to represent the position of at least a portion of the image content in the target image.

5. The method according to claim 4, wherein the target image is generated include: Determining text features of the description text; Generate a multimodal fusion feature based on the text feature and the image segmentation feature; as well as The target image is generated based on the position vector and the multimodal fusion feature.

6. The method according to claim 5, wherein generating the target image further comprises: include: Determining feature offset information corresponding to the position vector; Determining a shifted multimodal fusion feature based on the feature shift information and the multimodal fusion feature; as well as The target image is generated based on the multimodal fusion features.

7. The method according to claim 6, wherein the feature offset information includes feature offset information of multiple sizes, wherein determining the multimodal fusion feature include: Determine first feature offset information having the same size as the multimodal fusion feature among the feature offset information of the multiple sizes; Determine a shifted multimodal fusion feature based on the first feature offset information and the multimodal fusion feature; Upsampling the multimodal fusion features; Determine second feature offset information having the same size as the upsampled multimodal fusion feature among the feature offset information of the multiple sizes; as well as Based on the second feature offset information and the multimodal fusion feature, a multimodal fusion feature that has been offset again is determined.

8. The method according to claim 5, wherein the multimodal fusion is generated based on a self-attention mechanism, and the self-attention mechanism is used to fuse the text features and the image segmentation features according to the importance between the text features and the image segmentation features.

9. The method according to claim 5, further comprising: include: Based on the multimodal fusion feature and the image segmentation feature, an updated multimodal fusion feature is generated.

10. The method according to claim 1, further comprising: include: Acquire a sample data set, the sample data set including sample description text, location prior information corresponding to the sample description text, and a real image; Based on the sample description text and the position prior information, outputting a sample target image through a predetermined image generation model; The predetermined image generation model is trained based on the similarity label between the real image and the sample target image, the similarity label between the sample description text and the sample target image, the sample data set and the sample target image by contrast loss, adversarial loss and average error loss, wherein the contrast loss is the loss between the sample description text and the real image and the sample target image respectively, and the adversarial loss and the average error loss are the losses between the real image and the sample target image; as well as Identify the trained image generation model.

11. The method according to claim 10, wherein the sample data set further comprises a sample segmentation map, and training the predetermined image generation model further comprises: include: The predetermined image generation model is trained based on the sample segmentation map and the sample target image through a structural similarity loss, wherein the structural similarity loss is a loss between the sample segmentation map and the sample target image.

12. The method according to claim 10, wherein the contrast loss is used to train the predetermined image generation model based on the similarity between the description text and the target image, the similarity between the real image and the target image, the similarity label between the real image and the sample target image, and the similarity label between the sample description text and the sample target image.

13. An electronic device, include: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, the instructions causing the electronic device to perform operations when executed by the processor, the operations comprising: Get the description text used to describe the image content of the target image; Determining position a priori information, the position a priori information indicating a type of object in each of a plurality of image blocks of the target image, the position a priori information being used to locate a position of an object in the image content in the target image; and The target image is generated based on the description text and the location prior information. 14 . The electronic device according to claim 13 , wherein the position a priori information comprises at least a position vector, and the position vector is used to indicate the object category in each of the plurality of image blocks.

15. An electronic device according to claim 14, wherein the position vector is a combination of a first number of one-dimensional vectors of a first size and a first number of one-dimensional vectors of a second size corresponding to the first number of one-dimensional vectors of the first size, the value of the first size is equal to the first number, the first number is the number of object types, the value of the second size is equal to the total number of image blocks of the multiple image blocks, and each one-dimensional vector of the first size is combined with the corresponding one-dimensional vector of the second size to indicate whether a specific type of object exists in each of the multiple image blocks.

16. The electronic device according to claim 14, wherein determining the location prior information further comprises: include: Acquire a segmentation map, wherein the segmentation map includes at least a portion of the image content; Based on the segmentation map, the position prior information is determined to include image segmentation features in the segmentation map, where the image segmentation features are used to represent the position of at least a portion of the image content in the target image.

17. The electronic device according to claim 16, wherein the target image is generated include: Determining text features of the description text; Generate a multimodal fusion feature based on the text feature and the image segmentation feature; as well as The target image is generated based on the position vector and the multimodal fusion feature.

18. The electronic device according to claim 17, wherein generating the target image further comprises: include: Determining feature offset information corresponding to the position vector; Determining a shifted multimodal fusion feature based on the feature shift information and the multimodal fusion feature; as well as The target image is generated based on the multimodal fusion features.

19. The electronic device according to claim 18, wherein the feature offset information comprises feature offset information of multiple sizes, wherein determining the multimodal fusion feature include: Determine first feature offset information having the same size as the multimodal fusion feature among the feature offset information of the multiple sizes; Determine a shifted multimodal fusion feature based on the first feature offset information and the multimodal fusion feature; Upsampling the multimodal fusion features; Determine second feature offset information having the same size as the upsampled multimodal fusion feature among the feature offset information of the multiple sizes; as well as Based on the second feature offset information and the multimodal fusion feature, a multimodal fusion feature that has been offset again is determined.

20. A computer program product tangibly stored on a non-transitory computer readable medium and comprising computer executable instructions that when executed cause a computer to: Get the description text used to describe the image content of the target image; Determining position a priori information, the position a priori information indicating a type of object in each of a plurality of image blocks of the target image, the position a priori information being used to locate a position of an object in the image content in the target image; and The target image is generated based on the description text and the location prior information.