Image generation method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of image processing, and provides an image generation method and device, electronic equipment and a readable storage medium. The method comprises the steps of obtaining a to-be-processed image and a task description text for image generation based on the to-be-processed image; performing feature extraction processing on the to-be-processed image and the task description text to obtain image features and text features; performing routing correlation fusion on the image features and the text features to obtain image adjustment features; and the target image is generated based on the image adjustment features and the text features, so that the semantic integrating degree of the generated image and text description is improved, the coordinated presentation effect of an entity and a background in the image is enhanced, the overall visual quality of the generated image is improved, and the accuracy of multi-modal feature interaction is optimized.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and particularly relates to an image generation method and device, an electronic device and a readable storage medium. BACKGROUND

[0002] In the field of image generation, the splicing or weighted summation method is usually used when fusing image and text features, which leads to low semantic consistency between the generated image and the text description, poor coordination between entities and backgrounds, and difficult to meet the actual application requirements of visual quality. The correlation of features is insufficient in the cross-modal fusion process, and the above technical defects make it difficult for existing methods to generate high-quality images that meet both the text semantics and the coordination between image entities and backgrounds.

[0003] Therefore, there is a problem in the prior art that the generated image has low semantic consistency with the text and poor coordination between entities and backgrounds due to insufficient feature extraction and simple fusion method. SUMMARY

[0004] Therefore, the embodiments of the present disclosure provide an image generation method and device, an electronic device and a readable storage medium to solve the problem that the generated image has low semantic consistency with the text and poor coordination between entities and backgrounds due to insufficient feature extraction and simple fusion method in the prior art.

[0005] In a first aspect, the embodiments of the present disclosure provide an image generation method, comprising: obtaining a to-be-processed image and a task description text based on the to-be-processed image for image generation; performing feature extraction processing on the to-be-processed image and the task description text to obtain image features and text features; performing route correlation fusion on the image features and the text features to obtain image adjustment features; and generating a target image based on the image adjustment features and the text features.

[0006] In some embodiments, the image features include entity image features and background image features; and the performing route correlation fusion on the image features and the text features to obtain the image adjustment features comprises: performing feature mapping processing on the text features to obtain text isometric features; performing feature mapping processing on the entity image features and the background image features based on the text isometric features to obtain entity image isometric features and background image isometric features; performing feature fusion processing on the entity image isometric features and the background image isometric features with the text isometric features to obtain entity image fusion features and background image fusion features; and performing feature adjustment processing on the entity image fusion features and the background image fusion features to obtain the image adjustment features.

[0007] In some embodiments, the image features include entity image features and background image features; the feature extraction processing is performed on the to-be-processed image and the task description text to obtain image features and text features, including: performing feature extraction on the task description text to obtain text features; performing image entity segmentation processing on the to-be-processed image to obtain an entity image corresponding to the to-be-processed image and a background image corresponding to the to-be-processed image; and performing feature extraction processing on the entity image and the background image based on the text features to obtain entity image features and background image features.

[0008] In some embodiments, the image entity segmentation processing is performed on the to-be-processed image to obtain the background image corresponding to the to-be-processed image, including: performing image entity segmentation processing on the to-be-processed image to obtain the entity image corresponding to the to-be-processed image and a de-subject background image; performing background mask generation processing on the entity image to obtain a background mask; and performing image inpainting processing on the de-subject background image and the background mask to obtain the background image.

[0009] In some embodiments, the target image is generated based on the image adjustment features and the text features, including: performing global feature enhancement processing on a noise image to obtain global features corresponding to the noise image; and determining the target image according to the text features, the image adjustment features, and the global features.

[0010] In some embodiments, the image entity segmentation processing is performed on the to-be-processed image to obtain the entity image corresponding to the to-be-processed image and the background image corresponding to the to-be-processed image, including: performing target detection processing on the to-be-processed image to obtain a recognized object and label data corresponding to the recognized object; performing segmentation processing on the recognized object based on the label data corresponding to the recognized object to obtain a segmentation mask corresponding to the recognized object; and determining the entity image and the background image according to the segmentation mask corresponding to the recognized object.

[0011] In some embodiments, before the global feature enhancement processing is performed on the noise image to obtain the global features corresponding to the noise image, the method further includes: performing noise value generation processing on a blank image to obtain noise values corresponding to the blank image; and performing value filling processing on the blank image based on the noise values to obtain the noise image.

[0012] A second aspect of the embodiments of the present disclosure provides an image generation apparatus, including: an acquisition module configured to acquire a to-be-processed image and a task description text based on which image generation is performed on the to-be-processed image; a first processing module configured to perform feature extraction processing on the to-be-processed image and the task description text to obtain image features and text features; a second processing module configured to fuse the image features and the text features to obtain image adjustment features; and a third processing module configured to generate a target image based on the image adjustment features and the text features.

[0013] In a third aspect, the embodiment of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0014] In a fourth aspect, the embodiment of the present disclosure provides a readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the above method.

[0015] Compared with the prior art, the embodiment of the present disclosure has the beneficial effects that: by acquiring a to-be-processed image and a task description text, image features and text features are extracted respectively, the two types of features can be interactively fused by a routing correlation fusion mechanism to generate image adjustment features, and then a target image can be generated based on the fused adjustment features and the text features, so that the deep interaction between the image features and the text features is realized by the routing correlation fusion, the semantic fit degree of the generated image and the text description is improved, and the coordination and visual quality of entities and backgrounds in the generated image are improved by collaborative optimization of multi-modal features. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is a scene schematic diagram of an application scenario of the embodiment of the present disclosure;

[0018] Figure 2 is a flow schematic diagram of an image generation method provided by the embodiment of the present disclosure;

[0019] Figure 3 is a flow schematic diagram of another image generation method provided by the embodiment of the present disclosure;

[0020] Figure 4 is a processing flow schematic diagram of a routing correlation fusion module provided by the embodiment of the present disclosure;

[0021] Figure 5 is a structural schematic diagram of an image generation device provided by the embodiment of the present disclosure;

[0022] Figure 6 is a structural schematic diagram of an electronic device provided by the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In the following description, for the purposes of explanation, numerous specific details are set forth in order to thoroughly describe certain embodiments of the present disclosure. One skilled in the art, however, will appreciate that embodiments of the present disclosure can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present disclosure.

[0024] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0025] An image generation method and device according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 is a scene diagram of an application scenario of an embodiment of the present disclosure. The application scenario can include terminal devices 1, 2, and 3, a server 4, and a network 5.

[0027] The terminal devices 1, 2, and 3 can be hardware or software. When the terminal devices 1, 2, and 3 are hardware, they can be various electronic devices with a display screen and supporting communication with the server 4, including but not limited to a smart phone, a tablet computer, a laptop computer, and a desktop computer, etc. When the terminal devices 1, 2, and 3 are software, they can be installed in the above electronic devices. The terminal devices 1, 2, and 3 can be implemented as multiple software or software modules, or as a single software or software module, and the present disclosure does not limit this. Further, the terminal devices 1, 2, and 3 can have various applications installed thereon, such as a data processing application, an instant messaging tool, a social platform software, a search application, a shopping application, etc.

[0028] The server 4 can be a server providing various services, for example, a background server receiving a request sent by a terminal device establishing a communication connection therewith. The background server can receive and analyze the request sent by the terminal device, etc., and generate a processing result. The server 4 can be a single server, a server cluster composed of several servers, or a cloud computing service center, and the present disclosure does not limit this.

[0029] It should be noted that the server 4 can be hardware or software. When the server 4 is hardware, it can be various electronic devices that provide various services for the terminal devices 1, 2 and 3. When the server 4 is software, it can be multiple software or software modules that provide various services for the terminal devices 1, 2 and 3, or a single software or software module that provides various services for the terminal devices 1, 2 and 3, and the embodiments of the present disclosure do not limit this.

[0030] The network 5 can be a wired network using coaxial cables, twisted-pair cables and optical fibers, or a wireless network that can realize interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), Infrared, etc., and the embodiments of the present disclosure do not limit this.

[0031] The user can establish a communication connection with the server 4 through the terminal devices 1, 2 and 3 via the network 5 to receive or send information, etc. Specifically, the server 4 can obtain the to-be-processed image and the task description text via the terminal devices 1, 2 and 3, extract the image features and the text features respectively, and can generate the target image based on the fused adjustment features and the text features by the route relevance fusion mechanism to interact and fuse the two types of features.

[0032] It should be noted that the specific types, quantities and combinations of the terminal devices 1, 2 and 3, the server 4 and the network 5 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present disclosure do not limit this.

[0033] Figure 2 is a flowchart of an image generation method provided by an embodiment of the present disclosure. Figure 2 The image generation method of Figure 1 may be executed by the server. As Figure 2 shown, the image generation method includes:

[0034] S201, obtaining a to-be-processed image and a task description text for image generation based on the to-be-processed image.

[0035] Specifically, the to-be-processed image can be received through an input interface, and the task description text can be obtained through a natural language input interface to receive a text instruction. The text instruction can be used to represent semantic requirements for modifying, replacing or regenerating entities and / or backgrounds in the to-be-processed image. In this way, an accurate and standardized input basis is provided, the combination of the task description text and the to-be-processed image in the generation process is ensured, the understanding ability of the semantic instruction is enhanced, and the semantic consistency of the generated image, the task description text and the to-be-processed image elements is improved.

[0036] The to-be-processed image can be an original image that needs to be modified, replaced, or regenerated. The task description text can be an instruction input through natural language, which can be used to represent the semantic requirements for operating on entities and / or backgrounds in the to-be-processed image.

[0037] For example, a user uploads a scenic photo containing a beach and a coconut tree as a to-be-processed image, and inputs a task description text: "replace the coconut tree on the beach with a palm tree, and replace the white cloud in the sky with a sunset." The task description text can be used to represent that the user needs to modify the entity (coconut tree) and background (white cloud in the sky) in the image, and the modification content is to replace the coconut tree with a palm tree and replace the white cloud with a sunset.

[0038] S202, performing feature extraction processing on the to-be-processed image and the task description text to obtain image features and text features.

[0039] Specifically, the to-be-processed image can be subjected to image feature extraction processing, which can be realized by an image encoder and a feature optimizer connected in sequence. The image encoder can perform block processing and encoding on the to-be-processed image to obtain initial visual feature representation, capturing global information in the to-be-processed image. The feature optimizer can perform nonlinear transformation and attention mechanism processing on the initial visual feature representation to obtain image features. The task description text can be subjected to semantic encoding by a text encoder, which can obtain deep semantic information of the task description text through an embedding layer and an attention mechanism to obtain text features. The feature extraction processing can be independently performed in a feature space, ensuring that the image features and the text features each retain corresponding modal characteristics, thereby enhancing the expression ability and discrimination of the image features and the text features, improving the accuracy and efficiency of subsequent multi-modal feature fusion, and improving the semantic consistency and overall visual effect between the generated image and the text description.

[0040] The image features can be feature vectors or feature representations extracted from the to-be-processed image and used to represent the content of the to-be-processed image. The information represented by the image features includes visual information of the to-be-processed image, such as color, texture, shape, spatial layout, etc., which is not limited here. The information represented by the image features can be used to indicate the characteristics of elements such as entities and backgrounds in the to-be-processed image. The image features can be obtained by processing through an image encoder and a feature optimizer.

[0041] The text features can be feature vectors or feature representations extracted from the task description text and used to represent the semantics of the task description text. The information represented by the text features includes semantic information of the task description text, such as the meaning of words, the structure and / or semantic relationship of sentences, etc., which is not limited here. The text features can be obtained by processing through a text encoder.

[0042] For example, a user uploads a scenic photo containing a beach and coconut trees as the image to be processed, and inputs the task description text: "replace the coconut trees on the beach with palm trees, and replace the white clouds in the sky with sunset."

[0043] Image encoder: input the image to be processed into the Vision Transformer (ViT) image encoder, which can divide the image to be processed into multiple small blocks and convert each small block into an embedding vector. The global information in the image can be captured through the multi-head self-attention mechanism to obtain the initial visual feature representation.

[0044] Feature optimizer: the initial visual features output by the ViT can be input into the Q-former model, which further optimizes the features through nonlinear transformation and attention mechanism to obtain higher-level image features that can more effectively express the visual information of elements such as coconut trees, beaches, and skies in the image.

[0045] Text encoder: the task description text can be input into the Contrastive Language-Image Pre-training (CLIP) text encoder, which converts the words in the text into embedding vectors through an embedding layer, and then captures the semantic relationships in the text through the multi-head self-attention mechanism to obtain deep semantic information. The text features can be used to represent semantic information such as "coconut trees", "palm trees", "white clouds", and "sunset".

[0046] S203, fuse the image features and text features through routing correlation to obtain image adjustment features.

[0047] Specifically, the image features can be input into a Multi-Layer Perceptron (MLP) for nonlinear mapping to obtain first mapping features; the text features can be input into the MLP for nonlinear mapping to obtain second mapping features; the first mapping features and the second mapping features can be multiplied element by element to generate multiplication features; the first mapping features can be input as queries, the second mapping features can be input as keys and values through the mask cross-attention mechanism, and the attention calculation can be performed in combination with the multiplication features to obtain attention features; then the image features and the attention features can be added element by element to obtain enhanced features; the multiplication features can be input into an activation function for normalization, and the normalized result can be multiplied element by element with the enhanced features, and input into the MLP for feature adjustment to obtain image adjustment features. In this way, the correlation capturing ability between the image features and the text features is enhanced, the semantic expression accuracy of the feature fusion is improved, and the quality and accuracy of the subsequent image generation are improved.

[0048] The image adjustment feature can be an image feature obtained through a routing correlation fusion process, which fuses text feature semantic information, retains visual information of the image to be processed, and enhances semantic understanding of the text description through interaction with the text feature.

[0049] For example, the image feature representing the visual information of elements such as coconut trees, beaches, and skies in the image to be processed and the text feature representing semantic information such as “coconut tree”, “palm tree”, “white cloud”, and “sunset” are respectively input into the MLP for nonlinear mapping to obtain first mapping features and second mapping features; the two mapping features are multiplied element by element to generate a multiplication feature; the first mapping feature is taken as the query (Q), the second mapping feature is taken as the key (K) and the value (V), and the mask cross-attention calculation is performed in combination with the multiplication feature to obtain an attention feature; then the image feature and the attention feature can be added element by element to obtain an enhanced feature; the multiplication feature can be input into an activation function (such as Sigmoid) for normalization processing, and the normalized result and the enhanced feature are multiplied element by element, and input into the MLP for feature adjustment to obtain the image adjustment feature.

[0050] S204, generating a target image based on the image adjustment feature and the text feature.

[0051] Specifically, the noise image can be obtained by performing noise value generation processing on the blank image, and the noise image can be input into the diffusion model to capture global information of the input feature through the self-attention mechanism in the diffusion model; the output obtained by processing can be taken as the query, and the text feature can be taken as the key and the value to perform text cross-attention calculation to integrate the text semantics; then nonlinear transformation and feature extraction can be performed through the feedforward network; the input of the feedforward network and the entity adjustment feature can be added through the residual structure to generate a target image, thereby enhancing the semantic consistency between the generated image and the text description, improving the coordination and visual quality of the entities and the background in the image, and improving the accuracy and reliability of the overall image generation.

[0052] The target image can be an image generated by the image generation model according to the image to be processed and the task description text. The target image can be used to represent the semantic information of the task text description and the visual features of the image to be processed.

[0053] For example, a user uploads a scenic photo containing a beach and coconut trees, and inputs a task description text: "replace the coconut trees on the beach with palm trees, and replace the white clouds in the sky with sunset." A blank image with the same size as the target image can be created, and a random noise value (such as Gaussian distribution or uniform distribution noise) can be randomly assigned to each pixel position of the blank image to generate a random noise image; the generated noise image can be input into the diffusion model, the global information of the input features can be captured through the self-attention mechanism, the result obtained can be taken as the query (Q), the text features can be taken as the key (K) and the value (V), the text cross-attention calculation can be performed, the input of the feedforward network can be added to the image adjustment features through the nonlinear transformation and feature extraction of the feedforward network through the residual structure, and the target image can be generated, in which the original coconut trees are replaced with palm trees, and the white clouds in the sky are replaced with sunset, and these modifications are naturally integrated with the background of the original image, improving the semantic consistency and visual quality of the generated image.

[0054] According to the technical scheme provided by the embodiments of the present disclosure, by obtaining the to-be-processed image and the task description text, the to-be-processed image is processed by the image encoder and the feature optimizer respectively, and the task description is processed by the text encoder; the image features and the text features can be fused by routing correlation, which includes but is not limited to nonlinear mapping, multiplication operation, attention mechanism and normalization processing, and the image adjustment features are output; based on the image adjustment features and the text features, the diffusion model is used to combine the text cross-attention and the entity background adjustment calculation to gradually generate the target image, thereby enhancing the semantic consistency between the generated image and the text description, improving the matching degree, visual quality and overall reliability of the image entity and background.

[0055] In some embodiments, the image features include entity image features and background image features; the image features and the text features are fused by routing correlation to obtain the image adjustment features, including: performing feature mapping processing on the text features to obtain text isometric features; based on the text isometric features, performing feature mapping processing on the entity image features and the background image features to obtain entity image isometric features and background image isometric features; performing feature fusion processing on the entity image isometric features and the background image isometric features with the text isometric features respectively to obtain entity image fusion features and background image fusion features; performing feature adjustment processing on the entity image fusion features and the background image fusion features respectively to obtain the image adjustment features.

[0056] Specifically, the text feature can be processed by MLP for feature mapping, and the text feature is mapped to a text homodimensional feature matching the dimension of the image feature; the entity image feature and the background image feature can be independently processed by feature mapping, the entity image feature is processed by MLP to obtain an entity image homodimensional feature; the background image feature is processed by MLP to obtain a background image homodimensional feature; then the entity image homodimensional feature and the text homodimensional feature can be multiplied element by element to generate a multiplication feature; the entity image homodimensional feature can be used as a query vector, the text homodimensional feature can be used as a key vector and a value vector, and the multiplication feature can be combined to obtain an entity image fusion feature by mask cross attention; the background image part is processed in the same way as the entity part, that is, the background image fusion feature can be obtained by element-level multiplication and mask cross attention mechanism; then the entity image feature and the entity image fusion feature can be added to obtain an intermediate feature; the multiplication feature can be input into a Sigmoid activation function for normalization to generate a weight coefficient; the weight coefficient can be multiplied by the intermediate feature to obtain a weighted feature; the weighted feature is processed by MLP for nonlinear transformation and optimization to obtain an entity image adjustment feature.

[0057] The entity image feature can be a visual feature obtained from a region of the image to be processed, which can be used to represent a subject object in the image to be processed, including but not limited to feature information of a person or an object, etc. The entity image feature can be obtained by segmenting the subject object in the image to be processed by a segment anything model (SAM), and then extracted by a ViT. The entity image feature can be used to generate or modify the subject part in the image to be processed.

[0058] The background image feature can be a visual feature extracted from the image to be processed, which can be used to represent the visual feature of the background part in the image to be processed. The background image feature can be extracted separately by image segmentation and then extracted by a ViT. The background image feature can be used to generate or modify the background part in the image to be processed.

[0059] The text homodimensional feature can be a text feature obtained by feature mapping, which can be nonlinearly transformed by MLP. The text homodimensional feature can be used for fusion with the image feature.

[0060] The entity image homodimensional feature can be an entity image feature obtained by feature mapping. The dimension of the entity image homodimensional feature is the same as that of the text homodimensional feature. The entity image homodimensional feature can be nonlinearly transformed by MLP. The entity image homodimensional feature can be used for fusion with the text homodimensional feature.

[0061] The background image same-dimensional feature can be an entity image feature obtained through feature mapping processing, the dimension of the background image same-dimensional feature is the same as that of the text same-dimensional feature, the background image same-dimensional feature can be obtained through nonlinear transformation of the background image feature by an MLP, and can be used for fusion with the text same-dimensional feature.

[0062] The entity image fusion feature can be a feature obtained by fusing the entity image same-dimensional feature and the text same-dimensional feature, the entity image fusion feature can be used to represent visual information of the entity image and semantic information of the text, and the entity image fusion feature can be obtained by fusing the entity image same-dimensional feature and the text same-dimensional feature through element-level multiplication and mask cross-attention mechanism.

[0063] The background image fusion feature can be a new feature obtained by fusing the background image same-dimensional feature and the text same-dimensional feature, the background image fusion feature can be used to represent visual information of the background image and semantic information of the text, and the background image fusion feature can be obtained by fusing the background image same-dimensional feature and the text same-dimensional feature through element-level multiplication and mask cross-attention mechanism.

[0064] For example, the coconut tree (entity) and the background (beach and sky) in the image to be processed can be segmented by SAM respectively, and the segmented entity image and background image can be encoded by ViT respectively to obtain entity image features and background image features; the task description text can be encoded by a CLIP text encoder to obtain a text feature, and then the text feature can be nonlinearly transformed by an MLP to obtain a text same-dimensional feature, the entity image feature and the background image feature can be nonlinearly transformed by an MLP respectively to obtain an entity image same-dimensional feature and a background image same-dimensional feature, and then the entity image same-dimensional feature and the text same-dimensional feature can be multiplied element by element to obtain a multiplied result, the entity image same-dimensional feature can be taken as a query vector, the text same-dimensional feature can be taken as a key vector and a value vector, and the multiplied result can be calculated by mask cross-attention to obtain an entity image fusion feature, the background image same-dimensional feature can be processed in the same way to obtain a background image fusion feature; then the entity image feature and the entity image fusion feature can be added, the multiplied result can be normalized by a Sigmoid activation function to obtain a weight coefficient, and the weight coefficient and the addition result can be multiplied to obtain a weighted feature, and the weighted feature can be nonlinearly transformed by an MLP to obtain an entity image adjustment feature.

[0065] According to the technical scheme provided by the embodiment of the present disclosure, the text feature is mapped to a text homodimensional feature matching the image feature dimension through feature mapping processing of the text feature; the entity image feature and the background image feature can be independently subjected to feature mapping processing, the entity image homodimensional feature is obtained by processing the entity image feature through the MLP; the background image homodimensional feature is obtained by processing the background image feature through the MLP; then the entity image homodimensional feature and the text homodimensional feature can be multiplied at the element level to generate a multiplication feature; the entity image fusion feature is obtained by calculating through the mask cross attention; the processing of the background image part is the same as that of the entity part, that is, the background image fusion feature can be obtained through element-level multiplication and the mask cross attention mechanism; then the entity image feature and the entity image fusion feature can be added to obtain an intermediate feature; the weight coefficient can be generated by inputting the multiplication feature into the Sigmoid activation function for normalization processing; the weighted feature can be obtained by multiplying the weight coefficient and the intermediate feature; the entity image adjustment feature is obtained by nonlinear transformation and optimization of the weighted feature through the MLP, thereby improving the fusion accuracy of the image feature and the text feature and enhancing the quality of the generated image.

[0066] In some embodiments, the image feature includes an entity image feature and a background image feature; the feature extraction processing of the to-be-processed image and the task description text to obtain the image feature and the text feature includes: performing feature extraction on the task description text to obtain the text feature; performing image entity segmentation processing on the to-be-processed image to obtain an entity image corresponding to the to-be-processed image and a background image corresponding to the to-be-processed image; performing feature extraction on the entity image and the background image based on the text feature to obtain the entity image feature and the background image feature.

[0067] Specifically, the task description text can be subjected to feature extraction through a text encoder, the text encoder can convert the input task description text into a dense vector representation containing semantic information, that is, a text feature; then the image entity segmentation processing can be performed on the to-be-processed image, the target detection model (Grounding DINO) can be used to identify the target object related to the task description text in the to-be-processed image and the position corresponding to the target object, then the SAM can be used to segment the entity image and the background image according to the detection result, and then the entity image and the background image segmented can be subjected to feature extraction processing based on the text feature extracted: the entity image can be input into the ViT encoder and the Q-former model, combined with the text feature for fusion to obtain the entity image feature; the background image can be input into the ViT encoder and the Q-former model, combined with the text feature for fusion to obtain the background image feature.

[0068] The entity image feature can be a visual feature extracted from the to-be-processed image, can be used to represent the subject object feature in the to-be-processed image, and can be extracted by segmenting the subject object in the to-be-processed image by SAM, and then by ViT and Q-former.

[0069] The background image feature can be extracted from the to-be-processed image, and can be used to represent the visual feature of the background part in the to-be-processed image. The background image feature can be extracted by ViT and Q-former.

[0070] The entity image corresponding to the to-be-processed image can be a subject object part segmented from the to-be-processed image. The entity image can be obtained by Grounding DINO identifying the target object and its position related to the task description text in the to-be-processed image, and then segmented by SAM according to the detection result.

[0071] The background image corresponding to the to-be-processed image can be a background part segmented from the to-be-processed image. The background image can be obtained by separately extracting the background part after image segmentation.

[0072] For example, in the application scenario of the above embodiment, the task description text can be encoded by the CLIP text encoder to extract the text feature. The Grounding DINO target detection model can be used to identify the target object (such as a coconut tree) and its position related to the task description text in the to-be-processed image. The SAM image segmentation model can be used to segment the entity image (coconut tree) and the background image (beach and sky) according to the detection result. Then, the segmented entity image (coconut tree) can be input into the ViT encoder to extract the initial visual feature. The feature output by the ViT can be input into the Q-former model to be combined with the text feature to obtain the entity image feature. Then, the segmented background image (beach and sky) can be input into the ViT encoder to extract the initial visual feature. The feature output by the ViT can be input into the Q-former model to be combined with the text feature to obtain the background image feature.

[0073] According to the technical scheme provided by the embodiment of the present disclosure, the text feature is obtained by extracting the features of the task description text through the text encoder, and then the image entity segmentation processing can be performed on the to-be-processed image. The target object and the position corresponding to the target object in the to-be-processed image related to the task description text can be recognized through the Grounding DINO, and then the entity image and the background image can be obtained by the SAM according to the detection result. Furthermore, the feature extraction processing can be performed on the entity image and the background image obtained by segmentation based on the text feature extracted in the foregoing, so as to obtain the entity image feature and the background image feature. In this way, the extraction accuracy of the entity and background features is improved, and the quality and semantic consistency of the generated image are enhanced.

[0074] In some embodiments, the image entity segmentation processing is performed on the to-be-processed image to obtain the background image corresponding to the to-be-processed image, including: performing image entity segmentation processing on the to-be-processed image to obtain the entity image and the de-subject background image corresponding to the to-be-processed image; performing background mask generation processing on the entity image to obtain the background mask; and performing image completion processing on the de-subject background image and the background mask to obtain the background image.

[0075] Specifically, the entity object in the to-be-processed image and the boundary information corresponding to the entity object can be recognized through the target detection model. Based on the detection result, the entity region can be pixel-level segmented through the semantic segmentation model to separate the entity image containing the entity object and the de-subject background image after removing the entity subject. Furthermore, according to the contour and position information of the entity object in the entity image, a corresponding binary mask image can be generated. In the mask image, the entity region is marked as a specific value that needs to be retained or completed. The specific value can be used to indicate the missing area in the de-subject background image due to the removal of the entity. Furthermore, the missing area can be located according to the mask, and the context semantic information of the de-subject background image is used to generate content that is visually coherent with the surrounding background to fill the missing area, so as to obtain the background image.

[0076] The entity image corresponding to the to-be-processed image can be the subject object part segmented from the to-be-processed image. The entity image can recognize the subject object and the boundary information thereof in the to-be-processed image through the Grounding DINO, and then the subject region can be pixel-level segmented through the SAM.

[0077] The de-subject background image can be the background part remaining after removing the subject object from the to-be-processed image. The de-subject background image can be obtained by removing the subject object from the to-be-processed image after image segmentation.

[0078] The background mask can be a binary mask image, which can be used to indicate the missing area in the subject-removed background image due to the removal of the subject object, and can be generated according to the contour and position information of the subject object in the entity image. In the mask image, the subject area is marked as a specific value, indicating the area to be completed; and the background area is marked as another specific value, indicating the area to be retained.

[0079] For example, the subject object (palm tree) in the image to be processed and its boundary information can be identified by Grounding DINO; then the subject area can be segmented at the pixel level by SAM to separate the entity image containing the subject object and the subject-removed background image; then a corresponding binary mask image can be generated according to the contour and position information of the subject object (palm tree) in the entity image, in which the subject area is marked as 0, indicating the area to be completed; and the background area is marked as 1, indicating the area to be retained; and then the subject-removed background image can be completed by the image completion model (Stable Diffusion Inpainting), which can locate the missing area according to the background mask and generate content that is visually coherent with the surrounding background based on the context semantic information of the subject-removed background image to fill the missing area, thereby obtaining a complete background image.

[0080] According to the technical scheme provided by the embodiments of the present disclosure, the entity object in the image to be processed and the boundary information corresponding to the entity object are identified by the target detection model; the entity area can be segmented at the pixel level based on the detection result by the semantic segmentation model to separate the subject-removed background image; then a corresponding binary mask image can be generated according to the contour and position information of the entity object in the entity image, and then the missing area can be located according to the mask, and content that is visually coherent with the surrounding background can be generated based on the context semantic information of the subject-removed background image to fill the missing area, thereby obtaining a background image, which improves the completeness and naturalness of the background image.

[0081] In some embodiments, generating the target image based on the image adjustment feature and the text feature includes: performing global feature enhancement processing on the noise image to obtain global features corresponding to the noise image; and determining the target image based on the text feature, the image adjustment feature, and the global features.

[0082] Specifically, the global feature enhancement processing on the noise image can be performed by a self-attention module in the diffusion model, which can capture the global dependency between regions in the noise image, extract and strengthen the global context features of the noise image, and generate global features containing overall structure information; then the target image can be generated by iterative optimization of the diffusion model based on the text feature, the image adjustment feature (including the entity adjustment feature and the background adjustment feature), and the global features.

[0083] It should be further noted that the iterative optimization process of the diffusion model can include: processing the global feature through text cross attention, so that the global feature can interact with the text feature, and the text semantic information can be injected into the feature representation; the features after interaction can be interacted with the entity adjustment feature and the background adjustment feature respectively, and then the nonlinear transformation and residual connection optimization of the feedforward network can be used to gradually remove noise and refine the image content through multiple rounds of iteration to output the target image.

[0084] The noise image can be a blank image or a randomly generated image, and the noise image can include randomly distributed pixel values.

[0085] The global feature corresponding to the noise image can be a global feature representation extracted from the noise image, which can be used to represent the overall structure and context information of the noise image. The global feature can be obtained by performing global feature enhancement processing on the noise image through the diffusion model, capturing the global dependency between regions in the noise image, and extracting and strengthening the global context feature of the noise image.

[0086] For example, a blank image with the same size as the target image can be created, and a noise value such as a Gaussian distribution or a uniform distribution noise can be randomly assigned to each pixel position of the blank image to generate a completely random noise image. Then, the generated noise image can be input into the diffusion model to capture the global dependency between regions in the noise image through the self-attention mechanism, extract and strengthen the global context feature of the noise image, and generate a global feature containing overall structure information. Then, the global feature can be processed through text cross attention, so that the global feature interacts with the text feature, and the text semantic information can be injected into the feature representation. The features after interaction can be interacted with the entity adjustment feature and the background adjustment feature respectively, and further fused with the image feature and the text feature. Then, the nonlinear transformation and residual connection optimization of the feedforward network can be used to gradually remove noise and refine the image content. Through multiple rounds of iteration, the generated image is gradually optimized to ensure that the generated image is consistent with the input text description and image feature in terms of semantics and vision, and the target image is obtained. In the target image, the coconut trees on the beach are replaced with palm trees, the white clouds in the sky are replaced with sunset, and the overall image is natural and coordinated in vision.

[0087] According to the technical scheme provided by the embodiments of the present disclosure, the global feature enhancement processing of the noise image is performed through the self-attention module in the diffusion model to generate a global feature containing overall structure information. Then, the target image can be generated through the iterative optimization of the diffusion model by combining the text feature, the image adjustment feature, and the global feature. In this way, the global representation ability of the noise image is enhanced through the self-attention mechanism, and the text, image adjustment feature, and global feature are synergistically fused to improve the synergistic effect of multi-modal features in image generation.

[0088] In some embodiments, the image entity segmentation processing is performed on the to-be-processed image to obtain an entity image corresponding to the to-be-processed image and a background image corresponding to the to-be-processed image, including: performing target detection processing on the to-be-processed image to obtain a recognized object and label data corresponding to the recognized object; performing segmentation processing on the recognized object based on the label data corresponding to the recognized object to obtain a segmentation mask corresponding to the recognized object; and determining the entity image and the background image according to the segmentation mask corresponding to the recognized object.

[0089] Specifically, the to-be-processed image can be input into the Grounding DINO, the target entity object in the to-be-processed image is recognized by the target detection model, and the label data corresponding to each recognized object is output. The label data can be input into the SAM, the target entity object can be segmented at the pixel level according to the label data to generate a segmentation mask, wherein the region marked as foreground in the segmentation mask corresponds to the pixel position of the recognized object, and then the to-be-processed image can be divided based on the segmentation mask, the original image pixels corresponding to the foreground region in the mask are extracted as the entity image, and the original image pixels corresponding to the background region in the mask are extracted as the background image, thereby separating the entity image and the background image.

[0090] The recognized object can be a subject object or a target object recognized in the to-be-processed image by the target detection model.

[0091] The label data corresponding to the recognized object can be detailed information related to the recognized object output by the target detection model, including but not limited to a bounding box and / or category information, etc. The label data can be obtained by recognizing the target entity object in the to-be-processed image by the Grounding DINO, and the label data of each recognized object can be used to guide the SAM to perform pixel-level segmentation on the target object to generate a segmentation mask.

[0092] The segmentation mask corresponding to the recognized object can be a binary mask image, which can be used to represent the specific position of the recognized object in the image. The region marked as foreground in the mask corresponds to the pixel position of the recognized object, and the region marked as background corresponds to the pixel position of the background. The segmentation mask can be obtained by inputting the label data into the SAM and performing pixel-level segmentation processing on the target entity object according to the label data.

[0093] For example, in the foregoing application scenarios, the to-be-processed image can be input into the Grounding DINO, target entity objects (such as coconut trees) in the to-be-processed image and corresponding annotation data (such as a bounding box and class information) of the target entity objects are recognized, and then the annotation data can be input into the SAM, the target entity objects (coconut trees) are subjected to pixel-level segmentation processing according to the annotation data, and a segmentation mask is generated, wherein a region marked as foreground in the segmentation mask corresponds to a pixel position of a coconut tree, and a region marked as background corresponds to a pixel position of the background; then, according to the segmentation mask, the to-be-processed image can be divided into an entity image and a background image: the original image pixels corresponding to the foreground region in the mask can be extracted as the entity image (coconut tree); and the original image pixels corresponding to the background region in the mask can be extracted as the background image (beach and sky).

[0094] According to the technical scheme provided by the embodiments of the present disclosure, by performing target detection processing on the to-be-processed image, the target object and the corresponding text description in the to-be-processed image are recognized by the Grounding DINO model as the recognition object and the annotation data corresponding to the recognition object; based on the annotation data corresponding to the recognition object, the SAM is used to perform segmentation processing on the recognition object, and a segmentation mask corresponding to the recognition object is generated; and then, according to the segmentation mask corresponding to the recognition object, the entity image and the background image are separated by the mask application, so that the accuracy of image entity segmentation and the efficiency of background separation are improved, and the semantic consistency between the generated image and the task description text is ensured.

[0095] In some embodiments, before the global feature enhancement processing is performed on the noise map to obtain the global feature corresponding to the noise map, the method further includes: performing noise value generation processing on the blank image to obtain a noise value corresponding to the blank image; and performing value filling processing on the blank image based on the noise value to obtain the noise map.

[0096] Specifically, a blank image matching the size of the target generated image can be created, and the initial pixel value of the blank image can be zero or a preset background value, which is not limited herein; then, a noise value conforming to a specific probability distribution can be generated for each pixel position of the blank image through random noise generation; and then, the generated noise value can be mapped and assigned to the corresponding pixel position of the blank image one by one, covering the initial pixel value, so as to form a noise map in which each pixel point is filled with a random noise value.

[0097] The blank image can be an image matching the size of the target generated image, and the initial pixel value of the blank image can be zero or a preset background value, which is not limited herein. The blank image can be obtained by creating a matrix with the same size as the target image and initializing all pixel values of the matrix to zero or a preset background value.

[0098] The noise value corresponding to the blank image can be a random noise value generated for each pixel position, the noise value conforming to a specific probability distribution, including but not limited to a Gaussian distribution or a uniform distribution, the noise value can be generated for each pixel position by a random noise generator, and the noise value can be used to fill the blank image to generate a noise map as an initial input of the diffusion model.

[0099] For example, a blank image with the same size as the target image can be created, the initial pixel value of the blank image is zero, and then a random noise generator can be used to generate a noise value conforming to a specific probability distribution for each pixel position; and then the generated noise value can be mapped and assigned to the corresponding pixel position of the blank image one by one, covering the initial pixel value, to obtain a noise map in which each pixel point is filled with a random noise value.

[0100] According to the technical scheme provided by the embodiments of the present disclosure, the noise value corresponding to the blank image is obtained by performing noise value generation processing on the blank image; and the noise map is obtained by performing value assignment and filling processing on the blank image based on the noise value, thereby improving the accuracy and controllability of the initial noise condition setting of the image generation, and enhancing the stability of the diffusion model denoising process and the quality of the generated image.

[0101] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described here.

[0102] Figure 3 is another flowchart of an image generation method provided by the embodiments of the present disclosure. As shown in Figure 3 The image generation method comprises:

[0103] The image is generated by fusing the routing module and the multi-modal technology; the input image (to-be-processed image) can be divided into an entity (entity image) and a background part (background image), and the two parts are respectively completed and feature extracted, and the text features are extracted. Then, the image and text features are fused by using the routing correlation fusion module. Finally, a new image (target image) is generated according to these features by using the diffusion model.

[0104] 1. Image entity segmentation

[0105] The preprocessed input image is processed by using the Grounding DINO model and the SAM model; the Grounding DINO can identify the target objects in the image and their corresponding text descriptions, and the SAM can accurately segment the objects according to the detection results of the Grounding DINO, and then obtain the segmented entity image part and the background part.

[0106] Image entity segmentation can decompose an image into two parts: entity and background (de-backgrounding), providing a basis for subsequent feature extraction and image generation.

[0107] 2. Background part completion

[0108] The obtained background part is input into the Stable Diffusion Inpainting model to complete the background image after matting, thereby obtaining the completed background image.

[0109] Background mask generation: According to the obtained entity mask, generate a background mask, which can be used to indicate the area that needs to be completed.

[0110] Background completion: input the background part and the background mask into the Stable Diffusion Inpainting model, the model can complete the missing part according to the context information of the background, and obtain the completed background image.

[0111] The purpose of background part completion is to eliminate the blank area in the background after matting, making the background image more complete and natural. Stable Diffusion Inpainting model can generate content matching the background through the context information of the image to be processed, thereby realizing the completion of the background.

[0112] 3. Entity image feature extraction

[0113] Input the entity image into the ViT image encoder to obtain the image features. Then input these features into the Q-former model to obtain the extracted entity image features.

[0114] ViT encoding: input the entity image into the ViT image encoder, the encoder can divide the image into multiple small blocks and convert each small block into an embedding vector. Through the multi-head self-attention mechanism, ViT can capture the global information in the image to obtain the feature representation of the image.

[0115] Q-former processing: input the image features output by ViT into the Q-former model, Q-former can further process and convert the features to obtain the extracted entity image features.

[0116] ViT image encoder can capture global information in the image, while Q-former model can further optimize and extract these features.

[0117] 4. Background image feature extraction

[0118] The background image is input into the ViT image encoder to obtain image features. These features are then input into the Q-former model to obtain the extracted background image features.

[0119] Similar to entity image feature extraction, representative background image features can be obtained through the processing of the ViT and Q-former models.

[0120] 5. Text feature extraction

[0121] The text description is input into the CLIP text encoder to obtain text features.

[0122] The CLIP model is pre-trained on a large-scale image-text pair, which can learn the semantic correlation between text and image and obtain text features with semantic information.

[0123] 6. Entity adjustment feature extraction

[0124] The entity image features and text features are input into the routing correlation fusion module to obtain the fused entity adjustment features.

[0125] The purpose of entity adjustment feature extraction is to fuse entity image features and text features to obtain entity adjustment features with more semantic information and correlation. Through the routing correlation fusion module, the interaction information between features can be captured to improve the expression ability of entity features.

[0126] The input of the routing correlation fusion module is image features and text features, and the output is fused adjustment features.

[0127] 7. Background adjustment feature extraction

[0128] The background image features and text features are input into the routing correlation fusion module to obtain the fused background adjustment features.

[0129] Background adjustment feature extraction is to fuse background image features and text features to obtain background adjustment features with more semantic information and correlation. Similar to entity adjustment feature extraction, the routing correlation fusion module can improve the expression ability of background features.

[0130] 8. Diffusion model

[0131] The noise map is input into the diffusion model, which is composed of multiple identical sub-modules connected in series. The structure of each diffusion model sub-module is as follows:

[0132] 1) Self-attention module: input the noise map or the output of the previous sub-module into the self-attention module, which can capture the global information of the input features and enhance the expression ability of the features.

[0133] 2) Text Cross-Attention Module: The result of the previous step is taken as the query (Q), and the text features are taken as the key (K) and value (V) for cross-attention calculation. The text cross-attention module can integrate the semantic information of the text into the features, guiding the generation of the image.

[0134] 3) Entity Cross-Attention: The result of the previous step is taken as the query (Q), and the entity adjustment features are taken as the key (K) and value (V) for cross-attention calculation. The entity cross-attention module can integrate the feature information of the entity into the features, ensuring the accuracy and relevance of the generated image.

[0135] 4) Background Cross-Attention: The result of the previous step is taken as the query (Q), and the background adjustment features are taken as the key (K) and value (V) for cross-attention calculation. The background cross-attention module can integrate the feature information of the background into the features, making the generated image background more natural and reasonable.

[0136] 5) Feedforward Network: The result obtained in the previous step is input into the feedforward network, which is used for nonlinear transformation and feature extraction of the features.

[0137] 6) Residual Module: The input of the feedforward network is added to the entity adjustment features as the output of the diffusion model submodule. The residual module can alleviate the gradient vanishing problem and accelerate the training and convergence of the model.

[0138] Diffusion model iteration is the core step of image generation. Through multiple iterations, noise is gradually removed, and a new image that meets the text description and image features is generated. In each submodule, through self-attention, cross-attention, and feedforward network operations, text and image features can be effectively fused to improve the quality of the generated image.

[0139] 9. Generating an image

[0140] After multiple iterations of the diffusion model, noise is gradually removed, and a new image is generated based on the entities and background in the image, and adjusted in combination with the text description.

[0141] According to the technical scheme provided by the embodiments of the present disclosure, through segmentation, feature extraction, fusion, and diffusion model iteration, a new image that meets the text description and is related to the original image is generated. The routing correlation fusion module can effectively fuse image and text features, improving the quality and accuracy of the generated image; based on the stable diffusion model, the U-shaped network structure is innovatively designed for the diffusion model submodule architecture; global information is obtained by sequentially introducing self-attention, multi-modal features are integrated through text, entity, and background cross-attention, and the output is optimized by the residual structure after transformation by the feedforward network. The improved model can make more full use of multi-modal features, improving the matching degree of the generated image with the task description text, the entity of the image to be processed, and the background.

[0142] Figure 4 Figure 1 is a schematic diagram of a processing flow of a route relevance fusion module provided by an embodiment of the present disclosure.

[0143] As shown in Figure 4 , the processing flow of the route relevance fusion module includes:

[0144] 1) Feature mapping: input the entity image features into a multi-layer perception 1 (MLP1) to obtain feature 1; input the text features into a multi-layer perception 2 (MLP2) to obtain feature 2. The MLPs are used to map the features into the same feature space for subsequent fusion operations.

[0145] 2) Feature multiplication: multiply the feature 1 and the feature 2 to obtain feature 3. Through feature multiplication, the correlation and interaction information between the features can be captured.

[0146] 3) Mask cross-attention: input the feature 1 as a query (Q), the feature 2 as a key (K) and a value (V), and the feature 3 into a Mask cross-attention module together to obtain feature 4. The Mask cross-attention module can perform weighted summation on the features according to the correlation between the features to obtain more representative features.

[0147] 4) Feature addition: add the entity image features and the feature 4 to obtain feature 5. Through feature addition, the original entity image features can be retained, and the information obtained by the cross-attention module can be fused.

[0148] 5) Feature adjustment: input the feature 3 into a Sigmoid function, multiply the obtained result with the feature 5, and then pass through a multi-layer perception 3 (MLP3) to obtain the adjusted entity features. The Sigmoid function is used to normalize the feature 3 to control the weight of the feature. The MLP3 is used to further adjust and convert the feature.

[0149] According to the technical solution provided by the embodiment of the present disclosure, by mapping the image and text features through different MLPs respectively, multiplying the features, and then using the Mask cross-attention mechanism, and finally through feature addition, Sigmoid normalization and MLP adjustment, deep fusion is realized. Compared with the traditional splicing or element-by-element addition, this module can accurately capture the feature correlation and interaction information, and improve the semantic expression of the fused features, providing support for subsequent image feature adjustment.

[0150] The following is an apparatus embodiment of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0151] Figure 5is a schematic diagram of an image generation apparatus provided by an embodiment of the present disclosure. As shown in the figure, the image generation apparatus comprises: Figure 5

[0152] The acquisition module 501 is configured to acquire a to-be-processed image and a task description text based on which image generation is performed.

[0153] The first processing module 502 is configured to perform feature extraction processing on the to-be-processed image and the task description text to obtain image features and text features.

[0154] The second processing module 503 is configured to perform route correlation fusion on the image features and the text features to obtain image adjustment features.

[0155] The third processing module 504 is configured to generate a target image based on the image adjustment features and the text features.

[0156] According to the technical scheme provided by the embodiment of the present disclosure, by acquiring a to-be-processed image and a task description text, the to-be-processed image is processed by an image encoder and a feature optimizer, and the task description is processed by a text encoder; the image features and the text features can be fused in route correlation, which includes but is not limited to nonlinear mapping, multiplication operation, attention mechanism and normalization processing, and the image adjustment features are output; based on the image adjustment features and the text features, a diffusion model is used to combine text cross-attention and entity background adjustment calculation to gradually generate a target image, thereby enhancing the semantic consistency of the generated image and the text description, and improving the matching degree, visual quality and overall reliability of the image entity and background.

[0157] In some embodiments, the second processing module 503 is specifically configured to perform feature mapping processing on the text features to obtain text isometric features; perform feature mapping processing on the entity image features and the background image features based on the text isometric features to obtain entity image isometric features and background image isometric features; perform feature fusion processing on the entity image isometric features and the background image isometric features with the text isometric features respectively to obtain entity image fusion features and background image fusion features; and perform feature adjustment processing on the entity image fusion features and the background image fusion features respectively to obtain the image adjustment features.

[0158] In some embodiments, the first processing module 502 is specifically configured to perform feature extraction on the task description text to obtain text features; perform image entity segmentation processing on the to-be-processed image to obtain an entity image corresponding to the to-be-processed image and a background image corresponding to the to-be-processed image; and perform feature extraction processing on the entity image and the background image based on the text features to obtain entity image features and background image features.

[0159] ​In some embodiments, performing image entity segmentation on the image to be processed to obtain a background image corresponding to the image to be processed specifically involves performing image entity segmentation on the image to be processed to obtain an entity image and a background image with the subject removed; performing background mask generation on the entity image to obtain a background mask; and performing image completion processing on the background image with the subject removed and the background mask to obtain the background image.

[0160] In some embodiments, the third processing module 504 is specifically used to perform global feature enhancement processing on the noise map to obtain the global features corresponding to the noise map; and to determine the target image based on the text features, image adjustment features and global features.

[0161] In some embodiments, performing image entity segmentation on the image to be processed to obtain an entity image and a background image corresponding to the image to be processed is specifically used for: performing object detection processing on the image to be processed to obtain an object to be identified and its corresponding annotation data; performing segmentation processing on the object to be identified based on the annotation data to obtain a segmentation mask corresponding to the object; and determining the entity image and background image based on the segmentation mask corresponding to the object.

[0162] In some embodiments, the image generation apparatus is further configured to perform noise value generation processing on the blank image to obtain a noise value corresponding to the blank image; and perform value filling processing on the blank image based on the noise value to obtain a noise map.

[0163] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0164] Figure 6 This is a schematic diagram of the electronic device 6 provided in an embodiment of this disclosure. Figure 6 As shown, the electronic device 6 of this embodiment includes a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, it implements the steps in the various method embodiments described above. Alternatively, when the processor 601 executes the computer program 603, it implements the functions of each module / unit in the various device embodiments described above.

[0165] Electronic device 6 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 6 may include, but is not limited to, processor 601 and memory 602. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 6 and does not constitute a limitation on electronic device 6. It may include more or fewer components than shown, or different components.

[0166] The processor 601 can be a central processing unit (CPU), or can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0167] The memory 602 can be an internal storage unit of the electronic device 6, for example, a hard disk or a memory of the electronic device 6. The memory 602 can also be an external storage device of the electronic device 6, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. The memory 602 can also include both the internal storage unit and the external storage device of the electronic device 6. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit or module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0169] The integrated modules / units, if implemented in the form of software functional units and sold or used as independent products, can be stored in a readable storage medium (for example, a computer readable storage medium). Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program can include computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable storage medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0170] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure.

Claims

1. An image generation method, characterized in that, include: Acquire the image to be processed and the task description text for image generation based on the image to be processed; Feature extraction processing is performed on the image to be processed and the task description text to obtain image features and text features; The image features and text features are fused based on routing relevance to obtain image adjustment features; The target image is generated based on the image adjustment features and the text features.

2. The image generation method according to claim 1, characterized in that, The image features include entity image features and background image features; The step of fusing the image features and text features based on routing relevance to obtain image adjustment features includes: The text features are subjected to feature mapping processing to obtain text features of the same dimension; Based on the same-dimensional text features, feature mapping processing is performed on the entity image features and the background image features respectively to obtain same-dimensional entity image features and same-dimensional background image features. The same-dimensional features of the entity image and the same-dimensional features of the background image are respectively fused with the same-dimensional features of the text to obtain entity image fusion features and background image fusion features; The entity image fusion features and the background image fusion features are respectively subjected to feature adjustment processing to obtain the image adjustment features.

3. The image generation method according to claim 1, characterized in that, The image features include entity image features and background image features; The step of performing feature extraction processing on the image to be processed and the task description text to obtain image features and text features includes: Feature extraction is performed on the task description text to obtain the text features; The image to be processed is subjected to image entity segmentation processing to obtain the entity image corresponding to the image to be processed and the background image corresponding to the image to be processed. Based on the text features, feature extraction processing is performed on the entity image and the background image to obtain the entity image features and the background image features.

4. The image generation method according to claim 3, characterized in that, The step of performing image entity segmentation on the image to be processed to obtain the background image corresponding to the image to be processed includes: The image to be processed is subjected to image entity segmentation processing to obtain the entity image and the background-removed image corresponding to the image to be processed; The entity image is subjected to background mask generation processing to obtain a background mask; The background image is obtained by performing image completion processing on the background image with the removed subject and the background mask.

5. The image generation method according to claim 1, characterized in that, The step of generating a target image based on the image adjustment features and the text features includes: Global feature enhancement processing is performed on the noise map to obtain the global features corresponding to the noise map; The target image is determined based on the text features, the image adjustment features, and the global features.

6. The image generation method according to claim 3, characterized in that, The step of performing image entity segmentation processing on the image to be processed to obtain the entity image corresponding to the image to be processed and the background image corresponding to the image to be processed includes: The image to be processed is subjected to target detection processing to obtain the identified object and the corresponding annotation data of the identified object; Based on the labeled data corresponding to the identified object, the identified object is segmented to obtain the segmentation mask corresponding to the identified object; The entity image and the background image are determined based on the segmentation mask corresponding to the identified object.

7. The image generation method according to claim 5, characterized in that, Before performing global feature enhancement processing on the noise map to obtain the global features corresponding to the noise map, the method further includes: The blank image is subjected to noise value generation processing to obtain the noise value corresponding to the blank image; Based on the noise value, the blank image is filled with values ​​to obtain the noise map.

8. An image generation apparatus, characterized in that, include: The acquisition module is used to acquire the image to be processed and the task description text for image generation based on the image to be processed; The first processing module is used to perform feature extraction processing on the image to be processed and the task description text to obtain image features and text features; The second processing module is used to perform routing correlation fusion on the image features and the text features to obtain image adjustment features; The third processing module is used to generate a target image based on the image adjustment features and the text features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images