Data generation method and device, electronic equipment and storage medium

Abstract

The invention relates to a data generation method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: determining a data generation model, wherein the data generation model comprises a text processing network and an image processing network; performing feature extraction on the first reference text through a text processing network to obtain a first text feature; performing denoising on the first image feature by taking the first text feature as guidance through an image processing network to obtain a first intermediate feature, performing nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, and performing denoising on the first intermediate feature by taking the first text feature and the second intermediate feature as guidance to obtain a second image feature; and generating a target image based on the second image feature through the image processing network, the target image conforming to the description of the first reference text. According to the invention, the understanding capability and the generation capability of the multi-modal features can be improved in the data generation scene, and the accuracy of the generated target image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data generation method, device, electronic device, and storage medium. Background Art

[0002] With the development of computer technology, the ways of processing images are becoming more and more diverse. Currently, we can intelligently generate images or perform semantic understanding of images based on artificial intelligence technology. For example, we can generate images from text or text from images, which requires processing multimodal text features and image features.

[0003] In related technologies, in image generation tasks, dedicated processing networks are designed for text features and image features of different modalities, and two parallel processing networks are used to process text features and image features.

[0004] However, in this way, the separated architecture makes it difficult to achieve the fusion and synergy between cross-modal features, resulting in low feature perception and image generation capabilities during the image generation process, which in turn affects the accuracy of the generated images. Summary of the Invention

[0005] The present disclosure provides a data generation method, device, electronic device and storage medium, which can improve the accuracy of data generation.

[0006] According to one aspect of an embodiment of the present disclosure, a data generation method is provided, the method comprising: Determining a data generation model, wherein the data generation model includes a text processing network and an image processing network; Performing feature extraction on a first reference text through the text processing network to obtain a first text feature, where the first reference text is used to describe the image to be generated; Using the image processing network, using the first text feature as a guide, denoising the first image feature to obtain a first intermediate feature, performing a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, and using the first text feature and the second intermediate feature as a guide, denoising the first intermediate feature to obtain a second image feature; wherein the first image feature is any image feature containing noise, and the second intermediate feature is a feature obtained by enhancing the first intermediate feature; A target image is generated based on the second image features through the image processing network, and the target image is consistent with the description of the first reference text.

[0007] Optionally, the process of acquiring the first image feature includes: performing feature extraction on a first reference image through the image processing network to obtain a first initial image feature, adding a noise feature to the first initial image feature to obtain the first image feature, wherein the first reference image includes a reference object, the object in the target image matches the reference object, and the noise feature is used to characterize random noise; or The noise feature is determined as the first image feature.

[0008] Optionally, extracting features from the first reference text using the text processing network to obtain first text features includes: encoding the first reference text into a first initial text feature through a text encoding layer in the text processing network; The first initial text feature is causally encoded by a text feature processing layer in the text processing network to obtain the first text feature.

[0009] Optionally, performing causal attention encoding on the first initial text feature through a text feature processing layer in the text processing network to obtain the first text feature includes: Mapping the first initial text feature into a first query feature, a first key feature, and a first value feature through the text feature processing layer; By the text feature processing layer, masking is performed on the first query feature, the product of the transpose of the first key feature and the scaling factor based on a causal mask feature to obtain a first weight feature, wherein the first reference text includes n characters, and the causal mask feature is used to mask features of characters from i+1th to nth when determining the weight feature of the i-th character, where n is a positive integer and i is a positive integer not greater than n; The first weight feature is multiplied by the first value feature through the text feature processing layer to obtain the first text feature.

[0010] Optionally, the performing denoising on the first image feature using the image processing network and guided by the first text feature to obtain the first intermediate feature includes: Performing a nonlinear transformation on the first image feature through the image processing network to obtain a first enhanced feature, where the first enhanced feature is a feature after the first image feature is enhanced; The first image feature is denoised using the image processing network and guided by the first text feature and the first enhanced feature to obtain the first intermediate feature.

[0011] Optionally, the performing denoising on the first image feature using the image processing network and guided by the first text feature and the first enhanced feature to obtain the first intermediate feature includes: The first text feature, the first enhanced feature, and the first image feature are cross-attention encoded through the image feature processing layer in the image processing network to obtain the first intermediate feature.

[0012] Optionally, performing cross-attention encoding on the first text feature, the first enhanced feature, and the first image feature through an image feature processing layer in the image processing network to obtain the first intermediate feature includes: By means of the image feature processing layer, the first text feature is mapped into a first query sub-feature, a first key sub-feature, and a first value sub-feature, the first image feature is mapped into a second query sub-feature, and the first enhancement feature is mapped into a second key sub-feature and a second value sub-feature; The first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature are cross-fused through the image feature processing layer to obtain the first intermediate feature.

[0013] Optionally, cross-fusing the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature through the image feature processing layer to obtain the first intermediate feature includes: splicing the first query sub-feature and the second query sub-feature through the image feature processing layer to obtain a second query feature, splicing the first key sub-feature and the second key sub-feature to obtain a second key feature, and splicing the first value sub-feature and the second value sub-feature to obtain a second value feature; The image feature processing layer normalizes the product of the transpose of the second query feature and the second key feature and a scaling factor to obtain a second weight feature, and the second weight feature is multiplied by the second value feature to obtain the first intermediate feature.

[0014] Optionally, performing denoising on the first intermediate feature using the image processing network and the first text feature and the second intermediate feature as a guide to obtain the second image feature includes: The first text feature, the second intermediate feature, and the first intermediate feature are cross-attention encoded through the image feature processing layer in the image processing network to obtain the second image feature.

[0015] Optionally, performing cross-attention encoding on the first text feature, the second intermediate feature, and the first intermediate feature through an image feature processing layer in the image processing network to obtain the second image feature includes: Mapping the first text feature into a third query sub-feature, a third key sub-feature, and a third value sub-feature, mapping the first intermediate feature into a fourth query sub-feature, and mapping the second intermediate feature into a fourth key sub-feature and a fourth value sub-feature through the image feature processing layer; The third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature are cross-fused through the image feature processing layer to obtain the second image feature.

[0016] Optionally, cross-fusing the third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature, and the fourth value sub-feature through the image feature processing layer to obtain the second image feature includes: splicing the third query sub-feature and the fourth query sub-feature through the image feature processing layer to obtain a third query feature, splicing the third key sub-feature and the fourth key sub-feature to obtain a third key feature, and splicing the third value sub-feature and the fourth value sub-feature to obtain a third value feature; The image feature processing layer normalizes the product of the third query feature, the transpose of the third key feature, and the scaling factor to obtain a third weight feature, and the third weight feature is multiplied by the third value feature to obtain the second image feature.

[0017] Optionally, the image processing network includes N diffusion subnetworks, where N is an integer greater than 1; and the denoising of the first image feature using the image processing network and the first text feature as a guide to obtain the first intermediate feature includes: Using a first diffusion subnetwork in the image processing network, guided by the first text feature, denoising the first image feature to obtain a first first intermediate feature; The performing a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, and performing denoising on the first intermediate feature using the first text feature and the second intermediate feature as a guide to obtain a second image feature, includes: Performing a nonlinear transformation on the k-1th first intermediate feature through the k-th diffusion subnetwork in the image processing network to obtain the k-1th second intermediate feature; Through the kth diffusion subnetwork, guided by the first text feature and the k-1th second intermediate feature, the k-1th first intermediate feature is denoised to obtain the kth first intermediate feature; wherein k is an integer greater than 1 and not greater than N, and the second image feature is the Nth first intermediate feature.

[0018] Optionally, the method further includes: Acquire a second reference text and a second reference image, where the second reference text is used to describe the text to be generated; Performing feature extraction on the second reference text through the text processing network to obtain second text features; performing feature extraction on the second reference image through the image processing network to obtain a second initial image feature, adding a noise feature to the second initial image feature to obtain a third image feature, and denoising the third image feature using the second text feature as a guide to obtain a target intermediate feature, wherein the noise feature is used to characterize random noise; A target text is generated based on the target intermediate features through the text processing network, wherein the target text is consistent with the description of the second reference text, and the content described in the target text is consistent with the content of the second reference image.

[0019] Optionally, denoising the third image feature using the image processing network and the second text feature as a guide to obtain a target intermediate feature includes: Denoising the third image feature using the image processing network and the second text feature as a guide to obtain a third intermediate feature; The third intermediate feature is subjected to a nonlinear transformation through the image processing network to obtain the target intermediate feature, which is an enhanced feature of the third intermediate feature.

[0020] Optionally, generating a target text based on the target intermediate features through the text processing network includes: fusing the target intermediate feature and the second initial text feature through the text processing network to obtain a target text feature, where the second initial text feature is a feature obtained by encoding the second reference text; The target text is generated based on the target text features through the text processing network.

[0021] Optionally, the training process of the data generation model includes: Acquire a first sample reference text and a first sample reference image, where the first sample reference text is used to describe the first sample reference image; Performing feature extraction on the first sample reference text through the text processing network to obtain first sample text features; Performing feature extraction on the first sample reference image through the image processing network to obtain a first sample initial image feature, adding a noise feature to the first sample initial image feature to obtain a first sample image feature, denoising the first sample image feature using the first sample text feature as a guide to obtain a first sample intermediate feature, performing a nonlinear transformation on the first sample intermediate feature to obtain a second sample intermediate feature, denoising the first sample intermediate feature using the first sample text feature and the second sample intermediate feature as a guide to obtain a second sample image feature, wherein the noise feature is used to characterize random noise; The data generation model is trained based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature.

[0022] Optionally, the method further includes: Acquire a second sample reference text, a second sample reference image, and a target reference text, wherein the second sample reference text is used to describe the target reference text, and the target reference text is used to describe the second sample reference image; Performing feature extraction on the second sample reference text through the text processing network to obtain second sample text features; Performing feature extraction on the second sample reference image through the image processing network to obtain a second sample initial image feature, adding the noise feature to the second sample initial image feature to obtain a third sample image feature, and denoising the third sample image feature using the second sample text feature as a guide to obtain a target sample intermediate feature; Generate predicted text based on the intermediate features of the target sample through the text processing network; The training of the data generation model based on the at least one feature obtained by denoising, the first sample initial image feature, and the noise feature includes: Determining a first loss parameter based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature; determining a second loss parameter based on the predicted text and the target reference text; The data generation model is trained based on the first loss parameter and the second loss parameter.

[0023] According to another aspect of the embodiments of the present disclosure, there is provided a data generating device, the device comprising: A data preparation unit is configured to execute determination of a data generation model, wherein the data generation model includes a text processing network and an image processing network; a text processing unit configured to perform feature extraction on a first reference text through the text processing network to obtain a first text feature, wherein the first reference text is used to describe an image to be generated; The image processing unit is configured to perform, through the image processing network, denoising the first image feature using the first text feature as a guide to obtain a first intermediate feature, performing a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, and denoising the first intermediate feature using the first text feature and the second intermediate feature as a guide to obtain a second image feature; wherein the first image feature is any image feature containing noise, and the second intermediate feature is a feature obtained by enhancing the first intermediate feature; The image processing unit is further configured to generate a target image based on the second image feature through the image processing network, where the target image is consistent with the description of the first reference text.

[0024] Optionally, the image processing unit is further configured to perform: performing feature extraction on a first reference image through the image processing network to obtain a first initial image feature, adding a noise feature to the first initial image feature to obtain the first image feature, wherein the first reference image includes a reference object, the object in the target image matches the reference object, and the noise feature is used to characterize random noise; or The noise feature is determined as the first image feature.

[0025] Optionally, the text processing unit is configured to execute: encoding the first reference text into a first initial text feature through a text encoding layer in the text processing network; The first initial text feature is causally encoded by a text feature processing layer in the text processing network to obtain the first text feature.

[0026] Optionally, the text processing unit is configured to execute: Mapping the first initial text feature into a first query feature, a first key feature, and a first value feature through the text feature processing layer; By the text feature processing layer, masking is performed on the first query feature, the product of the transpose of the first key feature and the scaling factor based on a causal mask feature to obtain a first weight feature, wherein the first reference text includes n characters, and the causal mask feature is used to mask features of characters from i+1th to nth when determining the weight feature of the i-th character, where n is a positive integer and i is a positive integer not greater than n; The first weight feature is multiplied by the first value feature through the text feature processing layer to obtain the first text feature.

[0027] Optionally, the image processing unit is configured to perform: Performing a nonlinear transformation on the first image feature through the image processing network to obtain a first enhanced feature, where the first enhanced feature is a feature after the first image feature is enhanced; The first image feature is denoised using the image processing network and guided by the first text feature and the first enhanced feature to obtain the first intermediate feature.

[0028] Optionally, the image processing unit is configured to perform: The first text feature, the first enhanced feature, and the first image feature are cross-attention encoded through the image feature processing layer in the image processing network to obtain the first intermediate feature.

[0029] Optionally, the image processing unit is configured to perform: By means of the image feature processing layer, the first text feature is mapped into a first query sub-feature, a first key sub-feature, and a first value sub-feature, the first image feature is mapped into a second query sub-feature, and the first enhancement feature is mapped into a second key sub-feature and a second value sub-feature; The first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature are cross-fused through the image feature processing layer to obtain the first intermediate feature.

[0030] Optionally, the image processing unit is configured to perform: splicing the first query sub-feature and the second query sub-feature through the image feature processing layer to obtain a second query feature, splicing the first key sub-feature and the second key sub-feature to obtain a second key feature, and splicing the first value sub-feature and the second value sub-feature to obtain a second value feature; The image feature processing layer normalizes the product of the transpose of the second query feature and the second key feature and a scaling factor to obtain a second weight feature, and the second weight feature is multiplied by the second value feature to obtain the first intermediate feature.

[0031] Optionally, the image processing unit is configured to perform: The first text feature, the second intermediate feature, and the first intermediate feature are cross-attention encoded through the image feature processing layer in the image processing network to obtain the second image feature.

[0032] Optionally, the image processing unit is configured to perform: Mapping the first text feature into a third query sub-feature, a third key sub-feature, and a third value sub-feature, mapping the first intermediate feature into a fourth query sub-feature, and mapping the second intermediate feature into a fourth key sub-feature and a fourth value sub-feature through the image feature processing layer; The third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature are cross-fused through the image feature processing layer to obtain the second image feature.

[0033] Optionally, the image processing unit is configured to perform: splicing the third query sub-feature and the fourth query sub-feature through the image feature processing layer to obtain a third query feature, splicing the third key sub-feature and the fourth key sub-feature to obtain a third key feature, and splicing the third value sub-feature and the fourth value sub-feature to obtain a third value feature; The image feature processing layer normalizes the product of the third query feature, the transpose of the third key feature, and the scaling factor to obtain a third weight feature, and the third weight feature is multiplied by the third value feature to obtain the second image feature.

[0034] Optionally, the image processing network includes N diffusion subnetworks, where N is an integer greater than 1; and the image processing unit is configured to execute: Using a first diffusion subnetwork in the image processing network, guided by the first text feature, denoising the first image feature to obtain a first first intermediate feature; Performing a nonlinear transformation on the k-1th first intermediate feature through the k-th diffusion subnetwork in the image processing network to obtain the k-1th second intermediate feature; Through the kth diffusion subnetwork, guided by the first text feature and the k-1th second intermediate feature, the k-1th first intermediate feature is denoised to obtain the kth first intermediate feature; wherein k is an integer greater than 1 and not greater than N, and the second image feature is the Nth first intermediate feature.

[0035] Optionally, the device further comprises: The data preparation unit is further configured to acquire a second reference text and a second reference image, where the second reference text is used to describe the text to be generated; The text processing unit is further configured to perform feature extraction on the second reference text through the text processing network to obtain second text features; The image processing unit is further configured to perform feature extraction on the second reference image through the image processing network to obtain a second initial image feature, add a noise feature to the second initial image feature to obtain a third image feature, and denoise the third image feature using the second text feature as a guide to obtain a target intermediate feature, wherein the noise feature is used to characterize random noise; The text processing unit is further configured to generate a target text based on the target intermediate features through the text processing network, wherein the target text is consistent with the description of the second reference text, and the content described in the target text is consistent with the content of the second reference image.

[0036] Optionally, the image processing unit is configured to perform: Denoising the third image feature using the image processing network and the second text feature as a guide to obtain a third intermediate feature; The third intermediate feature is subjected to a nonlinear transformation through the image processing network to obtain the target intermediate feature, which is an enhanced feature of the third intermediate feature.

[0037] Optionally, the text processing unit is configured to execute: fusing the target intermediate feature and the second initial text feature through the text processing network to obtain a target text feature, where the second initial text feature is a feature obtained by encoding the second reference text; The target text is generated based on the target text features through the text processing network.

[0038] Optionally, the apparatus further comprises a model training unit configured to execute: Acquire a first sample reference text and a first sample reference image, where the first sample reference text is used to describe the first sample reference image; Performing feature extraction on the first sample reference text through the text processing network to obtain first sample text features; Performing feature extraction on the first sample reference image through the image processing network to obtain a first sample initial image feature, adding a noise feature to the first sample initial image feature to obtain a first sample image feature, denoising the first sample image feature using the first sample text feature as a guide to obtain a first sample intermediate feature, performing a nonlinear transformation on the first sample intermediate feature to obtain a second sample intermediate feature, denoising the first sample intermediate feature using the first sample text feature and the second sample intermediate feature as a guide to obtain a second sample image feature, wherein the noise feature is used to characterize random noise; The data generation model is trained based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature.

[0039] Optionally, the model training unit is further configured to execute: Acquire a second sample reference text, a second sample reference image, and a target reference text, wherein the second sample reference text is used to describe the target reference text, and the target reference text is used to describe the second sample reference image; Performing feature extraction on the second sample reference text through the text processing network to obtain second sample text features; Performing feature extraction on the second sample reference image through the image processing network to obtain a second sample initial image feature, adding the noise feature to the second sample initial image feature to obtain a third sample image feature, and denoising the third sample image feature using the second sample text feature as a guide to obtain a target sample intermediate feature; Generate predicted text based on the intermediate features of the target sample through the text processing network; Determining a first loss parameter based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature; determining a second loss parameter based on the predicted text and the target reference text; The data generation model is trained based on the first loss parameter and the second loss parameter.

[0040] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device including: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data generation method as described in the above aspects.

[0041] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the data generation method as described in the above aspects.

[0042] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the data generating method as described in the above aspects is implemented.

[0043] The method provided by the embodiment of the present disclosure processes multimodal text features and image features in a data generation scenario, uses the text features of the reference text as a condition to guide the denoising process of the image features, and the intermediate features generated in the denoising process are also used as a condition to guide the denoising process of the image features after feature enhancement, thereby achieving unified semantic understanding based on multimodal features, which is conducive to promoting the fusion and synergy of cross-modal features. In addition, feature enhancement belongs to the perception task, and feature denoising belongs to the generation task. In the perception task, the intermediate features generated in the generation task are enhanced, and in turn, the enhanced features in the perception task are used as a condition to guide the generation task, thereby achieving deep fusion between the perception task and the generation task. Therefore, the processing method of the present disclosure can improve the understanding and generation capabilities of multimodal features in the data generation scenario, thereby improving the accuracy of the target image finally generated.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0046] Figure 1 is a schematic diagram of a computer system according to an exemplary embodiment; Figure 2 is a structural diagram of a data generation model according to an exemplary embodiment; Figure 3 is a flow chart showing a data generation method according to an exemplary embodiment; Figure 4 is a flow chart showing another data generating method according to an exemplary embodiment; Figure 5 is a flow chart showing another data generating method according to an exemplary embodiment; Figure 6is a flowchart of a text generation method according to an exemplary embodiment; Figure 7 is a flowchart of a method for training a data generation model according to an exemplary embodiment; Figure 8 is a flow chart of yet another data generating method according to an exemplary embodiment; Figure 9 is a block diagram of a data generating device according to an exemplary embodiment; Figure 10 is a block diagram showing another data generating device according to an exemplary embodiment; Figure 11 is a block diagram of a terminal according to an exemplary embodiment; Figure 12 The figure is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0047] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0048] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0049] It should be noted that the user information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, and display), and signals involved in this disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the text and images involved in this disclosure were obtained with full authorization.

[0050] The disclosed embodiments provide a data generation method, the execution subject of which is an electronic device. Exemplarily, the electronic device is a terminal, which is a computer, a mobile phone, a tablet computer, or other terminals. Exemplarily, the electronic device is a server, which is a backend server or a cloud server that provides cloud computing and cloud storage services. The electronic device can generate a target image based on a reference text, or generate a target image based on a reference text and a reference image, or generate a target text based on a reference text and a reference image, etc. through a data generation model. The data generation model can be used to process any image-text related data generation task.

[0051] Figure 1 is a schematic diagram of a computer system according to an exemplary embodiment. Figure 1 , the computer system includes: a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0052] Terminal 101 has a client installed and running. This client can be a social application client, an online payment client, an online shopping client, a game client, a medical service client, a video client, or other types of clients. When terminal 101 runs the client, the client's user interface is displayed on the screen of terminal 101. Terminal 101 is the terminal used by the user.

[0053] Optionally, terminal 101 may generally refer to one of multiple terminals. Those skilled in the art will appreciate that the number of terminals may be greater or lesser. For example, there may be only one terminal, or there may be six, eight, or more terminals. The embodiments of this disclosure do not limit the number of terminals or device types.

[0054] Server 102 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 102 provides backend services for clients. Optionally, server 102 performs primary computing tasks, while terminal 101 performs secondary computing tasks; alternatively, server 102 performs secondary computing tasks, while terminal 101 performs primary computing tasks; alternatively, server 102 and terminal 101 utilize a distributed computing architecture for collaborative computing.

[0055] In one embodiment, terminal 101 sends an image generation request carrying reference text to server 102. Upon receiving the image generation request, server 102 obtains the reference text. Based on the reference text, server 102 generates a target image that matches the reference text using a data generation model. Server 102 then returns the target image to terminal 101, which can then display the target image to the user.

[0056] In another embodiment, the server 102 trains a data generation model and sends the data generation model to the terminal 101. The terminal 101 generates a target image that matches the reference text through the data generation model.

[0057] It should be noted that the above implementation environment is only an example. The method provided in the embodiment of the present disclosure can also be executed by the terminal 101 or the server 102 alone, or by other computer devices, and the embodiment of the present disclosure does not limit this.

[0058] Figure 2 is a structural diagram of a data generation model provided by an embodiment of the present disclosure, such as Figure 2 As shown, the data generation model includes a text processing network 201 and an image processing network 202, wherein the text processing network 201 is connected to the image processing network 202. In the image generation task, the output data of the text processing network 201 is used as the input data of the image processing network 202. In the image understanding task, the output data of the image processing network 202 is used as the input data of the text processing network 201.

[0059] In some embodiments, as Figure 2 As shown, the text processing network 201 includes a text encoding layer, a text feature processing layer, and a text decoding layer. The text encoding layer is connected to the text feature processing layer, and the output data of the text encoding layer serves as the input data of the text feature processing layer. The text feature processing layer is connected to the text decoding layer. The output data of the text feature processing layer serves as the input data of the text decoding layer.

[0060] The text encoding layer is used to encode text into text features, which can be embedded text vectors. The text feature processing layer is used to process text features to mine deeper text features. The text decoding layer is used to decode text features into text.

[0061] Optionally, the text processing network 201 is constructed based on the Transformer decoder architecture. For example, the Transformer decoder can be a decoder-only network. Exemplarily, the text feature processing layer in the text processing network 201 includes the causal attention layer, normalization layer, and feedforward layer in the decoder-only network. For example, the text feature processing layer includes a causal attention layer, a first normalization layer, a feedforward layer, and a second normalization layer connected in sequence.

[0062] In some embodiments, as Figure 2As shown, the image processing network 202 includes an image coding layer, N sequentially connected diffusion sub-models, and an image decoding layer. The image coding layer is connected to the first diffusion sub-model, and the output data of the image coding layer serves as the input data of the first diffusion sub-model. The image decoding layer is connected to the last diffusion sub-model, and the output data of the last diffusion sub-model serves as the input data of the image decoding layer.

[0063] The image encoding layer is used to encode the image into image features, the multiple diffusion sub-models are used to denoise the image features, and the image decoding layer is used to decode the image features into an image.

[0064] Alternatively, as Figure 2 As shown, each diffusion sub-model includes a perception layer and an image feature processing layer, and the perception layer and the image feature processing layer are connected. The input data of the perception layer of the kth diffusion sub-model is the output data of the image feature processing layer of the k-1th diffusion sub-model. The input data of the image feature processing layer of the kth diffusion sub-model includes the output data of the text feature processing layer of the text processing network 201, the output data of the perception layer of the kth diffusion sub-model, and the output data of the image feature processing layer of the k-1th diffusion sub-model. k is an integer greater than 1 and not greater than N.

[0065] Optionally, the image processing network 202 is constructed based on a Diffusion Transformer (DiT) architecture. Exemplarily, the perception layer in the diffusion sub-model includes a multi-layer perceptron in the DiT model, and the image feature processing layer in the diffusion sub-model includes a normalization layer, a linear transformation layer, a cross-attention layer, and a scale transformation layer in the DiT model.

[0066] The following embodiments are combined Figure 2 The data generation model shown can provide a detailed introduction to the data generation method. Figure 3 is a flow chart showing a data generation method according to an exemplary embodiment. Figure 2 The method is executed by an electronic device and includes the following steps.

[0067] In step 301, the electronic device determines a data generation model, where the data generation model includes a text processing network and an image processing network.

[0068] In the embodiment of the present disclosure, the data generation model includes a text processing network and an image processing network, which are interconnected. The data generation model can be used to generate images based on text or to generate text by understanding images. The structure of the data generation model can be seen in Figure 2 Schematic diagram of the structure.

[0069] In step 302, the electronic device extracts features from a first reference text through a text processing network to obtain first text features. The first reference text is used to describe an image to be generated.

[0070] In an image generation task, an electronic device obtains a first reference text. The task involves generating an image that matches the description in the first reference text. Using a text processing network, the electronic device extracts features from the first reference text to obtain first text features, which represent the semantic characteristics of the first reference text.

[0071] In step 303, the electronic device uses an image processing network to denoise the first image feature with the first text feature as a guide to obtain a first intermediate feature, performs a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, and denoises the first intermediate feature with the first text feature and the second intermediate feature as a guide to obtain a second image feature.

[0072] In the image generation task, the electronic device also obtains a first image feature, which is any image feature containing noise. The electronic device uses an image processing network to denoise the first image feature, guided by the first text feature, to obtain a first intermediate feature. The first intermediate feature contains the semantic information of the first text feature and the visual information of the first image feature. Denoising a feature as described in this disclosure includes reducing the noise of the feature.

[0073] To further denoise, the electronic device performs a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature. This second intermediate feature is an enhanced version of the first intermediate feature. That is, by perceiving the first intermediate feature, a deeper and more abstract second intermediate feature is obtained. Furthermore, guided by the first text feature and the second intermediate feature, the first intermediate feature is denoised to obtain the second image feature, thereby enhancing the denoising effect of the second image feature. Furthermore, feature enhancement is a perception task, while feature denoising is a generation task. In the perception task, the intermediate features generated in the generation task are enhanced, and in turn, the enhanced features in the perception task are used as conditions to guide the generation task. This achieves a deep fusion between the perception and generation tasks, and improves the second image feature's ability to represent semantic and visual information.

[0074] In step 304, the electronic device generates a target image based on the second image feature through an image processing network, and the target image is consistent with the description of the first reference text.

[0075] After determining the second image feature, the electronic device generates a target image based on the second image feature. The target image matches the description of the first reference text. For example, if the first reference text is "Generate an image of a running kitten," the target image will contain a running kitten.

[0076] The method provided by the embodiment of the present disclosure processes multimodal text features and image features in a data generation scenario, uses the text features of the reference text as a condition to guide the denoising process of the image features, and the intermediate features generated in the denoising process are also used as a condition to guide the denoising process of the image features after feature enhancement, thereby achieving unified semantic understanding based on multimodal features, which is conducive to promoting the fusion and synergy of cross-modal features. In addition, feature enhancement belongs to the perception task, and feature denoising belongs to the generation task. In the perception task, the intermediate features generated in the generation task are enhanced, and in turn, the enhanced features in the perception task are used as a condition to guide the generation task, thereby achieving deep fusion between the perception task and the generation task. Therefore, the processing method of the present disclosure can improve the understanding and generation capabilities of multimodal features in the data generation scenario, thereby improving the accuracy of the target image finally generated.

[0077] above Figure 3 The embodiment is only a brief introduction to the data generation method. The detailed process of the data generation method is described below. Figure 4 is a flow chart showing another data generation method according to an exemplary embodiment, the method being executed by an electronic device, see Figure 4 , the method includes the following steps.

[0078] In step 401, the electronic device determines a data generation model, where the data generation model includes a text processing network and an image processing network.

[0079] In the embodiment of the present disclosure, the data generation model includes a text processing network and an image processing network, which are interconnected. The data generation model can be used to generate images based on text or generate text by understanding images.

[0080] In the scenario of generating images based on text, the text features generated by the text processing network are used to assist the image processing network in generating images. Therefore, the output data of the text processing network is the input data of the image processing network.

[0081] In the scenario of generating text by understanding images, the image features generated by the image processing network are used to assist the text processing network in generating text. Therefore, the output data of the image processing network is the input data of the text processing network.

[0082] Among them, the structure of the data generation model is as follows Figure 2 As shown, no further details are given here.

[0083] In step 402, the electronic device obtains a first image feature, where the first image feature is any image feature containing noise.

[0084] In an image generation task, the electronic device obtains a first image feature, where the first image feature is any image feature containing noise.

[0085] In some embodiments, the image generation task includes two situations: generating an image based on text and generating an image based on text and pictures. Based on this, obtaining the first image feature includes the following two ways.

[0086] (1) Through the image processing network, the first reference image is subjected to feature extraction to obtain the first initial image feature, and the noise feature is added to the first initial image feature to obtain the first image feature. The first reference image includes a reference object, and the object in the target image is consistent with the reference object. The noise feature is used to characterize random noise.

[0087] In the case of generating an image based on text and graphics, in addition to providing a first reference text, a first reference image is also provided as a reference. Therefore, the electronic device performs feature extraction on the first reference image through an image processing network to obtain a first initial image feature of the first reference image, and then adds a noise feature to the first initial image feature to obtain a first image feature.

[0088] Alternatively, as Figure 2 As shown, the image processing network includes an image coding layer. The electronic device inputs a first reference image into the image coding layer, and the image coding layer encodes the first reference image and outputs a first initial image feature.

[0089] Optionally, the noise feature may be random Gaussian noise or the like.

[0090] (2) Determine the noise feature as the first image feature.

[0091] In the case of generating an image based on text, only the first reference text is provided, and no reference image is provided. Therefore, the electronic device directly obtains the noise feature of the random noise and uses the noise feature as the first image feature.

[0092] Optionally, the noise feature may be random Gaussian noise or the like.

[0093] In the disclosed embodiments, when a reference image is available, noise features are added to the image features of the reference image as the first initial image feature, enabling the task of generating a new image from an image and text. When no reference image is available, the noise features are directly used as the first initial image feature, enabling the task of generating an image from text, thereby improving the applicability of the data generation model.

[0094] In step 403, the electronic device extracts features from the first reference text through a text processing network to obtain first text features. The first reference text is used to describe the image to be generated.

[0095] In an image generation task, an electronic device obtains a first reference text. The task involves generating an image that matches the description in the first reference text. The electronic device inputs the first reference text into a text processing network, which extracts features from the first reference text and outputs first text features.

[0096] In some embodiments, as Figure 2 As shown, the text processing network includes a text encoding layer and a text feature processing layer. Step 403 includes: encoding the first reference text into a first initial text feature through the text encoding layer in the text processing network, and performing causal attention encoding on the first initial text feature through the text feature processing layer in the text processing network to obtain a first text feature.

[0097] The electronic device inputs the first reference text into the text encoding layer. The text encoding layer encodes the first reference text and outputs a first initial text feature, for example, an embedded text feature vector. The first initial text feature output by the text encoding layer is input into the text feature extraction layer. The text feature extraction layer performs causal attention encoding on the first initial text feature and outputs a first text feature.

[0098] Optionally, the text feature extraction layer includes a causal attention layer, a first normalization layer, a forward feedback layer and a second normalization layer connected in sequence. The electronic device inputs the first initial text feature into the causal attention layer, which is processed in sequence by the causal attention layer, the first normalization layer, the forward feedback layer and the second normalization layer, and finally the second normalization layer outputs the first text feature.

[0099] Alternatively, as Figure 2 As shown, the text processing network also includes a text decoding layer, which is used to decode text features into text. However, since there is no need to generate text in the image generation task, the embodiment of the present disclosure does not need to be processed by the text decoding layer, and the first text feature output by the text feature processing layer can be directly provided to the image processing network.

[0100] In this disclosed embodiment, a text encoding layer and a text feature processing layer are introduced into the text processing network, enabling deeper semantic understanding and contextual modeling of the reference text. The text encoding layer efficiently encodes the first reference text into an embedded feature representation, while the text feature processing layer utilizes an autoregressive language processing mechanism to capture the dependencies of words in the text on preceding words, thereby improving the representational capabilities of the first text features, thereby enabling text-driven image generation and enhancing image generation accuracy.

[0101] In some embodiments, the process of causal attention encoding includes: mapping the first initial text feature into a first query feature, a first key feature and a first value feature through a text feature processing layer; masking the product of the transpose of the first query feature and the first key feature and the scaling factor based on the causal mask feature through the text feature processing layer to obtain a first weight feature, the first reference text includes n characters, and the causal mask feature is used to mask the features of the i+1th character to the nth character when determining the weight feature of the i-th character, where n is a positive integer and i is a positive integer not greater than n; multiplying the first weight feature by the first value feature through the text feature processing layer to obtain a first text feature.

[0102] The text feature processing layer includes a first conversion matrix, a second conversion matrix, and a third conversion matrix, which serve as model parameters of the text feature processing layer. The first conversion matrix is multiplied by the first initial text feature to obtain the first query feature, the second conversion matrix is multiplied by the first initial text feature to obtain the first key feature, and the third conversion matrix is multiplied by the first initial text feature to obtain the first value feature.

[0103] The causal mask matrix is an n×n matrix, where n is the number of characters in the first reference text. The causal mask matrix is a lower triangular matrix that, when processing the features of the i-th character, retains only the feature path from the 1st character to the i-th character and masks the features from the i+1th character to the n-th character. Exemplarily, the element in the i-th row and j-th column of the causal mask matrix is a valid value when j is less than or equal to i, and a mask value when j is greater than i.

[0104] For example, the element in the i-th row and j-th column of the causal mask matrix is 0 when j is less than or equal to i, and is negative infinity when j is greater than i. Then, the product of the transpose of the first query feature and the first key feature and the scaling factor is determined, the product is added to the causal mask matrix, the result of the addition is normalized to obtain a first weight feature, and the first weight feature is multiplied by the first value feature to obtain a first text feature.

[0105] In the disclosed embodiments, a causal attention mechanism is introduced into the text processing network, and subsequent characters are masked using a causal mask. This allows for autoregressive modeling during text feature extraction, ensuring the sequential dependency and semantic rationality of the extracted text features. Furthermore, by mapping initial text features to query features, key features, and value features, combined with masking, it is possible to dynamically focus on valid information in the context, highlighting the importance of the current character in semantic understanding, thereby improving the representational capabilities of text features. This is beneficial for providing more sequentially logical text guidance for subsequent image generation, and enhancing the text consistency and expression accuracy of the generated images.

[0106] In step 404, the electronic device performs a nonlinear transformation on the first image feature through an image processing network to obtain a first enhanced feature, where the first enhanced feature is a feature after the first image feature is enhanced.

[0107] The electronic device performs a nonlinear transformation on the first image feature through an image processing network to obtain a first enhanced feature. That is, the electronic device obtains a deeper and more abstract first enhanced feature by perceiving the first image feature.

[0108] In some embodiments, as Figure 2 As shown, the image processing network includes a perception layer, and the electronic device inputs the first image feature into the perception layer, and the perception layer performs a nonlinear transformation on the first image feature to obtain a first enhanced feature.

[0109] Optionally, the perception layer may be an MLP (Multilayer Perceptron).

[0110] In step 405, the electronic device uses an image processing network to denoise the first image feature with the first text feature and the first enhanced feature as guidance to obtain a first intermediate feature.

[0111] In the image processing network, the electronic device uses the first text feature and the first enhancement feature as guidance to denoise the first image feature to obtain a first intermediate feature, which is an intermediate representation after denoising.

[0112] In this disclosed embodiment, an image processing network first performs nonlinear enhancement on the first image features to extract more discriminative visual information. Then, using both textual and enhanced features as dual guides, the first image features are denoised, effectively improving the stability of the generation process and the fidelity of image details. This approach achieves a deep fusion of textual semantic information and image structural information, achieving a better balance between content accuracy and visual quality in the generated images, thereby enhancing the expressiveness of image features and the denoising effect.

[0113] In some embodiments, the electronic device performs cross-attention encoding on the first text feature, the first enhanced feature, and the first image feature through an image feature processing layer in an image processing network to obtain a first intermediate feature.

[0114] like Figure 2 As shown, the image processing network includes an image feature processing layer, the electronic device inputs the first text feature, the first enhanced feature and the first image feature into the image feature processing layer, and the image feature processing layer outputs the first intermediate feature.

[0115] Optionally, the image feature processing layer includes a normalization layer, a linear transformation layer, a cross-attention layer, and a scale transformation layer connected in sequence. The first image feature is input into the normalization layer, the first text feature and the first enhanced feature are input into the cross-attention layer for processing in sequence, and finally the scale transformation layer outputs the first intermediate feature.

[0116] In the disclosed embodiment, the text features, the first enhanced features and the first image features are cross-attention encoded through the image feature processing layer in the image processing network, which can dynamically capture the correlation between the text semantics and the visual information, and accurately adjust the denoising process of the image features, thereby generating intermediate features that are more consistent with the text description and rich in details, improving the coordination and expression ability of multimodal information, and enhancing the semantic consistency and visual effect of image generation.

[0117] Optionally, the process of cross-attention encoding includes: mapping the first text feature into a first query sub-feature, a first key sub-feature and a first value sub-feature, mapping the first image feature into a second query sub-feature, and mapping the first enhancement feature into a second key sub-feature and a second value sub-feature through an image feature processing layer; and cross-fusing the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature and the second value sub-feature through an image feature processing layer to obtain a first intermediate feature.

[0118] The image feature processing layer includes a fourth conversion matrix, a fifth conversion matrix, a sixth conversion matrix, a seventh conversion matrix, an eighth conversion matrix, and a ninth conversion matrix. The conversion matrices are model parameters of the image feature processing layer. The fourth conversion matrix is multiplied by the first text feature to obtain a first query sub-feature, the fifth conversion matrix is multiplied by the first text feature to obtain a first key sub-feature, and the sixth conversion matrix is multiplied by the first text feature to obtain a first value sub-feature. The seventh conversion matrix is multiplied by the first image feature to obtain a second query sub-feature, the eighth conversion matrix is multiplied by the first enhancement feature to obtain a second key sub-feature, and the ninth conversion matrix is multiplied by the first enhancement feature to obtain a second value sub-feature.

[0119] In the embodiment of the present disclosure, the first query sub-feature, the first key sub-feature and the first value sub-feature are obtained through text feature mapping, and the second query sub-feature, the second key sub-feature and the second value sub-feature are obtained through image feature mapping. The first query sub-feature, the first key sub-feature and the first value sub-feature as well as the second query sub-feature, the second key sub-feature and the second value sub-feature are cross-fused, which can effectively capture the correlation information and complementary features between different modalities. This refined cross-fusion enhances the ability to jointly understand text semantics and visual information, and is conducive to ensuring the generation of richer and more semantically accurate intermediate features.

[0120] Optionally, the electronic device cross-fuses the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature and the second value sub-feature, including: splicing the first query sub-feature and the second query sub-feature through the image feature processing layer to obtain the second query feature, splicing the first key sub-feature and the second key sub-feature to obtain the second key feature, and splicing the first value sub-feature and the second value sub-feature to obtain the second value feature; normalizing the product of the transpose of the second query feature and the second key feature and the scaling factor through the image feature processing layer to obtain a second weight feature, and multiplying the second weight feature by the second value feature to obtain a first intermediate feature.

[0121] Exemplarily, the electronic device uses the following formulas (1) to (3) to determine the second query feature, the second key feature, and the second value feature.

[0122] ;Formula (1) ;Formula (2) ;Formula (3) in, represents the second query feature, represents the second key feature, represents the second value feature, represents the fourth transformation matrix, represents the fifth transformation matrix, represents the sixth transformation matrix, represents the seventh transformation matrix, represents the eighth transformation matrix, represents the ninth transformation matrix, represents the first text feature, represents the first image feature, represents the first enhancement feature, Represents a splicing operation.

[0123] In the embodiment of the present disclosure, the first query sub-feature, the first key sub-feature and the first value sub-feature of the text feature are spliced with the second query sub-feature, the second key sub-feature and the second value sub-feature of the image feature to form a unified multimodal feature representation, which is conducive to enhancing the ability to capture the complex relationship between text semantics and visual information, so that the generated intermediate features have rich text semantics and retain delicate visual details, laying the foundation for high-quality, multimodal fusion image generation.

[0124] In step 406, the electronic device performs a nonlinear transformation on the first intermediate feature through an image processing network to obtain a second intermediate feature, where the second intermediate feature is an enhanced version of the first intermediate feature.

[0125] The electronic device performs a nonlinear transformation on the first intermediate feature through an image processing network to obtain a second intermediate feature. That is, by perceiving the first intermediate feature, a deeper and more abstract second intermediate feature is obtained.

[0126] In some embodiments, as Figure 2 As shown, the image processing network includes a perception layer, and the electronic device inputs the first intermediate feature into the perception layer, and the perception layer performs a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature.

[0127] In the disclosed embodiment, the intermediate features generated in the process of generative modeling (denoising and generating images) are enhanced, and the enhanced features are used as conditions to further guide the next step of denoising of the intermediate features, thereby utilizing the intermediate features generated in the process of generative modeling to enhance perception capabilities, which is conducive to achieving a deep fusion of perception capabilities and generation capabilities and improving the accuracy of denoising.

[0128] In step 407, the electronic device uses an image processing network to denoise the first intermediate feature with the first text feature and the second intermediate feature as guidance to obtain the second image feature.

[0129] In the image processing network, the electronic device uses the first text feature and the second intermediate feature as a guide to denoise the first intermediate feature to obtain a second image feature, which is the denoised image feature.

[0130] In some embodiments, the electronic device performs cross-attention encoding on the first text feature, the second intermediate feature, and the first intermediate feature through an image feature processing layer in an image processing network to obtain a second image feature.

[0131] like Figure 2 As shown, the image processing network includes an image feature processing layer, the electronic device inputs the first text feature, the second intermediate feature and the first intermediate feature into the image feature processing layer, and the image feature processing layer outputs the second image feature.

[0132] In the disclosed embodiment, the first text feature, the second intermediate feature, and the first intermediate feature are cross-attention encoded through the image feature processing layer in the image processing network, which can dynamically capture the correlation between text semantics and visual information, and accurately adjust the denoising process of the image features, thereby generating intermediate features that are more consistent with the text description and rich in details, improving the coordination and expression ability of multimodal information, and enhancing the semantic consistency and visual effect of image generation.

[0133] Optionally, the cross-attention encoding process includes: mapping the first text feature to a third query sub-feature, a third key sub-feature and a third value sub-feature through an image feature processing layer, mapping the first intermediate feature to a fourth query sub-feature, and mapping the second intermediate feature to a fourth key sub-feature and a fourth value sub-feature; and cross-fusing the third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature through an image feature processing layer to obtain a second image feature.

[0134] The image feature processing layer includes a tenth conversion matrix, an eleventh conversion matrix, a twelfth conversion matrix, a thirteenth conversion matrix, a fourteenth conversion matrix, and a fifteenth conversion matrix, and the above conversion matrices are model parameters of the image feature processing layer. The tenth conversion matrix is multiplied by the first text feature to obtain a third query sub-feature, the eleventh conversion matrix is multiplied by the first text feature to obtain a third key sub-feature, and the twelfth conversion matrix is multiplied by the first text feature to obtain a third value sub-feature. The thirteenth conversion matrix is multiplied by the first intermediate feature to obtain a fourth query sub-feature, the fourteenth conversion matrix is multiplied by the second intermediate feature to obtain a fourth key sub-feature, and the fifteenth conversion matrix is multiplied by the second intermediate feature to obtain a fourth value sub-feature.

[0135] In the embodiment of the present disclosure, the third query sub-feature, the third key sub-feature and the third value sub-feature are obtained through text feature mapping, and the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature are obtained through image feature mapping. The third query sub-feature, the third key sub-feature and the third value sub-feature as well as the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature are cross-fused, which can effectively capture the correlation information and complementary features between different modalities. This refined cross-fusion enhances the ability to jointly understand text semantics and visual information, and is conducive to ensuring the generation of richer and more semantically accurate intermediate features.

[0136] Optionally, the electronic device cross-fuses the third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature, including: through the image feature processing layer, splicing the third query sub-feature and the fourth query sub-feature to obtain a third query feature, splicing the third key sub-feature and the fourth key sub-feature to obtain a third key feature, and splicing the third value sub-feature and the fourth value sub-feature to obtain a third value feature; through the image feature processing layer, normalizing the product of the transpose of the third query feature and the third key feature and the scaling factor to obtain a third weight feature, and multiplying the third weight feature by the third value feature to obtain a second image feature.

[0137] The process of the electronic device determining the third query feature, the third key feature, and the third value feature is similar to the process of determining the second query feature, the second key feature, and the second value feature in step 405. Please refer to formulas (1) to (3) above, and no further details will be given here.

[0138] In the disclosed embodiment, the third query sub-feature, the third key sub-feature and the third value sub-feature of the text feature are spliced with the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature of the image feature to form a unified multimodal feature representation, which is conducive to enhancing the ability to capture the complex relationship between text semantics and visual information, so that the generated intermediate features have rich text semantics and retain delicate visual details, laying the foundation for high-quality, multimodal fusion image generation.

[0139] In step 408, the electronic device generates a target image based on the second image feature through an image processing network, and the target image is consistent with the description of the first reference text.

[0140] After determining the second image feature, the electronic device generates a target image based on the second image feature, where the target image matches the description of the first reference text.

[0141] In some embodiments, the first image feature is obtained by adding a noise feature to the first initial image feature of the first reference image. In this case, the target image not only matches the description of the first reference text, but also the object in the target image matches the reference object. For example, if the first reference text is "Generate an image of a running kitten" and the first reference image is an image of a kitten, then the target image will contain a running kitten, and the kitten in the target image will be the same as the kitten in the first reference image.

[0142] In some embodiments, as Figure 2 As shown, the image processing network includes an image decoding layer. The electronic device inputs the second image feature into the image decoding layer, and the image decoding layer decodes the second image feature and outputs the target image.

[0143] The method provided by the embodiment of the present disclosure processes multimodal text features and image features in a data generation scenario, uses the text features of the reference text as a condition to guide the denoising process of the image features, and the intermediate features generated in the denoising process are also used as a condition to guide the denoising process of the image features after feature enhancement, thereby achieving unified semantic understanding based on multimodal features, which is conducive to promoting the fusion and synergy of cross-modal features. In addition, feature enhancement belongs to the perception task, and feature denoising belongs to the generation task. In the perception task, the intermediate features generated in the generation task are enhanced, and in turn, the enhanced features in the perception task are used as a condition to guide the generation task, thereby achieving deep fusion between the perception task and the generation task. Therefore, the processing method of the present disclosure can improve the understanding and generation capabilities of multimodal features in the data generation scenario, thereby improving the accuracy of the target image finally generated.

[0144] Figure 5 is a flow chart showing another data generation method according to an exemplary embodiment, the method is executed by an electronic device, see Figure 5 , the method includes the following steps.

[0145] In step 501, the electronic device determines a data generation model, where the data generation model includes a text processing network and an image processing network. The image processing network includes N diffusion subnetworks and an image decoding layer, where N is an integer greater than 1.

[0146] Among them, the structure of the data generation model is as follows Figure 2 As shown, no further details are given here.

[0147] In step 502, the electronic device obtains a first image feature, where the first image feature is any image feature containing noise.

[0148] The process of obtaining the first image feature in step 502 is the same as the process of obtaining the first image feature in step 402 above, and will not be repeated here.

[0149] In step 503, the electronic device extracts features from the first reference text through a text processing network to obtain first text features. The first reference text is used to describe the image to be generated.

[0150] The process of obtaining the first text feature in step 503 is the same as the process of obtaining the first text feature in step 403 above, and will not be repeated here.

[0151] In step 504, the electronic device uses the first diffusion subnetwork in the image processing network and the first text feature as a guide to denoise the first image feature to obtain the first first intermediate feature.

[0152] The electronic device inputs the first text feature and the first image feature into the first diffusion sub-network, and the first diffusion sub-network outputs the first first intermediate feature.

[0153] In some embodiments, as Figure 2 As shown, the first diffusion subnetwork includes a perception layer and an image feature processing layer. The electronic device inputs the first image feature into the perception layer, which performs a nonlinear transformation on the first image feature. The perception layer then outputs a first enhanced feature. The first text feature, the first enhanced feature, and the first image feature are input into the image feature processing layer. The image feature processing layer, guided by the first text feature and the first enhanced feature, performs denoising on the first image feature. The image feature processing layer then outputs a first intermediate feature.

[0154] Optionally, the electronic device performs cross-attention encoding on the first text feature, the first enhanced feature, and the first image feature through the image feature processing layer to obtain a first first intermediate feature. The process of obtaining the first first intermediate feature is similar to the process of obtaining the first intermediate feature in step 405 above and is not further described here.

[0155] In step 505, the electronic device performs a nonlinear transformation on the k-1th first intermediate feature through the k-1th diffusion subnetwork in the image processing network to obtain the k-1th second intermediate feature, and denoises the k-1th first intermediate feature using the first text feature and the k-1th second intermediate feature as a guide to obtain the k-1th first intermediate feature, where k is an integer greater than 1 and not greater than N.

[0156] The computer device inputs the first text feature and the k-1th first intermediate feature into the kth diffusion sub-network, and the kth diffusion sub-network outputs the kth first intermediate feature.

[0157] In some embodiments, as Figure 2 As shown, the k-th diffusion subnetwork includes a perception layer and an image feature processing layer. The electronic device inputs the k-1th first intermediate feature into the perception layer, which performs a nonlinear transformation on the k-1th first intermediate feature. The perception layer then outputs the k-1th second intermediate feature. The first text feature, the k-1th first intermediate feature, and the k-1th second intermediate feature are then input into the image feature processing layer. The image feature processing layer, guided by the first text feature and the k-1th second intermediate feature, performs denoising on the k-1th first intermediate feature. The image feature processing layer then outputs the k-th first intermediate feature.

[0158] Optionally, the electronic device performs cross-attention encoding on the first text feature, the k-1th first intermediate feature, and the k-1th second intermediate feature through the image feature processing layer to obtain the kth first intermediate feature. The process of obtaining the kth first intermediate feature is similar to the process of obtaining the first intermediate feature in step 405 above and is not further described here.

[0159] In step 506 , the electronic device uses the Nth first intermediate feature as the second image feature.

[0160] Among them, the Nth first intermediate feature is also the first intermediate feature output by the last diffusion sub-network in the N diffusion sub-networks, and the last first intermediate feature serves as the second image feature.

[0161] In step 507, the electronic device generates a target image based on the second image feature through the image decoding layer in the image processing network, and the target image is consistent with the description of the first reference text.

[0162] The process of generating the target image in step 507 is the same as the process of generating the target image in step 408 above, and will not be repeated here.

[0163] The method provided by the embodiment of the present disclosure processes multimodal text features and image features in a data generation scenario, uses the text features of the reference text as a condition to guide the denoising process of the image features, and the intermediate features generated in the denoising process are also used as a condition to guide the denoising process of the image features after feature enhancement, thereby achieving unified semantic understanding based on multimodal features, which is conducive to promoting the fusion and synergy of cross-modal features. In addition, feature enhancement belongs to the perception task, and feature denoising belongs to the generation task. In the perception task, the intermediate features generated in the generation task are enhanced, and in turn, the enhanced features in the perception task are used as a condition to guide the generation task, thereby achieving deep fusion between the perception task and the generation task. Therefore, the processing method of the present disclosure can improve the understanding and generation capabilities of multimodal features in the data generation scenario, thereby improving the accuracy of the target image finally generated.

[0164] Furthermore, multiple diffusion subnetworks sequentially utilize text features and intermediate features from the previous layer for joint guidance, gradually refining image features and improving denoising effectiveness and feature expression capabilities. Furthermore, by enhancing the intermediate features generated by the previous diffusion subnetwork, these features in turn guide the generation process of the current diffusion subnetwork. As a result, multiple diffusion subnetworks alternately perform perception and generation tasks, achieving a deep fusion of these two tasks. Therefore, by introducing a cascaded structure of multiple diffusion subnetworks, a multi-stage iterative optimization process for image feature denoising and enhancement is achieved.

[0165] The above embodiment illustrates the process of generating a target image through a data generation model. In addition to the image generation task, the data generation model can also be used to understand the image and generate the target text. The detailed process is shown in the following Figure 6 Example of . Figure 6 is a flowchart of a text generation method according to an exemplary embodiment, the method is executed by an electronic device, see Figure 6 , the method includes the following steps.

[0166] In step 601, the electronic device obtains a second reference text and a second reference image, where the second reference text is used to describe the text to be generated.

[0167] In the task of generating text from an image, the second reference text is used to constrain the text to be generated. For example, the second reference text can be used to indicate the number of words, style, etc. of the text to be generated.

[0168] In step 602, the electronic device extracts features from the second reference text through the text processing network in the data generation model to obtain second text features.

[0169] In some embodiments, as Figure 2 As shown, the text processing network includes a text encoding layer and a text feature processing layer. Step 602 includes: encoding the second reference text into a second initial text feature through the text encoding layer in the text processing network, and performing causal attention encoding on the second initial text feature through the text feature processing layer in the text processing network to obtain a second text feature.

[0170] The process of obtaining the second text feature in step 602 is similar to the process of obtaining the first text feature in step 403 , and will not be described in detail here.

[0171] In step 603, the electronic device extracts features from the second reference image through the image processing network in the data generation model to obtain second initial image features, adds noise features to the second initial image features to obtain third image features, and denoises the third image features guided by the second text features to obtain target intermediate features.

[0172] The noise feature is used to characterize random noise. The electronic device inputs the second text feature and the second reference image into the image processing network to obtain the target intermediate feature generated by the image processing network.

[0173] In some embodiments, as Figure 2 As shown, the image processing network includes an image coding layer. The electronic device inputs the second reference image into the image coding layer, and the image coding layer encodes the second reference image and outputs a second initial image feature.

[0174] In some embodiments, the electronic device uses the second text feature as a guide to denoise the third image feature to obtain a target intermediate feature, including: using an image processing network to denoise the third image feature using the second text feature as a guide to obtain a third intermediate feature; using an image processing network to perform a nonlinear transformation on the third intermediate feature to obtain a target intermediate feature, where the target intermediate feature is an enhanced feature of the third intermediate feature.

[0175] Optionally, through an image processing network, a nonlinear transformation is performed on the third image feature to obtain a second enhanced feature, which is an enhanced feature of the third image feature. The second enhanced feature is guided by the second text feature and the second enhanced feature to obtain a third intermediate feature. The third intermediate feature is nonlinearly transformed to obtain a fourth intermediate feature, which is an enhanced feature of the third intermediate feature. The third intermediate feature is denoised using the second text feature and the fourth intermediate feature as guidance to obtain a fifth intermediate feature. The fifth intermediate feature is nonlinearly transformed to obtain a target intermediate feature. In other words, multi-level denoising is required to obtain the target intermediate feature.

[0176] The process of obtaining the fifth intermediate feature in the embodiment of the present disclosure is similar to the process of obtaining the second image feature in the above embodiment.

[0177] In the disclosed embodiment, the image features are first denoised using the second text features as a guide, and then the denoised image features are enhanced. This not only enhances the responsiveness of the image features to the text semantics, but also effectively retains the key details of the image content, improves the expression quality of the target intermediate features, and makes the subsequently generated target text more accurate and semantically consistent.

[0178] In step 604, the electronic device generates a target text based on the target intermediate features through a text processing network, where the target text matches the description of the second reference text, and the content described in the target text matches the content of the second reference image.

[0179] After determining the target intermediate features, the electronic device generates target text based on the target intermediate features. The target text matches the description of the second reference text, and the content described in the target text matches the content of the second reference image. For example, if the second reference text indicates the number of words to be generated, the number of words in the target text matches the number of words indicated by the second reference text.

[0180] In some embodiments, the electronic device fuses the target intermediate features and the second initial text features through a text processing network to obtain target text features, where the second initial text features are features obtained by encoding the second reference text; and generates target text based on the target text features through the text processing network.

[0181] The process of obtaining the second initial text feature is the same as the process of obtaining the first initial text feature in the above embodiment, and will not be repeated here.

[0182] Alternatively, as Figure 2 As shown, the text processing network includes a text feature processing layer. The electronic device inputs the target intermediate feature and the second initial text feature into the text feature processing layer, and the text feature processing layer performs causal attention encoding on the target intermediate feature and the second initial text feature to obtain the target text feature.

[0183] Among them, the electronic device maps the target intermediate feature into the fifth query sub-feature, the fifth key sub-feature and the fifth value sub-feature through the text feature processing layer, maps the second initial text feature into the sixth query sub-feature, the sixth key sub-feature and the sixth value sub-feature, and cross-integrates the fifth query sub-feature, the fifth key sub-feature, the fifth value sub-feature, the sixth query sub-feature, the sixth key sub-feature and the sixth value sub-feature to obtain the target text feature.

[0184] Exemplarily, through the text feature processing layer, the fifth query sub-feature and the sixth query sub-feature are spliced to obtain the fourth query feature, the fifth key sub-feature and the sixth key sub-feature are spliced to obtain the fourth key feature, and the fifth value sub-feature and the sixth value sub-feature are spliced to obtain the fourth value feature; through the text feature processing layer, the product of the transpose of the fourth query feature and the fourth key feature and the scaling factor is normalized to obtain the fourth weight feature, and the fourth weight feature is multiplied by the fourth value feature to obtain the target text feature.

[0185] Alternatively, as Figure 2 As shown, the text processing network includes a text decoding layer. The electronic device inputs the target text features into the text decoding layer, and the target text features are decoded by the text decoding layer to obtain the target text.

[0186] In the disclosed embodiment, during the text generation process, the target intermediate features obtained by image processing and the initial text features obtained by text encoding are fused to construct target text features with highly unified semantics and visual information, thereby achieving deep coordination between visual semantics and language semantics. This ensures that the generated text not only closely fits the details of the reference image, but also more accurately continues the expression intent of the original text, which helps to improve the coherence, accuracy and diversity of text generation, thereby improving the generation quality of the target text.

[0187] According to the method provided by the embodiments of the present disclosure, the text processing network and the image processing network in the data generation model can also be used to execute the path of generating text from images. Therefore, the data generation model can realize both image generation and image understanding based on the ability of bidirectional generation of images and texts, which significantly improves the image generation and image understanding capabilities of the data generation model, provides greater flexibility and practicality for the multimodal data generation model, and broadens the application space of the data generation model in fields such as intelligent content generation.

[0188] The following Figure 7 The embodiment of the present invention illustrates the training process of the data generation model. Figure 7 is a flowchart of a method for training a data generation model according to an exemplary embodiment, the method being executed by an electronic device, see Figure 7 , the method includes the following steps.

[0189] In step 701, the electronic device obtains a first sample reference text and a first sample reference image, where the first sample reference text is used to describe the first sample reference image.

[0190] In step 702, the electronic device extracts features from the first sample reference text through a text processing network to obtain first sample text features.

[0191] The process of obtaining the sample text features in step 702 is similar to the process of obtaining the first text features in step 403 above, and will not be repeated here.

[0192] In step 703, the electronic device extracts features from the first sample reference image through an image processing network to obtain first sample initial image features, adds noise features to the first sample initial image features to obtain first sample image features, denoises the first sample image features using the first sample text features as a guide to obtain first sample intermediate features, performs a nonlinear transformation on the first sample intermediate features to obtain second sample intermediate features, and denoises the first sample intermediate features using the first sample text features and the second sample intermediate features as a guide to obtain second sample image features.

[0193] The noise feature is used to characterize random noise. The process of obtaining the second sample image feature in step 703 is the same as the process of obtaining the second image feature in the above embodiments, and will not be repeated here.

[0194] In step 704, the electronic device trains the data generation model based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature.

[0195] The electronic device determines a first loss parameter based on at least one feature obtained by denoising, the initial image feature of the first sample and the noise feature, and trains the data generation model with the goal of reducing the first loss parameter.

[0196] Optionally, the electronic device uses the following formula (4) to determine the first loss parameter.

[0197] ;Formula (4) in, Represents the first loss parameter, which is the Flow Matching MSE loss parameter for generating supervision. represents the initial image feature of the first sample, Represents the noise feature, t represents the time step, representing the process from clean to fully noisy image, It represents the denoised image features at time t, Indicates the denoising direction from the image features at time t back to the initial image features of the first sample, Indicates the true denoising direction. Indicates MSE mean square error.

[0198] In some embodiments, the electronic device further obtains a second sample reference text, a second sample reference image, and a target reference text, wherein the second sample reference text is used to describe the target reference text, and the target reference text is used to describe the second sample reference image; the second sample reference text is subjected to feature extraction through a text processing network to obtain second sample text features; the second sample reference image is subjected to feature extraction through an image processing network to obtain second sample initial image features, noise features are added to the second sample initial image features to obtain third sample image features, and the third sample image features are denoised using the second sample text features as a guide to obtain target sample intermediate features; and the text processing network is used to generate predicted text based on the target sample intermediate features. The process of generating predicted text is similar to the process of generating target text in step 604 above and will not be repeated here.

[0199] Then, step 705 includes: determining a first loss parameter based on at least one feature obtained by denoising, the initial image feature of the first sample and the noise feature; determining a second loss parameter based on the predicted text and the target reference text; and training the data generation model based on the first loss parameter and the second loss parameter.

[0200] The target reference text is real text, while the predicted text is generated by the data generation model based on the second sample reference image. The smaller the difference between the predicted text generated by the data generation model and the target reference text, the more accurate the data generation model. Therefore, when training the data generation model, the training goal also includes reducing the difference between the predicted text and the target reference text.

[0201] Optionally, the electronic device performs a weighted summation of the first loss parameter and the second loss parameter to obtain a third loss parameter, and trains the data generation model with the goal of reducing the third loss parameter.

[0202] Optionally, the electronic device uses the following formula (5) and formula (6) to determine the second loss parameter and the third loss parameter respectively.

[0203] ;Formula (5) ;Formula (6) in, represents the second loss parameter, N represents the number of characters in the character table, V represents the number of characters in the text, Indicates whether the jth character in the sample reference text belongs to the true label of the i-th character in the character table, represents the predicted probability that the jth character in the predicted text belongs to the i-th character in the vocabulary.

[0204] in, represents the third loss parameter, Represents the weight factor.

[0205] In the embodiment of the present disclosure, based on the difference between the predicted text and the sample reference text in the perception task and the difference between the predicted image and the sample reference image in the generation task, the data generation model is collaboratively trained to simultaneously improve the perception ability and generation ability of the data generation model.

[0206] The method provided by the embodiment of the present disclosure processes multimodal text features and image features during the training process of the data generation model, uses the sample text features of the sample reference text as a condition to guide the denoising process of the sample image features, and the sample intermediate features generated in the denoising process are also used as a condition to guide the denoising process of the sample image features after feature enhancement, thereby achieving unified semantic understanding based on multimodal features, which is conducive to promoting the fusion and synergy of cross-modal features. In addition, feature enhancement belongs to the perception task, and feature denoising belongs to the generation task. In the perception task, the intermediate features generated in the generation task are enhanced, and in turn, the enhanced features in the perception task are used as a condition to guide the generation task, thereby achieving deep fusion between the perception task and the generation task, which is conducive to improving the data generation model's ability to understand and generate multimodal features, thereby improving the accuracy of the data generation model.

[0207] Figure 8 is a flow chart of another data generation method according to an exemplary embodiment. Figure 8 As shown in the figure, in the image generation task, the query features, key features, and value features of the text features are transferred to the query features, key features, and value features of the image features, thereby cross-fusing the text features and image features, and generating images guided by the text features to achieve the image generation task. In the image understanding task, the query features, key features, and value features of the image features learned through the attention mechanism are enhanced by a multi-layer perceptron, and the enhanced query features, key features, and value features are transferred to the query features, key features, and value features of the text features, thereby cross-fusing the text features and image features, and generating text guided by the image features to achieve the image understanding task.

[0208] In the disclosed embodiments, the data generation model utilizes a dynamic routing mechanism based on sequence type, through the collaborative work of the text processing network and the image processing network. This mechanism enables adaptive switching between different modeling strategies when processing mixed-modal sequences. Causal modeling is performed on the autoregressive decoding path of the text processing network, which is activated by text, while continuous signal processing is performed on the diffusion denoising path of the image processing network, which is activated by images.

[0209] In addition, in order to achieve information interaction between modalities, a cross-modal attention mechanism is introduced, allowing the text features extracted by the text processing network to be used as conditions to guide the generation process of the image processing network. At the same time, the image features extracted by the image processing network can also enhance the semantic understanding of the text processing network through the attention mechanism. This bidirectional information flow design ensures the coordination and consistency between different processing networks.

[0210] Moreover, the multi-level intermediate features generated by generative modeling provide a unified semantic space for cross-modal understanding, enabling the data generation model to achieve deeper semantic alignment and cross-modal reasoning in complex scenarios where images and texts are intertwined, thereby possessing powerful multimodal understanding and generation capabilities under a unified framework.

[0211] In addition, the intermediate features in the generative modeling process are used as enhanced signals for perceptual modeling to achieve feature unification and deep fusion of perceptual tasks and generation tasks in the image processing network, which is conducive to promoting the coordinated improvement of perceptual tasks and generation tasks. Figure 9 FIG is a block diagram of a data generating device according to an exemplary embodiment. Figure 9 The device includes a data preparation unit 901, a text processing unit 902 and an image processing unit 903.

[0212] A data preparation unit 901 is configured to determine a data generation model, where the data generation model includes a text processing network and an image processing network; The text processing unit 902 is configured to perform feature extraction on a first reference text through a text processing network to obtain a first text feature, where the first reference text is used to describe the image to be generated; The image processing unit 903 is configured to perform, through an image processing network, denoising the first image feature using the first text feature as a guide to obtain a first intermediate feature, performing a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, and denoising the first intermediate feature using the first text feature and the second intermediate feature as a guide to obtain a second image feature; wherein the first image feature is any image feature containing noise, and the second intermediate feature is a feature obtained by enhancing the first intermediate feature; The image processing unit 903 is further configured to generate a target image based on the second image feature through an image processing network, where the target image is consistent with the description of the first reference text.

[0213] The device provided by the embodiment of the present disclosure processes multimodal text features and image features in a data generation scenario, uses the text features of the reference text as a condition to guide the denoising process of the image features, and the intermediate features generated in the denoising process are also used as a condition to guide the denoising process of the image features after feature enhancement, thereby achieving unified semantic understanding based on multimodal features, which is conducive to promoting the fusion and synergy of cross-modal features. In addition, feature enhancement belongs to the perception task, and feature denoising belongs to the generation task. In the perception task, the intermediate features generated in the generation task are enhanced, and in turn, the enhanced features in the perception task are used as a condition to guide the generation task, thereby achieving deep fusion between the perception task and the generation task. Therefore, the processing method of the present disclosure can improve the understanding and generation capabilities of multimodal features in the data generation scenario, thereby improving the accuracy of the target image finally generated.

[0214] In one possible implementation, see Figure 10 The image processing unit 903 is further configured to perform: Performing feature extraction on the first reference image through an image processing network to obtain a first initial image feature, adding a noise feature to the first initial image feature to obtain a first image feature, wherein the first reference image includes a reference object, the object in the target image matches the reference object, and the noise feature is used to characterize random noise; or A noise feature is determined as a first image feature.

[0215] In one possible implementation, see Figure 10 , the text processing unit 902 is configured to execute: encoding the first reference text into a first initial text feature through a text encoding layer in a text processing network; The first initial text feature is causally encoded by the text feature processing layer in the text processing network to obtain the first text feature.

[0216] In one possible implementation, see Figure 10 , the text processing unit 902 is configured to execute: Mapping the first initial text feature into a first query feature, a first key feature, and a first value feature through a text feature processing layer; Through the text feature processing layer, masking is performed on the product of the transpose of the first query feature and the first key feature and the scaling factor based on the causal mask feature to obtain a first weight feature, where the first reference text includes n characters, and the causal mask feature is used to mask features of characters from i+1 to n when determining the weight feature of the i-th character, where n is a positive integer and i is a positive integer not greater than n; The first weight feature is multiplied by the first value feature through the text feature processing layer to obtain the first text feature.

[0217] In one possible implementation, see Figure 10 , the image processing unit 903 is configured to perform: Performing a nonlinear transformation on the first image feature through an image processing network to obtain a first enhanced feature, where the first enhanced feature is a feature obtained by enhancing the first image feature; Through the image processing network, guided by the first text feature and the first enhanced feature, the first image feature is denoised to obtain a first intermediate feature.

[0218] In one possible implementation, see Figure 10 , the image processing unit 903 is configured to perform: Through the image feature processing layer in the image processing network, the first text feature, the first enhanced feature and the first image feature are cross-attention encoded to obtain the first intermediate feature.

[0219] In one possible implementation, see Figure 10 , the image processing unit 903 is configured to perform: By means of an image feature processing layer, the first text feature is mapped into a first query sub-feature, a first key sub-feature, and a first value sub-feature, the first image feature is mapped into a second query sub-feature, and the first enhancement feature is mapped into a second key sub-feature and a second value sub-feature; Through the image feature processing layer, the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature and the second value sub-feature are cross-fused to obtain a first intermediate feature.

[0220] In one possible implementation, see Figure 10 , the image processing unit 903 is configured to perform: Through the image feature processing layer, the first query sub-feature and the second query sub-feature are concatenated to obtain a second query feature, the first key sub-feature and the second key sub-feature are concatenated to obtain a second key feature, and the first value sub-feature and the second value sub-feature are concatenated to obtain a second value feature; The image feature processing layer normalizes the product of the transposition of the second query feature and the second key feature and the scaling factor to obtain a second weight feature, and the second weight feature is multiplied by the second value feature to obtain a first intermediate feature.

[0221] In one possible implementation, see Figure 10 , the image processing unit 903 is configured to perform: Through the image feature processing layer in the image processing network, the first text feature, the second intermediate feature and the first intermediate feature are cross-attention encoded to obtain the second image feature.

[0222] In one possible implementation, see Figure 10 , the image processing unit 903 is configured to perform: Mapping the first text feature into a third query sub-feature, a third key sub-feature, and a third value sub-feature through an image feature processing layer, mapping the first intermediate feature into a fourth query sub-feature, and mapping the second intermediate feature into a fourth key sub-feature and a fourth value sub-feature; The third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature are cross-fused through the image feature processing layer to obtain a second image feature.

[0223] In one possible implementation, see Figure 10 , the image processing unit 903 is configured to perform: Through the image feature processing layer, the third query sub-feature and the fourth query sub-feature are concatenated to obtain a third query feature, the third key sub-feature and the fourth key sub-feature are concatenated to obtain a third key feature, and the third value sub-feature and the fourth value sub-feature are concatenated to obtain a third value feature; The image feature processing layer normalizes the product of the transposition of the third query feature and the third key feature and the scaling factor to obtain a third weight feature, and the third weight feature is multiplied by the third value feature to obtain a second image feature.

[0224] In one possible implementation, see Figure 10 The image processing network includes N diffusion subnetworks, where N is an integer greater than 1; the image processing unit 903 is configured to perform: The first diffusion subnetwork in the image processing network is used to denoise the first image feature with the first text feature as a guide to obtain the first first intermediate feature; Through the k-th diffusion subnetwork in the image processing network, a nonlinear transformation is performed on the k-1th first intermediate feature to obtain the k-1th second intermediate feature; Through the kth diffusion subnetwork, guided by the first text feature and the k-1th second intermediate feature, the k-1th first intermediate feature is denoised to obtain the kth first intermediate feature; where k is an integer greater than 1 and not greater than N, and the second image feature is the Nth first intermediate feature.

[0225] In one possible implementation, see Figure 10 , the device further comprises: The data preparation unit 901 is further configured to obtain a second reference text and a second reference image, where the second reference text is used to describe the text to be generated; The text processing unit 902 is further configured to perform feature extraction on the second reference text through the text processing network to obtain second text features; The image processing unit 903 is further configured to perform feature extraction on the second reference image through the image processing network to obtain a second initial image feature, add a noise feature to the second initial image feature to obtain a third image feature, and denoise the third image feature using the second text feature as a guide to obtain a target intermediate feature, where the noise feature is used to characterize random noise; The text processing unit 902 is further configured to generate a target text based on the target intermediate features through the text processing network, where the target text is consistent with the description of the second reference text, and the content described in the target text is consistent with the content of the second reference image.

[0226] In one possible implementation, see Figure 10 , the image processing unit 903 is configured to perform: Through the image processing network, guided by the second text feature, the third image feature is denoised to obtain the third intermediate feature; Through the image processing network, the third intermediate feature is nonlinearly transformed to obtain the target intermediate feature, which is the feature after the third intermediate feature is enhanced.

[0227] In one possible implementation, see Figure 10 , the text processing unit 902 is configured to execute: The target intermediate features and the second initial text features are fused through the text processing network to obtain the target text features, where the second initial text features are features obtained by encoding the second reference text; Generate target text based on target text features through text processing network.

[0228] In one possible implementation, see Figure 10 The apparatus further includes a model training unit 904 configured to execute: Acquire a first sample reference text and a first sample reference image, where the first sample reference text is used to describe the first sample reference image; Performing feature extraction on the first sample reference text through a text processing network to obtain first sample text features; Through the image processing network, feature extraction is performed on the first sample reference image to obtain the first sample initial image feature, noise features are added to the first sample initial image feature to obtain the first sample image feature, the first sample text feature is used as a guide, the first sample image feature is denoised to obtain the first sample intermediate feature, the first sample intermediate feature is nonlinearly transformed to obtain the second sample intermediate feature, the first sample text feature and the second sample intermediate feature are used as a guide, the first sample intermediate feature is denoised to obtain the second sample image feature, and the noise feature is used to characterize random noise; The data generation model is trained based on at least one feature obtained by denoising, the first sample initial image feature and the noise feature.

[0229] In one possible implementation, see Figure 10 , the model training unit 904 is configured to perform: Acquire a second sample reference text, a second sample reference image, and a target reference text, wherein the second sample reference text is used to describe the target reference text, and the target reference text is used to describe the second sample reference image; Extracting features of the second sample reference text through a text processing network to obtain features of the second sample text; Through the image processing network, feature extraction is performed on the second sample reference image to obtain the second sample initial image feature, noise features are added to the second sample initial image feature to obtain the third sample image feature, and the third sample image feature is denoised using the second sample text feature as a guide to obtain the target sample intermediate feature; Generate predicted text based on the intermediate features of the target sample through the text processing network; Determine a first loss parameter based on at least one feature obtained by denoising, a feature of the first sample initial image, and a noise feature; and determine a second loss parameter based on the predicted text and the target reference text; The data generation model is trained based on the first loss parameter and the second loss parameter.

[0230] Regarding the apparatus in the above embodiment, the specific manner in which each unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0231] An embodiment of the present disclosure provides an electronic device, comprising: a processor and a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement the above-mentioned data generation method.

[0232] In some embodiments, the electronic device includes a terminal. Figure 11This is a block diagram illustrating the structure of a terminal 1100 according to an exemplary embodiment. Terminal 1100 may be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 1100 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other similar names.

[0233] The terminal 1100 includes a processor 1101 and a memory 1102 .

[0234] Processor 1101 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 1101 may be implemented in hardware using at least one of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. Memory 1102 may include one or more computer-readable storage media, which may be non-transitory. Memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 1102 is used to store at least one program code, which is executed by processor 1101 to implement the data generation method provided in the method embodiments of the present disclosure.

[0235] In some embodiments, terminal 1100 may also optionally include a peripheral device interface 1103 and at least one peripheral device. Processor 1101, memory 1102, and peripheral device interface 1103 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1103 via a bus, signal lines, or circuit boards. Specifically, the peripheral devices include a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, and a power supply 1107.

[0236] The peripheral device interface 1103 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1101 and the memory 1102. In some embodiments, the processor 1101, the memory 1102, and the peripheral device interface 1103 are integrated on the same chip or circuit board. In other embodiments, any one or two of the processor 1101, the memory 1102, and the peripheral device interface 1103 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0237] RF circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. RF circuit 1104 optionally includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. RF circuit 1104 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, RF circuit 1104 may also include circuitry related to Near Field Communication (NFC), although this disclosure is not limiting in this regard.

[0238] Display screen 1105 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, or any combination thereof. When display screen 1105 is a touch screen display, it is also capable of collecting touch signals on or above the surface of display screen 1105. This touch signal may be input as a control signal to processor 1101 for processing. Display screen 1105 may also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1105, disposed on the front panel of electronic device 1100. In other embodiments, there may be at least two display screens 1105, disposed on different surfaces of electronic device 1100 or in a foldable design. In other embodiments, display screen 1105 may be a flexible display, disposed on a curved or foldable surface of electronic device 1100. Display screen 1105 may also be configured as a non-rectangular, irregular shape, i.e., a special-shaped screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0239] The camera assembly 1106 is used to capture images or videos. Optionally, the camera assembly 1106 includes a front camera and a rear camera. The front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0240] Power supply 1107 is used to power the various components of electronic device 1100. Power supply 1107 can be AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 1107 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0241] Those skilled in the art will understand that Figure 11The structure shown in the figure does not constitute a limitation on the terminal 1100, and the terminal 1100 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0242] In some embodiments, the electronic device includes a server. Figure 12 This is a schematic diagram of the structure of a server according to an exemplary embodiment. The server 1200 may vary significantly due to different configurations or performance. It may include one or more processors (Central Processing Units, CPUs) 1201 and one or more memories 1202. The memories 1202 store at least one computer program, which is loaded and executed by the processor 1201 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which are not described in detail here.

[0243] In an exemplary embodiment, a computer-readable storage medium is also provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps of the above-described data generation method. For example, the computer-readable storage medium may be a ROM (Read Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, or an optical data storage device.

[0244] In an exemplary embodiment, a computer program product is further provided, including a computer program, which implements the steps in the above data generation method when executed by a processor of an electronic device.

[0245] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0246] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data generation method, characterized in that: The method comprises: Determining a data generation model, wherein the data generation model includes a text processing network and an image processing network; Performing feature extraction on a first reference text through the text processing network to obtain a first text feature, where the first reference text is used to describe the image to be generated; Using the image processing network, using the first text feature as a guide, denoising the first image feature to obtain a first intermediate feature, performing a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, and using the first text feature and the second intermediate feature as a guide, denoising the first intermediate feature to obtain a second image feature; wherein the first image feature is any image feature containing noise, and the second intermediate feature is a feature obtained by enhancing the first intermediate feature; A target image is generated based on the second image features through the image processing network, and the target image is consistent with the description of the first reference text.

2. The data generation method according to claim 1, wherein: The process of acquiring the first image feature includes: performing feature extraction on a first reference image through the image processing network to obtain a first initial image feature, adding a noise feature to the first initial image feature to obtain the first image feature, wherein the first reference image includes a reference object, the object in the target image matches the reference object, and the noise feature is used to characterize random noise; or The noise feature is determined as the first image feature.

3. The data generation method according to claim 1, wherein: The step of extracting features from the first reference text through the text processing network to obtain first text features includes: encoding the first reference text into a first initial text feature through a text encoding layer in the text processing network; The first initial text feature is causally encoded by a text feature processing layer in the text processing network to obtain the first text feature.

4. The data generation method according to claim 3, wherein: The step of performing causal attention encoding on the first initial text feature through a text feature processing layer in the text processing network to obtain the first text feature includes: Mapping the first initial text feature into a first query feature, a first key feature, and a first value feature through the text feature processing layer; By the text feature processing layer, masking is performed on the first query feature, the product of the transpose of the first key feature and the scaling factor based on a causal mask feature to obtain a first weight feature, wherein the first reference text includes n characters, and the causal mask feature is used to mask features of characters from i+1th to nth when determining the weight feature of the i-th character, where n is a positive integer and i is a positive integer not greater than n; The first weight feature is multiplied by the first value feature through the text feature processing layer to obtain the first text feature.

5. The data generation method according to claim 1, wherein: The step of performing denoising on the first image feature using the image processing network and taking the first text feature as a guide to obtain a first intermediate feature includes: Performing a nonlinear transformation on the first image feature through the image processing network to obtain a first enhanced feature, where the first enhanced feature is a feature after the first image feature is enhanced; The first image feature is denoised using the image processing network and guided by the first text feature and the first enhanced feature to obtain the first intermediate feature.

6. The data generation method according to claim 5, characterized in that: The step of performing denoising on the first image feature using the image processing network and guided by the first text feature and the first enhanced feature to obtain the first intermediate feature includes: The first text feature, the first enhanced feature, and the first image feature are cross-attention encoded through the image feature processing layer in the image processing network to obtain the first intermediate feature.

7. The data generation method according to claim 6, characterized in that: The step of performing cross-attention encoding on the first text feature, the first enhanced feature, and the first image feature through the image feature processing layer in the image processing network to obtain the first intermediate feature includes: By means of the image feature processing layer, the first text feature is mapped into a first query sub-feature, a first key sub-feature, and a first value sub-feature, the first image feature is mapped into a second query sub-feature, and the first enhancement feature is mapped into a second key sub-feature and a second value sub-feature; The first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature are cross-fused through the image feature processing layer to obtain the first intermediate feature.

8. The data generation method according to claim 7, characterized in that: The cross-fusion of the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature through the image feature processing layer to obtain the first intermediate feature includes: splicing the first query sub-feature and the second query sub-feature through the image feature processing layer to obtain a second query feature, splicing the first key sub-feature and the second key sub-feature to obtain a second key feature, and splicing the first value sub-feature and the second value sub-feature to obtain a second value feature; The image feature processing layer normalizes the product of the transpose of the second query feature and the second key feature and a scaling factor to obtain a second weight feature, and the second weight feature is multiplied by the second value feature to obtain the first intermediate feature.

9. The data generation method according to claim 1, wherein: The first intermediate feature is denoised using the image processing network and guided by the first text feature and the second intermediate feature to obtain a second image feature, including: The first text feature, the second intermediate feature, and the first intermediate feature are cross-attention encoded through the image feature processing layer in the image processing network to obtain the second image feature.

10. The data generation method according to any one of claims 1 to 9, characterized in that: The image processing network includes N diffusion subnetworks, where N is an integer greater than 1. The first image feature is denoised using the image processing network and guided by the first text feature to obtain a first intermediate feature, including: Using a first diffusion subnetwork in the image processing network, guided by the first text feature, denoising the first image feature to obtain a first first intermediate feature; The performing a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, and performing denoising on the first intermediate feature using the first text feature and the second intermediate feature as a guide to obtain a second image feature, includes: Performing a nonlinear transformation on the k-1th first intermediate feature through the k-th diffusion subnetwork in the image processing network to obtain the k-1th second intermediate feature; Through the kth diffusion subnetwork, guided by the first text feature and the k-1th second intermediate feature, the k-1th first intermediate feature is denoised to obtain the kth first intermediate feature; wherein k is an integer greater than 1 and not greater than N, and the second image feature is the Nth first intermediate feature.

11. The data generation method according to any one of claims 1 to 9, characterized in that: The method further comprises: Acquire a second reference text and a second reference image, where the second reference text is used to describe the text to be generated; Performing feature extraction on the second reference text through the text processing network to obtain second text features; performing feature extraction on the second reference image through the image processing network to obtain a second initial image feature, adding a noise feature to the second initial image feature to obtain a third image feature, and denoising the third image feature using the second text feature as a guide to obtain a target intermediate feature, wherein the noise feature is used to characterize random noise; A target text is generated based on the target intermediate features through the text processing network, wherein the target text is consistent with the description of the second reference text, and the content described in the target text is consistent with the content of the second reference image.

12. The data generation method according to claim 11, characterized in that: Denoising the third image feature using the image processing network and the second text feature as a guide to obtain a target intermediate feature includes: Denoising the third image feature using the image processing network and the second text feature as a guide to obtain a third intermediate feature; The third intermediate feature is subjected to a nonlinear transformation through the image processing network to obtain the target intermediate feature, which is an enhanced feature of the third intermediate feature.

13. The data generation method according to claim 11, characterized in that: Generating a target text based on the target intermediate features through the text processing network includes: fusing the target intermediate feature and the second initial text feature through the text processing network to obtain a target text feature, where the second initial text feature is a feature obtained by encoding the second reference text; The target text is generated based on the target text features through the text processing network.

14. The data generation method according to any one of claims 1 to 9, characterized in that: The training process of the data generation model includes: Acquire a first sample reference text and a first sample reference image, where the first sample reference text is used to describe the first sample reference image; Performing feature extraction on the first sample reference text through the text processing network to obtain first sample text features; Performing feature extraction on the first sample reference image through the image processing network to obtain a first sample initial image feature, adding a noise feature to the first sample initial image feature to obtain a first sample image feature, denoising the first sample image feature using the first sample text feature as a guide to obtain a first sample intermediate feature, performing a nonlinear transformation on the first sample intermediate feature to obtain a second sample intermediate feature, denoising the first sample intermediate feature using the first sample text feature and the second sample intermediate feature as a guide to obtain a second sample image feature, wherein the noise feature is used to characterize random noise; The data generation model is trained based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature.

15. The data generation method according to claim 14, characterized in that: The method further comprises: Acquire a second sample reference text, a second sample reference image, and a target reference text, wherein the second sample reference text is used to describe the target reference text, and the target reference text is used to describe the second sample reference image; Performing feature extraction on the second sample reference text through the text processing network to obtain second sample text features; Performing feature extraction on the second sample reference image through the image processing network to obtain a second sample initial image feature, adding the noise feature to the second sample initial image feature to obtain a third sample image feature, and denoising the third sample image feature using the second sample text feature as a guide to obtain a target sample intermediate feature; Generate predicted text based on the intermediate features of the target sample through the text processing network; The training of the data generation model based on the at least one feature obtained by denoising, the first sample initial image feature, and the noise feature includes: Determining a first loss parameter based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature; determining a second loss parameter based on the predicted text and the target reference text; The data generation model is trained based on the first loss parameter and the second loss parameter.

16. A data generating device, characterized in that: The device comprises: A data preparation unit is configured to execute determination of a data generation model, wherein the data generation model includes a text processing network and an image processing network; a text processing unit configured to perform feature extraction on a first reference text through the text processing network to obtain a first text feature, wherein the first reference text is used to describe an image to be generated; The image processing unit is configured to perform, through the image processing network, denoising the first image feature using the first text feature as a guide to obtain a first intermediate feature, performing a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, and denoising the first intermediate feature using the first text feature and the second intermediate feature as a guide to obtain a second image feature; wherein the first image feature is any image feature containing noise, and the second intermediate feature is a feature obtained by enhancing the first intermediate feature; The image processing unit is further configured to generate a target image based on the second image feature through the image processing network, where the target image is consistent with the description of the first reference text.

17. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data generation method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data generation method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data generation method according to any one of claims 1 to 15 is implemented.

Images