Data generation method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a data generation method and device, electronic equipment and storage medium, belonging to the technical field of computers. The method comprises: determining a data generation model, the data generation model comprising a text processing network and an image processing network; performing feature extraction on a first reference text through the text processing network to obtain first text features; performing denoising on first image features guided by the first text features through the image processing network to obtain first intermediate features, performing nonlinear transformation on the first intermediate features to obtain second intermediate features, and performing denoising on the first intermediate features guided by the first text features and the second intermediate features to obtain second image features; and generating a target image based on the second image features through the image processing network, the target image being consistent with the description of the first reference text. The present disclosure can improve the understanding and generation capabilities of multi-modal features in a data generation scenario, and improve the accuracy of the generated target image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a data generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of computer technology, the processing mode of images is more and more diverse. At present, images can be intelligently generated based on artificial intelligence technology or semantically understood, such as generating images from text or generating text from images, which all need to process multi-modal text features and image features.

[0003] In the related art, in the image generation task, a special processing network is designed for text features and image features of different modalities, and two parallel processing networks are used to process text features and image features.

[0004] However, in this way, the separation architecture is difficult to realize the fusion and synergy between cross-modal features, resulting in low feature perception ability and image generation ability in the image generation process, and further affecting the accuracy of the generated image. SUMMARY

[0005] The present disclosure provides a data generation method, device, electronic equipment and storage medium, which can improve the accuracy of data generation.

[0006] According to an aspect of an embodiment of the present disclosure, a data generation method is provided, the method comprising:

[0007] determining a data generation model, the data generation model comprising a text processing network and an image processing network;

[0008] extracting features of a first reference text through the text processing network to obtain first text features, the first reference text being used to describe an image to be generated;

[0009] guiding the first text features to denoise first image features through the image processing network to obtain first intermediate features, performing nonlinear transformation on the first intermediate features to obtain second intermediate features, guiding the first text features and the second intermediate features to denoise the first intermediate features to obtain second image features, wherein the first image features are any image features containing noise, and the second intermediate features are enhanced features of the first intermediate features;

[0010] generating a target image based on the second image features through the image processing network, the target image being consistent with the description of the first reference text.

[0011] Optionally, the process of obtaining the first image features comprises:

[0012] extracting, by the image processing network, a feature of the first reference image to obtain a first initial image feature, adding a noise feature to the first initial image feature to obtain the first image feature, the first reference image comprising a reference object, an object in the target image being consistent with the reference object, the noise feature being used to represent random noise; or

[0013] determining the noise feature as the first image feature.

[0014] Optionally, the extracting, by the text processing network, a feature of the first reference text to obtain a first text feature comprises:

[0015] encoding, by a text encoding layer in the text processing network, the first reference text into a first initial text feature;

[0016] performing, by a text feature processing layer in the text processing network, causal attention encoding on the first initial text feature to obtain the first text feature.

[0017] Optionally, the performing, by the text feature processing layer in the text processing network, causal attention encoding on the first initial text feature to obtain the first text feature comprises:

[0018] mapping, by the text feature processing layer, the first initial text feature into a first query feature, a first key feature and a first value feature;

[0019] performing, by the text feature processing layer, mask processing on a product of a transpose of the first query feature and a scaling factor based on a causal mask feature, to obtain a first weight feature, the first reference text comprising n characters, the causal mask feature being used to perform mask processing on features of (i+1)th character to nth character when determining a weight feature of ith character, n being a positive integer, i being a positive integer not greater than n;

[0020] multiplying, by the text feature processing layer, the first weight feature and the first value feature to obtain the first text feature.

[0021] Optionally, the denoising, by the image processing network, the first image feature guided by the first text feature to obtain a first intermediate feature comprises:

[0022] performing, by the image processing network, nonlinear transformation on the first image feature to obtain a first enhanced feature, the first enhanced feature being a feature of the first image feature after enhancement;

[0023] The image processing network is used to denoise the first image feature based on the first text feature and the first enhanced feature as guidance, to obtain the first intermediate feature.

[0024] Optionally, the image processing network is used to denoise the first image feature based on the first text feature and the first enhanced feature as guidance, to obtain the first intermediate feature, including:

[0025] The image feature processing layer in the image processing network is used to cross-attention encode the first text feature, the first enhanced feature and the first image feature, to obtain the first intermediate feature.

[0026] Optionally, the image feature processing layer in the image processing network is used to cross-attention encode the first text feature, the first enhanced feature and the first image feature, to obtain the first intermediate feature, including:

[0027] The image feature processing layer is used to map the first text feature into a first query sub-feature, a first key sub-feature and a first value sub-feature, map the first image feature into a second query sub-feature, and map the first enhanced feature into a second key sub-feature and a second value sub-feature.

[0028] The image feature processing layer is used to cross-fuse the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature and the second value sub-feature, to obtain the first intermediate feature.

[0029] Optionally, the image feature processing layer is used to cross-fuse the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature and the second value sub-feature, to obtain the first intermediate feature, including:

[0030] The image feature processing layer is used to concatenate the first query sub-feature and the second query sub-feature to obtain a second query feature, concatenate the first key sub-feature and the second key sub-feature to obtain a second key feature, and concatenate the first value sub-feature and the second value sub-feature to obtain a second value feature.

[0031] The image feature processing layer is used to normalize a product of a transpose of the second query feature and a scaling factor to obtain a second weight feature, and multiply the second weight feature and the second value feature to obtain the first intermediate feature.

[0032] Optionally, the image processing network is used to denoise the first intermediate feature based on the first text feature and the second intermediate feature, to obtain a second image feature, including:

[0033] The image feature processing layer in the image processing network is used to cross-attention encode the first text feature, the second intermediate feature and the first intermediate feature, to obtain the second image feature.

[0034] Optionally, the image feature processing layer in the image processing network is used to cross-attention encode the first text feature, the second intermediate feature and the first intermediate feature, to obtain the second image feature, including:

[0035] The image feature processing layer is used to map the first text feature into a third query sub-feature, a third key sub-feature and a third value sub-feature, map the first intermediate feature into a fourth query sub-feature, and map the second intermediate feature into a fourth key sub-feature and a fourth value sub-feature.

[0036] The image feature processing layer is used to cross-fuse the third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature, to obtain the second image feature.

[0037] Optionally, the image feature processing layer in the image processing network is used to cross-fuse the third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature, to obtain the second image feature, including:

[0038] The image feature processing layer is used to concatenate the third query sub-feature and the fourth query sub-feature to obtain a third query feature, concatenate the third key sub-feature and the fourth key sub-feature to obtain a third key feature, and concatenate the third value sub-feature and the fourth value sub-feature to obtain a third value feature.

[0039] The image feature processing layer is used to normalize a product of a transpose of the third query feature and a scaling factor to obtain a third weight feature, and multiply the third weight feature and the third value feature to obtain the second image feature.

[0040] Optionally, the image processing network includes N diffusion sub-networks, N being an integer greater than 1; and the image processing network is used to denoise the first image feature based on the first text feature, to obtain a first intermediate feature, including:

[0041] The first image feature is denoised by a first diffusion sub-network in the image processing network, to obtain a first intermediate feature;

[0042] The first intermediate feature is nonlinearly transformed to obtain a second intermediate feature, and the first intermediate feature is denoised based on the first text feature and the second intermediate feature to obtain a second image feature, including:

[0043] The k-1th intermediate feature is nonlinearly transformed by a kth diffusion sub-network in the image processing network to obtain a k-1th second intermediate feature;

[0044] The k-1th intermediate feature is denoised by the kth diffusion sub-network based on the first text feature and the k-1th second intermediate feature to obtain a kth intermediate feature, wherein k is an integer greater than 1 and not greater than N, and the second image feature is the Nth intermediate feature.

[0045] Optionally, the method further includes:

[0046] A second reference text and a second reference image are obtained, and the second reference text is used to describe a text to be generated;

[0047] The second reference text is feature extracted by the text processing network to obtain a second text feature;

[0048] The second reference image is feature extracted by the image processing network to obtain a second initial image feature, a noise feature is added to the second initial image feature to obtain a third image feature, and the third image feature is denoised based on the second text feature to obtain a target intermediate feature, and the noise feature is used to represent random noise;

[0049] The target text is generated based on the target intermediate feature by the text processing network, the target text is consistent with the description of the second reference text, and the content described by the target text is consistent with the content of the second reference image.

[0050] Optionally, the third image feature is denoised based on the second text feature by the image processing network to obtain a target intermediate feature, including:

[0051] The third image feature is denoised based on the second text feature by the image processing network to obtain a third intermediate feature;

[0052] The third intermediate feature is nonlinearly transformed by the image processing network to obtain the target intermediate feature, which is an enhanced feature of the third intermediate feature.

[0053] Optionally, the generating the target text based on the target intermediate feature by the text processing network comprises:

[0054] The target intermediate feature and a second initial text feature are fused by the text processing network to obtain a target text feature, the second initial text feature being a feature obtained by encoding the second reference text;

[0055] The target text is generated based on the target text feature by the text processing network.

[0056] Optionally, the training process of the data generation model comprises:

[0057] A first sample reference text and a first sample reference image are obtained, the first sample reference text being used to describe the first sample reference image;

[0058] The first sample reference text is feature-extracted by the text processing network to obtain a first sample text feature;

[0059] The first sample reference image is feature-extracted by the image processing network to obtain a first sample initial image feature, a noise feature is added to the first sample initial image feature to obtain a first sample image feature, the first sample image feature is denoised with the first sample text feature as a guide to obtain a first sample intermediate feature, the first sample intermediate feature is nonlinearly transformed to obtain a second sample intermediate feature, the first sample intermediate feature is denoised with the first sample text feature and the second sample intermediate feature as guides to obtain a second sample image feature, the noise feature being used to represent random noise;

[0060] The data generation model is trained based on at least one feature obtained by denoising, the first sample initial image feature and the noise feature.

[0061] Optionally, the method further comprises:

[0062] A second sample reference text, a second sample reference image and a target reference text are obtained, the second sample reference text being used to describe the target reference text, the target reference text being used to describe the second sample reference image;

[0063] The second sample reference text is feature-extracted by the text processing network to obtain a second sample text feature;

[0064] extracting, by the image processing network, a feature of the second sample reference image to obtain a second sample initial image feature, adding the noise feature in the second sample initial image feature to obtain a third sample image feature, and denoising the third sample image feature based on the second sample text feature to obtain a target sample intermediate feature;

[0065] generating, by the text processing network, a predicted text based on the target sample intermediate feature;

[0066] training the data generation model based on the at least one feature obtained by denoising, the first sample initial image feature, and the noise feature, including:

[0067] determining a first loss parameter based on the at least one feature obtained by denoising, the first sample initial image feature, and the noise feature, and determining a second loss parameter based on the predicted text and the target reference text;

[0068] training the data generation model based on the first loss parameter and the second loss parameter.

[0069] According to still another aspect of the embodiments of the present disclosure, a data generation apparatus is provided, and the apparatus includes:

[0070] a data preparation unit configured to determine a data generation model including a text processing network and an image processing network;

[0071] a text processing unit configured to extract a feature of a first reference text based on the text processing network to obtain a first text feature, the first reference text being used to describe an image to be generated;

[0072] an image processing unit configured to denoise a first image feature based on the first text feature by the image processing network to obtain a first intermediate feature, perform a nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, and denoise the first intermediate feature based on the first text feature and the second intermediate feature to obtain a second image feature, wherein the first image feature is any image feature containing noise, and the second intermediate feature is an enhanced feature of the first intermediate feature;

[0073] the image processing unit is further configured to generate a target image based on the second image feature by the image processing network, the target image being consistent with the description of the first reference text.

[0074] Optionally, the image processing unit is further configured to:

[0075] extracting, by the image processing network, a feature of the first reference image to obtain a first initial image feature, adding a noise feature in the first initial image feature to obtain the first image feature, the first reference image including a reference object, an object in the target image being consistent with the reference object, the noise feature being used to represent random noise; or,

[0076] determining the noise feature as the first image feature.

[0077] Optionally, the text processing unit is configured to perform:

[0078] encoding, by a text encoding layer in the text processing network, the first reference text into a first initial text feature;

[0079] performing, by a text feature processing layer in the text processing network, causal attention encoding on the first initial text feature to obtain the first text feature.

[0080] Optionally, the text processing unit is configured to perform:

[0081] mapping, by the text feature processing layer, the first initial text feature into a first query feature, a first key feature, and a first value feature;

[0082] masking, by the text feature processing layer, a product of a transpose of the first key feature and a scaling factor based on a causal mask feature to obtain a first weight feature, the first reference text including n characters, the causal mask feature being used to mask features of (i+1)th character to nth character when determining a weight feature of ith character, n being a positive integer, i being a positive integer not greater than n;

[0083] multiplying, by the text feature processing layer, the first weight feature and the first value feature to obtain the first text feature.

[0084] Optionally, the image processing unit is configured to perform:

[0085] performing, by the image processing network, nonlinear transformation on the first image feature to obtain a first enhanced feature, the first enhanced feature being a feature of the first image feature after enhancement;

[0086] performing, by the image processing network, denoising on the first image feature based on the first text feature and the first enhanced feature as guidance to obtain the first intermediate feature.

[0087] Optionally, the image processing unit is configured to perform:

[0088] The first text feature, the first enhanced feature, and the first image feature are cross-attention encoded by an image feature processing layer in the image processing network to obtain the first intermediate feature.

[0089] Optionally, the image processing unit is configured to perform:

[0090] The first text feature is mapped into a first query sub-feature, a first key sub-feature, and a first value sub-feature by the image feature processing layer, the first image feature is mapped into a second query sub-feature, and the first enhanced feature is mapped into a second key sub-feature and a second value sub-feature;

[0091] The first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature are cross-fused by the image feature processing layer to obtain the first intermediate feature.

[0092] Optionally, the image processing unit is configured to perform:

[0093] The first query sub-feature and the second query sub-feature are spliced by the image feature processing layer to obtain a second query feature, the first key sub-feature and the second key sub-feature are spliced to obtain a second key feature, and the first value sub-feature and the second value sub-feature are spliced to obtain a second value feature;

[0094] The second query feature, a transpose of the second key feature multiplied by a scaling factor, is normalized by the image feature processing layer to obtain a second weight feature, and the second weight feature is multiplied by the second value feature to obtain the first intermediate feature.

[0095] Optionally, the image processing unit is configured to perform:

[0096] The first text feature, the second intermediate feature, and the first intermediate feature are cross-attention encoded by an image feature processing layer in the image processing network to obtain the second image feature.

[0097] Optionally, the image processing unit is configured to perform:

[0098] The first text feature is mapped into a third query sub-feature, a third key sub-feature, and a third value sub-feature by the image feature processing layer, the first intermediate feature is mapped into a fourth query sub-feature, and the second intermediate feature is mapped into a fourth key sub-feature and a fourth value sub-feature;

[0099] The third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature, and the fourth value sub-feature are cross-fused by the image feature processing layer to obtain the second image feature.

[0100] Optionally, the image processing unit is configured to perform:

[0101] The third query sub-feature and the fourth query sub-feature are spliced by the image feature processing layer to obtain a third query feature, the third key sub-feature and the fourth key sub-feature are spliced to obtain a third key feature, and the third value sub-feature and the fourth value sub-feature are spliced to obtain a third value feature.

[0102] The third query feature, a transpose of the third key feature, and a product of the third key feature and a scaling factor are normalized by the image feature processing layer to obtain a third weight feature, and the third weight feature and the third value feature are multiplied to obtain the second image feature.

[0103] Optionally, the image processing network includes N diffusion sub-networks, N being an integer greater than 1; and the image processing unit is configured to perform:

[0104] The first image feature is denoised by the first diffusion sub-network in the image processing network using the first text feature as a guide to obtain a first first intermediate feature;

[0105] The k-1th first intermediate feature is nonlinearly transformed by the kth diffusion sub-network in the image processing network to obtain a k-1th second intermediate feature;

[0106] The k-1th first intermediate feature is denoised by the kth diffusion sub-network using the first text feature and the k-1th second intermediate feature as guides to obtain a kth first intermediate feature; k is an integer greater than 1 and not greater than N, and the second image feature is an Nth first intermediate feature.

[0107] Optionally, the apparatus further includes:

[0108] The data preparation unit is further configured to perform obtaining a second reference text and a second reference image, the second reference text being used to describe a text to be generated;

[0109] The text processing unit is further configured to perform feature extraction on the second reference text by the text processing network to obtain a second text feature;

[0110] The image processing unit is further configured to perform feature extraction on the second reference image by the image processing network to obtain a second initial image feature, add a noise feature to the second initial image feature to obtain a third image feature, and denoise the third image feature guided by the second text feature to obtain a target intermediate feature, where the noise feature is used to represent random noise.

[0111] The text processing unit is further configured to perform, by the text processing network, generation of a target text based on the target intermediate feature, where the target text is consistent with the description of the second reference text, and the content described by the target text is consistent with the content of the second reference image.

[0112] Optionally, the image processing unit is configured to perform:

[0113] denoising the third image feature guided by the second text feature by the image processing network to obtain a third intermediate feature;

[0114] performing nonlinear transformation on the third intermediate feature by the image processing network to obtain the target intermediate feature, where the target intermediate feature is an enhanced feature of the third intermediate feature.

[0115] Optionally, the text processing unit is configured to perform:

[0116] fusion of the target intermediate feature and a second initial text feature by the text processing network to obtain a target text feature, where the second initial text feature is a feature obtained by encoding the second reference text;

[0117] generation of the target text based on the target text feature by the text processing network.

[0118] Optionally, the apparatus further comprises a model training unit configured to perform:

[0119] obtaining a first sample reference text and a first sample reference image, where the first sample reference text is used to describe the first sample reference image;

[0120] performing feature extraction on the first sample reference text by the text processing network to obtain a first sample text feature;

[0121] The image processing network is used for performing feature extraction on the first sample reference image to obtain a first sample initial image feature, adding a noise feature to the first sample initial image feature to obtain a first sample image feature, performing denoising on the first sample image feature with the first sample text feature as guidance to obtain a first sample intermediate feature, performing nonlinear transformation on the first sample intermediate feature to obtain a second sample intermediate feature, performing denoising on the first sample intermediate feature with the first sample text feature and the second sample intermediate feature as guidance to obtain a second sample image feature, and the noise feature is used for representing random noise.

[0122] The data generation model is trained based on at least one feature obtained through denoising, the first sample initial image feature, and the noise feature.

[0123] Optionally, the model training unit is further configured to perform:

[0124] obtaining a second sample reference text, a second sample reference image, and a target reference text, the second sample reference text being used for describing the target reference text, and the target reference text being used for describing the second sample reference image;

[0125] The text processing network is used for performing feature extraction on the second sample reference text to obtain a second sample text feature.

[0126] The image processing network is used for performing feature extraction on the second sample reference image to obtain a second sample initial image feature, adding the noise feature to the second sample initial image feature to obtain a third sample image feature, and performing denoising on the third sample image feature with the second sample text feature as guidance to obtain a target sample intermediate feature.

[0127] The text processing network is used for generating a predicted text based on the target sample intermediate feature.

[0128] A first loss parameter is determined based on at least one feature obtained through denoising, the first sample initial image feature, and the noise feature, and a second loss parameter is determined based on the predicted text and the target reference text.

[0129] The data generation model is trained based on the first loss parameter and the second loss parameter.

[0130] According to still another aspect of the embodiments of the present disclosure, an electronic device is provided, which includes:

[0131] a processor;

[0132] a memory for storing processor-executable instructions.

[0133] The processor is configured to execute the instructions to implement the data generation method according to the above aspect.

[0134] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, and instructions in the computer readable storage medium are executed by a processor of an electronic device, so that the electronic device can perform the data generation method according to the above aspect.

[0135] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, and the computer program product comprises a computer program, and the computer program is executed by a processor to implement the data generation method according to the above aspect.

[0136] The method provided by the embodiments of the present disclosure processes the text features and image features of multiple modalities in a data generation scenario, uses the text features of the reference text as a condition to guide the denoising process of the image features, and uses the intermediate features generated in the denoising process as a condition to guide the denoising process of the image features after feature enhancement, thereby achieving unified semantic understanding based on the features of multiple modalities, and facilitating the fusion and synergistic effect of cross-modal features. Moreover, the feature enhancement belongs to a perception task, and the feature denoising belongs to a generation task. The intermediate features generated in the generation task are enhanced in the perception task, and the enhanced features in the perception task are used as a condition to guide the generation task, thereby realizing the deep fusion between the perception task and the generation task. Therefore, the processing manner of the present disclosure can improve the understanding ability and generation ability of the multi-modal features in the data generation scenario, and further improve the accuracy of the finally generated target image.

[0137] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0138] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0139] Figure 1 is a schematic diagram of a computer system according to an exemplary embodiment;

[0140] Figure 2 is a structural schematic diagram of a data generation model according to an exemplary embodiment;

[0141] Figure 3 is a flowchart of a data generation method according to an exemplary embodiment;

[0142] Figure 4is a flowchart of another data generation method according to an example embodiment;

[0143] Figure 5 is a flowchart of yet another data generation method according to an example embodiment;

[0144] Figure 6 is a flowchart of a text generation method according to an example embodiment;

[0145] Figure 7 is a flowchart of a training method of a data generation model according to an example embodiment;

[0146] Figure 8 is a flowchart of still another data generation method according to an example embodiment;

[0147] Figure 9 is a block diagram of a data generation apparatus according to an example embodiment;

[0148] Figure 10 is a block diagram of another data generation apparatus according to an example embodiment;

[0149] Figure 11 is a block diagram of a terminal according to an example embodiment;

[0150] Figure 12 is a block diagram of a server according to an example embodiment. DETAILED DESCRIPTION

[0151] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0152] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above description of the drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present disclosure are all authorized by the user or fully authorized by all parties. The collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the texts and images involved in the present disclosure are obtained under full authorization.

[0154] The embodiment of the present disclosure provides a data generation method, and the execution subject is an electronic device. Illustratively, the electronic device is a terminal, and the terminal is a computer, a mobile phone, a tablet computer or other terminals. Illustratively, the electronic device is a server, and the server is a background server or a cloud server providing cloud computing and cloud storage services, etc. The electronic device can generate a target image based on a reference text, or generate a target image based on a reference text and a reference image, or generate a target text based on a reference text and a reference image, etc. by using a data generation model, and the data generation model can be used to process any data generation task related to texts and images.

[0155] Figure 1 is a schematic diagram of a computer system according to an exemplary embodiment, referring to Figure 1 The computer system includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected through a wireless or wired network.

[0156] The terminal 101 is installed and runs a client, which can be a social application client, an online payment client, a network shopping client, a game client, a medical service client, a video client, etc. When the terminal 101 runs the client, the user interface of the client is displayed on the screen of the terminal 101. The terminal 101 is a terminal used by a user.

[0157] Optionally, the terminal 101 can generally refer to one of a plurality of terminals, and those skilled in the art can know that the number of the above terminals can be more or less. For example, the above terminals can be only one, or the above terminals can be 6 or 8 or more. The number and type of the terminal are not limited in the embodiment of the present disclosure.

[0158] The server 102 includes at least one of a server, a plurality of servers, a cloud computing platform and a virtualization center. The server 102 is used to provide background services for the client. Optionally, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or the server 102 and the terminal 101 adopt a distributed computing architecture for collaborative computing.

[0159] In an embodiment, the terminal 101 sends an image generation request carrying a reference text to the server 102, and the server 102 obtains the reference text upon receiving the image generation request. The server 102 generates a target image matching the reference text based on the reference text through a data generation model, and returns the target image to the terminal 101, which can display the target image to the user.

[0160] In another embodiment, the server 102 trains a data generation model and distributes the data generation model to the terminal 101, and the terminal 101 generates a target image matching the reference text through the data generation model.

[0161] It should be noted that the above implementation environment is only an example, and the method provided by the embodiments of the present disclosure can also be executed by the terminal 101 or the server 102 alone, or by other computer devices, which are not limited by the embodiments of the present disclosure.

[0162] Figure 2 FIG. 1 is a structural diagram of a data generation model provided by the embodiments of the present disclosure, as shown in FIG. 1, the data generation model includes a text processing network 201 and an image processing network 202, and the text processing network 201 is connected with the image processing network 202. In an image generation task, the output data of the text processing network 201 is used as the input data of the image processing network 202, and in an image understanding task, the output data of the image processing network 202 is used as the input data of the text processing network 201. Figure 2 In some embodiments, as shown in FIG. 2, the text processing network 201 includes a text encoding layer, a text feature processing layer, and a text decoding layer. The text encoding layer is connected with the text feature processing layer, and the output data of the text encoding layer is used as the input data of the text feature processing layer. The text feature processing layer is connected with the text decoding layer, and the output data of the text feature processing layer is used as the input data of the text decoding layer.

[0163] Figure 2 The text encoding layer is used to encode the text into text features, which can be embedded text vectors. The text feature processing layer is used to process the text features to mine deeper text features. The text decoding layer is used to decode the text features into text.

[0164] The text encoding layer is used to encode the text into text features, which can be embedded text vectors. The text feature processing layer is used to process the text features to mine deeper text features. The text decoding layer is used to decode the text features into text.

[0165] ​Optionally, the text processing network 201 is a network constructed based on a Transformer decoder architecture, for example, the Transformer decoder can be Decoder-Only. For example, the text feature processing layer in the text processing network 201 includes a causal attention layer, a normalization layer and a forward feedback layer in the Decoder-Only. For example, the text feature processing layer includes a causal attention layer, a first normalization layer, a forward feedback layer and a second normalization layer connected in sequence.

[0166] In some embodiments, as shown in Figure 2 , the image processing network 202 includes an image encoding layer, N diffusion sub-models connected in sequence, and an image decoding layer. The image encoding layer is connected with the first diffusion sub-model, and the output data of the image encoding layer is used as the input data of the first diffusion sub-model. The image decoding layer is connected with the last diffusion sub-model, and the output data of the last diffusion sub-model is used as the input data of the image decoding layer.

[0167] , the image encoding layer is used to encode an image into image features, the plurality of diffusion sub-models are used to denoise the image features, and the image decoding layer is used to decode the image features into an image.

[0168] Optionally, as shown in Figure 2 , each diffusion sub-model includes a perception layer and an image feature processing layer, and the perception layer and the image feature processing layer are connected. The input data of the perception layer of the kth diffusion sub-model is the output data of the image feature processing layer of the (k-1)th diffusion sub-model, and the input data of the image feature processing layer of the kth diffusion sub-model includes the output data of the text feature processing layer of the text processing network 201, the output data of the perception layer of the kth diffusion sub-model, and the output data of the image feature processing layer of the (k-1)th diffusion sub-model, k is an integer greater than 1 and not greater than N.

[0169] Optionally, the image processing network 202 is a network constructed based on a DiT (Diffusion Transformer) architecture. For example, the perception layer in the diffusion sub-model includes a multi-layer perceptron in the DiT model, and the image feature processing layer in the diffusion sub-model includes a normalization layer, a linear transformation layer, a cross-attention layer, and a scale transformation layer in the DiT model.

[0170] The following embodiments are combined with the data generation model shown in Figure 2 , and the data generation method can be described in detail. Figure 3 is a flowchart of a data generation method according to an example embodiment, referring to Figure 2 , the method is performed by an electronic device and includes the following steps.

[0171] In step 301, the electronic device determines a data generation model, the data generation model comprising a text processing network and an image processing network.

[0172] In the embodiments of the present disclosure, the data generation model comprises the text processing network and the image processing network, and the text processing network and the image processing network are connected with each other. The data generation model can be used to generate an image based on a text or to generate a text based on an image. The structure of the data generation model can refer to the structural schematic diagram of the data generation model in Figure 2 .

[0173] In step 302, the electronic device extracts features of a first reference text through the text processing network to obtain first text features, the first reference text being used to describe an image to be generated.

[0174] In the image generation task, the electronic device obtains a first reference text, the image generation task being a task of generating an image consistent with the description of the first reference text. The electronic device extracts features of the first reference text through the text processing network to obtain first text features, the first text features being used to represent semantic features of the first reference text.

[0175] In step 303, the electronic device guides the first text features to denoise the first image features to obtain first intermediate features, performs nonlinear transformation on the first intermediate features to obtain second intermediate features, and guides the first text features and the second intermediate features to denoise the first intermediate features to obtain second image features.

[0176] In the image generation task, the electronic device further obtains first image features, the first image features being any image features containing noise. The electronic device guides the first text features to denoise the first image features through the image processing network to obtain the first intermediate features, so that the first intermediate features contain semantic information of the first text features and visual information of the first image features. In the present disclosure, the denoising of the features includes the noise reduction of the features.

[0177] To further denoise, the electronic device performs nonlinear transformation on the first intermediate feature to obtain a second intermediate feature, which is an enhanced feature of the first intermediate feature, that is, a more deep and abstract second intermediate feature is obtained by perceiving the first intermediate feature. Further, guided by the first text feature and the second intermediate feature, the first intermediate feature is denoised to obtain a second image feature, thereby enhancing the denoising effect of the second image feature. And, feature enhancement belongs to the perception task, and feature denoising belongs to the generation task. In the perception task, the intermediate feature generated in the generation task is enhanced, and in turn, the enhanced feature in the perception task is used as a condition to guide the generation task, realizing the deep fusion between the perception task and the generation task, and improving the representation ability of the second image feature to semantic information and visual information.

[0178] In step 304, the electronic device generates a target image based on the second image feature through the image processing network, and the target image corresponds to the description of the first reference text.

[0179] After determining the second image feature, the electronic device generates a target image based on the second image feature, and the target image corresponds to the description of the first reference text. For example, the first reference text is "generate an image of a cat running", and the content of the target image is a cat running.

[0180] The method provided by the embodiments of the present disclosure processes multi-modal text features and image features in a data generation scenario, uses the text features of the reference text as a condition to guide the denoising process of the image features, and uses the intermediate features generated in the denoising process as a condition to guide the denoising process of the image features after feature enhancement, thereby realizing unified semantic understanding based on multi-modal features, and facilitating the fusion and synergy of cross-modal features. And, feature enhancement belongs to the perception task, and feature denoising belongs to the generation task. In the perception task, the intermediate feature generated in the generation task is enhanced, and in turn, the enhanced feature in the perception task is used as a condition to guide the generation task, realizing the deep fusion between the perception task and the generation task. Therefore, the processing method of the present disclosure can improve the understanding and generation ability of multi-modal features in the data generation scenario, thereby improving the accuracy of the finally generated target image.

[0181] The above Figure 3 Embodiments are only a brief introduction to the data generation method, and the detailed process of the data generation method is described below. Figure 4 is a flowchart of another data generation method according to an example embodiment, which is performed by an electronic device, see Figure 4 The method includes the following steps.

[0182] In step 401, the electronic device determines a data generation model, the data generation model comprising a text processing network and an image processing network.

[0183] In the embodiments of the present disclosure, the data generation model comprises the text processing network and the image processing network, and the text processing network and the image processing network are connected to each other. The data generation model can be used to generate an image based on text or to generate text based on image understanding, etc.

[0184] In the scenario of generating an image based on text, the text processing network generates text features to assist the image processing network in generating an image, so the output data of the text processing network is the input data of the image processing network.

[0185] In the scenario of generating text based on image understanding, the image processing network generates image features to assist the text processing network in generating text, so the output data of the image processing network is the input data of the text processing network.

[0186] The structure of the data generation model is shown in Figure 2 and will not be described here.

[0187] In step 402, the electronic device obtains first image features, the first image features being any image features containing noise.

[0188] In the image generation task, the electronic device obtains first image features, the first image features being any image features containing noise.

[0189] In some embodiments, the image generation task includes two cases of generating an image based on text and generating an image based on text and image, and then based on this, obtaining the first image features includes the following two ways.

[0190] (1) The first reference image is extracted by the image processing network to obtain the first initial image features, and noise features are added to the first initial image features to obtain the first image features. The first reference image includes a reference object, and the object in the target image is consistent with the reference object. The noise features are used to represent random noise.

[0191] In the case of generating an image based on text and image, in addition to the first reference text, the first reference image is also provided as a reference. Therefore, the electronic device extracts the first reference image by the image processing network to obtain the first initial image features of the first reference image, and then adds noise features to the first initial image features to obtain the first image features.

[0192] Optionally, as shown in Figure 2As shown, the image processing network includes an image coding layer. The electronic device inputs a first reference image into the image coding layer, and the image coding layer encodes the first reference image and outputs a first initial image feature.

[0193] Optionally, the noise feature may be random Gaussian noise or the like.

[0194] (2) Determine the noise feature as the first image feature.

[0195] In the case of generating an image based on text, only the first reference text is provided, and no reference image is provided. Therefore, the electronic device directly obtains the noise feature of the random noise and uses the noise feature as the first image feature.

[0196] Optionally, the noise feature may be random Gaussian noise or the like.

[0197] In the disclosed embodiments, when a reference image is available, noise features are added to the image features of the reference image as the first initial image feature, enabling the task of generating a new image from an image and text. When no reference image is available, the noise features are directly used as the first initial image feature, enabling the task of generating an image from text, thereby improving the applicability of the data generation model.

[0198] In step 403, the electronic device extracts features from the first reference text through a text processing network to obtain first text features. The first reference text is used to describe the image to be generated.

[0199] In an image generation task, an electronic device obtains a first reference text. The task involves generating an image that matches the description in the first reference text. The electronic device inputs the first reference text into a text processing network, which extracts features from the first reference text and outputs first text features.

[0200] In some embodiments, as Figure 2 As shown, the text processing network includes a text encoding layer and a text feature processing layer. Step 403 includes: encoding the first reference text into a first initial text feature through the text encoding layer in the text processing network, and performing causal attention encoding on the first initial text feature through the text feature processing layer in the text processing network to obtain a first text feature.

[0201] The electronic device inputs the first reference text into the text encoding layer. The text encoding layer encodes the first reference text and outputs a first initial text feature, for example, an embedded text feature vector. The first initial text feature output by the text encoding layer is input into the text feature extraction layer. The text feature extraction layer performs causal attention encoding on the first initial text feature and outputs a first text feature.

[0202] Optionally, the text feature extraction layer includes a causal attention layer, a first normalization layer, a forward feedback layer and a second normalization layer connected in sequence. The electronic device inputs the first initial text feature into the causal attention layer, which is processed in sequence by the causal attention layer, the first normalization layer, the forward feedback layer and the second normalization layer, and finally the second normalization layer outputs the first text feature.

[0203] Alternatively, as Figure 2 As shown, the text processing network also includes a text decoding layer, which is used to decode text features into text. However, since there is no need to generate text in the image generation task, the embodiment of the present disclosure does not need to be processed by the text decoding layer, and the first text feature output by the text feature processing layer can be directly provided to the image processing network.

[0204] In this disclosed embodiment, a text encoding layer and a text feature processing layer are introduced into the text processing network, enabling deeper semantic understanding and contextual modeling of the reference text. The text encoding layer efficiently encodes the first reference text into an embedded feature representation, while the text feature processing layer utilizes an autoregressive language processing mechanism to capture the dependencies of words in the text on preceding words, thereby improving the representational capabilities of the first text features, thereby enabling text-driven image generation and enhancing image generation accuracy.

[0205] In some embodiments, the process of causal attention encoding includes: mapping the first initial text feature into a first query feature, a first key feature and a first value feature through a text feature processing layer; masking the product of the transpose of the first query feature and the first key feature and the scaling factor based on the causal mask feature through the text feature processing layer to obtain a first weight feature, the first reference text includes n characters, and the causal mask feature is used to mask the features of the i+1th character to the nth character when determining the weight feature of the i-th character, where n is a positive integer and i is a positive integer not greater than n; multiplying the first weight feature by the first value feature through the text feature processing layer to obtain a first text feature.

[0206] The text feature processing layer includes a first conversion matrix, a second conversion matrix, and a third conversion matrix, which serve as model parameters of the text feature processing layer. The first conversion matrix is ​​multiplied by the first initial text feature to obtain the first query feature, the second conversion matrix is ​​multiplied by the first initial text feature to obtain the first key feature, and the third conversion matrix is ​​multiplied by the first initial text feature to obtain the first value feature.

[0207] wherein the causal mask matrix is an n*n matrix, and n can be the number of characters in the first reference text. The causal mask matrix is a lower triangular matrix, which is configured to reserve only the feature paths from the first character to the i th character and mask the features of the i+1 th character to the n th character when processing the features of the i th character. For example, the element in the i th row and the j th column of the causal mask matrix is a valid value when j is less than or equal to i, and is a mask value when j is greater than i.

[0208] For example, the element in the i th row and the j th column of the causal mask matrix is 0 when j is less than or equal to i, and is negative infinity when j is greater than i. The product of the first query feature, the transpose of the first key feature and the scaling factor is determined, the product is added to the causal mask matrix, the result of the addition is normalized to obtain the first weight feature, and then the first weight feature is multiplied by the first value feature to obtain the first text feature.

[0209] In the embodiments of the present disclosure, by introducing the causal attention mechanism in the text processing network, the subsequent characters are masked by using the causal mask, so that the autoregressive modeling is performed in the text feature extraction process, and the sequence dependency and semantic rationality of the extracted text feature are ensured. Moreover, by mapping the initial text feature into a query feature, a key feature and a value feature, and combining the mask processing, the effective information in the context can be dynamically focused, the important role of the current character in semantic understanding is highlighted, so that the representation ability of the text feature is improved, which is beneficial to providing more time sequence logic text guidance for subsequent image generation, and improving the text consistency and expression accuracy of the generated image.

[0210] In step 404, the electronic device performs nonlinear transformation on the first image feature through the image processing network to obtain a first enhanced feature. The first enhanced feature is a feature after enhancement of the first image feature.

[0211] The electronic device performs nonlinear transformation on the first image feature through the image processing network to obtain a first enhanced feature, that is, a more deep and abstract first enhanced feature is obtained by perceiving the first image feature.

[0212] In some embodiments, as shown in Figure 2 The image processing network includes a perception layer, and the electronic device inputs the first image feature into the perception layer to perform nonlinear transformation on the first image feature through the perception layer to obtain the first enhanced feature.

[0213] Optionally, the perception layer can be an MLP (Multilayer Perceptron, multi-layer perceptron).

[0214] In step 405, the electronic device performs denoising on the first image feature by using the first text feature and the first enhanced feature as guidance through the image processing network to obtain a first intermediate feature.

[0215] The electronic device performs denoising on the first image feature by using the first text feature and the first enhanced feature as guidance in the image processing network to obtain a first intermediate feature, which is an intermediate representation after denoising.

[0216] In the embodiments of the present disclosure, the first image feature is first nonlinearly enhanced through the image processing network to extract more discriminative visual information, and then the first image feature is denoised by combining the text feature and the enhanced feature as double guidance, which effectively improves the stability of the generation process and the fidelity of image details. The scheme realizes deep fusion of text semantic information and image structural information, so that the generated image achieves a better balance between content accuracy and visual quality, thereby enhancing the image feature expression and denoising effect.

[0217] In some embodiments, the electronic device performs cross-attention encoding on the first text feature, the first enhanced feature, and the first image feature through an image feature processing layer in the image processing network to obtain the first intermediate feature.

[0218] As shown in Figure 2 The image processing network includes an image feature processing layer, and the electronic device inputs the first text feature, the first enhanced feature, and the first image feature into the image feature processing layer, and the image feature processing layer outputs the first intermediate feature.

[0219] Optionally, the image feature processing layer includes a normalization layer, a linear transformation layer, a cross-attention layer, and a scale transformation layer connected in sequence, the first image feature is input into the normalization layer, the first text feature and the first enhanced feature are input into the cross-attention layer for processing in sequence, and finally the scale transformation layer outputs the first intermediate feature.

[0220] In the embodiments of the present disclosure, the text feature, the first enhanced feature, and the first image feature are cross-attention encoded through the image feature processing layer in the image processing network, which can dynamically capture the correlation between text semantics and visual information, accurately adjust the denoising process of the image feature, and thus generate an intermediate feature that is more consistent with the text description and rich in details, thereby improving the coordination and expression ability of multi-modal information and enhancing the semantic consistency and visual effect of image generation.

[0221] Optionally, the process of cross-attention encoding comprises: mapping, by the image feature processing layer, the first text feature into a first query sub-feature, a first key sub-feature, and a first value sub-feature, mapping the first image feature into a second query sub-feature, and mapping the first enhanced feature into a second key sub-feature and a second value sub-feature; and cross-fusing, by the image feature processing layer, the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature to obtain the first intermediate feature.

[0222] The image feature processing layer comprises a fourth conversion matrix, a fifth conversion matrix, a sixth conversion matrix, a seventh conversion matrix, an eighth conversion matrix, and a ninth conversion matrix, and the conversion matrices are model parameters of the image feature processing layer. The fourth conversion matrix is multiplied by the first text feature to obtain the first query sub-feature, the fifth conversion matrix is multiplied by the first text feature to obtain the first key sub-feature, and the sixth conversion matrix is multiplied by the first text feature to obtain the first value sub-feature. The seventh conversion matrix is multiplied by the first image feature to obtain the second query sub-feature, the eighth conversion matrix is multiplied by the first enhanced feature to obtain the second key sub-feature, and the ninth conversion matrix is multiplied by the first enhanced feature to obtain the second value sub-feature.

[0223] In the embodiments of the present disclosure, the first query sub-feature, the first key sub-feature, and the first value sub-feature are obtained by text feature mapping, the second query sub-feature, the second key sub-feature, and the second value sub-feature are obtained by image feature mapping, and the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature are cross-fused, which can effectively capture the associated information and complementary features between different modalities. This fine cross-fusion improves the joint understanding ability of text semantics and visual information, and is beneficial to generating more rich and semantically accurate intermediate features.

[0224] Optionally, the electronic device cross-fuses the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature, comprising: splicing, by the image feature processing layer, the first query sub-feature and the second query sub-feature to obtain a second query feature, splicing the first key sub-feature and the second key sub-feature to obtain a second key feature, and splicing the first value sub-feature and the second value sub-feature to obtain a second value feature; and normalizing, by the image feature processing layer, a product of a transpose of the second query feature and the second key feature and a scaling factor to obtain a second weight feature, and multiplying the second weight feature and the second value feature to obtain the first intermediate feature.

[0225] For example, the electronic device uses the following formulas (1) to (3) to determine the second query feature, the second key feature, and the second value feature.

[0226] Equation (1)

[0227] Equation (2)

[0228] Equation (3)

[0229] wherein, denotes the second query feature, denotes the second key feature, denotes the second value feature, denotes the fourth conversion matrix, denotes the fifth conversion matrix, denotes the sixth conversion matrix, denotes the seventh conversion matrix, denotes the eighth conversion matrix, denotes the ninth conversion matrix, denotes the first text feature, denotes the first image feature, denotes the first enhanced feature, denotes the splicing operation.

[0230] In the embodiments of the present disclosure, the first query sub-feature, the first key sub-feature and the first value sub-feature of the text feature are spliced with the second query sub-feature, the second key sub-feature and the second value sub-feature of the image feature to form a unified multi-modal feature representation, which is beneficial to enhance the capturing ability of complex associations between text semantics and visual information, so that the generated intermediate feature has rich text semantics and retains delicate visual details, thereby laying a foundation for high-quality and multi-modal fusion image generation.

[0231] In step 406, the electronic device performs a non-linear transformation on the first intermediate feature through an image processing network to obtain a second intermediate feature, and the second intermediate feature is an enhanced feature of the first intermediate feature.

[0232] The electronic device performs a non-linear transformation on the first intermediate feature through an image processing network to obtain a second intermediate feature, that is, a more deep and abstract second intermediate feature is obtained by perceiving the first intermediate feature.

[0233] In some embodiments, as shown in Figure 2 the image processing network includes a perception layer, and the electronic device inputs the first intermediate feature into the perception layer to perform a non-linear transformation on the first intermediate feature through the perception layer to obtain the second intermediate feature.

[0234] In the disclosed embodiment, the intermediate features generated in the process of generative modeling (denoising and generating images) are enhanced, and the enhanced features are used as conditions to further guide the next step of denoising of the intermediate features, thereby utilizing the intermediate features generated in the process of generative modeling to enhance perception capabilities, which is conducive to achieving a deep fusion of perception capabilities and generation capabilities and improving the accuracy of denoising.

[0235] In step 407, the electronic device uses an image processing network to denoise the first intermediate feature with the first text feature and the second intermediate feature as guidance to obtain the second image feature.

[0236] In the image processing network, the electronic device uses the first text feature and the second intermediate feature as a guide to denoise the first intermediate feature to obtain a second image feature, which is the denoised image feature.

[0237] In some embodiments, the electronic device performs cross-attention encoding on the first text feature, the second intermediate feature, and the first intermediate feature through an image feature processing layer in an image processing network to obtain a second image feature.

[0238] like Figure 2 As shown, the image processing network includes an image feature processing layer, the electronic device inputs the first text feature, the second intermediate feature and the first intermediate feature into the image feature processing layer, and the image feature processing layer outputs the second image feature.

[0239] In the disclosed embodiment, the first text feature, the second intermediate feature, and the first intermediate feature are cross-attention encoded through the image feature processing layer in the image processing network, which can dynamically capture the correlation between text semantics and visual information, and accurately adjust the denoising process of the image features, thereby generating intermediate features that are more consistent with the text description and rich in details, improving the coordination and expression ability of multimodal information, and enhancing the semantic consistency and visual effect of image generation.

[0240] Optionally, the cross-attention encoding process includes: mapping the first text feature to a third query sub-feature, a third key sub-feature and a third value sub-feature through an image feature processing layer, mapping the first intermediate feature to a fourth query sub-feature, and mapping the second intermediate feature to a fourth key sub-feature and a fourth value sub-feature; and cross-fusing the third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature through an image feature processing layer to obtain a second image feature.

[0241] The image feature processing layer includes a tenth conversion matrix, an eleventh conversion matrix, a twelfth conversion matrix, a thirteenth conversion matrix, a fourteenth conversion matrix, and a fifteenth conversion matrix, and the conversion matrices are model parameters of the image feature processing layer. The tenth conversion matrix is multiplied by the first text feature to obtain a third query sub-feature, the eleventh conversion matrix is multiplied by the first text feature to obtain a third key sub-feature, and the twelfth conversion matrix is multiplied by the first text feature to obtain a third value sub-feature. The thirteenth conversion matrix is multiplied by the first intermediate feature to obtain a fourth query sub-feature, the fourteenth conversion matrix is multiplied by the second intermediate feature to obtain a fourth key sub-feature, and the fifteenth conversion matrix is multiplied by the second intermediate feature to obtain a fourth value sub-feature.

[0242] In the embodiments of the present disclosure, the third query sub-feature, the third key sub-feature, and the third value sub-feature are obtained through text feature mapping, the fourth query sub-feature, the fourth key sub-feature, and the fourth value sub-feature are obtained through image feature mapping, and the third query sub-feature, the third key sub-feature, and the third value sub-feature are cross-fused with the fourth query sub-feature, the fourth key sub-feature, and the fourth value sub-feature. The cross-fusion can effectively capture the associated information and complementary features between different modalities, the fine cross-fusion improves the joint understanding ability of text semantics and visual information, and is beneficial to generating more rich and accurate semantic intermediate features.

[0243] Optionally, the electronic device cross-fuses the third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature, and the fourth value sub-feature, including: through the image feature processing layer, splicing the third query sub-feature and the fourth query sub-feature to obtain a third query feature, splicing the third key sub-feature and the fourth key sub-feature to obtain a third key feature, and splicing the third value sub-feature and the fourth value sub-feature to obtain a third value feature; through the image feature processing layer, normalizing the product of the transpose of the third query feature and the third key feature and a scaling factor to obtain a third weight feature, and multiplying the third weight feature and the third value feature to obtain a second image feature.

[0244] The process of determining the third query feature, the third key feature, and the third value feature by the electronic device is the same as the process of determining the second query feature, the second key feature, and the second value feature in the above step 405, and can be referred to the above formulas (1) to (3), which will not be described here.

[0245] In the embodiments of the present disclosure, the third query sub-feature, the third key sub-feature and the third value sub-feature of the text feature are spliced with the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature of the image feature to form a unified multi-modal feature representation, which is beneficial to enhance the capturing ability of the complex association between the text semantics and the visual information, so that the generated intermediate feature has rich text semantics and retains delicate visual details, thereby laying a foundation for high-quality and multi-modal fusion image generation.

[0246] In step 408, the electronic device generates a target image based on the second image feature through an image processing network, and the target image is consistent with the description of the first reference text.

[0247] After determining the second image feature, the electronic device generates a target image based on the second image feature, and the target image is consistent with the description of the first reference text.

[0248] In some embodiments, the first image feature is obtained by adding a noise feature to the first initial image feature of the first reference image, and then the target image is not only consistent with the description of the first reference text, but also the object in the target image is consistent with the reference object. For example, the first reference text is "generate an image of a cat running", and the first reference image is an image including a cat, then the content of the target image is a cat running, and the cat in the target image is the same as the cat in the first reference image.

[0249] In some embodiments, as shown in Figure 2 The image processing network includes an image decoding layer, and the electronic device inputs the second image feature into the image decoding layer, decodes the second image feature by the image decoding layer, and outputs the target image.

[0250] The method provided by the embodiments of the present disclosure processes the multi-modal text feature and image feature in the data generation scenario, uses the text feature of the reference text as a condition to guide the denoising process of the image feature, and uses the intermediate feature generated in the denoising process as a condition to guide the denoising process of the image feature after feature enhancement, thereby realizing unified semantic understanding based on multi-modal features, and facilitating the fusion and synergy of cross-modal features. Moreover, feature enhancement belongs to a perception task, and feature denoising belongs to a generation task. The intermediate feature generated in the generation task is enhanced in the perception task, and the enhanced feature in the perception task is used as a condition to guide the generation task, thereby realizing the deep fusion between the perception task and the generation task. Therefore, the processing manner of the present disclosure can improve the understanding ability and generation ability of the multi-modal feature in the data generation scenario, and further improve the accuracy of the finally generated target image.

[0251] Figure 5is a flowchart of yet another data generation method according to an exemplary embodiment, which is performed by an electronic device, see Figure 5 The method comprises the following steps.

[0252] In step 501, the electronic device determines a data generation model, the data generation model comprising a text processing network and an image processing network, the image processing network comprising N diffusion sub-networks and an image decoding layer, N being an integer greater than 1.

[0253] The structure of the data generation model is shown in Figure 2 , which will not be repeated here.

[0254] In step 502, the electronic device obtains a first image feature, which is any image feature containing noise.

[0255] The process of obtaining the first image feature in step 502 is the same as the process of obtaining the first image feature in step 402 described above, which will not be repeated here.

[0256] In step 503, the electronic device extracts features from a first reference text through the text processing network to obtain a first text feature, the first reference text being used to describe an image to be generated.

[0257] The process of obtaining the first text feature in step 503 is the same as the process of obtaining the first text feature in step 403 described above, which will not be repeated here.

[0258] In step 504, the electronic device denoises the first image feature with the first text feature as a guide through the first diffusion sub-network in the image processing network to obtain a first intermediate feature.

[0259] The electronic device inputs the first text feature and the first image feature into the first diffusion sub-network, and the first diffusion sub-network outputs the first intermediate feature.

[0260] In some embodiments, as shown in Figure 2 , the first diffusion sub-network comprises a perception layer and an image feature processing layer, the electronic device inputs the first image feature into the perception layer, and performs a nonlinear transformation on the first image feature through the perception layer, the perception layer outputs a first enhanced feature. The first text feature, the first enhanced feature and the first image feature are input into the image feature processing layer, and the image feature processing layer denoises the first image feature with the first text feature and the first enhanced feature as a guide, the image feature processing layer outputs the first intermediate feature.

[0261] Optionally, the electronic device performs cross-attention encoding on the first text feature, the first enhanced feature, and the first image feature through the image feature processing layer to obtain a first first intermediate feature. The process of obtaining the first first intermediate feature is similar to the process of obtaining the first intermediate feature in step 405 above and is not further described here.

[0262] In step 505, the electronic device performs a nonlinear transformation on the k-1th first intermediate feature through the k-1th diffusion subnetwork in the image processing network to obtain the k-1th second intermediate feature, and denoises the k-1th first intermediate feature using the first text feature and the k-1th second intermediate feature as a guide to obtain the k-1th first intermediate feature, where k is an integer greater than 1 and not greater than N.

[0263] The computer device inputs the first text feature and the k-1th first intermediate feature into the kth diffusion sub-network, and the kth diffusion sub-network outputs the kth first intermediate feature.

[0264] In some embodiments, as Figure 2 As shown, the k-th diffusion subnetwork includes a perception layer and an image feature processing layer. The electronic device inputs the k-1th first intermediate feature into the perception layer, which performs a nonlinear transformation on the k-1th first intermediate feature. The perception layer then outputs the k-1th second intermediate feature. The first text feature, the k-1th first intermediate feature, and the k-1th second intermediate feature are then input into the image feature processing layer. The image feature processing layer, guided by the first text feature and the k-1th second intermediate feature, performs denoising on the k-1th first intermediate feature. The image feature processing layer then outputs the k-th first intermediate feature.

[0265] Optionally, the electronic device performs cross-attention encoding on the first text feature, the k-1th first intermediate feature, and the k-1th second intermediate feature through the image feature processing layer to obtain the kth first intermediate feature. The process of obtaining the kth first intermediate feature is similar to the process of obtaining the first intermediate feature in step 405 above and is not further described here.

[0266] In step 506 , the electronic device uses the Nth first intermediate feature as the second image feature.

[0267] Among them, the Nth first intermediate feature is also the first intermediate feature output by the last diffusion sub-network in the N diffusion sub-networks, and the last first intermediate feature serves as the second image feature.

[0268] In step 507, the electronic device generates a target image based on the second image feature through the image decoding layer in the image processing network, and the target image is consistent with the description of the first reference text.

[0269] The process of generating the target image in this step 507 is the same as the process of generating the target image in the above step 408, and will not be repeated here.

[0270] The method provided by the embodiments of the present disclosure processes the text features and image features of multiple modalities in a data generation scenario, uses the text features of the reference text as a condition to guide the denoising process of the image features, and uses the intermediate features generated in the denoising process after feature enhancement as a condition to guide the denoising process of the image features, thereby achieving unified semantic understanding based on multiple modal features, which is conducive to promoting the fusion and synergy of cross-modal features. Moreover, feature enhancement belongs to a perception task, and feature denoising belongs to a generation task. The intermediate features generated in the generation task are enhanced in the perception task, and the enhanced features in the perception task are used as a condition to guide the generation task, thereby realizing the deep fusion between the perception task and the generation task. Therefore, the processing manner of the present disclosure can improve the understanding and generation capabilities of the multi-modal features in the data generation scenario, thereby improving the accuracy of the finally generated target image.

[0271] Moreover, the multiple diffusion sub-networks use the text features and the intermediate features of the previous layer to guide in turn, gradually refine the image features, and improve the denoising effect and feature expression capability. Moreover, the intermediate features generated in the previous diffusion sub-network are enhanced, and the enhanced features are used to guide the generation process of the current diffusion sub-network, thereby realizing the deep fusion between the perception task and the generation task. Therefore, by introducing the cascade structure of the multiple diffusion sub-networks, a multi-stage iterative optimization process of image feature denoising and enhancement is realized.

[0272] The above embodiments illustrate the process of generating a target image by a data generation model. In addition to the image generation task, the data generation model can also be used to generate a target text by understanding an image. For details, refer to the following Figure 6 embodiments. Figure 6 is a flowchart of a text generation method according to an example embodiment, which is performed by an electronic device, as shown in Figure 6 , the method includes the following steps.

[0273] In step 601, the electronic device obtains a second reference text and a second reference image, and the second reference text is used to describe the text to be generated.

[0274] In the task of generating a text based on an image, the second reference text is used to constrain the text to be generated. For example, the second reference text can be used to indicate the number of words, style, etc. of the text to be generated.

[0275] In step 602, the electronic device performs feature extraction on the second reference text by a text processing network in the data generation model to obtain second text features.

[0276] In some embodiments, as shown in Figure 2 the text processing network includes a text encoding layer and a text feature processing layer, and step 602 includes: encoding the second reference text into second initial text features by the text encoding layer in the text processing network, and performing causal attention encoding on the second initial text features by the text feature processing layer in the text processing network to obtain the second text features.

[0277] In step 602, the electronic device performs feature extraction on the second reference text by a text processing network in the data generation model to obtain second text features.

[0278] In step 603, the electronic device performs feature extraction on the second reference image by an image processing network in the data generation model to obtain second initial image features, adds noise features to the second initial image features to obtain third image features, and performs denoising on the third image features guided by the second text features to obtain target intermediate features.

[0279] In step 603, the electronic device performs feature extraction on the second reference image by an image processing network in the data generation model to obtain second initial image features, adds noise features to the second initial image features to obtain third image features, and performs denoising on the third image features guided by the second text features to obtain target intermediate features.

[0280] In some embodiments, as shown in Figure 2 the image processing network includes an image encoding layer, and the electronic device inputs the second reference image into the image encoding layer to encode the second reference image by the image encoding layer to output the second initial image features.

[0281] In some embodiments, the electronic device performs denoising on the third image features guided by the second text features to obtain the target intermediate features, including: performing denoising on the third image features guided by the second text features by the image processing network to obtain third intermediate features; and performing nonlinear transformation on the third intermediate features by the image processing network to obtain the target intermediate features, the target intermediate features being enhanced features of the third intermediate features.

[0282] Optionally, the third image feature is subjected to a non-linear transformation by the image processing network to obtain a second enhanced feature, the second enhanced feature being an enhanced feature of the third image feature, the third image feature is subjected to denoising with the second text feature and the second enhanced feature as guidance to obtain a third intermediate feature, the third intermediate feature is subjected to a non-linear transformation to obtain a fourth intermediate feature, the fourth intermediate feature being an enhanced feature of the third intermediate feature, the third intermediate feature is subjected to denoising with the second text feature and the fourth intermediate feature as guidance to obtain a fifth intermediate feature, and the fifth intermediate feature is subjected to a non-linear transformation to obtain the target intermediate feature. That is, the image feature needs to be subjected to multi-level denoising to obtain the target intermediate feature.

[0283] In the embodiments of the present disclosure, the process of obtaining the fifth intermediate feature is the same as the process of obtaining the second image feature in the above embodiments.

[0284] In the embodiments of the present disclosure, the image feature is first denoised with the second text feature as guidance, and then the denoised image feature is enhanced. This not only enhances the response capability of the image feature to the text semantics, but also effectively preserves the key details of the image content, improves the expression quality of the target intermediate feature, and makes the subsequently generated target text more accurate and semantically consistent.

[0285] In step 604, the electronic device generates a target text based on the target intermediate feature by a text processing network, the target text being consistent with the description of the second reference text, and the content described by the target text being consistent with the content of the second reference image.

[0286] After determining the target intermediate feature, the electronic device generates a target text based on the target intermediate feature, the target text being consistent with the description of the second reference text, and the content described by the target text being consistent with the content of the second reference image. For example, the second reference text is used to indicate the number of words to be generated, and the number of words of the target text is consistent with the number of words indicated by the second reference text.

[0287] In some embodiments, the electronic device fuses the target intermediate feature and a second initial text feature by a text processing network to obtain a target text feature, the second initial text feature being a feature obtained by encoding the second reference text, and generates the target text based on the target text feature by the text processing network.

[0288] The process of obtaining the second initial text feature is the same as the process of obtaining the first initial text feature in the above embodiments, and will not be repeated here.

[0289] Optionally, as Figure 2As shown, the text processing network includes a text feature processing layer. The electronic device inputs the target intermediate feature and the second initial text feature into the text feature processing layer, and the text feature processing layer performs causal attention encoding on the target intermediate feature and the second initial text feature to obtain the target text feature.

[0290] Among them, the electronic device maps the target intermediate feature into the fifth query sub-feature, the fifth key sub-feature and the fifth value sub-feature through the text feature processing layer, maps the second initial text feature into the sixth query sub-feature, the sixth key sub-feature and the sixth value sub-feature, and cross-integrates the fifth query sub-feature, the fifth key sub-feature, the fifth value sub-feature, the sixth query sub-feature, the sixth key sub-feature and the sixth value sub-feature to obtain the target text feature.

[0291] Exemplarily, through the text feature processing layer, the fifth query sub-feature and the sixth query sub-feature are spliced ​​to obtain the fourth query feature, the fifth key sub-feature and the sixth key sub-feature are spliced ​​to obtain the fourth key feature, and the fifth value sub-feature and the sixth value sub-feature are spliced ​​to obtain the fourth value feature; through the text feature processing layer, the product of the transpose of the fourth query feature and the fourth key feature and the scaling factor is normalized to obtain the fourth weight feature, and the fourth weight feature is multiplied by the fourth value feature to obtain the target text feature.

[0292] Alternatively, as Figure 2 As shown, the text processing network includes a text decoding layer. The electronic device inputs the target text features into the text decoding layer, and the target text features are decoded by the text decoding layer to obtain the target text.

[0293] In the disclosed embodiment, during the text generation process, the target intermediate features obtained by image processing and the initial text features obtained by text encoding are fused to construct target text features with highly unified semantics and visual information, thereby achieving deep coordination between visual semantics and language semantics. This ensures that the generated text not only closely fits the details of the reference image, but also more accurately continues the expression intent of the original text, which helps to improve the coherence, accuracy and diversity of text generation, thereby improving the generation quality of the target text.

[0294] According to the method provided by the embodiments of the present disclosure, the text processing network and the image processing network in the data generation model can also be used to execute the path of generating text from images. Therefore, the data generation model can realize both image generation and image understanding based on the ability of bidirectional generation of images and texts, which significantly improves the image generation and image understanding capabilities of the data generation model, provides greater flexibility and practicality for the multimodal data generation model, and broadens the application space of the data generation model in fields such as intelligent content generation.

[0295] The following Figure 7The embodiments of the present application illustrate the training process of the data generation model. Figure 7 is a flowchart of a training method of a data generation model according to an exemplary embodiment, which is executed by an electronic device, referring to Figure 7 , the method comprises the following steps.

[0296] In step 701, the electronic device obtains a first sample reference text and a first sample reference image, and the first sample reference text is used to describe the first sample reference image.

[0297] In step 702, the electronic device extracts features from the first sample reference text through a text processing network to obtain a first sample text feature.

[0298] The process of obtaining the sample text feature in step 702 is the same as the process of obtaining the first text feature in step 403, and will not be repeated here.

[0299] In step 703, the electronic device extracts features from the first sample reference image through an image processing network to obtain a first sample initial image feature, adds a noise feature to the first sample initial image feature to obtain a first sample image feature, denoises the first sample image feature guided by the first sample text feature to obtain a first sample intermediate feature, performs a nonlinear transformation on the first sample intermediate feature to obtain a second sample intermediate feature, denoises the first sample intermediate feature guided by the first sample text feature and the second sample intermediate feature to obtain a second sample image feature.

[0300] The noise feature is used to represent random noise. The process of obtaining the second sample image feature in step 703 is the same as the process of obtaining the second image feature in the above embodiments, and will not be repeated here.

[0301] In step 704, the electronic device trains the data generation model based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature.

[0302] The electronic device determines a first loss parameter based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature, and trains the data generation model with the goal of reducing the first loss parameter.

[0303] Optionally, the electronic device uses the following formula (4) to determine the first loss parameter.

[0304] ; formula (4)

[0305] wherein, denotes the first loss parameter, is a generated supervised Flow Matching MSE loss parameter. denotes the first sample initial image feature, denotes the noise feature, t denotes a time step, represents a process of an image from clean to completely noisy, denotes the image feature after denoising at t, denotes the denoising direction from the image feature at t to the first sample initial image feature, denotes the real denoising direction. denotes the MSE mean square error.

[0306] In some embodiments, the electronic device further obtains a second sample reference text, a second sample reference image, and a target reference text, the second sample reference text is used to describe the target reference text, and the target reference text is used to describe the second sample reference image; extracts a second sample text feature by performing feature extraction on the second sample reference text through a text processing network; extracts a second sample initial image feature by performing feature extraction on the second sample reference image through an image processing network, adds a noise feature to the second sample initial image feature to obtain a third sample image feature, and performs denoising on the third sample image feature to obtain a target sample intermediate feature, guided by the second sample text feature; and generates a predicted text based on the target sample intermediate feature through the text processing network. The process of generating the predicted text is the same as the process of generating the target text in step 604, and will not be described here.

[0307] Then, the step 705 includes: determining a first loss parameter based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature; determining a second loss parameter based on the predicted text and the target reference text; and training the data generation model based on the first loss parameter and the second loss parameter.

[0308] The target reference text is a real text, and the predicted text is a text generated by the data generation model based on the second sample reference image. The smaller the difference between the predicted text generated by the data generation model and the target reference text, the more accurate the data generation model is. Therefore, when training the data generation model, the training target also includes reducing the difference between the predicted text and the target reference text.

[0309] Optionally, the electronic device performs weighted summation on the first loss parameter and the second loss parameter to obtain a third loss parameter. The data generation model is trained to reduce the third loss parameter.

[0310] Optionally, the electronic device uses the following formula (5) and formula (6) to determine the second loss parameter and the third loss parameter, respectively.

[0311] Equation (5)

[0312] Equation (6)

[0313] wherein, denotes a second loss parameter, N denotes the number of characters in the character table, V denotes the number of characters in the text, denotes whether the jth character in the sample reference text belongs to the true label of the ith character in the character table, denotes the predicted probability that the jth character in the predicted text belongs to the ith character in the vocabulary.

[0314] wherein, denotes a third loss parameter, denotes a weight factor.

[0315] In the embodiments of the present disclosure, based on the difference between the predicted text and the sample reference text in the perception task and the difference between the predicted image and the sample reference image in the generation task, the data generation model is collaboratively trained, and the perception ability and the generation ability of the data generation model can be simultaneously improved.

[0316] The method provided by the embodiments of the present disclosure processes the text features and the image features of the multi-modal in the training process of the data generation model, uses the sample text features of the sample reference text as a condition to guide the denoising process of the sample image features, and uses the intermediate features generated in the denoising process as a condition to guide the denoising process of the sample image features after feature enhancement, thereby achieving unified semantic understanding based on multi-modal features, which is conducive to promoting the fusion and synergistic effect of cross-modal features. Moreover, the feature enhancement belongs to the perception task, and the feature denoising belongs to the generation task. The intermediate features generated in the generation task are enhanced in the perception task, and the enhanced features in the perception task are used as a condition to guide the generation task, thereby realizing the deep fusion between the perception task and the generation task. This is conducive to improving the understanding ability and the generation ability of the data generation model for multi-modal features, and thus improves the accuracy of the data generation model.

[0317] Figure 8 is a flowchart of still another data generation method according to an exemplary embodiment, as shown in Figure 8As shown, in the image generation task, the query feature, key feature and value feature of the text feature flow to the query feature, key feature and value feature of the image feature, so as to cross-fuse the text feature and the image feature, generate an image guided by the text feature, and realize the image generation task. In the image understanding task, for the query feature, key feature and value feature of the image feature learned through the attention mechanism, the query feature, key feature and value feature after enhancement by the multi-layer perception flow to the query feature, key feature and value feature of the text feature, so as to cross-fuse the text feature and the image feature, generate a text guided by the image feature, and realize the image understanding task.

[0318] In the embodiments of the present disclosure, through the cooperative work of the text processing network and the image processing network, the data generation model adopts a dynamic routing mechanism based on sequence types, and can adaptively switch different modeling strategies when processing mixed modal sequences. The autoregressive decoding path of the text processing network is activated for causal modeling when the text is activated, and the diffusion denoising path of the image processing network is activated for continuous signal processing when the image is activated.

[0319] In addition, in order to realize the information interaction between the modal, a cross-modal attention mechanism is also introduced, which allows the text features extracted by the text processing network to guide the generation process of the image processing network as a condition, and the image features extracted by the image processing network can also enhance the semantic understanding of the text processing network through the attention mechanism. The design of this two-way information flow ensures the coordination and consistency between different processing networks.

[0320] In addition, the multi-level intermediate features generated by generative modeling provide a unified semantic space for cross-modal understanding, so that the data generation model can still achieve deeper semantic alignment and cross-modal reasoning in the complex scene of mixed text and images, thereby simultaneously having strong multi-modal understanding and generation ability in a unified framework.

[0321] In addition, the intermediate features in the generative modeling process are used as an enhanced signal for perceptual modeling, realizing the feature unification and deep fusion of the perception task and the generation task in the image processing network, which is conducive to promoting the collaborative improvement of the perception task and the generation task.

[0322] Figure 9 is a block diagram of a data generation apparatus according to an exemplary embodiment. Referring to Figure 9 The apparatus includes a data preparation unit 901, a text processing unit 902 and an image processing unit 903.

[0323] The data preparation unit 901 is configured to perform determining a data generation model, the data generation model including a text processing network and an image processing network;

[0324] The text processing unit 902 is configured to perform feature extraction on the first reference text by using a text processing network to obtain first text features, the first reference text being used to describe an image to be generated.

[0325] The image processing unit 903 is configured to perform denoising on the first image features by using an image processing network with the first text features as guidance to obtain first intermediate features, perform nonlinear transformation on the first intermediate features to obtain second intermediate features, and perform denoising on the first intermediate features with the first text features and the second intermediate features as guidance to obtain second image features, wherein the first image features are any image features containing noise, and the second intermediate features are enhanced features of the first intermediate features.

[0326] The image processing unit 903 is further configured to perform generation of a target image based on the second image features by using the image processing network, the target image being consistent with the description of the first reference text.

[0327] The apparatus provided by the embodiments of the present disclosure processes multi-modal text features and image features in a data generation scenario, uses text features of a reference text as a condition to guide the denoising process of image features, and uses the intermediate features generated in the denoising process as a condition to guide the denoising process of image features after feature enhancement, thereby achieving unified semantic understanding based on multi-modal features, which is conducive to promoting the fusion and synergy of cross-modal features. Moreover, feature enhancement belongs to a perception task, and feature denoising belongs to a generation task. The intermediate features generated in the generation task are enhanced in the perception task, and the enhanced features in the perception task are used as a condition to guide the generation task, thereby achieving deep fusion between the perception task and the generation task. Therefore, the processing manner of the present disclosure can improve the understanding and generation capabilities of multi-modal features in a data generation scenario, thereby improving the accuracy of the finally generated target image.

[0328] In a possible implementation manner, referring to Figure 10 The image processing unit 903 is further configured to perform:

[0329] perform feature extraction on the first reference image by using the image processing network to obtain first initial image features, and add noise features to the first initial image features to obtain the first image features, the first reference image including a reference object, an object in the target image being consistent with the reference object, and the noise features being used to represent random noise; or

[0330] determine the noise features as the first image features.

[0331] In a possible implementation manner, referring to Figure 10 The text processing unit 902 is configured to perform:

[0332] encode the first reference text into first initial text features through a text encoding layer in the text processing network;

[0333] encode the first initial text features into first text features through a text feature processing layer in the text processing network.

[0334] In a possible implementation, referring to Figure 10 , the text processing unit 902 is configured to perform:

[0335] map the first initial text features into first query features, first key features, and first value features through the text feature processing layer;

[0336] mask process, through the text feature processing layer, a product of a transpose of the first query features and the first key features and a scaling factor based on a causal mask feature to obtain first weight features, the first reference text including n characters, the causal mask feature being used to mask process features of (i+1)th character to nth character when determining a weight feature of ith character, n being a positive integer, and i being a positive integer not greater than n;

[0337] multiply, through the text feature processing layer, the first weight features and the first value features to obtain the first text features.

[0338] In a possible implementation, referring to Figure 10 , the image processing unit 903 is configured to perform:

[0339] perform, through the image processing network, nonlinear transformation on the first image features to obtain first enhanced features, the first enhanced features being enhanced features of the first image features;

[0340] perform, through the image processing network, denoising on the first image features with the first text features and the first enhanced features as guidance to obtain the first intermediate features.

[0341] In a possible implementation, referring to Figure 10 , the image processing unit 903 is configured to perform:

[0342] perform, through an image feature processing layer in the image processing network, cross-attention encoding on the first text features, the first enhanced features, and the first image features to obtain the first intermediate features.

[0343] In a possible implementation, referring to Figure 10 , the image processing unit 903 is configured to perform:

[0344] The first text feature is mapped to a first query sub-feature, a first key sub-feature and a first value sub-feature by the image feature processing layer, the first image feature is mapped to a second query sub-feature, and the first enhanced feature is mapped to a second key sub-feature and a second value sub-feature.

[0345] The first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature and the second value sub-feature are cross-fused by the image feature processing layer to obtain the first intermediate feature.

[0346] In a possible implementation manner, referring to Figure 10 The image processing unit 903 is configured to perform:

[0347] The first query sub-feature and the second query sub-feature are spliced by the image feature processing layer to obtain a second query feature, the first key sub-feature and the second key sub-feature are spliced to obtain a second key feature, and the first value sub-feature and the second value sub-feature are spliced to obtain a second value feature.

[0348] The second query feature and the product of the transpose of the second key feature and the scaling factor are normalized by the image feature processing layer to obtain a second weight feature, and the second weight feature is multiplied by the second value feature to obtain the first intermediate feature.

[0349] In a possible implementation manner, referring to Figure 10 The image processing unit 903 is configured to perform:

[0350] The first text feature, the second intermediate feature and the first intermediate feature are cross-attention encoded by the image feature processing layer in the image processing network to obtain the second image feature.

[0351] In a possible implementation manner, referring to Figure 10 The image processing unit 903 is configured to perform:

[0352] The first text feature is mapped to a third query sub-feature, a third key sub-feature and a third value sub-feature by the image feature processing layer, the first intermediate feature is mapped to a fourth query sub-feature, and the second intermediate feature is mapped to a fourth key sub-feature and a fourth value sub-feature.

[0353] The third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature and the fourth value sub-feature are cross-fused by the image feature processing layer to obtain the second image feature.

[0354] In a possible implementation manner, referring to Figure 10 The image processing unit 903 is configured to perform:

[0355] The third query feature is obtained by splicing the third query sub-feature and the fourth query sub-feature through the image feature processing layer, the third key feature is obtained by splicing the third key sub-feature and the fourth key sub-feature, and the third value feature is obtained by splicing the third value sub-feature and the fourth value sub-feature.

[0356] The third weight feature is obtained by normalizing the product of the transpose of the third key feature and the scaling factor through the image feature processing layer, and the second image feature is obtained by multiplying the third weight feature and the third value feature.

[0357] In a possible implementation manner, referring to Figure 10 , the image processing network includes N diffusion sub-networks, N is an integer greater than 1; and the image processing unit 903 is configured to perform:

[0358] The first image feature is denoised by the first diffusion sub-network in the image processing network to obtain the first first intermediate feature.

[0359] The k-1th second intermediate feature is obtained by performing nonlinear transformation on the k-1th first intermediate feature through the kth diffusion sub-network in the image processing network.

[0360] The kth first intermediate feature is obtained by denoising the k-1th first intermediate feature guided by the first text feature and the k-1th second intermediate feature through the kth diffusion sub-network, wherein k is an integer greater than 1 and not greater than N, and the second image feature is the Nth first intermediate feature.

[0361] In a possible implementation manner, referring to Figure 10 , the apparatus further includes:

[0362] The data preparation unit 901 is further configured to perform: obtaining a second reference text and a second reference image, the second reference text being used to describe a text to be generated;

[0363] The text processing unit 902 is further configured to perform: obtaining a second text feature by performing feature extraction on the second reference text through a text processing network;

[0364] The image processing unit 903 is further configured to perform: obtaining a second initial image feature by performing feature extraction on the second reference image through an image processing network, adding a noise feature to the second initial image feature to obtain a third image feature, and denoising the third image feature guided by the second text feature to obtain a target intermediate feature, the noise feature being used to represent random noise.

[0365] The text processing unit 902 is further configured to perform, by the text processing network, generating a target text based on the target intermediate feature, the target text being consistent with the description of the second reference text, and the content described by the target text being consistent with the content of the second reference image.

[0366] In a possible implementation, referring to Figure 10 , the image processing unit 903 is configured to perform:

[0367] de-noising the third image feature guided by the second text feature to obtain a third intermediate feature by the image processing network;

[0368] performing a non-linear transformation on the third intermediate feature to obtain a target intermediate feature by the image processing network, the target intermediate feature being an enhanced feature of the third intermediate feature.

[0369] In a possible implementation, referring to Figure 10 , the text processing unit 902 is configured to perform:

[0370] fusing the target intermediate feature and a second initial text feature to obtain a target text feature by the text processing network, the second initial text feature being a feature obtained by encoding the second reference text;

[0371] generating the target text based on the target text feature by the text processing network.

[0372] In a possible implementation, referring to Figure 10 , the apparatus further includes a model training unit 904 configured to perform:

[0373] obtaining a first sample reference text and a first sample reference image, the first sample reference text being used to describe the first sample reference image;

[0374] extracting a first sample text feature from the first sample reference text by the text processing network;

[0375] extracting a first sample initial image feature from the first sample reference image by the image processing network, adding a noise feature to the first sample initial image feature to obtain a first sample image feature, de-noising the first sample image feature guided by the first sample text feature to obtain a first sample intermediate feature, performing a non-linear transformation on the first sample intermediate feature to obtain a second sample intermediate feature, de-noising the first sample intermediate feature guided by the first sample text feature and the second sample intermediate feature to obtain a second sample image feature, the noise feature being used to represent random noise;

[0376] training the data generation model based on at least one feature obtained by de-noising, the first sample initial image feature, and the noise feature.

[0377] In a possible implementation, referring to Figure 11 The model training unit 904 is configured to perform the following:

[0378] obtain a second sample reference text, a second sample reference image, and a target reference text, the second sample reference text being used to describe the target reference text, and the target reference text being used to describe the second sample reference image;

[0379] perform feature extraction on the second sample reference text through the text processing network to obtain a second sample text feature;

[0380] perform feature extraction on the second sample reference image through the image processing network to obtain a second sample initial image feature, add a noise feature to the second sample initial image feature to obtain a third sample image feature, and perform denoising on the third sample image feature based on the second sample text feature to obtain a target sample intermediate feature;

[0381] generate a predicted text based on the target sample intermediate feature through the text processing network;

[0382] determine a first loss parameter based on the at least one feature obtained after denoising, the first sample initial image feature, and the noise feature, and determine a second loss parameter based on the predicted text and the target reference text;

[0383] train the data generation model based on the first loss parameter and the second loss parameter.

[0384] As to the apparatus in the above-described embodiments, the specific manners in which various units perform operations have been described in detail in the embodiments of the method, and thus will not be described here in detail.

[0385] The electronic device includes a processor and a memory for storing instructions executable by the processor. The processor is configured to execute the instructions to implement the above-described data generation method.

[0386] In some embodiments, the electronic device includes a terminal. Figure 11is a structural block diagram of a terminal 1100 according to an exemplary embodiment. The terminal 1100 can be a portable mobile terminal such as a smartphone, a tablet computer, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer, or a desktop computer. The terminal 1100 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0387] The terminal 1100 includes a processor 1101 and a memory 1102.

[0388] The processor 1101 can include one or more processing cores, such as a 4-core processor, an 8-core processor, or the like. The processor 1101 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1101 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. The memory 1102 can include one or more computer-readable storage media, which can be non-transitory. The memory 1102 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash memory devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one piece of program code for being executed by the processor 1101 to implement the data generation method provided by the method embodiment in the present disclosure.

[0389] In some embodiments, the terminal 1100 can also optionally include a peripheral device interface 1103 and at least one peripheral device. The processor 1101, the memory 1102, and the peripheral device interface 1103 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1103 through a bus, a signal line, or a circuit board. Specifically, the peripheral devices include a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, and a power supply 1107.

[0390] The peripheral interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 1101 and the memory 1102. In some embodiments, the processor 1101, the memory 1102 and the peripheral interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1101, the memory 1102 and the peripheral interface 1103 can be implemented on a separate chip or circuit board, and the present disclosure does not limit the embodiments in this regard.

[0391] The radio frequency circuit 1104 is used to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1104 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1104 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1104 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1104 can also include NFC (Near Field Communication) related circuitry, and the present disclosure does not limit the embodiments in this regard.

[0392] The display screen 1105 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1105 is a touch display screen, the display screen 1105 is further configured to capture touch signals on or above the surface of the display screen 1105. The touch signals can be input to the processor 1101 as control signals for processing. In this case, the display screen 1105 can also be configured to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1105 can be one, disposed on the front panel of the electronic device 1100; in other embodiments, the display screen 1105 can be at least two, respectively disposed on different surfaces of the electronic device 1100 or in a folding design; in other embodiments, the display screen 1105 can be a flexible display screen, disposed on a curved surface or a folding surface of the electronic device 1100. Even, the display screen 1105 can also be disposed in an irregular shape other than a rectangle, i.e., a notched screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0393] The camera assembly 1106 is configured to capture images or videos. Optionally, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the electronic device, and the rear-facing camera is disposed on the back of the electronic device. In some embodiments, the rear-facing camera is at least two, which is any one of a main camera, a depth-of-field camera, a wide-angle camera, and a long-focus camera, to realize the background blur function of the main camera and the depth-of-field camera, the panorama shooting and VR (Virtual Reality) shooting function of the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1106 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0394] The power supply 1107 is configured to supply power to various components in the electronic device 1100. The power supply 1107 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1107 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery that is charged through a wired line, and the wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0395] Those skilled in the art can understand that, Figure 12The structure shown in the figure does not constitute a limitation on the terminal 1100, and can include more or fewer components than shown, or combine certain components, or have a different arrangement of components.

[0396] In some embodiments, the electronic device comprises a server. ​ Fig. 12 is a structural schematic diagram of a server according to an example embodiment. The server 1200 can have a large difference in configuration or performance, and can include one or more processors (Central Processing Units, CPUs) 1201 and one or more memories 1202, wherein the memory 1202 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1201 to implement the method provided by each of the above-mentioned method embodiments. Of course, the server can also have a wired or wireless network interface, a keyboard, and an input and output interface, etc., so as to perform input and output, and the server can also include other components for realizing the functions of the device, which are not described here.

[0397] In an example embodiment, a computer readable storage medium is also provided, which, when the instructions in the computer readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the steps in the above-mentioned data generation method. For example, the computer readable storage medium can be a ROM (Read Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0398] In an example embodiment, a computer program product is also provided, which includes a computer program, and the computer program, when executed by a processor of an electronic device, implements the steps in the above-mentioned data generation method.

[0399] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0400] It should be understood that the present disclosure is not limited to the precise structures as herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data generation method, characterized in that: The method comprises: Determining a data generation model, wherein the data generation model includes a text processing network and an image processing network, wherein the image processing network includes N diffusion subnetworks and an image decoding layer, where N is an integer greater than 1; Performing feature extraction on a first reference text through the text processing network to obtain a first text feature, where the first reference text is used to describe the image to be generated; performing a nonlinear transformation on the first image feature through a first diffusion sub-network in the image processing network to obtain a first enhanced feature, mapping the first text feature to a first query sub-feature, a first key sub-feature, and a first value sub-feature, mapping the first image feature to a second query sub-feature, mapping the first enhanced feature to a second key sub-feature and a second value sub-feature, cross-fusing the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature to obtain a first first intermediate feature, wherein the first enhanced feature is a feature obtained by enhancing the first image feature, and the first image feature is any image feature containing noise; Performing a nonlinear transformation on the k-1th first intermediate feature through the k-th diffusion sub-network in the image processing network to obtain the k-1th second intermediate feature, mapping the first text feature to a third query sub-feature, a third key sub-feature, and a third value sub-feature, mapping the k-1th first intermediate feature to a fourth query sub-feature, mapping the k-1th second intermediate feature to a fourth key sub-feature and a fourth value sub-feature, and cross-fusing the third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature, and the fourth value sub-feature to obtain the k-th first intermediate feature; wherein the k-1th second intermediate feature is an enhanced feature of the k-1th first intermediate feature, the Nth first intermediate feature is a second image feature, and k is an integer greater than 1 and not greater than N; A target image is generated based on the second image features through an image decoding layer in the image processing network, and the target image is consistent with the description of the first reference text.

2. The data generation method according to claim 1, wherein: The process of acquiring the first image feature includes: performing feature extraction on a first reference image through the image processing network to obtain a first initial image feature, adding a noise feature to the first initial image feature to obtain the first image feature, wherein the first reference image includes a reference object, the object in the target image matches the reference object, and the noise feature is used to characterize random noise; or The noise feature is determined as the first image feature.

3. The data generation method according to claim 1, wherein: The step of extracting features from the first reference text through the text processing network to obtain first text features includes: encoding the first reference text into a first initial text feature through a text encoding layer in the text processing network; The first initial text feature is causally encoded by a text feature processing layer in the text processing network to obtain the first text feature.

4. The data generation method according to claim 3, characterized in that The step of performing causal attention encoding on the first initial text feature through a text feature processing layer in the text processing network to obtain the first text feature includes: Mapping the first initial text feature into a first query feature, a first key feature, and a first value feature through the text feature processing layer; By the text feature processing layer, masking is performed on the first query feature, the product of the transpose of the first key feature and the scaling factor based on a causal mask feature to obtain a first weight feature, wherein the first reference text includes n characters, and the causal mask feature is used to mask features of characters from i+1th to nth when determining the weight feature of the i-th character, where n is a positive integer and i is a positive integer not greater than n; The first weight feature is multiplied by the first value feature through the text feature processing layer to obtain the first text feature.

5. The data generation method according to claim 1, wherein: The cross-fusion of the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature to obtain a first first intermediate feature includes: Concatenate the first query sub-feature and the second query sub-feature to obtain a second query feature, concatenate the first key sub-feature and the second key sub-feature to obtain a second key feature, and concatenate the first value sub-feature and the second value sub-feature to obtain a second value feature; Normalize the product of the transpose of the second query feature and the second key feature and the scaling factor to obtain a second weight feature, and multiply the second weight feature by the second value feature to obtain the first first intermediate feature.

6. The data generation method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire a second reference text and a second reference image, where the second reference text is used to describe the text to be generated; Performing feature extraction on the second reference text through the text processing network to obtain second text features; performing feature extraction on the second reference image through the image processing network to obtain a second initial image feature, adding a noise feature to the second initial image feature to obtain a third image feature, and denoising the third image feature using the second text feature as a guide to obtain a target intermediate feature, wherein the noise feature is used to characterize random noise; A target text is generated based on the target intermediate features through the text processing network, wherein the target text is consistent with the description of the second reference text, and the content described in the target text is consistent with the content of the second reference image.

7. The data generation method according to claim 6, characterized in that: Denoising the third image feature using the image processing network and the second text feature as a guide to obtain a target intermediate feature includes: Denoising the third image feature using the image processing network and the second text feature as a guide to obtain a third intermediate feature; The third intermediate feature is subjected to a nonlinear transformation through the image processing network to obtain the target intermediate feature, which is an enhanced feature of the third intermediate feature.

8. The data generation method according to claim 6, characterized in that: Generating a target text based on the target intermediate features through the text processing network includes: fusing the target intermediate feature and the second initial text feature through the text processing network to obtain a target text feature, where the second initial text feature is a feature obtained by encoding the second reference text; The target text is generated based on the target text features through the text processing network.

9. The data generation method according to any one of claims 1 to 5, characterized in that: The training process of the data generation model includes: Acquire a first sample reference text and a first sample reference image, where the first sample reference text is used to describe the first sample reference image; Performing feature extraction on the first sample reference text through the text processing network to obtain first sample text features; Performing feature extraction on the first sample reference image through the image processing network to obtain a first sample initial image feature, adding a noise feature to the first sample initial image feature to obtain a first sample image feature, denoising the first sample image feature using the first sample text feature as a guide to obtain a first sample intermediate feature, performing a nonlinear transformation on the first sample intermediate feature to obtain a second sample intermediate feature, denoising the first sample intermediate feature using the first sample text feature and the second sample intermediate feature as a guide to obtain a second sample image feature, wherein the noise feature is used to characterize random noise; The data generation model is trained based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature.

10. The data generation method according to claim 9, characterized in that: The method further comprises: Acquire a second sample reference text, a second sample reference image, and a target reference text, wherein the second sample reference text is used to describe the target reference text, and the target reference text is used to describe the second sample reference image; Performing feature extraction on the second sample reference text through the text processing network to obtain second sample text features; Performing feature extraction on the second sample reference image through the image processing network to obtain a second sample initial image feature, adding the noise feature to the second sample initial image feature to obtain a third sample image feature, and denoising the third sample image feature using the second sample text feature as a guide to obtain a target sample intermediate feature; Generate predicted text based on the intermediate features of the target sample through the text processing network; The training of the data generation model based on the at least one feature obtained by denoising, the first sample initial image feature, and the noise feature includes: Determining a first loss parameter based on at least one feature obtained by denoising, the first sample initial image feature, and the noise feature; determining a second loss parameter based on the predicted text and the target reference text; The data generation model is trained based on the first loss parameter and the second loss parameter.

11. A data generating device, characterized in that: The device comprises: A data preparation unit is configured to execute determination of a data generation model, wherein the data generation model includes a text processing network and an image processing network, wherein the image processing network includes N diffusion subnetworks and an image decoding layer, where N is an integer greater than 1; a text processing unit configured to perform feature extraction on a first reference text through the text processing network to obtain a first text feature, wherein the first reference text is used to describe an image to be generated; an image processing unit configured to perform a nonlinear transformation on the first image feature through a first diffusion sub-network in the image processing network to obtain a first enhanced feature, map the first text feature to a first query sub-feature, a first key sub-feature, and a first value sub-feature, map the first image feature to a second query sub-feature, map the first enhanced feature to a second key sub-feature and a second value sub-feature, and cross-fuse the first query sub-feature, the first key sub-feature, the first value sub-feature, the second query sub-feature, the second key sub-feature, and the second value sub-feature to obtain a first first intermediate feature, wherein the first enhanced feature is a feature after the first image feature is enhanced, and the first image feature is any image feature containing noise; The image processing unit is further configured to perform a nonlinear transformation on the k-1th first intermediate feature through the k-th diffusion sub-network in the image processing network to obtain a k-1th second intermediate feature, map the first text feature to a third query sub-feature, a third key sub-feature, and a third value sub-feature, map the k-1th first intermediate feature to a fourth query sub-feature, map the k-1th second intermediate feature to a fourth key sub-feature and a fourth value sub-feature, and cross-fuse the third query sub-feature, the third key sub-feature, the third value sub-feature, the fourth query sub-feature, the fourth key sub-feature, and the fourth value sub-feature to obtain a k-th first intermediate feature; wherein the k-1th second intermediate feature is a feature after the k-1th first intermediate feature is enhanced, the Nth first intermediate feature is a second image feature, and k is an integer greater than 1 and not greater than N; The image processing unit is further configured to execute an image decoding layer in the image processing network to generate a target image based on the second image features, where the target image is consistent with the description of the first reference text.

12. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data generation method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data generating method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data generation method according to any one of claims 1 to 10 is implemented.

Images