Target model training method, multi-modal data processing method, device and equipment

Abstract

The invention provides a target model training method, a multi-modal data processing method, a device thereof and equipment, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and is applied to a content generation scene based on artificial intelligence. According to the specific implementation scheme, the method comprises the steps of inputting sample data into a preset model to obtain initial multi-modal features of the sample data; and under the condition that the parameters of the image-text encoder of the preset model are fixed, performing model training on N layers of diffusion networks in the preset model by using the initial multi-modal features and the preset noise features to obtain a target model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and is applied to scenarios such as artificial intelligence-based content generation (AIGC). Background Art

[0002] Text to Image (T2I) diffusion models have made significant progress in recent years. This model technology has demonstrated remarkable capabilities in generating high-fidelity images from text descriptions. Through the continuous innovation of deep learning and artificial intelligence technology, the existing model technology is able to transform simple text descriptions into detailed and realistic images. This transformation not only provides a powerful tool for creative workers, but also brings users an unprecedented visual experience. Summary of the invention

[0003] The present invention provides a target model training method, a multimodal data processing method, and a device and equipment thereof.

[0004] According to one aspect of the present disclosure, a target model training method is provided, comprising:

[0005] Input the sample data into a preset model to obtain initial multimodal features of the sample data; wherein the sample data at least includes sample text and at least one entity image, the sample text includes at least one entity text, and the entity image is an image corresponding to the entity text; the initial multimodal features include image features of the entity image and text features of the entity text, and the image features of the entity image are after the text features of the entity text corresponding to the entity image;

[0006] When the parameters of the image-text encoder of the preset model are fixed, the N-layer diffusion network in the preset model is trained using the initial multimodal features and the preset noise features to obtain a target model, where N is a positive integer.

[0007] According to another aspect of the present disclosure, there is provided a multimodal data processing method, comprising:

[0008] Inputting the multimodal prompt information into a pre-trained target model to obtain target multimodal features of the multimodal prompt information, wherein the multimodal prompt information at least includes a target text and at least one target entity image, the target text includes at least one target entity text, and the target entity image is an image corresponding to the target entity text; the target multimodal features include image features of the target entity image and text features of the target entity text, and the image features of the target entity image are after the text features of the target entity text corresponding to the target entity image;

[0009] The target multimodal features are diffused using an N-layer target diffusion network in the target model to obtain a target inference result including an image and / or text associated with the multimodal prompt information; wherein N is a positive integer.

[0010] According to another aspect of the present disclosure, a target model training device is provided, comprising:

[0011] A training unit, used for inputting sample data into a preset model to obtain initial multimodal features of the sample data; wherein the sample data at least includes sample text and at least one entity image, the sample text includes at least one entity text, and the entity image is an image corresponding to the entity text; the initial multimodal features include image features of the entity image and text features of the entity text, and the image features of the entity image are after the text features of the entity text corresponding to the entity image; when the parameters of the image-text encoder of the preset model are fixed, the N-layer diffusion network in the preset model is trained using the initial multimodal features and the preset noise features to obtain a target model; wherein N is a positive integer;

[0012] A storage unit is used to output the target model.

[0013] According to another aspect of the present disclosure, there is provided a multimodal data processing device, comprising:

[0014] An inference unit is used to input the multimodal prompt information into a pre-trained target model to obtain a target multimodal feature of the multimodal prompt information, wherein the multimodal prompt information at least includes a target text and at least one target entity image, the target text includes at least one target entity text, and the target entity image is an image corresponding to the target entity text; the target multimodal feature includes an image feature of the target entity image and a text feature of the target entity text, and the image feature of the target entity image is after the text feature of the target entity text corresponding to the target entity image; the target multimodal feature is diffused using an N-layer target diffusion network in the target model to obtain a target inference result associated with the multimodal prompt information and including an image and / or text; wherein N is a positive integer;

[0015] An output unit is used to output the target reasoning result.

[0016] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0022] In this way, the disclosed solution utilizes the generated initial multimodal features and preset noise features to train the N-layer diffusion network in the preset model, so that the target model obtained after training can process multimodal information, enriching the use scenarios of the model. At the same time, it also effectively improves the model's ability to control details, providing strong support for improving the controllability of image generation.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0025] Figure 1 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 1 ;

[0026] Figure 2 is a schematic diagram of the initial multimodal features according to the embodiment of the present application Figure 1 ;

[0027] FIG3( a ) is a schematic diagram of a diffusion network according to an embodiment of the present application;

[0028] FIG3( b ) is a schematic diagram of diffusion networks connected in series according to an embodiment of the present application;

[0029] Figure 4 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 2 ;

[0030] FIG5(a) is a schematic diagram of the initial multimodal features according to an embodiment of the present application. Figure 2 ;

[0031] FIG5( b ) is a schematic diagram of instruction information according to an embodiment of the present application;

[0032] FIG6( a ) is a schematic flowchart 3 of a model training method according to an embodiment of the present application;

[0033] FIG6( b ) is a third schematic diagram of initial multimodal features according to an embodiment of the present application;

[0034] FIG6( c ) is a schematic diagram of a scenario in a specific example of a model training method according to an embodiment of the present application;

[0035] FIG. 7( a ) is a schematic diagram of the structure of a dual information flow diffusion network according to another embodiment of the present application;

[0036] FIG7( b ) is a schematic diagram of a dual information flow diffusion network connected in series according to another embodiment of the present application;

[0037] Figure 8 This is a schematic flow chart of a multimodal data processing method according to an embodiment of the present application. Figure 1 ;

[0038] Fig. 9 This is a schematic flow chart of a multimodal data processing method according to an embodiment of the present application. Figure 2 ;

[0039] Fig.10 is a schematic diagram of an application flow of a multimodal data processing method in a specific example according to an embodiment of the present application;

[0040] Fig.11 is a structural schematic diagram of a model training device according to an embodiment of the present application;

[0041] Fig.12 is a structural diagram of a multimodal data processing device according to an embodiment of the present application;

[0042] Fig.13 It is a block diagram of an electronic device used to implement the model training method or multimodal data processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0044] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0045] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0046] The following is a description of the related technologies of the embodiments of the present disclosure. The following related technologies are optional solutions that can be arbitrarily combined with the technical solutions of the embodiments of the present disclosure, and they all belong to the protection scope of the embodiments of the present disclosure.

[0047] The text-to-image (T2I) diffusion model has made significant progress in recent years. This model technology has demonstrated remarkable capabilities in generating high-fidelity images from text descriptions. The existing T2I diffusion model can intelligently generate highly relevant images based on the text prompts provided by the user. The emergence of these methods has greatly enriched the visual expression methods, making the conversion from text to image more direct and efficient. This conversion not only provides a powerful tool for creative workers, but also brings users an unprecedented visual experience.

[0048] Although these methods have achieved remarkable achievements in image generation, they still face a fundamental challenge. This challenge stems from the inherent brevity of text descriptions. When trying to describe a scene with complex details, specific entities, or nuances in short text, it is often found that due to the limitations of language expression, the generated image may not fully capture all the details and essence of the original description.

[0049] To overcome this challenge and further improve the controllability of image generation, the disclosed solution provides a new method, namely, inputting a universal visual language (VL) to generate images faithfully. Here, the universal visual language may include richer descriptive information, such as product images, photos, sketches, etc., which can provide more context and detailed guidance for the image generation model. In this way, the content and style of the generated image can be more accurately controlled, thereby meeting the user's higher customization needs.

[0050] Here, before explaining the disclosed solution in detail, some concepts that may be involved in model training are briefly introduced:

[0051] Fine-tuning: Given a model to be trained, fine-tune the model based on it. Compared with training the model from scratch, fine-tuning can save a lot of computing resources and computing time, and can improve computing efficiency and accuracy.

[0052] Latent space: A representation method for image compression that extracts important features from an image and removes unimportant features.

[0053] Part 1: Detailed description of the model training method:

[0054] The present disclosure proposes a model training method for multimodal information, so that the model can have the ability to process multimodal information, thereby improving the controllability of the model in image generation, and thereby effectively improving the user experience.

[0055] It should be noted that the preset model of the disclosed scheme is a model based on the multimodal diffusion transformer MMDiT (Multimodal Diffusion Transformer, or MultiModal-DiT for short) architecture. Compared with the model of the U-net (U-Net) architecture, the disclosed scheme can not only support text-to-image tasks, but also support multimodal tasks (tasks of generating text and / or images based on input multimodal information). In this way, the application capability of the model in various scenarios is improved.

[0056] Specifically, Figure 1 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0057] Further, the method includes at least part of the following contents. Figure 1 As shown, including:

[0058] Step S101: input sample data into a preset model to obtain initial multimodal features of the sample data.

[0059] For example, in one example, the image-text encoder in the preset model can be used to obtain the initial multimodal features of the sample data.

[0060] Here, in one example, the sample data includes at least sample text and at least one entity image (for the convenience of subsequent description, it can be recorded as M1 entity images). Further, the sample text also includes at least one entity text (for the convenience of subsequent description, it can be recorded as M2 entity texts). Here, the entity image is an image corresponding to the entity text.

[0061] Here, M1 and M2 are positive integers greater than or equal to 1; further, in one example, the values ​​of M1 and M2 may be different, for example, M1 is less than M2, in which case, it may correspond to a scenario where not all sample texts correspond to an entity image. Alternatively, in another example, the values ​​of M1 and M2 are the same, in which case, it may correspond to a scenario where one entity text corresponds to one entity image.

[0062] Here, it should be noted that the entity text and entity image can be one-to-one or one-to-many, and the disclosed solution does not limit this. Accordingly, the disclosed solution does not limit the values ​​of M1 and M2.

[0063] Furthermore, in one example, the initial multimodal features include image features of an entity image and text features of an entity text, wherein the image features of the entity image are after the text features of the entity text corresponding to the entity image; in other words, in this example, the initial multimodal features are at least obtained by inserting the image features of the entity image after the text features of the entity text corresponding to the entity image.

[0064] Step S102: When the parameters of the image-text encoder of the preset model are fixed, the N-layer diffusion network in the preset model is trained (e.g., fine-tuned) using the initial multimodal features and the preset noise features to obtain a target model. For example, a target model including at least N target diffusion networks is obtained.

[0065] Here, the target model can infer, based on the input multimodal prompt information, a target inference result including an image and / or text associated with the multimodal prompt information.

[0066] Furthermore, N is a positive integer greater than or equal to 1.

[0067] It should be noted that the training method of the disclosed solution can be mainly applied to the scenario of training a diffusion network. In this case, step S101 can be specifically as follows: inputting the sample data required for the current step into the preset model, so as to obtain the initial multimodal features of the sample data used in the current step using the image-text encoder in the preset model. Correspondingly, step S102 can be specifically as follows: using the initial multimodal features and the preset noise features required for the current step, model training is performed on the N-layer diffusion network in the preset model to complete the current step training.

[0068] Here, the "current step" may be any step in the total number of steps (also referred to as the preset number of steps) required to train the diffusion network (or diffusion model). In other words, the disclosed solution may be applied to any step in the training of the diffusion network.

[0069] Furthermore, it should be noted that the "step" here is a concept in the diffusion network training process, which can refer to the discrete time points (or discrete steps) experienced in the process of converting the original data from the original state to the noisy state (which can be called the forward process, for example, in the forward process, the model gradually adds noise to the data, and this forward process can be divided into multiple discrete steps, each step can add a certain amount of noise to the data), and recovering from the noisy state to the original state (or original data) (reverse process, for example, in the reverse process, the model can try to gradually remove the noise to restore to the original data, and this reverse process can also be divided into multiple steps).

[0070] In this way, the disclosed solution utilizes the generated initial multimodal features and preset noise features to train the N-layer diffusion network in the preset model, so that the target model obtained after training can process multimodal information, enriching the use scenarios of the model. At the same time, it also effectively improves the model's ability to control details, providing strong support for improving the controllability of image generation.

[0071] In addition, since the disclosed solution makes full use of the multimodal features of mixed image and text input, that is, it makes full use of the universal visual language, the expressed features have richer descriptive information, and these descriptive information can provide more context and detailed guidance for reasoning, such as image generation, thereby further meeting the user's higher reasoning needs. At the same time, it can also enable the trained model to have multi-entity processing and multi-entity generation capabilities, thereby further enhancing the user experience.

[0072] Here, in one example, the above-mentioned preset model can be a generative model. For example, it can specifically be an image generation model, etc. The solution of the present disclosure does not make specific limitations thereto.

[0073] It should be noted that when the sample data is specifically a sample text containing multiple entity texts, as well as multiple entity images, and each entity text has a corresponding entity image, based on the normal browsing order of the texts in the sample text, the initial multi-modal feature can be expressed as: [the text feature of sub-text 1 (which can be understood as non-entity text) in the sample text, the text feature of entity text 1, the image feature of the entity image corresponding to entity text 1; the text feature of sub-text 2 (which can be understood as non-entity text) in the sample text, the text feature of entity text 2, the image feature of the entity image corresponding to entity text 2;...]; it can be understood that the above feature representation is only an exemplary illustration, and in actual applications, the specific positions of the text features of non-entity texts and entity texts in the initial multi-modal feature can be determined based on the positions of non-entity texts and entity texts in the sample text.

[0074] For example, as Figure 2 shown, the sample text included in the sample data is "Lin Daiyu took a selfie in the Forbidden City". The entity texts included in this sample text are: "Lin Daiyu" (which can be denoted as entity text 1) and "the Forbidden City" (which can be denoted as entity text 2), and the non-entity texts included in this sample text are "in" and "took a selfie"; further, the sample data also includes entity image 1 corresponding to "Lin Daiyu" (i.e., entity text 1), and entity image 2 corresponding to "the Forbidden City" (i.e., entity text 2). At this time, the image feature 1 extracted from entity image 1 can be inserted after the text feature 1 of "Lin Daiyu", and the image feature 2 extracted from entity image 2 can be inserted after the text feature 3 of "the Forbidden City", to obtain the initial multi-modal feature represented in the following way:

[0075] [the text feature 1 of "Lin Daiyu", the image feature 1 of the entity image corresponding to "Lin Daiyu"; the text feature 2 of "in"; the text feature 3 of "the Forbidden City", the image feature 2 of the entity image corresponding to "the Forbidden City"; the text feature 4 of "took a selfie"].

[0076] In this way, through the above text-image alignment strategy, information of different modalities can be aligned, realizing a text-image mixed feature representation, providing strong support for improving the controllability of image generation in the future.

[0077] It should be noted that, in the above example, in order to more clearly distinguish the text features from the image features in the initial multimodal features, the disclosed solution can also set a header field (head) before the inserted image features (for example, the head), for example, Figure 2 Taking the example shown as an example, the head field can be spliced ​​before image feature 1, and the head field can be spliced ​​before image feature 2, so as to distinguish the text features from the image features in the initial multimodal features, thereby providing strong support for further improving the controllability of image generation and improving the reasoning effect of the model.

[0078] Furthermore, in a specific example, the feature expression method of the image features of the entity image obtained by using the image-text encoder includes at least one of the following: position features of the entity image, segmentation features of the segmentation map corresponding to the entity image, image features of the cropped (Crop) map of the entity image, image depth features of the entity image, local features of the entity image (for example, facial features, and further, facial features).

[0079] That is to say, when constructing the initial multimodal features, the types of image features inserted can be in various forms. Thus, compared with the existing use of a single red, green, and blue (RGB) feature, the disclosed solution can support features in more expression forms, enrich the expression forms of the initial multimodal features, and enhance the breadth of model application. Moreover, it can also effectively enhance the semantic understanding ability of the model, and has better generalization ability when facing different types of inputs, laying the foundation for the subsequent generation of high-quality reasoning results that meet user needs.

[0080] Furthermore, in a specific example, the feature expression of the image feature of the entity image may be determined in the following manner; the initial multimodal features of the sample data obtained above may specifically include:

[0081] Method 1: Utilize the image-text encoder in the preset model, and based on the sample features of the sample data (for example, based on the task features of the generation task indicated by the sample data), select one or more of the above-mentioned feature expressions to obtain the image features of the entity image contained in the sample data, and then obtain the initial multimodal features.

[0082] Method 2: Using the image-text encoder in the preset model, randomly select one or more of the above feature expressions to obtain the image features of the entity images contained in the sample data.

[0083] It should be noted that when the above-mentioned method 1 or method 2 is adopted and multiple methods are selected from the above-mentioned feature expression methods, one sample data can be expanded to obtain multiple initial multimodal features. In this way, the sample size of the model training is effectively enriched, thereby providing strong support for improving the training efficiency of the model training.

[0084] In addition, it should be noted that, during the model training phase, the above two methods may be selected to be executed, and the above methods selected for different sample data may be the same or different, and the disclosed solution does not impose any restrictions on this.

[0085] Furthermore, it can be understood that in order to further improve the generalization ability of the model, the feature expressions selected for different sample data can be different, so that the trained target model can take into account any reasoning requirements, further enrich the usage scenarios, and thus further improve the user experience.

[0086] In addition, it should be noted that the present disclosure does not impose any specific restrictions on how to select the above-mentioned feature expressions. For example, in the scene, it can be adjusted based on actual scene requirements or training requirements.

[0087] Furthermore, it should be noted that the entity image described in the disclosed solution can be specifically an image in any form, such as a segmentation map, a position coordinate map, a crop map or a line drawing, etc., and the disclosed solution does not impose any restrictions on this.

[0088] In this way, the disclosed scheme can select appropriate image feature expressions according to the actual situation of the sample data to construct diversified initial multimodal features. Compared with the existing use of a single RGB feature, the disclosed scheme can support features in more expression forms and thus have diversified initial multimodal features. In this way, the semantic understanding ability of the model can be enhanced in subsequent model training, thereby improving the controllability of the model in generating image content. In addition, the use of different feature expressions is also conducive to better handling more complex generation tasks, thereby further improving the reasoning efficiency of the model, thereby laying the foundation for further improving the user experience.

[0089] In a specific example of the disclosed scheme, each diffusion network in the N-layer diffusion network is a dual information flow diffusion network; further, in an example, the dual information flow diffusion network may specifically include: a first information flow for processing initial multimodal features (for example, in an example, it may be specifically a conditional information flow (ConditionFlow), or, it may also be called a multimodal conditional information flow), and a second information flow for compressing the initial multimodal features based on preset noise features to represent the initial multimodal features in a hidden space (for example, in an example, it may be specifically a latent space flow (Latent Flow)).

[0090] Here, due to the adoption of a dual information flow architecture, one branch of the dual information flow can also effectively process multimodal features. This enhances the semantic understanding ability and controllability of the model. At the same time, it ensures that the model can effectively support the input of multimodal information and can complete generation tasks under multimodal conditions (for example, image generation tasks, or text reasoning tasks, etc.), thereby further improving the reasoning ability of the model and enriching the usage scenarios of the model.

[0091] In addition, it should be noted that the feature dimension of the features represented in the hidden space is smaller than the initial multimodal features. In this way, the reasoning effect is effectively improved while the reasoning efficiency is taken into account.

[0092] For example, in one example, as shown in FIG3(a), the first information flow is used to process the input initial multimodal features, and the second information flow is used to compress the initial multimodal features based on the preset noise features. For example, in one example, the initial multimodal features with a feature dimension of 128×128×3 can be specifically mapped to the hidden space to compress the initial multimodal features in the hidden space and obtain new features with a feature dimension of 32×32×3. In addition, the preset noise features can also be introduced in the above process, so as to effectively enhance the reasoning ability of the model, so that it can learn data features more effectively and generate high-quality reasoning results.

[0093] In this way, the disclosed solution can use the dual information flow diffusion network for model reasoning, and one branch in the dual information flow can also effectively process multimodal features, thus ensuring that the model can effectively support the input of multimodal information and enable the model to understand the generation task from different perspectives, thereby retaining more details and semantic information in the subsequent task generation process, and efficiently generating reasoning results that meet user expectations. In addition, noise is also introduced in another branch of the dual information flow, so that the diversity and stability of the model in content generation can be improved.

[0094] Furthermore, in a specific example, when the value of N is greater than or equal to 2, the N diffusion networks are connected in series; further, as shown in FIG3(b), the output corresponding to the first information flow in the i-th (at this time, i is a natural number greater than 0 and less than or equal to N-1) diffusion network among the N diffusion networks is the input of the first information flow in the i+1-th diffusion network among the N diffusion networks; the output of the second information flow in the i-th diffusion network among the N diffusion networks is the input of the second information flow in the i+1-th diffusion network among the N diffusion networks. In this way, the N diffusion networks connected in series are used to improve the representation ability of the model, thereby improving the overall performance of the model. At the same time, the generalization ability and robustness of the model can also be effectively improved, laying a foundation for further improving the user experience.

[0095] Figure 4 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 to Figure 3(a) , the related contents of the method shown in FIG3(b) can also be applied to this example, and the related contents will not be described in detail in this example.

[0096] Further, the method includes at least part of the following contents. Figure 4 As shown, including:

[0097] Step S401: input sample data into a preset model to obtain initial multimodal features of the sample data.

[0098] Here, the sample data at least includes sample text and at least one entity image. Further, the sample text includes at least one entity text, and the entity image is an image corresponding to the entity text.

[0099] Furthermore, the initial multimodal features include image features of the entity image and text features of the entity text, and the image features of the entity image are after the text features of the entity text corresponding to the entity image; that is, in one example, the initial multimodal features are at least obtained by inserting the image features of the entity image after the text features of the entity text corresponding to the entity image.

[0100] Here, for the relevant content about the sample data, please refer to the above description, which will not be repeated here.

[0101] Step S402: inserting instruction information into the initial multimodal feature.

[0102] For instance, in one example, when it is determined based on sample data that image adjustment needs to be performed on at least part of the entity image, for example, adjustment needs to be made to a partial area of ​​a specified entity image, or image adjustment needs to be performed on several specified entity images, etc. At this time, instruction information can be inserted into the initial multimodal feature, for example, instruction information can be inserted into the feature sequence representing the initial multimodal feature.

[0103] For example, for sample data of the task generation type, if the generation task indicated by the sample data requires image adjustment of at least part of the entity images (for example, adjusting a partial area of ​​a specified entity image, or image adjustment of several specified entity images among all entity images, etc.), instruction information can be inserted into the initial multimodal feature.

[0104] Here, the instruction information may specifically include the image features of the entity image to be adjusted and the adjustment instructions. Furthermore, the adjustment instructions may specifically include adding, deleting, changing attributes, swapping, scaling, and moving adjustment instructions. In this way, introducing instruction information in model training can effectively improve the multimodal instruction capability of the model.

[0105] Step S403: When the parameters of the image-text encoder of the preset model are fixed, the N-layer diffusion network in the preset model is trained using the initial multimodal features (i.e., the initial multimodal features with instruction information spliced ​​(or inserted) in step S402) and the preset noise features to obtain a target model.

[0106] Here, the target model can infer a target inference result including an image and / or text associated with the multimodal prompt information based on the input multimodal prompt information. Further, N is a positive integer greater than or equal to 1.

[0107] Here, for the relevant explanations of the diffusion network and the target model, please refer to the above description and will not be repeated here.

[0108] In this way, the disclosed scheme can add instruction information to the initial multimodal features obtained, and use the initial multimodal features after adding the instruction information to perform model training on the preset model. In this way, the instruction information can be used to effectively improve the instruction capability of the model, thereby improving the model's ability to understand the generation tasks indicated by the sample data, so that the subsequently generated images are more in line with the generation tasks, achieving precise control of the generated image content, thereby further improving the user experience.

[0109] Further, in a specific example, the instruction information may be inserted in the following manner; specifically, the above-mentioned inserting instruction information into the initial multimodal feature (such as the above-mentioned step S402) specifically includes:

[0110] The instruction information is concatenated at the head of the feature sequence representing the initial multimodal features.

[0111] For example, Figure 2Taking the feature sequence of the initial multimodal feature represented as an example, when the generation task indicated by the sample data requires image adjustment of part of the entity image, as shown in FIG5(a), the instruction information can be spliced ​​at the head of the feature sequence represented by the initial multimodal feature. Further, as shown in FIG5(b), the instruction information can be represented by two sequences, one of which is a text feature sequence corresponding to the adjustment instruction (for example, it can be recorded as InstructText Tokens), and the other is an image feature sequence corresponding to the image feature of the entity image to be adjusted (for example, it can be recorded as Global Image Tokens).

[0112] Furthermore, it should be noted that Figure 2 Similar to the example shown, in order to distinguish text features from image features, a header field head can also be added before the image features (for example, Global Image Tokens) to distinguish the text features from the image features in the instruction information, thereby providing strong support for further improving the controllability of image generation and improving the reasoning effect of the model.

[0113] In this way, the disclosed scheme provides a splicing and refinement scheme for initial multimodal features and instruction information, which effectively improves the instruction capability of the model. At the same time, it can also achieve precise control of the generated image content during model training, thereby laying the foundation for the subsequent generation of images that meet user needs.

[0114] FIG6(a) is a schematic flow chart 3 of a model training method according to an embodiment of the present application. The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 The relevant contents of the method shown in FIG. 5 can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0115] Further, the method includes at least part of the following contents. As shown in FIG6(a), it includes:

[0116] Step S601: input sample data into a preset model to obtain initial multimodal features of the sample data.

[0117] Here, the sample data at least includes sample text and at least one entity image. Further, the sample text includes at least one entity text, and the entity image is an image corresponding to the entity text; further, the initial multimodal features include image features of the entity image and text features of the entity text, and the image features of the entity image are after the text features of the entity text corresponding to the entity image.

[0118] Here, for the relevant content about the sample data, please refer to the above description, which will not be repeated here.

[0119] Step S602: performing a masking operation (eg, a random masking operation) on some features of the initial multimodal features (or the initial multimodal features concatenated with instruction information) to obtain masked initial multimodal features.

[0120] For example, taking the feature sequence of the initial multimodal feature represented by FIG5(a) as an example, at this time, some features in the feature sequence representing the initial multimodal feature can be randomly masked, for example, some tokens (words) in the initial multimodal feature are randomly masked with a certain probability, as shown in FIG6(b), the text feature 1 in the initial multimodal feature is masked (corresponding to mask situation 1), or the image feature 1 in the initial multimodal feature is masked (corresponding to mask situation 2), and then the masked initial multimodal feature is obtained. Here, since the disclosed scheme can mask at least part of the features in the initial multimodal feature, so that finer-grained features can be processed in the subsequent first information flow, and then finer-grained modeling can be performed, so that the model's ability to control details is effectively improved, and the model's ability to learn fine-grained tasks is improved, thereby laying a foundation for improving the accuracy of the reasoning results.

[0121] Step S603: When the parameters of the image-text encoder of the preset model are fixed, the masked initial multimodal features are used as the input of the first information flow of the first diffusion network in the N-layer diffusion network, and the preset noise features are used as the input of the second information flow of the first diffusion network in the N-layer diffusion network, so as to use the N-layer diffusion network to perform feature diffusion and obtain the overall prediction result.

[0122] Here, the total prediction result may specifically include a first prediction result and a second prediction result.

[0123] Furthermore, the first prediction result is the result output by the first information flow of the last diffusion network in the N-layer diffusion network, and the first prediction result can at least represent the predicted mask position corresponding to the predicted random mask operation. That is to say, the first prediction result is used to predict the position of the mask operation. For example, in one example, the first prediction result represents the feature targeted by the mask operation, and / or represents the specific position of the feature targeted by the mask operation in the initial multimodal feature before the mask operation. In this way, feature reconstruction is achieved, which provides strong support for the model to learn more fine-grained generation tasks, achieve more fine-grained modeling, and then have the ability to generate multimodal information (for example, given an entity text, the bounding box (Bbox) of the entity indicated by the entity text is generated, the face identification (Face ID), or given an entity image, the text information of the entity image is generated, etc.).

[0124] Further, the second prediction result is the result output by the second information flow of the last diffusion network in the N-layer diffusion network, and the second prediction result can specifically represent the prediction result of the generation task indicated by the predicted sample data in the hidden space. In other words, the second prediction result is used to represent the expression of the prediction result in the hidden space, and the prediction result refers to the generation result required by the generation task indicated by the sample data. For example, in one example, the prediction result represents the prediction result output by the model in the current step in the hidden space.

[0125] Step S604: Based on the first prediction result and the second prediction result, obtain the loss value of the target loss function.

[0126] Here, the objective loss function is used to calculate the difference between the predicted mask position and the actual mask position, and to calculate the difference between the predicted result of the generation task indicated by the sample data in the hidden space and the actual result in the hidden space (for example, the actual result of the current step in the hidden space).

[0127] Here, the actual result in the hidden space can be understood as the theoretical result that needs to be achieved in the current step.

[0128] Step S605: Based on the loss value of the target loss function, at least part of the adjustable network parameters in the N-layer diffusion network are adjusted until the number of iterative training times (for example, the number of iterative training times of the current step) and / or the loss value of the target loss function meets the model training requirements. At this time, the training of the current step is completed, and the target model is obtained.

[0129] It should be noted that, in the scenario of training the diffusion network, the above steps S601 to S605 are specifically applied to any step in the total number of steps required for training the diffusion network (or diffusion model). Accordingly, steps S601 to S605 are specifically as follows:

[0130] Input the sample data required for the current step into the preset model, so as to obtain the initial multimodal features of the sample data used in the current step by using the image-text encoder in the preset model;

[0131] Performing a masking operation on some features in the initial multimodal features (or the initial multimodal features concatenated with instruction information) to obtain masked initial multimodal features;

[0132] When the parameters of the image-text encoder are fixed, the masked initial multimodal features are used as the input of the first information flow of the first diffusion network in the N-layer diffusion network, and the preset noise features required for the current step are used as the input of the second information flow of the first diffusion network in the N-layer diffusion network, so as to use the N-layer diffusion network for feature diffusion and obtain the total prediction result corresponding to the current step training;

[0133] Based on the total prediction results, the loss value of the target loss function is obtained, which is also the loss value of the current step training;

[0134] Based on the loss value of the current step training, at least part of the adjustable network parameters in the N-layer diffusion network are adjusted to complete the current step training, and then the target model is obtained after completing all steps of training.

[0135] For example, as shown in FIG6(c), first, the image-text encoder in the preset model is used to obtain the initial multimodal features, and a random masking operation is performed on some features of the initial multimodal features to obtain the masked initial multimodal features; secondly, the masked initial multimodal features are used as the input of the first information flow of the first diffusion network in the N-layer diffusion network in the preset model, and the preset noise features are used as the input of the second information flow of the first diffusion network, so as to use the N-layer diffusion network to perform feature diffusion, and then obtain the result output by the first information flow of the last diffusion network in the N-layer diffusion network (i.e., the first prediction result), and the result output by the second information flow of the last diffusion network in the N-layer diffusion network (i.e., the second prediction result); and calculate the loss value of the target loss function based on the first prediction result and the second prediction result; finally, based on the loss value of the target loss function, adjust at least some of the adjustable network parameters in the N-layer diffusion network in the preset model, so as to complete the training of the current step, and then obtain the target model after completing the training of all steps.

[0136] In this way, the disclosed scheme provides a refined scheme for training the N-layer diffusion network in the preset model to efficiently train the target model. Moreover, the training method effectively improves the model's learning ability for fine-grained tasks and improves the model's ability to control details, thereby improving the model's controllability over image content generation, making the resulting image content more consistent with the generation task indicated by the user, meeting the user's higher customization needs, and further improving the user experience.

[0137] Furthermore, in a specific example of the present disclosure, as shown in FIG7(a), the dual information flow diffusion network described above may include: a first conditional information flow module and a second conditional information flow module corresponding to the first information flow, and a first latent space flow module and a second latent space flow module corresponding to the second information flow.

[0138] Furthermore, in this example, in the dual information flow diffusion network, the input of the first information flow is the input of the first conditional information flow module, and the output of the second conditional information flow module is the output of the first information flow; accordingly, the input of the second information flow is the input of the first latent space flow module, and the output of the second latent space flow module is the output of the second information flow.

[0139] For example, for the first-layer dual information flow diffusion network in an N-layer dual information flow diffusion network, the input of the first information flow is the initial multimodal feature, and the input of the second information flow is the preset noise feature. At this time, the initial multimodal feature is the input of the first conditional information flow module in the first-layer dual information flow diffusion network, and the preset noise feature is the input of the first latent layer spatial flow module in the first-layer dual information flow diffusion network; accordingly, the output of the second conditional information flow module in the first-layer dual information flow diffusion network is the output of the first information flow in the first-layer dual information flow diffusion network, and the output of the second latent layer spatial flow module in the first-layer dual information flow diffusion network is the output of the second information flow in the first-layer dual information flow diffusion network.

[0140] Here, the first conditional information flow module in the above example can mainly include a normalization (Layernorm) layer, a modular operation (Mod) layer, and a linear (Linear) layer from top to bottom (for example, information is processed from top to bottom); Correspondingly, the first latent spatial flow module can also mainly include a normalization (Layernorm) layer, a modular operation (Mod) layer, and a linear (Linear) layer from top to bottom (for example, information is processed from top to bottom).

[0141] Furthermore, the second conditional information flow module in the above example may mainly include from top to bottom (for example, information is processed from top to bottom): linear layer, normalization layer, modular operation layer, multi-layer perceptron (Multi-Layer Perceptron, MLP), etc. Correspondingly, the second latent spatial flow module may also mainly include from top to bottom (for example, information is processed from top to bottom): linear layer, normalization layer, modular operation layer, multi-layer perceptron (Multi-Layer Perceptron, MLP), etc.

[0142] It should be noted that the above is only an exemplary description. In actual applications, the processing layers used in each information flow module and each spatial flow module can be set according to actual needs, and the present disclosure does not limit this.

[0143] In this way, the disclosed scheme provides a specific architecture of a dual information flow diffusion network, wherein one branch in the architecture can effectively process multimodal features, so that the model can understand the generation task from different perspectives, thereby retaining more details and semantic information in the subsequent task generation process. In addition, it can also achieve precise control of image content and reduce information loss, thereby ensuring that the generated image can meet user expectations, thereby improving the user experience; another branch in the architecture introduces noise to enhance the diversity and stability of the model in content generation, thereby effectively improving the flexibility and scalability of the entire model, thereby providing strong support for further improving the user experience.

[0144] Furthermore, in a specific example, the above-mentioned feature diffusion using the N-layer diffusion network, as shown in FIG7(b), may specifically include:

[0145] The following steps are adopted, and the i-th (here, i is an integer greater than 0 and less than or equal to N) layer diffusion network in the N-layer diffusion network (or the N-layer dual information flow diffusion network) is used to perform feature diffusion. In other words, the processing logic of the i-th layer diffusion network is as follows:

[0146] Step S701: the output of the first information flow in the i-1th layer diffusion network (that is, the output of the first information flow in the previous layer diffusion network of the current layer diffusion network, for example, it can be specifically the output of the second conditional information flow module in the i-1th layer diffusion network) is input into the first conditional information flow module in the i-th layer diffusion network (which can be understood as the current layer diffusion network) for processing, and the output of the second information flow in the i-1th layer diffusion network (that is, the output of the second information flow in the previous layer diffusion network of the current layer diffusion network, for example, the output of the second latent layer information flow module in the i-1th layer diffusion network) is input into the first latent layer spatial flow module in the i-th layer diffusion network for processing.

[0147] Step S702: performing element-level operations on the features processed by the first conditional information flow module in the i-th diffusion network and the features processed by the first latent spatial flow module in the i-th diffusion network to concatenate the features processed by the first conditional information flow module and the features processed by the first latent spatial flow module.

[0148] Step S703: Perform self-attention processing on the spliced ​​features to obtain fused features.

[0149] Step S704: Input the features in the fused features corresponding to the first information flow to the second conditional information flow module in the i-th diffusion network to obtain the output of the second conditional information flow module; and input the features in the fused features corresponding to the second information flow to the second latent spatial flow module in the i-th diffusion network to obtain the output of the second latent spatial flow module.

[0150] It should be pointed out that, in this example, when the value of i is 1, the input of the first conditional information flow module in the i-1th diffusion network is the masked initial multimodal feature (or the masked initial multimodal feature with instruction information spliced ​​therein), and the input of the first latent spatial flow module in the i-1th diffusion network is the preset noise feature.

[0151] In this way, the disclosed scheme provides a specific scheme for feature diffusion using an N-layer dual information flow diffusion network, which can effectively capture and disseminate key features and details in text and / or images, so that the model can understand the generation task from different angles, and thus retain more details and semantic information in the subsequent task generation process. In addition, it can also achieve precise control of image content, thereby ensuring that the generated image meets user expectations. At the same time, it can also improve the diversity and stability of the model in content generation, effectively improving the flexibility and scalability of the entire model, thereby providing strong support for further meeting users' higher customization needs and improving user experience.

[0152] Part 2: Detailed description of the model usage scenarios:

[0153] Figure 8 This is a schematic flow chart of a multimodal data processing method according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0154] Further, the method includes at least part of the following contents. Figure 8 As shown, including:

[0155] Step S801: input the multimodal prompt information into a pre-trained target model to obtain target multimodal features of the multimodal prompt information.

[0156] For example, in one example, after the multimodal prompt information is input into a pre-trained (or trained) target model, the target multimodal features of the multimodal prompt information can be obtained using the graphic encoder in the target model.

[0157] Here, in one example, the multimodal prompt information includes at least a target text and at least one target entity image (for the convenience of subsequent description, it can be recorded as M3 target entity images here). Further, the target text contains at least one target entity text (for the convenience of subsequent description, it can be recorded as M3 target entity text here). At this time, the target entity image is an image corresponding to the target entity text.

[0158] Here, M3 and M4 are positive integers greater than or equal to 1; further, in one example, the values ​​of M3 and M4 may be different, for example, M3 is less than M4, in which case, it may correspond to a scenario where not all target texts correspond to a target entity image. Alternatively, in another example, the values ​​of M3 and M4 are the same, in which case, it may correspond to a scenario where one target entity text corresponds to one target entity image.

[0159] Here, it should be noted that the target entity text and the target entity image can be one-to-one or one-to-many, and the disclosed solution does not limit this. Accordingly, the disclosed solution does not limit the values ​​of M3 and M4.

[0160] Furthermore, in one example, the target multimodal features include image features of a target entity image and text features of a target entity text, and the image features of the target entity image are after the text features of the target entity text corresponding to the target entity image; in other words, in this example, the target multimodal features are at least obtained by inserting the image features of the target entity image after the text features of the target entity text corresponding to the target entity image.

[0161] Furthermore, the target model in this example is trained based on any of the above model training methods.

[0162] Step S802: using the N-layer target diffusion network in the target model to perform feature diffusion on the target multimodal features, so as to obtain a target inference result including an image and / or text associated with the multimodal prompt information.

[0163] Here, N is a positive integer greater than or equal to 1.

[0164] In this way, the disclosed solution can utilize the target model to infer, based on the input multimodal prompt information, a target inference result containing images and / or texts associated with the multimodal prompt information. Compared with the existing solutions, the target model of the disclosed solution has multi-entity processing capabilities and multi-entity generation capabilities, which enriches the user experience and improves the user experience.

[0165] Moreover, since the disclosed solution makes full use of the multi-modal prompt information of mixed image and text input, in the scenario where image generation is required, the disclosed solution provides more context and detailed guidance, thereby effectively improving the controllability of image content generation and further meeting the user's higher reasoning needs.

[0166] It should be noted that, for the model training phase described above, the above model training method can be mainly applied to the training of the “current step” of the diffusion network. In other words, the above model training method can be mainly applied to the training of any step of the diffusion network. For the model usage phase of this example, it is necessary to use the target diffusion network to complete all steps before the target inference result described above can be obtained.

[0167] In addition, it should be noted that, when the multimodal prompt information specifically includes a sample text containing multiple target entity texts and multiple target entity images, and each target entity text has a corresponding target entity image, based on the normal browsing order of the text in the target text, the target multimodal features can be represented as: [text features of subtext 1 (which can be understood as target non-entity text) in the target text, text features of target entity text 1, and image features of the target entity image corresponding to target entity text 1; text features of subtext 2 (which can be understood as target non-entity text) in the target text, text features of target entity text 2, and image features of the target entity image corresponding to target entity text 2; ...]; It can be understood that the above feature representation method is only an exemplary description. In actual applications, the specific positions of the text features of the target non-entity text and the text features of the target entity text in the target multimodal features can be determined based on the positions of the target non-entity text and the target entity text in the target text.

[0168] Here, it should be noted that in the above example, see Figure 2 In the example shown, the image-text alignment strategy is used to align the different modal information in the multimodal prompt information, thereby achieving a mixed feature representation of images and texts, providing strong support for the subsequent improvement of the controllability of image generation.

[0169] Furthermore, it should be noted that in the above example, in order to more clearly distinguish the text features and image features in the target multimodal features, the disclosed solution can also set a header field (head) before the inserted image features (for example, the head), so as to distinguish the text features and image features in the target multimodal features, thereby providing strong support for further improving the controllability of image generation and improving the reasoning effect of the model. For details, please refer to Figure 2 The examples shown are not repeated here.

[0170] Further, in a specific example, the target multimodal feature can be obtained in the following manner; specifically, the above-mentioned step of obtaining the target multimodal feature of the multimodal prompt information (the above-mentioned step S801) specifically includes:

[0171] Using the image-text encoder in the target model and based on the data features of the multimodal prompt information (for example, based on the task features of the target generation task indicated by the multimodal prompt information), one or more of the following feature expressions are selected to obtain the image features of the target entity image contained in the multimodal prompt information, so as to obtain the target multimodal features:

[0172] The position features of the target entity image, the segmentation features of the segmentation map corresponding to the target entity image, the image features of the crop map of the target entity image, the image depth features of the target entity image, and the local features of the target entity image (for example, facial features, and further, human face features).

[0173] That is to say, the disclosed scheme can determine the expression method of image features suitable for the task requirements based on the task requirements of the input multimodal prompt information. In other words, the disclosed scheme can flexibly support various forms of image features, thereby enriching the expression form of the target multimodal features and having better generalization ability when facing different types of task requirements, laying the foundation for the subsequent generation of high-quality reasoning results that meet user needs, and further laying the foundation for further improving the user experience.

[0174] It should be noted that the target entity image described in the disclosed solution can be specifically an image in any form, such as a segmentation map, a position coordinate map, a crop map or a line drawing, and the disclosed solution does not impose any restrictions on this.

[0175] In this way, the disclosed scheme can select appropriate expression methods of image features according to the actual situation of the input multimodal prompt information to construct diversified target multimodal features. Compared with the existing use of single RGB features, the disclosed scheme can support more expression forms of features, thereby obtaining diversified target multimodal features. This is conducive to better handling more complex generation tasks, laying the foundation for further meeting users' higher reasoning needs and further improving user experience.

[0176] Furthermore, in a specific example, before the target multimodal features are diffused using the N-layer target diffusion network in the target model, the target multimodal features may be adjusted based on the target generation task indicated by the multimodal prompt information, specifically including:

[0177] Target instruction information is inserted into the target multimodal feature, wherein the target instruction information includes image features of the target entity image to be adjusted and a target adjustment instruction.

[0178] For instance, in one example, when it is determined based on the target multimodal features that at least part of the target entity image needs to be adjusted, for example, a partial area of ​​a specified target entity image needs to be adjusted, or image adjustments need to be performed on several specified target entity images, etc. At this time, target instruction information can be inserted into the feature sequence representing the target multimodal features, and then the N-layer target diffusion network in the target model can be used to perform feature diffusion on the target multimodal features with the target instruction information inserted.

[0179] For instance, in one example, if the multimodal prompt information is prompt information of task generation type information, at this time, when the target generation task indicated by the multimodal prompt information requires image adjustment of at least part of the target entity image (for example, adjusting a partial area of ​​a specified target entity image, or, image adjustment of several specified target entity images, etc.), the target instruction information can be inserted into the target multimodal feature.

[0180] Here, the target instruction information includes the image features of the target entity image to be adjusted and the target adjustment instruction. Furthermore, the target adjustment instruction can be specifically an adjustment instruction such as adding, deleting, changing attributes, exchanging, scaling and moving. In this way, the target instruction information is effectively used to instruct the reasoning process of the model, thereby improving the model's ability to understand the target generation task indicated by the multimodal prompt information, making the subsequently generated image more consistent with the target generation task, achieving precise control of the generated image content, and further improving the user experience.

[0181] Furthermore, in a specific example, the target instruction information may be inserted in the following manner; specifically, the above-mentioned inserting the target instruction information into the target multimodal feature specifically includes:

[0182] The target instruction information is spliced ​​at the head of the target feature sequence representing the target multimodal features.

[0183] For example, similar to the example in FIG5(a), when the target generation task indicated by the multimodal prompt information requires image adjustment of part of the target entity image, the target instruction information can be spliced ​​at the head of the target feature sequence represented by the target multimodal feature. Further, similar to the example shown in FIG5(b), the target instruction information can be represented by two sequences, one of which is a text feature sequence corresponding to the target adjustment instruction (for example, it can be recorded as Instruct Text Tokens), and the other is an image feature sequence corresponding to the image features of the target entity image to be adjusted (for example, it can be recorded as Global Image Tokens).

[0184] Furthermore, it should be noted that Figure 2 Similar to the example shown, in order to distinguish text features from image features, a header field head can also be added before the image features (for example, Global Image Tokens) to distinguish the text features from the image features in the target instruction information, thereby providing strong support for further improving the controllability of image generation and improving the reasoning effect of the model.

[0185] In this way, the disclosed scheme provides a splicing and refinement scheme of target multimodal features and target command information, which effectively improves the command capability of the model. At the same time, it can also achieve precise control of the generated image content during model reasoning, thereby laying the foundation for the subsequent generation of images that meet user needs.

[0186] In a specific example of the disclosed scheme, each target diffusion network in the N-layer target diffusion network is a dual information flow target diffusion network; further, in one example, the dual information flow target diffusion network may specifically include: a first target information flow for processing target multimodal features (for example, the first target information flow may be specifically a conditional information flow (Condition Flow), or may also be referred to as a first target multimodal conditional information flow), and a second target information flow for compressing the target multimodal features based on preset target noise features to represent the target multimodal features in a hidden space (for example, in one example, may be specifically a latent space flow (Latent Flow)).

[0187] Here, due to the adoption of a dual information flow architecture, and a branch of the dual information flow can effectively process multimodal features, the semantic understanding ability and controllability of the model are enhanced. At the same time, it effectively ensures that the model can efficiently complete target generation tasks (such as image generation tasks, or text reasoning tasks, etc.) under multimodal conditions, thereby further improving the model's reasoning ability and enriching the model's usage scenarios.

[0188] In addition, it should be noted that the feature dimension of the features represented in the latent space is smaller than the target multimodal features. In this way, the reasoning effect is effectively improved while the reasoning efficiency is taken into account.

[0189] For example, in one example, similar to the example shown in FIG3 (a), in a dual information flow target diffusion network, the first target information flow is used to process the input target multimodal features (or target multimodal features spliced ​​with target instruction information), and the second target information flow is used to compress the target multimodal features based on the preset target noise features. For example, in one example, the target multimodal features with a feature dimension of 128×128×3 can be specifically mapped to the hidden space to compress the target multimodal features in the hidden space and obtain new features with a feature dimension of 32×32×3. In addition, the preset target noise features can also be introduced in the above process, so as to effectively enhance the reasoning ability of the model, so that it can learn data features more effectively and generate high-quality reasoning results.

[0190] In this way, the disclosed solution can use the dual information flow target diffusion network for model reasoning, and one branch in the dual information flow can also effectively process multimodal features, thus ensuring that the model can effectively support the input of multimodal information and enable the model to understand the generation task from different perspectives, thereby retaining more details and semantic information in the subsequent task generation process, and efficiently generating reasoning results that meet user expectations. In addition, noise is also introduced in another branch of the dual information flow, so that the diversity and stability of the model in content generation can be improved.

[0191] Further, in a specific example, when the value of N is greater than or equal to 2, the N target diffusion networks are connected in series; further, similar to the example shown in FIG3(b), the output corresponding to the first target information flow in the i-th (i is a natural number greater than or equal to 0 and less than or equal to N-1) target diffusion network in the N target diffusion networks is the input of the first target information flow in the i+1-th target diffusion network in the N target diffusion networks; the output of the second target information flow in the i-th target diffusion network in the N target diffusion networks is the input of the second target information flow in the i+1-th target diffusion network in the N target diffusion networks. In this way, the N target diffusion networks connected in series are used to improve the representation ability of the model, thereby improving the overall performance of the model. At the same time, the generalization ability and robustness of the model can also be effectively improved, laying a foundation for further improving the user experience.

[0192] Fig. 9 This is a schematic flow chart of a multimodal data processing method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 8 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0193] Further, the method includes at least part of the following contents. Fig. 9 As shown, including:

[0194] Step S901: inputting multimodal prompt information into a target model to obtain target multimodal features of the multimodal prompt information.

[0195] Here, the target model is trained based on the above model training method.

[0196] Furthermore, the multimodal prompt information includes at least a target text and at least one target entity image, the target text contains at least one target entity text, and the target entity image is an image corresponding to the target entity text; the target multimodal features include image features of the target entity image and text features of the target entity text, and the image features of the target entity image are after the text features of the target entity text corresponding to the target entity image.

[0197] Here, the relevant content about the multimodal prompt information can refer to the above examples, which will not be repeated here.

[0198] Step S902: Using the target multimodal features (or the target multimodal features spliced ​​with target instruction information) as the input of the first target information flow of the first target diffusion network in the N-layer target diffusion network, and using the preset target noise features as the input of the second target information flow of the first target diffusion network in the N-layer target diffusion network, so as to use the N-layer target diffusion network to perform feature diffusion and obtain an initial inference result.

[0199] Here, the initial reasoning result is obtained based on the result output by the second target information flow of the last layer of the target diffusion network in the N-layer target diffusion network, and represents the predicted result of the target generation task in the hidden space, that is, the initial reasoning result represents the expression of the prediction result in the hidden space, and the prediction result here specifically refers to the generation result required for the target generation task indicated by the multimodal prompt information.

[0200] Step S903: When the initial inference result includes image features, the image features in the initial inference result are decoded by using the image decoder in the target model to obtain the target inference result.

[0201] For example, if Fig.10 As shown, first, the multimodal prompt information input by the target object is input into the target model, so as to obtain the target multimodal features by using the image-text encoder in the target model; secondly, the target multimodal features are used as the input of the first target information flow of the first target diffusion network in the N-layer target diffusion network, and the preset target noise features are used as the input of the second target information flow of the first target diffusion network in the N-layer target diffusion network, so as to use the N-layer target diffusion network to perform feature diffusion. For example, after performing feature diffusion for a preset number of steps, the output result of the second target information flow of the last target diffusion network in the N-layer target diffusion network (that is, the output result of the second target information flow of the last target diffusion network in the last step) is obtained, that is, the initial reasoning result; finally, the final target reasoning result is obtained based on the initial reasoning result, or, when the initial reasoning result contains image features, the image features in the initial reasoning result are decoded by using the image decoder in the target model to obtain the target reasoning result.

[0202] In this way, the disclosed solution provides a refined solution for reasoning using an N-layer target diffusion network in a target model, which achieves precise control over the generation of image content, so that the obtained image content is more closely aligned with the generation task indicated by the user, thereby satisfying the user's higher customization needs and further improving the user experience.

[0203] Furthermore, in a specific example of the present disclosure, the dual information flow target diffusion network described above may include: a first target conditional information flow module and a second target conditional information flow module corresponding to the first target information flow, and a first target latent space flow module and a second target latent space flow module corresponding to the second target information flow.

[0204] Further, similar to the example shown in Figure 7(a), in this example, in the dual information flow target diffusion network, the input of the first target information flow is the input of the first target conditional information flow module, and the output of the second target conditional information flow module is the output of the first target information flow; accordingly, the input of the second target information flow is the input of the first target latent space flow module, and the output of the second target latent space flow module is the output of the second target information flow.

[0205] For example, for the first-layer dual information flow target diffusion network in an N-layer dual information flow target diffusion network, the input of the first target information flow is the target multimodal feature, and the input of the second target information flow is the preset target noise feature. At this time, the target multimodal feature is the input of the first target conditional information flow module in the first-layer dual information flow target diffusion network, and the preset target noise feature is the input of the first target latent layer spatial flow module in the first-layer dual information flow target diffusion network; accordingly, the output of the second target conditional information flow module in the first-layer dual information flow target diffusion network is the output of the first target information flow in the first-layer dual information flow target diffusion network, and the output of the second target latent layer spatial flow module in the first-layer dual information flow target diffusion network is the output of the second target information flow in the first-layer dual information flow target diffusion network.

[0206] In this way, the disclosed scheme provides a specific architecture of a dual information flow target diffusion network, wherein one branch in the architecture can effectively process multimodal features, so that the model can understand the generation task from different perspectives, thereby retaining more details and semantic information in the subsequent task generation process. In addition, it can also achieve precise control of image content and reduce information loss, thereby ensuring that the generated image can meet user expectations, thereby improving the user experience; another branch in the architecture introduces noise to enhance the diversity and stability of the model in content generation, thereby effectively improving the flexibility and scalability of the entire model, thereby providing strong support for further improving the user experience.

[0207] Further, in a specific example, the above-mentioned feature diffusion using the N-layer target diffusion network, similar to the example shown in FIG. 7( b ), may specifically include:

[0208] The following steps are adopted, and the i-th (here, i is an integer greater than 0 and less than or equal to N) layer target diffusion network in the N-layer target diffusion network (or the N-layer dual information flow target diffusion network) is used to perform feature diffusion. In other words, the processing logic of the i-th layer target diffusion network is as follows:

[0209] Step S1101: the output of the first target information flow in the i-1th target diffusion network (that is, the output of the first target information flow in the target diffusion network of the previous layer of the current target diffusion network, for example, the output of the second target conditional information flow module in the i-1th target diffusion network) is input into the first target conditional information flow module in the i-th target diffusion network (which can be understood as the current target diffusion network) for processing, and the output of the second target information flow in the i-1th target diffusion network (that is, the output of the second target information flow in the target diffusion network of the previous layer of the current target diffusion network, for example, the output of the second target latent layer information flow module in the i-1th target diffusion network) is input into the first target latent layer spatial flow module in the i-th target diffusion network for processing.

[0210] Step S1102: performing element-level operations on features processed by the first target conditional information flow module in the i-th target diffusion network and features processed by the first target latent spatial flow module in the i-th target diffusion network to concatenate the features processed by the first target conditional information flow module and the features processed by the first target latent spatial flow module.

[0211] Step S1103: Perform self-attention processing on the spliced ​​features to obtain target fusion features.

[0212] Step S1104: Input the features in the target fusion features corresponding to the first target information flow to the second target conditional information flow module in the i-th layer target diffusion network to obtain the output of the second target conditional information flow module; and input the features in the target fusion features corresponding to the second target information flow to the second target latent space flow module in the i-th layer target diffusion network to obtain the output of the second target latent space flow module.

[0213] It should be pointed out that, in this example, when the value of i is 1, the input of the first target conditional information flow module in the i-1th layer target diffusion network is the target multimodal feature (or the target multimodal feature spliced ​​with target instruction information), and the input of the first target latent layer spatial flow module in the i-1th layer target diffusion network is the preset target noise feature.

[0214] In addition, it should be noted that, for the last target diffusion network, the output of the second target latent spatial flow module in the last target diffusion network can be directly used as the initial reasoning result.

[0215] In this way, the disclosed scheme provides a specific scheme for feature diffusion using an N-layer dual information flow target diffusion network, which can effectively capture and propagate key features and details in text and / or images, so that the model can understand the generation task from different perspectives, and thus retain more details and semantic information in the subsequent task generation process. In addition, it can also achieve precise control of image content, thereby ensuring that the generated image meets user expectations. At the same time, it can also improve the diversity and stability of the model in content generation, thereby providing strong support for further meeting users' higher customization needs and improving user experience.

[0216] The disclosed solution also provides a model training device, such as Fig.11 As shown, including:

[0217] The training unit 1101 is used to input the sample data into the preset model, so as to obtain the initial multimodal features of the sample data by using the image-text encoder in the preset model; wherein the sample data at least includes a sample text and at least one entity image, the sample text includes at least one entity text, and the entity image is an image corresponding to the entity text; the initial multimodal features include image features of the entity image and text features of the entity text, and the image features of the entity image are after the text features of the entity text corresponding to the entity image; when the parameters of the image-text encoder of the preset model are fixed, the N-layer diffusion network in the preset model is trained by using the initial multimodal features and the preset noise features to obtain a target model; wherein N is a positive integer;

[0218] The storage unit 1102 is used to output the target model.

[0219] In a specific example of the disclosed solution, the training unit is further used to:

[0220] Insert instruction information into the initial multimodal features, wherein the instruction information includes image features of the entity image to be adjusted and adjustment instructions. For example, in one example, before the N-layer diffusion network in the preset model is trained using the initial multimodal features and the preset noise features, and when image adjustment of the entity image is required, the instruction information is inserted into the initial multimodal features.

[0221] In a specific example of the disclosed solution, the training unit is specifically used to:

[0222] The instruction information is concatenated at the head of the feature sequence representing the initial multimodal features.

[0223] In a specific example of the present disclosure, the N-layer diffusion networks are all dual information flow diffusion networks, and the dual information flow diffusion networks include: a first information flow for processing initial multimodal features, and a second information flow for compressing the initial multimodal features based on preset noise features to represent the initial multimodal features in the hidden layer space.

[0224] In a specific example of the disclosed scheme, when the value of N is greater than or equal to 2, the N diffusion networks are connected in series; wherein the output corresponding to the first information flow in the i-th diffusion network among the N diffusion networks is the input of the first information flow in the i+1-th diffusion network among the N diffusion networks; and the output of the second information flow in the i-th diffusion network among the N diffusion networks is the input of the second information flow in the i+1-th diffusion network among the N diffusion networks.

[0225] In a specific example of the disclosed solution, the training unit is further used to:

[0226] A masking operation is performed on some features of the initial multimodal features to obtain masked initial multimodal features.

[0227] In a specific example of the disclosed solution, the training unit is specifically used to:

[0228] The masked initial multimodal features are used as the input of the first information flow of the first diffusion network in the N-layer diffusion network, and the preset noise features are used as the input of the second information flow of the first diffusion network in the N-layer diffusion network, so as to use the N-layer diffusion network to perform feature diffusion and obtain a total prediction result; wherein the total prediction result includes a first prediction result and a second prediction result, the first prediction result is the result output by the first information flow of the last diffusion network in the N-layer diffusion network, at least indicating the predicted mask position corresponding to the predicted mask operation; the second prediction result is the result output by the second information flow of the last diffusion network in the N-layer diffusion network, indicating the prediction result of the generation task indicated by the predicted sample data in the hidden space;

[0229] Based on the first prediction result and the second prediction result, a loss value of a target loss function is obtained; wherein the target loss function is used to calculate the difference between the predicted mask position and the actual mask position, and to calculate the difference between the predicted result of the generation task indicated by the sample data in the hidden space and the actual result in the hidden space;

[0230] At least some of the adjustable network parameters in the N-layer diffusion network are adjusted based on the loss value of the target loss function.

[0231] In a specific example of the disclosed solution, the dual information flow diffusion network includes: a first conditional information flow module and a second conditional information flow module corresponding to the first information flow, and a first latent space flow module and a second latent space flow module corresponding to the second information flow;

[0232] Wherein, the input of the first information flow is the input of the first conditional information flow module, and the output of the second conditional information flow module is the output of the first information flow;

[0233] The input of the second information stream is the input of the first latent spatial stream module, and the output of the second latent spatial stream module is the output of the second information stream.

[0234] In a specific example of the disclosed solution, the training unit is specifically used to:

[0235] The following steps are adopted and feature diffusion is performed using the i-th diffusion network in the N-layer diffusion network:

[0236] The output of the first information flow in the i-1th diffusion network is input into the first conditional information flow module in the i-th diffusion network for processing, and the output of the second information flow in the i-1th diffusion network is input into the first latent spatial flow module in the i-th diffusion network for processing;

[0237] Performing element-wise operations on the features processed by the first conditional information flow module in the i-th diffusion network and the features processed by the first latent spatial flow module in the i-th diffusion network to concatenate the features processed by the first conditional information flow module and the features processed by the first latent spatial flow module;

[0238] Perform self-attention processing on the concatenated features to obtain fused features;

[0239] The features in the fused features corresponding to the first information flow are input into the second conditional information flow module in the i-th diffusion network to obtain the output of the second conditional information flow module; and the features in the fused features corresponding to the second information flow are input into the second latent spatial flow module in the i-th diffusion network to obtain the output of the second latent spatial flow module.

[0240] In a specific example of the present disclosure, the feature expression method of the image features of the entity image includes at least one of the following: position features of the entity image, segmentation features of the segmentation map corresponding to the entity image, image features of the cropped map of the entity image, image depth features of the entity image, and local features of the entity image.

[0241] In a specific example of the disclosed solution, the training unit is specifically used to:

[0242] Using the image-text encoder in the preset model and based on the sample features of the sample data, one or more of the above feature expressions are selected to obtain the image features of the entity image contained in the sample data, so as to obtain the initial multimodal features of the sample data;

[0243] Alternatively, one or more of the above feature expressions are randomly selected to obtain image features of the entity images contained in the sample data, so as to obtain initial multimodal features of the sample data.

[0244] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0245] The disclosed solution also provides a multi-modal data processing device, such as Fig.12 As shown, including:

[0246] The inference unit 1201 is used to input the multimodal prompt information into the pre-trained target model to obtain the target multimodal features of the multimodal prompt information, wherein the multimodal prompt information at least includes a target text and at least one target entity image, the target text includes at least one target entity text, and the target entity image is an image corresponding to the target entity text; the target multimodal features include image features of the target entity image and text features of the target entity text, and the image features of the target entity image are after the text features of the target entity text corresponding to the target entity image; the target multimodal features are diffused using an N-layer target diffusion network in the target model to obtain a target inference result associated with the multimodal prompt information and including an image and / or text; wherein N is a positive integer;

[0247] The output unit 1202 is used to output the target reasoning result.

[0248] In a specific example of the disclosed solution, the inference unit is further used to: insert target instruction information into the target multimodal feature, wherein the target instruction information includes the image feature of the target entity image to be adjusted and the target adjustment instruction. For example, in one example, when image adjustment is required for at least part of the target entity image, the target instruction information is inserted into the target multimodal feature.

[0249] In a specific example of the disclosed solution, the reasoning unit is specifically used to:

[0250] The target instruction information is spliced ​​at the head of the target feature sequence representing the target multimodal features.

[0251] In a specific example of the present disclosure, the N-layer target diffusion networks are all dual information flow target diffusion networks, and the dual information flow target diffusion networks include: a first target information flow for processing target multimodal features, and a second target information flow for compressing the target multimodal features based on preset target noise features to represent the target multimodal features in the hidden space.

[0252] In a specific example of the disclosed scheme, when the value of N is greater than or equal to 2, the N target diffusion networks are connected in series; wherein the output corresponding to the first target information flow in the i-th target diffusion network among the N target diffusion networks is the input of the first target information flow in the i+1-th target diffusion network among the N target diffusion networks; and the output of the second target information flow in the i-th target diffusion network among the N target diffusion networks is the input of the second target information flow in the i+1-th target diffusion network among the N target diffusion networks.

[0253] In a specific example of the disclosed solution, the reasoning unit is specifically used to:

[0254] The target multimodal feature is used as the input of the first target information flow of the first target diffusion network in the N-layer target diffusion network, and the preset target noise feature is used as the input of the second target information flow of the first target diffusion network in the N-layer target diffusion network, so as to use the N-layer target diffusion network to perform feature diffusion and obtain an initial reasoning result, wherein the initial reasoning result is obtained based on the result output by the second target information flow of the last target diffusion network in the N-layer target diffusion network, and represents the predicted prediction result of the target generation task in the hidden space;

[0255] In the case where the initial inference result includes image features, the image features in the initial inference result are decoded using an image decoder in the target model to obtain the target inference result.

[0256] In a specific example of the disclosed solution, the dual information flow target diffusion network includes: a first target conditional information flow module and a second target conditional information flow module corresponding to the first target information flow, and a first target latent space flow module and a second target latent space flow module corresponding to the second target information flow;

[0257] Wherein, the input of the first target information flow is the input of the first target condition information flow module, and the output of the second target condition information flow module is the output of the first target information flow;

[0258] The input of the second target information stream is the input of the first target latent spatial stream module, and the output of the second target latent spatial stream module is the output of the second target information stream.

[0259] In a specific example of the disclosed solution, the reasoning unit is specifically used to:

[0260] The following steps are adopted and feature diffusion is performed using the i-th target diffusion network in the N-layer target diffusion network:

[0261] The output of the first target information flow in the i-1th layer target diffusion network is input into the first target conditional information flow module in the i-th layer target diffusion network for processing, and the output of the second target information flow in the i-1th layer target diffusion network is input into the first target latent space flow module in the i-th layer target diffusion network for processing;

[0262] Performing element-wise operations on the features processed by the first target conditional information flow module in the i-th target diffusion network and the features processed by the first target latent spatial flow module in the i-th target diffusion network to concatenate the features processed by the first target conditional information flow module and the features processed by the first target latent spatial flow module;

[0263] Perform self-attention processing on the concatenated features to obtain the target fusion features;

[0264] The features in the target fusion features corresponding to the first target information flow are input into the second target conditional information flow module in the i-th layer target diffusion network to obtain the output of the second target conditional information flow module; and the features in the target fusion features corresponding to the second target information flow are input into the second target latent layer spatial flow module in the i-th layer target diffusion network to obtain the output of the second target latent layer spatial flow module.

[0265] In a specific example of the disclosed solution, the reasoning unit is specifically used to:

[0266] By using the image-text encoder in the target model and based on the data features of the multimodal prompt information, one or more of the following feature expressions are selected to obtain the image features of the target entity image contained in the multimodal prompt information, so as to obtain the target multimodal features:

[0267] The position features of the target entity image, the segmentation features of the segmentation map corresponding to the target entity image, the image features of the cropped map of the target entity image, the image depth features of the target entity image, and the local features of the target entity image.

[0268] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0269] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0270] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0271] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0272] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0273] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0274] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as a model training method or a multimodal data processing method. For example, in some embodiments, the model training method or the multimodal data processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the model training method or the multimodal data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the model training method or the multimodal data processing method in any other suitable manner (for example, by means of firmware).

[0275] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0276] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0277] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0278] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0279] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0280] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0281] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0282] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A target model training method, comprising: Input the sample data into a preset model to obtain initial multimodal features of the sample data; wherein the sample data at least includes sample text and at least one entity image, the sample text includes at least one entity text, and the entity image is an image corresponding to the entity text; the initial multimodal features include image features of the entity image and text features of the entity text, and the image features of the entity image are after the text features of the entity text corresponding to the entity image; When the parameters of the image-text encoder of the preset model are fixed, the N-layer diffusion network in the preset model is trained using the initial multimodal features and the preset noise features to obtain a target model, where N is a positive integer.

2. The method according to claim 1, further comprising: Insert instruction information into the initial multimodal features, wherein the instruction information includes image features of the entity image to be adjusted and adjustment instructions.

3. The method according to claim 2, wherein: The inserting instruction information into the initial multimodal feature includes: The instruction information is concatenated at the head of the feature sequence representing the initial multimodal features.

4. The method according to any one of claims 1 to 3, wherein: The N-layer diffusion networks are all dual information flow diffusion networks, which include: a first information flow for processing initial multimodal features, and a second information flow for compressing the initial multimodal features based on preset noise features to represent the initial multimodal features in the hidden layer space.

5. The method according to claim 4, wherein: When the value of N is greater than or equal to 2, the N diffusion networks are connected in series; wherein the output corresponding to the first information flow in the i-th diffusion network among the N diffusion networks is the input of the first information flow in the i+1-th diffusion network among the N diffusion networks; and the output of the second information flow in the i-th diffusion network among the N diffusion networks is the input of the second information flow in the i+1-th diffusion network among the N diffusion networks.

6. The method according to claim 4 or 5, further comprising: A masking operation is performed on some features of the initial multimodal features to obtain masked initial multimodal features.

7. The method according to claim 6, wherein: The method of using the initial multimodal features and the preset noise features to perform model training on the N-layer diffusion network in the preset model includes: The masked initial multimodal features are used as the input of the first information flow of the first diffusion network in the N-layer diffusion network, and the preset noise features are used as the input of the second information flow of the first diffusion network in the N-layer diffusion network, so as to use the N-layer diffusion network to perform feature diffusion and obtain a total prediction result; wherein the total prediction result includes a first prediction result and a second prediction result, the first prediction result is the result output by the first information flow of the last diffusion network in the N-layer diffusion network, at least indicating the predicted mask position corresponding to the predicted mask operation; the second prediction result is the result output by the second information flow of the last diffusion network in the N-layer diffusion network, indicating the prediction result of the generation task indicated by the predicted sample data in the hidden space; Based on the first prediction result and the second prediction result, a loss value of a target loss function is obtained; wherein the target loss function is used to calculate the difference between the predicted mask position and the actual mask position, and to calculate the difference between the predicted result of the generation task indicated by the sample data in the hidden space and the actual result in the hidden space; At least some of the adjustable network parameters in the N-layer diffusion network are adjusted based on the loss value of the target loss function.

8. The method according to claim 7, wherein: The dual information flow diffusion network comprises: A first conditional information flow module and a second conditional information flow module corresponding to the first information flow, and a first latent spatial flow module and a second latent spatial flow module corresponding to the second information flow; Wherein, the input of the first information flow is the input of the first conditional information flow module, and the output of the second conditional information flow module is the output of the first information flow; The input of the second information stream is the input of the first latent spatial stream module, and the output of the second latent spatial stream module is the output of the second information stream.

9. The method according to claim 8, wherein: The feature diffusion is performed using an N-layer diffusion network, including: The following steps are adopted and feature diffusion is performed using the i-th diffusion network in the N-layer diffusion network: The output of the first information flow in the i-1th diffusion network is input into the first conditional information flow module in the i-th diffusion network for processing, and the output of the second information flow in the i-1th diffusion network is input into the first latent spatial flow module in the i-th diffusion network for processing; Performing element-wise operations on the features processed by the first conditional information flow module in the i-th diffusion network and the features processed by the first latent spatial flow module in the i-th diffusion network to concatenate the features processed by the first conditional information flow module and the features processed by the first latent spatial flow module; Perform self-attention processing on the concatenated features to obtain fused features; The features in the fused features corresponding to the first information flow are input into the second conditional information flow module in the i-th diffusion network to obtain the output of the second conditional information flow module; and the features in the fused features corresponding to the second information flow are input into the second latent spatial flow module in the i-th diffusion network to obtain the output of the second latent spatial flow module.

10. The method according to any one of claims 1 to 9, wherein: The feature expression method of the image feature of the entity image includes at least one of the following: the position feature of the entity image, the segmentation feature of the segmentation map corresponding to the entity image, the image feature of the cropped map of the entity image, the image depth feature of the entity image, and the local feature of the entity image.

11. The method according to claim 10, wherein: The obtaining of initial multimodal features of the sample data includes: Using the image-text encoder in the preset model and based on the sample features of the sample data, one or more of the above feature expressions are selected to obtain the image features of the entity image contained in the sample data, so as to obtain the initial multimodal features of the sample data; Alternatively, one or more of the above feature expressions are randomly selected to obtain image features of the entity images contained in the sample data, so as to obtain initial multimodal features of the sample data.

12. A multimodal data processing method, comprising: Inputting the multimodal prompt information into a pre-trained target model to obtain target multimodal features of the multimodal prompt information, wherein the multimodal prompt information at least includes a target text and at least one target entity image, the target text includes at least one target entity text, and the target entity image is an image corresponding to the target entity text; the target multimodal features include image features of the target entity image and text features of the target entity text, and the image features of the target entity image are after the text features of the target entity text corresponding to the target entity image; The target multimodal features are diffused using an N-layer target diffusion network in the target model to obtain a target inference result including an image and / or text associated with the multimodal prompt information; wherein N is a positive integer.

13. The method according to claim 12, further comprising: Target instruction information is inserted into the target multimodal feature, wherein the target instruction information includes image features of the target entity image to be adjusted and a target adjustment instruction.

14. The method according to claim 13, wherein: The inserting target instruction information into the target multimodal feature includes: The target instruction information is spliced ​​at the head of the target feature sequence representing the target multimodal features.

15. The method according to any one of claims 12 to 14, wherein: The N-layer target diffusion network is a dual information flow target diffusion network, which includes: a first target information flow for processing target multimodal features, and a second target information flow for compressing the target multimodal features based on preset target noise features to represent the target multimodal features in a hidden layer space.

16. The method according to claim 15, wherein: When the value of N is greater than or equal to 2, the N target diffusion networks are connected in series; wherein, the output corresponding to the first target information flow in the i-th target diffusion network among the N target diffusion networks is the input of the first target information flow in the i+1-th target diffusion network among the N target diffusion networks; and the output of the second target information flow in the i-th target diffusion network among the N target diffusion networks is the input of the second target information flow in the i+1-th target diffusion network among the N target diffusion networks.

17. The method according to claim 15, wherein: The step of utilizing the N-layer target diffusion network in the target model to perform feature diffusion on the target multimodal features to obtain a target inference result including an image and / or text associated with the multimodal prompt information includes: The target multimodal feature is used as the input of the first target information flow of the first target diffusion network in the N-layer target diffusion network, and the preset target noise feature is used as the input of the second target information flow of the first target diffusion network in the N-layer target diffusion network, so as to use the N-layer target diffusion network to perform feature diffusion and obtain an initial reasoning result, wherein the initial reasoning result is obtained based on the result output by the second target information flow of the last target diffusion network in the N-layer target diffusion network, and represents the predicted prediction result of the target generation task in the hidden space; In the case where the initial inference result includes image features, the image features in the initial inference result are decoded using an image decoder in the target model to obtain the target inference result.

18. The method according to claim 17, wherein: The dual information flow target diffusion network includes: a first target conditional information flow module and a second target conditional information flow module corresponding to the first target information flow, and a first target latent space flow module and a second target latent space flow module corresponding to the second target information flow; Wherein, the input of the first target information flow is the input of the first target condition information flow module, and the output of the second target condition information flow module is the output of the first target information flow; The input of the second target information stream is the input of the first target latent spatial stream module, and the output of the second target latent spatial stream module is the output of the second target information stream.

19. The method according to claim 18, wherein: The using the N-layer target diffusion network to perform feature diffusion includes: The following steps are adopted and feature diffusion is performed using the i-th layer target diffusion network in the N-layer target diffusion network: The output of the first target information flow in the i-1th layer target diffusion network is input into the first target conditional information flow module in the i-th layer target diffusion network for processing, and the output of the second target information flow in the i-1th layer target diffusion network is input into the first target latent space flow module in the i-th layer target diffusion network for processing; Performing element-wise operations on the features processed by the first target conditional information flow module in the i-th target diffusion network and the features processed by the first target latent spatial flow module in the i-th target diffusion network to concatenate the features processed by the first target conditional information flow module and the features processed by the first target latent spatial flow module; Perform self-attention processing on the concatenated features to obtain the target fusion features; The features in the target fusion features corresponding to the first target information flow are input into the second target conditional information flow module in the i-th layer target diffusion network to obtain the output of the second target conditional information flow module; and the features in the target fusion features corresponding to the second target information flow are input into the second target latent layer spatial flow module in the i-th layer target diffusion network to obtain the output of the second target latent layer spatial flow module.

20. The method according to any one of claims 13 to 19, wherein: The obtaining of the target multimodal features of the multimodal prompt information includes: By using the image-text encoder in the target model and based on the data features of the multimodal prompt information, one or more of the following feature expressions are selected to obtain the image features of the target entity image contained in the multimodal prompt information, so as to obtain the target multimodal features: The position features of the target entity image, the segmentation features of the segmentation map corresponding to the target entity image, the image features of the cropped map of the target entity image, the image depth features of the target entity image, and the local features of the target entity image.

21. A target model training device, comprising: A training unit, used for inputting sample data into a preset model to obtain initial multimodal features of the sample data; wherein the sample data at least includes sample text and at least one entity image, the sample text includes at least one entity text, and the entity image is an image corresponding to the entity text; the initial multimodal features include image features of the entity image and text features of the entity text, and the image features of the entity image are after the text features of the entity text corresponding to the entity image; when the parameters of the image-text encoder of the preset model are fixed, the N-layer diffusion network in the preset model is trained using the initial multimodal features and the preset noise features to obtain a target model; wherein N is a positive integer; A storage unit is used for the target model.

22. The device according to claim 21, wherein The training unit is further used for: Insert instruction information into the initial multimodal features, wherein the instruction information includes image features of the entity image to be adjusted and adjustment instructions.

23. The device according to claim 22, wherein: The training unit is specifically used for: The instruction information is concatenated at the head of the feature sequence representing the initial multimodal features.

24. The device according to any one of claims 21 to 23, wherein: The N-layer diffusion networks are all dual information flow diffusion networks, which include: a first information flow for processing initial multimodal features, and a second information flow for compressing the initial multimodal features based on preset noise features to represent the initial multimodal features in the hidden layer space.

25. The device according to claim 24, wherein When the value of N is greater than or equal to 2, the N diffusion networks are connected in series; wherein the output corresponding to the first information flow in the i-th diffusion network among the N diffusion networks is the input of the first information flow in the i+1-th diffusion network among the N diffusion networks; and the output of the second information flow in the i-th diffusion network among the N diffusion networks is the input of the second information flow in the i+1-th diffusion network among the N diffusion networks.

26. The device according to claim 24 or 25, wherein: The training unit is further used for: A masking operation is performed on some features of the initial multimodal features to obtain masked initial multimodal features.

27. The device according to claim 26, wherein The training unit is specifically used for: The masked initial multimodal features are used as the input of the first information flow of the first diffusion network in the N-layer diffusion network, and the preset noise features are used as the input of the second information flow of the first diffusion network in the N-layer diffusion network, so as to use the N-layer diffusion network to perform feature diffusion and obtain a total prediction result; wherein the total prediction result includes a first prediction result and a second prediction result, the first prediction result is the result output by the first information flow of the last diffusion network in the N-layer diffusion network, at least indicating the predicted mask position corresponding to the predicted mask operation; the second prediction result is the result output by the second information flow of the last diffusion network in the N-layer diffusion network, indicating the prediction result of the generation task indicated by the predicted sample data in the hidden space; Based on the first prediction result and the second prediction result, a loss value of a target loss function is obtained; wherein the target loss function is used to calculate the difference between the predicted mask position and the actual mask position, and to calculate the difference between the predicted result of the generation task indicated by the sample data in the hidden space and the actual result in the hidden space; At least some of the adjustable network parameters in the N-layer diffusion network are adjusted based on the loss value of the target loss function.

28. The device according to claim 27, wherein The dual information flow diffusion network includes: a first conditional information flow module and a second conditional information flow module corresponding to the first information flow, and a first latent space flow module and a second latent space flow module corresponding to the second information flow; Wherein, the input of the first information flow is the input of the first conditional information flow module, and the output of the second conditional information flow module is the output of the first information flow; The input of the second information stream is the input of the first latent spatial stream module, and the output of the second latent spatial stream module is the output of the second information stream.

29. The device according to claim 28, wherein The training unit is specifically used for: The following steps are adopted and feature diffusion is performed using the i-th diffusion network in the N-layer diffusion network: The output of the first information flow in the i-1th diffusion network is input into the first conditional information flow module in the i-th diffusion network for processing, and the output of the second information flow in the i-1th diffusion network is input into the first latent spatial flow module in the i-th diffusion network for processing; Performing element-wise operations on the features processed by the first conditional information flow module in the i-th diffusion network and the features processed by the first latent spatial flow module in the i-th diffusion network to concatenate the features processed by the first conditional information flow module and the features processed by the first latent spatial flow module; Perform self-attention processing on the concatenated features to obtain fused features; Inputting the features in the fused features corresponding to the first information flow into the second conditional information flow module in the i-th layer diffusion network to obtain the output of the second conditional information flow module; And, the features corresponding to the second information flow in the fused features are input into the second latent spatial flow module in the i-th diffusion network to obtain the output of the second latent spatial flow module.

30. The device according to any one of claims 21 to 29, wherein: The feature expression method of the image feature of the entity image includes at least one of the following: the position feature of the entity image, the segmentation feature of the segmentation map corresponding to the entity image, the image feature of the cropped map of the entity image, the image depth feature of the entity image, and the local feature of the entity image.

31. The device according to claim 30, wherein The training unit is specifically used for: Using the image-text encoder in the preset model and based on the sample features of the sample data, one or more of the above feature expressions are selected to obtain the image features of the entity image contained in the sample data, so as to obtain the initial multimodal features of the sample data; Alternatively, one or more of the above feature expressions are randomly selected to obtain image features of the entity images contained in the sample data, so as to obtain initial multimodal features of the sample data.

32. A multimodal data processing device, comprising: An inference unit is used to input the multimodal prompt information into a pre-trained target model to obtain a target multimodal feature of the multimodal prompt information, wherein the multimodal prompt information at least includes a target text and at least one target entity image, the target text includes at least one target entity text, and the target entity image is an image corresponding to the target entity text; the target multimodal feature includes an image feature of the target entity image and a text feature of the target entity text, and the image feature of the target entity image is after the text feature of the target entity text corresponding to the target entity image; the target multimodal feature is diffused using an N-layer target diffusion network in the target model to obtain a target inference result associated with the multimodal prompt information and including an image and / or text; wherein N is a positive integer; An output unit is used to output the target reasoning result.

33. The device according to claim 32, wherein: The reasoning unit is further used for: Target instruction information is inserted into the target multimodal feature, wherein the target instruction information includes image features of the target entity image to be adjusted and a target adjustment instruction.

34. The device according to claim 33, wherein: The reasoning unit is specifically used for: The target instruction information is spliced ​​at the head of the target feature sequence representing the target multimodal features.

35. The device according to any one of claims 32 to 34, wherein: The N-layer target diffusion network is a dual information flow target diffusion network, which includes: a first target information flow for processing target multimodal features, and a second target information flow for compressing the target multimodal features based on preset target noise features to represent the target multimodal features in a hidden layer space.

36. The device according to claim 35, wherein When the value of N is greater than or equal to 2, the N target diffusion networks are connected in series; wherein, the output corresponding to the first target information flow in the i-th target diffusion network among the N target diffusion networks is the input of the first target information flow in the i+1-th target diffusion network among the N target diffusion networks; and the output of the second target information flow in the i-th target diffusion network among the N target diffusion networks is the input of the second target information flow in the i+1-th target diffusion network among the N target diffusion networks.

37. The device according to claim 35, wherein: The reasoning unit is specifically used for: The target multimodal feature is used as the input of the first target information flow of the first target diffusion network in the N-layer target diffusion network, and the preset target noise feature is used as the input of the second target information flow of the first target diffusion network in the N-layer target diffusion network, so as to use the N-layer target diffusion network to perform feature diffusion and obtain an initial reasoning result, wherein the initial reasoning result is obtained based on the result output by the second target information flow of the last target diffusion network in the N-layer target diffusion network, and represents the predicted prediction result of the target generation task in the hidden space; In the case where the initial inference result includes image features, the image features in the initial inference result are decoded using an image decoder in the target model to obtain the target inference result.

38. The device according to claim 37, wherein The dual information flow target diffusion network includes: a first target conditional information flow module and a second target conditional information flow module corresponding to the first target information flow, and a first target latent space flow module and a second target latent space flow module corresponding to the second target information flow; Wherein, the input of the first target information flow is the input of the first target condition information flow module, and the output of the second target condition information flow module is the output of the first target information flow; The input of the second target information stream is the input of the first target latent spatial stream module, and the output of the second target latent spatial stream module is the output of the second target information stream.

39. The device according to claim 38, wherein The reasoning unit is specifically used for: The following steps are adopted and feature diffusion is performed using the i-th layer target diffusion network in the N-layer target diffusion network: The output of the first target information flow in the i-1th layer target diffusion network is input into the first target conditional information flow module in the i-th layer target diffusion network for processing, and the output of the second target information flow in the i-1th layer target diffusion network is input into the first target latent space flow module in the i-th layer target diffusion network for processing; Performing element-wise operations on the features processed by the first target conditional information flow module in the i-th target diffusion network and the features processed by the first target latent spatial flow module in the i-th target diffusion network to concatenate the features processed by the first target conditional information flow module and the features processed by the first target latent spatial flow module; Perform self-attention processing on the concatenated features to obtain the target fusion features; Input the features in the target fusion features corresponding to the first target information flow into the second target condition information flow module in the i-th layer target diffusion network to obtain the output of the second target condition information flow module; And, the features corresponding to the second target information flow in the target fusion features are input into the second target latent space flow module in the i-th layer target diffusion network to obtain the output of the second target latent space flow module.

40. The device according to any one of claims 33 to 39, wherein: The reasoning unit is specifically used for: By using the image-text encoder in the target model and based on the data features of the multimodal prompt information, one or more of the following feature expressions are selected to obtain the image features of the target entity image contained in the multimodal prompt information, so as to obtain the target multimodal features: The position features of the target entity image, the segmentation features of the segmentation map corresponding to the target entity image, the image features of the cropped map of the target entity image, the image depth features of the target entity image, and the local features of the target entity image.

41. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 20.

42. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-20.

43. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 20.

Images