Multi-modal data generation method and multi-modal model training method

Abstract

The disclosure provides a multi-modal data generation method and a multi-modal model training method, relates to the technical field of artificial intelligence, and in particular to the technical fields of deep learning, AIGC, large language models, large multi-modal models, and the like. The multi-modal data generation method comprises: inputting a query data sequence into a multi-modal model to obtain a plurality of tokens in a response data sequence output by the model in turn, wherein a current token is generated by: in response to determining that the current token belongs to a first data modality, inputting the query data sequence and the current response data sequence into the multi-modal model, so that the multi-modal model generates the current token based on the query data sequence and the current response data sequence; and in response to determining that the current token belongs to a second data modality, inputting the query data sequence and the current response data sequence into the multi-modal model, so that the multi-modal model denoises an initial token sequence based on the query data sequence and the current response data sequence to generate a result token sequence.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of deep learning, AIGC (artificial intelligence generated content), large language models, and large multimodal models. Specifically, it relates to a multimodal data generation method and apparatus, a multimodal model training method and apparatus, electronic equipment, computer-readable storage media, and computer program products. Background Technology

[0002] Large Language Model (LLM) is a deep learning model trained on a large amount of text data, which can achieve the understanding and generation of natural language text.

[0003] Large Multimodal Models (LMMs) are an extension of large language models that can process data from multiple modalities, such as text and images, simultaneously, enabling cross-modal data recognition and understanding.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a method and apparatus for generating multimodal data, a method and apparatus for training multimodal models, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of this disclosure, a method for generating multimodal data is provided, comprising: acquiring a query data sequence, wherein the query data sequence includes at least one data segment, each data segment corresponding to a data modality; and inputting the query data sequence into a multimodal model to obtain a plurality of tags in a response data sequence sequentially output by the multimodal model, wherein a current tag among the plurality of tags is generated by the following operation: in response to determining that the current tag belongs to a first data modality, inputting the query data sequence and the current response data sequence into the multimodal model, so that the multimodal model can generate multimodal data based on the query data sequence. The current tag is generated from the sequence and the current response data sequence, wherein the unit data value of the first data modality is discrete; or in response to determining that the current tag belongs to the second data modality, the query data sequence and the current response data sequence are input into the multimodal model so that the multimodal model denoises the initial tag sequence based on the query data sequence and the current response data sequence to generate a result tag sequence, wherein the unit data value of the second data modality is continuous, the initial tag sequence includes a preset number of initial tags, and the result tag sequence includes the preset number of tags starting from the current tag.

[0007] According to one aspect of this disclosure, a method for training a multimodal model is provided, comprising: acquiring a sample data sequence, wherein the sample data sequence includes at least one data segment, each data segment corresponding to a data modality; inputting the sample data sequence into the multimodal model to obtain multiple labels in a prediction data sequence sequentially output by the multimodal model, wherein a current label among the multiple labels is generated by the following operation: in response to determining that the current label belongs to a first data modality, inputting the sample data sequence and the current prediction data sequence into the multimodal model, so that the multimodal model generates the current label based on the sample data sequence and the current prediction data sequence. In the first data modality, the value of the unit data is discrete; or in response to determining that the current label belongs to the second data modality, the sample data sequence and the current predicted data sequence are input into the multimodal model so that the multimodal model denoises the initial label sequence based on the sample data sequence and the current predicted data sequence to generate a result label sequence, wherein the value of the unit data of the second data modality is continuous, the initial label sequence includes a preset number of initial labels, and the result label sequence includes the preset number of labels starting from the current label; and the parameters of the multimodal model are adjusted based on the difference between the predicted data sequence and the target data sequence corresponding to the sample data sequence.

[0008] According to one aspect of this disclosure, a multimodal data generation apparatus is provided, comprising: an acquisition module configured to acquire an inquiry data sequence, wherein the inquiry data sequence includes at least one data segment, each data segment corresponding to a data modality; and an output module configured to input the inquiry data sequence into a multimodal model to obtain a plurality of tags in a response data sequence sequentially output by the multimodal model, wherein a current tag among the plurality of tags is generated by the following operation: in response to determining that the current tag belongs to a first data modality, the inquiry data sequence and the current response data sequence are input into the multimodal model so that the multimodal model... The model generates the current tag based on the query data sequence and the current response data sequence, wherein the unit data value of the first data modality is discrete; or in response to determining that the current tag belongs to the second data modality, the query data sequence and the current response data sequence are input into the multimodal model so that the multimodal model denoises the initial tag sequence based on the query data sequence and the current response data sequence to generate a result tag sequence, wherein the unit data value of the second data modality is continuous, the initial tag sequence includes a preset number of initial tags, and the result tag sequence includes the preset number of tags starting from the current tag.

[0009] According to one aspect of this disclosure, a training apparatus for a multimodal model is provided, comprising: an acquisition module configured to acquire a sample data sequence, wherein the sample data sequence includes at least one data segment, each data segment corresponding to a data modality; and an output module configured to input the sample data sequence into the multimodal model to obtain a plurality of labels in a prediction data sequence sequentially output by the multimodal model, wherein a current label among the plurality of labels is generated by the following operation: in response to determining that the current label belongs to a first data modality, the sample data sequence and the current prediction data sequence are input into the multimodal model, so that the multimodal model generates the current label based on the sample data sequence and the current prediction data sequence. The multimodal model includes a pre-labeling module, wherein the unit data values ​​of the first data modality are discrete; or, in response to determining that the current label belongs to the second data modality, the sample data sequence and the current predicted data sequence are input into the multimodal model so that the multimodal model denoises the initial label sequence based on the sample data sequence and the current predicted data sequence to generate a result label sequence, wherein the unit data values ​​of the second data modality are continuous, the initial label sequence includes a preset number of initial labels, and the result label sequence includes the preset number of labels starting from the current label; and an adjustment module configured to adjust the parameters of the multimodal model based on the difference between the predicted data sequence and the target data sequence corresponding to the sample data sequence.

[0010] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods of any of the above aspects.

[0011] According to one aspect of this disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method of any of the above aspects.

[0012] According to one or more embodiments of this disclosure, the multimodal model has general multimodal data understanding and generation capabilities, improving the content generation effect and capability range of the multimodal model, thereby enabling the multimodal model to handle diverse user tasks more flexibly and efficiently.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram of an exemplary system in which various methods described herein may be implemented according to embodiments of this disclosure is shown;

[0016] Figure 2 A flowchart of a multimodal data generation method according to an embodiment of the present disclosure is shown;

[0017] Figure 3 A schematic diagram of a multimodal data generation process according to an embodiment of the present disclosure is shown;

[0018] Figure 4 A flowchart illustrating a training method for a multimodal model according to an embodiment of the present disclosure is shown;

[0019] Figure 5 A schematic diagram of a noisy sample data sequence according to an embodiment of the present disclosure is shown;

[0020] Figure 6 A schematic diagram of a predicted data sequence output by a multimodal model according to an embodiment of the present disclosure is shown;

[0021] Figure 7 A schematic diagram illustrating the training process of a multimodal model according to an embodiment of the present disclosure is shown;

[0022] Figure 8 A schematic diagram of the mask matrix for the attention mechanism in a multimodal model according to an embodiment of the present disclosure is shown;

[0023] Figure 9 A schematic diagram is shown illustrating how a multimodal model, according to an embodiment of the present disclosure, generates a result label sequence through multiple inverse diffusion operations;

[0024] Figure 10 A schematic diagram of sample data sequences of different modalities according to some embodiments of the present disclosure is shown;

[0025] Figure 11 A schematic diagram of sample data sequences for different modalities and tasks according to embodiments of the present disclosure is shown;

[0026] Figure 12 A structural block diagram of a multimodal data generation apparatus according to an embodiment of the present disclosure is shown;

[0027] Figure 13A structural block diagram of a training apparatus for a multimodal model according to an embodiment of the present disclosure is shown; and

[0028] Figure 14 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0031] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof. "More than" means two or more.

[0032] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] Current large multimodal models (such as GPT-4V, Flamingo, MiniGPT-4, Gemini, etc.) typically possess multimodal data understanding capabilities, for example, they can simultaneously understand text and image content. However, they lack multimodal data generation capabilities; for instance, they can only generate text, not simultaneously generate multimodal content such as text and images. For data generation tasks involving modalities other than text (such as images and audio), separate models are usually required. For example, for image generation tasks, models such as diffusion models, generative adversarial networks (GANs), and variational autoencoders (VAEs) can be used; for speech synthesis tasks, dedicated speech synthesis models can be employed.

[0034] As can be seen from the above, the multimodal large models in related technologies only have the ability to generate data in a single modality (i.e., text modality), and do not have the ability to generate multimodal data. This results in a limited number of tasks that multimodal large models can handle, and they cannot efficiently meet the diverse task processing needs of users.

[0035] To address the aforementioned issues, this disclosure provides a method for generating multimodal data based on a multimodal model, as well as a method for training the multimodal model.

[0036] The embodiments of this disclosure utilize a unified multimodal model to achieve multimodal data generation. For discrete modalities such as natural language text, code, and protein sequences, the multimodal model uses an autoregressive approach to sequentially generate each tag for that modality. For continuous modalities such as images and audio, the multimodal model treats multiple tags for that modality as a whole and generates them through diffusion.

[0037] The multimodal model of this disclosure integrates the autoregressive generation process of discrete data and the diffusion generation process of continuous data, giving it a general multimodal data understanding and generation capability, improving the content generation effect and capability range, and thus enabling more flexible and efficient handling of diverse user tasks.

[0038] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0039] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0040] In embodiments of this disclosure, client devices 101, 102, 103, 104, 105, and 106, and server 120, may run one or more services or software applications that enable the execution of multimodal data generation methods or multimodal model training methods.

[0041] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105, and / or 106 under a Software as a Service (SaaS) model.

[0042] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0043] Client devices 101, 102, 103, 104, 105, and / or 106 can provide interfaces that enable users of the client devices to interact with them. The client devices can also output information to the user through these interfaces. Although Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0044] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, in-vehicle devices, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux, or Linux-like operating systems; or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0045] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, Wi-Fi), and / or any combination of these and / or other networks.

[0046] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0047] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0048] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0049] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0050] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0051] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0052] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0053] According to some embodiments, server 120 can execute the multimodal model training method of the present disclosure embodiments to obtain a trained multimodal model. Further, server 120 can utilize the trained multimodal model to execute the multimodal data generation method of the present disclosure embodiments to provide multimodal data generation services to client devices 101-106. For example, a user can submit their data generation request to server 120 through client devices 101-106, such as "compose a poem on the theme of bamboo and accompany it with a painting." Server 120, by calling the trained multimodal model, generates the poem "The temple is lush with bamboo, the evening bell tolls faintly" and an image depicting a bamboo forest, and returns the poem and image to client devices 101-106.

[0054] According to some embodiments, server 120 can execute the multimodal model training method of the present disclosure embodiments to obtain a trained multimodal model. The trained multimodal model can be deployed to client devices 101-106. Client devices 101-106 can utilize the locally deployed trained multimodal model to execute the multimodal model training method of the present disclosure embodiments to provide multimodal data generation services to users.

[0055] According to some embodiments, client devices 101-106 can also execute the multimodal model training method of the present disclosure embodiments to obtain a trained multimodal model. This typically requires client devices 101-106 to have high hardware configuration and computing power.

[0056] Figure 2 A flowchart of a multimodal data generation method 200 according to an embodiment of the present disclosure is shown. The execution entity of each step of method 200 is typically a server, for example... Figure 1 The server 120 shown. In some embodiments, the entity executing method 200 may also be a client device, such as... Figure 1 The client devices shown are 101-106.

[0057] like Figure 2 As shown, method 200 includes steps S210-S220.

[0058] In step S210, a query data sequence is obtained. The query data sequence includes at least one data segment, and each data segment in the at least one data segment corresponds to a data modality.

[0059] In step S220, the query data sequence is input into the multimodal model to obtain multiple tags in the response data sequence output sequentially by the multimodal model. The current tag among the multiple tags is generated through step S221 or S222.

[0060] In step S221, in response to determining that the current tag belongs to the first data modality, the query data sequence and the current response data sequence are input into the multimodal model so that the multimodal model generates the current tag based on the query data sequence and the current response data sequence. The unit data values ​​of the first data modality are discrete.

[0061] In step S222, in response to determining that the current tag belongs to the second data modality, the query data sequence and the current response data sequence are input into the multimodal model so that the multimodal model can denoise the initial tag sequence based on the query data sequence and the current response data sequence to generate the result tag sequence. Here, the unit data values ​​of the second data modality are continuous, the initial tag sequence includes a preset number of initial tags, and the result tag sequence includes a preset number of tags starting from the current tag.

[0062] According to embodiments of this disclosure, multimodal data generation is achieved using a unified multimodal model. For discrete modalities such as natural language text, code, and protein sequences, the multimodal model sequentially generates each tag for that modality using an autoregressive approach. For continuous modalities such as images and audio, the multimodal model diffuses and generates multiple tags for that modality as a whole.

[0063] The multimodal model of this disclosure integrates the autoregressive generation process of discrete data and the diffusion generation process of continuous data, and has a general multimodal data understanding and generation capability, which improves the content generation effect and capability range, thereby enabling more flexible and efficient handling of diverse user tasks.

[0064] Data modality refers to the format or type of data, such as natural language text, tables, code, SMILES (Simplified Molecular Input Line Entry System) molecular formulas, protein sequences, images, videos, audio, point cloud data acquired by radar, etc.

[0065] In the embodiments of this disclosure, data modes are divided into two categories—a first data mode and a second data mode—based on whether the values ​​of the unit data of a data mode are discrete. It should be noted that the unit data of a data mode refers to the basic constituent unit of the data in that data mode. Examples of unit data for some data modes are shown in Table 1 below.

[0066] Table 1. Unit data for different data modalities

[0067]

[0068]

[0069] In the embodiments of this disclosure, a data mode in which the value of unit data is discrete is denoted as the first data mode, that is, the value of unit data in the first data mode is discrete. For the first data mode, the set of values ​​for unit data can be a non-uniformly distributed finite set, that is, the number of elements in the set of values ​​is finite, and the distribution of elements is non-uniform.

[0070] The first data modality can be, for example, natural language text, tables, code, SMILE molecular formulas, protein sequences, etc.

[0071] According to some embodiments, the first data modality can be divided into different granularities. For example, the aforementioned natural language text, tables, codes, SMILE molecular formulas, protein sequences, and other data modalities can each be considered as a first data modality (fine-grained). Alternatively, the aforementioned natural language text and codes can be uniformly denoted as the "unformatted text modality," tables as the "formatted text modality," and SMILE molecular formulas and protein sequences as the "chemical text modality" (medium-grained). Furthermore, since the aforementioned natural language text, tables, codes, SMILE molecular formulas, and protein sequences can all be represented as text sequences, these data modalities can be uniformly denoted as the "text modality."

[0072] In embodiments of this disclosure, a data mode in which the value of unit data is continuous is denoted as the second data mode, that is, the value of unit data in the second data mode is continuous. The set of values ​​for unit data in the second data mode can be an infinite set; for example, the distance from the sampling point to the radar can be any value. The set of values ​​for unit data in the second data mode can also be a uniformly distributed finite set, that is, the number of elements in the set of values ​​is finite and the elements are uniformly distributed. For example, for an 8-bit pixel, the set of pixel values ​​for a single pixel is a uniformly distributed finite set {0, 1, 2, ..., 255}.

[0073] The second data modality can be, for example, images, videos, audio, point cloud data, etc.

[0074] According to some embodiments, the second data modalities can be divided into different granularities. For example, the aforementioned data modalities such as images, videos, audio, and point cloud data can each be treated as a separate second data modal (fine-grained). Another example is that images and videos can be collectively referred to as the "image modality," while audio and point cloud data can be treated as separate second data modalities. Yet another example is that since the aforementioned images, videos, audio, and point cloud data can all be represented as images (specifically, audio data can be converted into spectrograms, and point cloud data can be converted into depth maps), these data modalities can be collectively referred to as the "image modality."

[0075] In step S210, a query data sequence is obtained. The query data sequence includes at least one data segment, each data segment corresponding to a first data modality or a second data modality.

[0076] According to some embodiments, the query data sequence can be a single-modal data sequence that includes only one data segment, such as a plain text sequence, a single image, an audio segment, etc.

[0077] According to other embodiments, the query data sequence can be a multimodal data sequence comprising multiple data segments. For example, the query data sequence can be a data sequence in the form of "text-image", "audio-text", "text-image-text", etc.

[0078] According to some embodiments, a query data sequence can be obtained by processing an initial query data sequence input by a user. Specifically, the initial query data sequence can be segmented into at least one initial data segment according to a data modality. Each initial data segment corresponds to a data modality. Subsequently, modality label pairs indicating the data modality of each initial data segment are added to each initial data segment to obtain the query data sequence.

[0079] According to some embodiments, modal tag pairs include a modal data start tag and a modal data end tag indicating the same data modality. For example, text modal tag pairs... <text>< / text> Image modality tag pairs Audio modality tag pairs <audio>< / audio> wait.

[0080] According to some embodiments, a query data sequence can be obtained through the following steps: First, an initial query data sequence input by the user is acquired. The initial query data sequence includes at least one initial data segment, each of the at least one initial data segment corresponding to a data modality. Subsequently, for each initial data segment, a modality data start label indicating the data modality of the initial data segment is added before the initial data segment, and a modality data end label indicating the data modality of the initial data segment is added after the initial data segment to obtain the query data sequence.

[0081] For example, the initial query data sequence input by the user might be "Does □ match 'A knight is coming soon'?", where □ represents the image specified by the user. By adding modal label pairs to this initial query data sequence, the query data sequence " □ <text> Does it match "A knight is coming soon"?< / text> ".

[0082] According to the above embodiments, the query data sequence can include modality data start / end labels to indicate the data modality, thereby better guiding the multimodal model to understand and generate multimodal data.

[0083] After obtaining the query data sequence through step S210, in step S220, the query data sequence is input into the trained multimodal model to obtain multiple tokens in the response data sequence output sequentially by the multimodal model.

[0084] According to some embodiments, the multimodal model is a Transformer model that includes only a decoder. In this embodiment, the multimodal model employs a Decoder-Only Transformer Decoder structure, excluding the encoder.

[0085] According to the above embodiments, the query data sequence does not need to be encoded, but is directly input into the multimodal model. This enables the multimodal model to directly process multimodal data, enhancing the fusion efficiency and effect of multimodal knowledge, thereby improving the understanding and generation capabilities of multimodal data.

[0086] According to some embodiments, data segments of the first data modality (e.g., text) can be directly input into the multimodal model. Data segments of the second data modality (e.g., images, audio, etc.) can be segmented into multiple sub-segments; for example, a single image can be segmented into multiple sub-images of the same size, or a single audio segment can be segmented into multiple audio segments of the same length. Then, each sub-segment is input into the multimodal model separately; for example, the pixel values ​​of each sub-image and the amplitude values ​​in the spectrograms corresponding to each audio segment are input into the multimodal model.

[0087] Figure 3 A schematic diagram of a multimodal data generation process implemented using a multimodal model 300 according to an embodiment of the present disclosure is shown. The multimodal model 300 is a unified multilingual multimodal multitask transformer model, which includes only decoders in the transformer and does not include encoders.

[0088] like Figure 3 As shown, the query data sequence " <text> Images of starry skies and trees< / text> "Input multimodal model 300, output response data sequence" <text> Starry sky and trees< / text> (Image label 311)(Image label 312)(Image label 313)(Image label 314) <text> This scene is located in the countryside.< / text>"In this context, image markers 311 to 314 constitute the generated result image 310.

[0089] In embodiments of this disclosure, the multimodal model generates each label in the response data sequence using a semi-autoregressive approach. Specifically, the multimodal model integrates the autoregressive generation process for discrete data and the diffusion generation process for continuous data. For the discrete modality, i.e., the first data modality, each label of this modality is generated sequentially using a fully autoregressive approach (step S221). For the continuous modality, i.e., the second data modality, the multiple labels of this modality are generated as a whole through diffusion (step S222).

[0090] According to some embodiments, the multiple tags output by the multimodal model include data content tags and modality tags. Figure 3 Taking the response data sequence output by the multimodal model 300 as an example, in this response data sequence, "starry sky", "and", "tree", "311", "312", "313", "314", "this scene", "located in", and "countryside" are data content markers, and "", "", "", " " and " are modal tags.

[0091] In embodiments of this disclosure, a multimodal model sequentially generates and outputs multiple markers in a response data sequence. The multimodal marker is generated based on the data modality of the current marker.

[0092] According to some embodiments, the multiple tags output by the multimodal model include modal label pairs. A modal label pair includes a modal data start label and a modal data end label indicating the same data modality. For example, Figure 3 In the text modal, and are the start label and end label of the modal data, respectively. and are the start label and end label of the modal data for the image modality, respectively.

[0093] According to some embodiments, corresponding to the ability of a multimodal model to output modal label pairs, the data modality to which the current label belongs can be determined based on the last modal data start label in the current response data sequence.

[0094] According to some embodiments, both the modal data start label and the modal data end label are themselves markers of the first data modality, for example, both being markers of the text modality. In response to determining that the current response data sequence is empty, it is determined that the current marker belongs to the first data modality. Thus, the multimodal model can initiate the multimodal data generation process in an autoregressive manner and output the modal data start label as the first marker in the response data sequence.

[0095] According to some embodiments, when the current response data sequence is not empty, in response to determining that the start label of the last modality data in the current response data sequence indicates a first data modality, it is determined that the current label belongs to the first data modality. Then, step S221 is executed, in which the query data sequence and the current response data sequence are input into the multimodal model so that the multimodal model generates the current label in an autoregressive manner.

[0096] For example, in Figure 3 In the embodiment shown, the current response data sequence is " <text>"Starry Sky and", the last modal data in it starts with the label "Starry Sky and". <text>Therefore, it is determined that the current tag belongs to the text modality, and the current tag "tree" is generated through autoregression.

[0097] Alternatively, in response to determining that the last modal data start tag in the current response data sequence indicates a second data modality, it is determined that the current tag belongs to the second data modality. Then, step S222 is executed, in which the query data sequence, the current response data sequence, and the initial tag sequence are input into the multimodal model, so that the multimodal model generates a preset number of tags starting from the current tag through diffusion generation, with the initial tag sequence as the starting point.

[0098] For example, in Figure 3 In the illustrated embodiment, the current response data sequence is " <text> Starry sky and trees< / text> The last modal data in the dataset starts with a label of ". Therefore, the current marker (i.e., "311") is determined to belong to the image modality. The multimodal model starts with an initial marker sequence including four random noise markers, and generates four markers starting with the current marker "311" through diffusion generation, i.e., the resulting marker sequence [311, 312, 313, 314]. This resulting marker sequence represents the generated result image.

[0099] According to some embodiments, after generating the result label sequence, the multimodal model sequentially generates a modal data end label (e.g.) indicating the second data modality and a modal data start label (e.g.) indicating the first data modality. <text>This allows for the recovery of the autoregressive pattern after generating the resulting label sequence of the second data modality, enabling the generation of subsequent labels one by one.

[0100] According to some embodiments, in step S222, the multimodal model can perform a preset number of back-diffusion operations (i.e., denoising operations) on the initial labeled sequence based on the query data sequence and the current response data sequence to generate a result labeled sequence. Each back-diffusion operation includes:

[0101] Input the query data sequence, the current response data sequence, the current tag sequence, and the identifier of the current backdiffusion round into the multimodal model so that the multimodal model can generate a denoised tag sequence.

[0102] In this process, the current label sequence of the first inverse diffusion operation is the initial label sequence, the current label sequence of the second and subsequent inverse diffusion operations is the denoised label sequence generated by the previous inverse diffusion operation, and the denoised label sequence generated by the last inverse diffusion operation is the result label sequence.

[0103] According to the above embodiments, the initial label sequence is denoised multiple times using diffusion generation logic, and finally a denoised and clear result label sequence is generated, which can ensure the data generation effect of the second data modality (e.g., image, audio).

[0104] According to some embodiments, the initial label sequence can be a random noise sequence. For example, multiple random noise labels can be generated using a Gaussian distribution, and then these multiple random noise labels can be combined to obtain the initial label sequence.

[0105] In embodiments of this disclosure, the initial tag sequence includes a preset number of initial tags. According to some embodiments, this preset number may be a preset fixed value, such as 4, 6, etc.

[0106] According to other embodiments, the preset quantity can also be a label of the first data modality output by the multimodal model, such as a parameter label of the text modality.<token_num=N> The value N of the modality parameter label is the preset quantity. According to some embodiments, this preset quantity can be calculated based on the label of the first data modality output by the multimodal model. For example, the parameter label of the text modality output by the multimodal model.<img_size=W*H> (refer to Figure 3 The parameter label indicates the dimensions of both the initial image and the resulting image. Based on the parameter label...<img_size=W*H> The amount of pixel data in the initial or result image can be determined. By dividing this amount of data by the amount of data of a single preset image marker, the number of image markers can be obtained, which is the preset number mentioned above.

[0107] According to some embodiments, the preset number of the above-mentioned reverse diffusion operation can be a preset fixed value, such as 50, 100, etc.

[0108] According to other embodiments, the preset number of times can also be a label of the first data modality output by the multimodal model, such as a parameter label of the text modality.<diffusion=T> The value T of this parameter label is the preset number of back diffusion cycles.

[0109] According to the above embodiments, the preset number of initial tags and the preset number of reverse diffusion operations included in the initial tag sequence can both be tags output by the multimodal model, thereby improving the flexibility of data generation.

[0110] According to embodiments of this disclosure, a method for training a multimodal model is also provided. By executing this method, a trained multimodal model can be obtained. The trained multimodal model can be used to implement the multimodal data generation method 200 described above.

[0111] Figure 4 A flowchart of a multimodal model training method 400 according to an embodiment of the present disclosure is shown. The execution entity of method 400 is typically a server. In some embodiments, the execution entity of method 400 may also be a client device, which typically requires the client device to have high hardware configuration and computing power. Figure 4 As shown, method 400 includes steps S410-S430.

[0112] In step S410, a sample data sequence is obtained. The sample data sequence includes at least one data segment, and each data segment in the at least one data segment corresponds to a data mode.

[0113] In step S420, the sample data sequence is input into the multimodal model to obtain multiple labels in the prediction data sequence output sequentially by the multimodal model. The current label among the multiple labels is generated through the following steps S421 or S422.

[0114] In step S421, in response to determining that the current label belongs to the first data mode, the sample data sequence and the current predicted data sequence are input into the multimodal model so that the multimodal model generates the current label based on the sample data sequence and the current predicted data sequence. The unit data values ​​of the first data mode are discrete.

[0115] In step S242, in response to determining that the current label belongs to the second data modality, the sample data sequence and the current predicted data sequence are input into the multimodal model so that the multimodal model can denoise the initial label sequence based on the sample data sequence and the current predicted data sequence to generate the result label sequence. Here, the unit data values ​​of the second data modality are continuous, the initial label sequence includes a preset number of initial labels, and the result label sequence includes a preset number of labels starting from the current label.

[0116] In step S430, the parameters of the multimodal model are adjusted based on the difference between the predicted data sequence and the target data sequence corresponding to the sample data sequence.

[0117] According to embodiments of this disclosure, multimodal data generation is achieved using a unified multimodal model. For discrete modalities such as natural language text, code, and protein sequences, the multimodal model sequentially generates each tag for that modality using an autoregressive approach. For continuous modalities such as images and audio, the multimodal model diffuses and generates multiple tags for that modality as a whole.

[0118] The multimodal model of this disclosure integrates the autoregressive generation process of discrete data and the diffusion generation process of continuous data, and has a general multimodal data understanding and generation capability, which improves the content generation effect and capability range, thereby enabling more flexible and efficient handling of diverse user tasks.

[0119] In embodiments of this disclosure, the sample data sequence includes modal tag pairs. Each modal tag pair includes a modal data start tag and a modal data end tag indicating the same data modality. For example, a text modal tag pair. <text>< / text> Image modality tag pairs Audio modality tag pairs <audio>< / audio> The data modality corresponding to each data segment in the sample data sequence is identified by a modality start label set at the beginning of the data segment and a modality end label set at the end of the data segment.

[0120] In embodiments of this disclosure, the sample data sequence may contain noise. The target data sequence corresponding to the sample data sequence is the noise-free, clear data sequence corresponding to the noisy sample data sequence; that is, the target data sequence is the target for generating the sample data sequence.

[0121] According to some embodiments, the sample data sequence can be generated through the following steps S401-S403.

[0122] In step S401, the first data sequence is obtained.

[0123] In step S402, in response to the first data sequence comprising only data segments of the first data modality, for example, if the first data sequence is plain text, then the first data sequence is directly used as a single-modality sample data sequence. Since no noise is added to the first data sequence, the target data sequence corresponding to the sample data sequence is the sample data sequence itself.

[0124] In step S403, in response to the first data sequence including at least a first data segment of a second data modality—for example, the first data sequence being image unimodal data or multimodal data composed of text and images—multiple noise addition operations (i.e., forward diffusion) are performed on the first data segment to obtain multiple second data segments corresponding to the aforementioned multiple noise addition operations, thereby obtaining multiple second data sequences corresponding to the aforementioned multiple noise addition operations. The aforementioned noise addition operation may, for example, be adding random noise conforming to a Gaussian distribution. It is understood that each second data sequence includes a corresponding second data segment. Each of the aforementioned multiple second data sequences can serve as a sample data sequence for training a multimodal model. In practice, one or more second data sequences can be arbitrarily sampled from the aforementioned multiple second data sequences as sample data sequences. The target data sequence corresponding to this sample data sequence is the aforementioned first data sequence.

[0125] According to the above embodiments, for discrete single-modal data such as text, the multimodal model processes it through a fully autoregressive approach, without the need for noise addition. The generation target of the multimodal model is the same as the input. For single-modal / multimodal data containing continuous modalities, the multimodal model processes it through diffusion generation, which requires adding noise to the original data (i.e., the first data sequence). The input of the multimodal model is the data with added noise (i.e., the second data sequence), and the generation target of the model is the clear original data.

[0126] Figure 5 A schematic diagram of a noisy sample data sequence according to an embodiment of the present disclosure is shown. Figure 5 As shown, the first data sequence 510 includes image 511, that is, a first data segment including the second data modality. The number of noise addition operations is determined by... <diffusion>Labeling. It should be noted that image 511 is a clear image without noise.

[0127] like Figure 5 As shown, by adding noise to image 511, a noisy image 521 is obtained, thereby obtaining the second data sequence 520.

[0128] By adding noise to the noisy image 521, a noisy image 531 is obtained, thereby obtaining the second data sequence 530.

[0129] By adding noise to the noisy image 531, a noisy image 541 is obtained, thereby obtaining the second data sequence 540.

[0130] The second data sequence 520-540 can all be used as sample data sequences for training the multimodal model. The target data sequence corresponding to the sample data sequence is the first data sequence 510.

[0131] By inputting sample data sequences 520, 530, or 540 into a multimodal model, the model generates and outputs predicted data sequences using a semi-autoregressive approach. We expect these predicted data sequences to be as similar as possible to the target data sequence 510.

[0132] Figure 6 The diagram shows the predicted data sequences 620, 630, and 640 output by the multimodal model 600 after the sample data sequences 520, 530, and 540 are input into the multimodal model 600, respectively.

[0133] In embodiments of this disclosure, the multimodal model generates each label in the predicted data sequence using a semi-autoregressive approach. Specifically, the multimodal model integrates the autoregressive generation process for discrete data and the diffusion generation process for continuous data. For the discrete mode, i.e., the first data mode, each label of this mode is generated sequentially using a fully autoregressive approach (step S421). For the continuous mode, i.e., the second data mode, the multiple labels of this mode are generated as a whole through diffusion (step S422).

[0134] According to some embodiments, the multimodal model is a Transformer model that includes only a decoder. In this embodiment, the multimodal model employs a Decoder-Only Transformer Decoder structure, excluding the encoder.

[0135] According to the above embodiments, the query data sequence does not need to be encoded, but is directly input into the multimodal model. This enables the multimodal model to directly process multimodal data, enhancing the fusion efficiency and effect of multimodal knowledge, thereby improving the understanding and generation capabilities of multimodal data.

[0136] According to some embodiments, the decoder of the multimodal model includes an attention layer, and the aforementioned semi-autoregressive data generation method of the multimodal model can be implemented by controlling the attention mechanism employed by the attention layer. Specifically, a mask can be used to control the information of other tags that need to be focused on when generating each tag in the predicted data sequence. When generating the current tag for a first data modality (e.g., a text modality), all currently generated tags (i.e., all tags in the current response sequence) are visible to the current tag, while other ungenerated tags are not visible to the current tag. This allows the model to focus only on historical information when generating the current tag, avoiding attention to future information. When generating the current tag for a second data modality (e.g., an image modality), multiple tags belonging to this modality, starting from the current tag, are treated as a single tag at the control layer for attention calculation.

[0137] Figure 7 A schematic diagram of the training process of a multimodal model 700 according to an embodiment of the present disclosure is shown. The multimodal model 700 is a unified multilingual multimodal multitask transformer model, which includes only the decoder in the transformer and does not include the encoder.

[0138] like Figure 7 As shown, after inputting the sample data sequence 710 into the multimodal model 700, the predicted data sequence 720 output by the multimodal model 700 is obtained. For the text modality labels t1–t7 and t12–t17, a fully autoregressive method is used for generation. For the image modality labels t8–t11, a diffusion generation process is used. The autoregressive generation of the text modality and the diffusion generation of the image modality form the semi-autoregressive generation mechanism of the multimodal model 700. This semi-autoregressive generation mechanism is implemented by controlling the attention mechanism in the decoder of the multimodal model 700.

[0139] Figure 8 A schematic diagram of the mask matrix 800 used in the decoder's attention mechanism is shown. Figure 8 As shown, the text modality tags t1-t7 and t12-t17 are generated sequentially, with each tag based on all currently generated tags. For example, tag t6 is generated based on the already generated tags t1-t5, and tag t16 is generated based on the already generated tags t1-t15. The image modality tags t8-t11 are generated simultaneously as a single, unified tag at the control level, as shown... Figure 8 As shown in the black rectangle in the image.

[0140] According to some embodiments, in step S422, the multimodal model can perform a preset number of inverse diffusion operations (i.e., denoising operations) on the initial labeled sequence based on the sample data sequence and the current predicted data sequence to generate the resulting labeled sequence. Each inverse diffusion operation includes:

[0141] Input the sample data sequence, the current predicted data sequence, the current labeled sequence, and the identifier of the current backdiffusion round into the multimodal model so that the multimodal model can generate a denoised labeled sequence.

[0142] In this process, the current label sequence of the first inverse diffusion operation is the initial label sequence, the current label sequence of the second and subsequent inverse diffusion operations is the denoised label sequence generated by the previous inverse diffusion operation, and the denoised label sequence generated by the last inverse diffusion operation is the result label sequence.

[0143] According to the above embodiments, the initial label sequence is denoised multiple times using diffusion generation logic, and finally a denoised and clear result label sequence is generated, which can ensure the data generation effect of the second data modality (e.g., image, audio).

[0144] Figure 9 A schematic diagram is shown illustrating how a multimodal model, according to an embodiment of the present disclosure, generates a resulting label sequence through multiple inverse diffusion operations. Figure 9 In the embodiment shown, the multimodal model 900 generates the resulting label sequence 940 by performing three reverse diffusion operations on the initial label sequence 910.

[0145] Specifically, in Step 1, the multimodal model 900 performs the first backdiffusion operation on the initial labeled sequence 910. At this time, the backdiffusion round is identified as 3, i.e.<diffusion=3> The denoised label sequence 920 is generated.

[0146] In Step 2, the multimodal model 900 performs a further back-diffusion operation on the denoised labeled sequence 920. The back-diffusion round is marked as 2 at this point.<diffusion=2> The denoised label sequence 930 is generated after two rounds of noise reduction.

[0147] In Step 3, the multimodal model 900 performs a further back-diffusion operation on the labeled sequence 930 after two denoising operations. At this point, the back-diffusion round is marked as 1.<diffusion=1> The resulting label sequence 940 is generated after three rounds of denoising.

[0148] After obtaining the result labeled sequence 940, the identifier of the reverse diffusion round is set to 0, i.e.<diffusion=0> .

[0149] It is understood that, in the embodiments of this disclosure, parameter labels<diffusion=t> This indicates the number of backdiffusion operations to be performed. For each backdiffusion operation performed in a multimodal model, the value of t is decremented by one. This can be achieved using parameter labels.<diffusion=t> This distinguishes between data understanding tasks and data generation tasks. If t is a positive integer, it indicates that the multimodal model is performing the data generation task of the second data modality, such as generating an image. If t is 0, it indicates that the multimodal model has completed image generation and can continue to maintain the current state.<diffusion=0> And based on this condition, continue to perform the data generation task of the first data modality or the data understanding task of the second data modality, such as image classification, image similarity judgment, visual question answering, etc.

[0150] According to some embodiments, the initial label sequence can be obtained by segmenting a second data segment in the sample data sequence. For example, the image in the sample data sequence (i.e., the second data segment) is segmented into multiple sub-images of the same size, and all pixel values ​​within a sub-image are used as one or more labels to obtain the initial label sequence. For example, refer to... Figure 7 The initial label sequence is the sequence composed of labels t8 to t11 at the model input end.

[0151] According to some embodiments, the sample data sequence includes at least two of the following:

[0152] 1. A first sample data sequence that includes only data fragments of the first data modality. That is, the first sample data sequence is discrete single-modal data, such as plain text data.

[0153] 2. A second sample data sequence that includes only data segments of the second data modality. That is, the second sample data sequence is single-modal data of continuous modality, such as single-modal data such as images and audio.

[0154] 3. A third sample data sequence comprising interleaved data segments of the first and second data modalities. That is, the third sample data sequence is interleaved multimodal data. For example, the third sample data sequence could be a text-image sequence, an image-text sequence, a text-image-text sequence, a text-audio sequence, etc.

[0155] According to the above embodiments, by using sample data composed of different modalities to perform multi-task joint training on the multimodal model, the model can fully learn single-modal and cross-modal knowledge, thereby improving the model's ability to understand and generate multimodal data.

[0156] Figure 10 A schematic diagram of sample data sequences of different modalities according to some embodiments of the present disclosure is shown. Figure 10 The sample data sequences include unimodal data of text, image, and speech, as well as interleaved multimodal data such as image-text, text-image, speech-text, and text-speech. In this embodiment, the sample data sequences use a formal markup language as prompts to fuse different tasks across different modalities, employing a unified approach for sequential modeling and joint multi-task training of the multimodal model. This enables the model to fully learn unimodal and cross-modal knowledge, improving its multimodal data understanding and generation capabilities.

[0157] It should be noted that, Figure 10 Parameter labels in the sample data sequence<diffusion=t> This indicates the number of reverse diffusion operations to be performed.

[0158] Figure 11 A schematic diagram of sample data sequences for different modalities and tasks according to some embodiments of the present disclosure is shown. Figure 11 The sample data sequences include unimodal text data, as well as multimodal data such as image-text, audio-text, text-text-image, and text-text-image-text. These sample data sequences can be used to model specific tasks such as natural language question answering, intent recognition, image-text matching, and multimodal joint generation, thereby enabling multimodal models to fully learn knowledge in specific tasks and improve their performance on specific multimodal data processing tasks.

[0159] It should be noted that, Figure 11 The parameter labels are omitted.<diffusion=t> .

[0160] According to some embodiments, it is possible to utilize Figure 10 The multimodal model is pre-trained using sample data sequences from different modalities to enable it to fully learn unimodal and cross-modal knowledge, thus giving it general multimodal data understanding and generation capabilities. Based on this pre-training, the multimodal model can then utilize... Figure 11 The multimodal model is further fine-tuned using sample data sequences of different modalities for specific tasks, thereby improving the model's performance in processing specific multimodal data processing tasks.

[0161] According to some embodiments, method 400 can be performed by multiple electronic devices. Any one of the multiple electronic devices is configured to train a multimodal model using a sequence of sample data of the same type. For example, each electronic device can be configured to utilize... Figure 10 or Figure 11 The multimodal model is trained using any type of sample data sequence. The loss value of the multimodal model is the sum of the local loss values ​​of the multiple electronic devices. According to this embodiment, distributed training of the multimodal model in a data-parallel manner can improve training efficiency.

[0162] According to embodiments of this disclosure, a multimodal data generation apparatus is also provided. Figure 12 A structural block diagram of a multimodal data generation apparatus 1200 according to an embodiment of the present disclosure is shown. Figure 12 As shown, the device 1200 includes an acquisition module 1210 and an output module 1220.

[0163] The acquisition module 1210 is configured to acquire a query data sequence, wherein the query data sequence includes at least one data segment, and each data segment in the at least one data segment corresponds to a data modality.

[0164] Output module 1220 is configured to input the query data sequence into a multimodal model to obtain multiple tags in the response data sequence output sequentially by the multimodal model, wherein the current tag among the multiple tags is generated through the following operation:

[0165] In response to determining that the current tag belongs to a first data modality, the query data sequence and the current response data sequence are input into the multimodal model, so that the multimodal model generates the current tag based on the query data sequence and the current response data sequence, wherein the unit data value of the first data modality is discrete; or

[0166] In response to determining that the current tag belongs to the second data modality, the query data sequence and the current response data sequence are input into the multimodal model so that the multimodal model denoises the initial tag sequence based on the query data sequence and the current response data sequence to generate a result tag sequence, wherein the unit data values ​​of the second data modality are continuous, the initial tag sequence includes a preset number of initial tags, and the result tag sequence includes the preset number of tags starting from the current tag.

[0167] According to embodiments of this disclosure, multimodal data generation is achieved using a unified multimodal model. For discrete modalities such as natural language text, code, and protein sequences, the multimodal model sequentially generates each tag for that modality using an autoregressive approach. For continuous modalities such as images and audio, the multimodal model diffuses and generates multiple tags for that modality as a whole.

[0168] The multimodal model of this disclosure integrates the autoregressive generation process of discrete data and the diffusion generation process of continuous data, and has a general multimodal data understanding and generation capability, which improves the content generation effect and capability range, thereby enabling more flexible and efficient handling of diverse user tasks.

[0169] It should be understood that Figure 12 The various modules and units of the device 1200 shown can be used with reference to Figure 2 The steps in method 200 described correspond to each other. Therefore, the operations, features, and advantages described above for method 200 also apply to apparatus 1200 and its included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0170] According to embodiments of this disclosure, a training apparatus for a multimodal model is also provided. Figure 13 A structural block diagram of a multimodal model training apparatus 1300 according to an embodiment of the present disclosure is shown. Figure 13 As shown, the device 1300 includes an acquisition module 1310, an output module 1320, and an adjustment module 1330.

[0171] The acquisition module 1310 is configured to acquire a sample data sequence, wherein the sample data sequence includes at least one data segment, and each data segment in the at least one data segment corresponds to a data modality.

[0172] Output module 1320 is configured to input the sample data sequence into the multimodal model to obtain multiple labels in the prediction data sequence sequentially output by the multimodal model, wherein the current label among the multiple labels is generated through the following operation:

[0173] In response to determining that the current label belongs to a first data modality, the sample data sequence and the current predicted data sequence are input into the multimodal model so that the multimodal model generates the current label based on the sample data sequence and the current predicted data sequence, wherein the unit data values ​​of the first data modality are discrete; or

[0174] In response to determining that the current label belongs to the second data modality, the sample data sequence and the current predicted data sequence are input into the multimodal model so that the multimodal model denoises the initial label sequence based on the sample data sequence and the current predicted data sequence to generate a result label sequence, wherein the unit data values ​​of the second data modality are continuous, the initial label sequence includes a preset number of initial labels, and the result label sequence includes the preset number of labels starting from the current label.

[0175] The adjustment module 1330 is configured to adjust the parameters of the multimodal model based on the difference between the predicted data sequence and the target data sequence corresponding to the sample data sequence.

[0176] According to embodiments of this disclosure, multimodal data generation is achieved using a unified multimodal model. For discrete modalities such as natural language text, code, and protein sequences, the multimodal model sequentially generates each tag for that modality using an autoregressive approach. For continuous modalities such as images and audio, the multimodal model diffuses and generates multiple tags for that modality as a whole.

[0177] The multimodal model of this disclosure integrates the autoregressive generation process of discrete data and the diffusion generation process of continuous data, and has a general multimodal data understanding and generation capability, which improves the content generation effect and capability range, thereby enabling more flexible and efficient handling of diverse user tasks.

[0178] It should be understood that Figure 13 The various modules and units of the device 1300 shown can be used with reference to Figure 4 The steps in method 400 described correspond to each other. Therefore, the operations, features, and advantages described above for method 400 also apply to apparatus 1300 and its included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0179] Although specific functions have been discussed with reference to specific modules above, it should be noted that the functions of the various modules discussed in this article can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module.

[0180] It should also be understood that this article can describe various technologies in the general context of software and hardware components or program modules. The above regarding... Figure 12 , Figure 13 The described units can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuit. For example, in some embodiments, one or more of modules 1210-1220, 1310-1330 can be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components of other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0181] According to embodiments of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the multimodal data generation method and / or multimodal model training method of the embodiments of the present disclosure.

[0182] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the multimodal data generation method and / or multimodal model training method of the present disclosure embodiments is also provided.

[0183] According to embodiments of the present disclosure, a computer program product is also provided, including computer program instructions that, when executed by a processor, implement the multimodal data generation method and / or the multimodal model training method of the present disclosure embodiments.

[0184] refer to Figure 14 The present invention describes a structural block diagram of an electronic device 1400 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0185] like Figure 14 As shown, the electronic device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. The RAM 1403 may also store various programs and data required for the operation of the electronic device 1400. The computing unit 1401, ROM 1402, and RAM 1403 are interconnected via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0186] Multiple components in electronic device 1400 are connected to I / O interface 1405, including: input unit 1406, output unit 1407, storage unit 1408, and communication unit 1409. Input unit 1406 can be any type of device capable of inputting information to electronic device 1400. Input unit 1406 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 1407 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1408 may include, but is not limited to, a hard disk and an optical disk. The communication unit 1409 allows the electronic device 1400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, 802.11 devices, Wi-Fi devices, WiMAX devices, cellular communication devices and / or the like.

[0187] The computing unit 1401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1401 performs the various methods and processes described above, such as method 200 or method 400. For example, in some embodiments, method 200 and method 400 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1400 via ROM 1402 and / or communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of method 200 and method 400 described above may be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to perform method 200 or method 400 by any other suitable means (e.g., by means of firmware).

[0188] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0189] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0190] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0191] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0192] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0193] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0194] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0195] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of this disclosure is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.< / diffusion> < / text> < / text> < / text>

Claims

1. A method for generating multimodal data, comprising: Obtain a query data sequence, wherein the query data sequence includes at least one data segment, and each data segment in the at least one data segment corresponds to a data modality; and The query data sequence is input into a multimodal model to obtain multiple tags in the response data sequence output sequentially by the multimodal model, wherein the current tag among the multiple tags is generated through the following operation: In response to determining that the current tag belongs to a first data modality, the query data sequence and the current response data sequence are input into the multimodal model so that the multimodal model generates the current tag based on the query data sequence and the current response data sequence, wherein the unit data values ​​of the first data modality are discrete, and wherein the first data modality includes at least one of natural language text, tables, code, SMILE molecular formulas, or protein sequences; or In response to determining that the current tag belongs to the second data modality, the query data sequence and the current response data sequence are input into the multimodal model so that the multimodal model denoises the initial tag sequence based on the query data sequence and the current response data sequence to generate a result tag sequence, wherein the unit data values ​​of the second data modality are continuous, the initial tag sequence includes a preset number of initial tags, and the result tag sequence includes the preset number of tags starting from the current tag, wherein the second data modality includes at least one of image, video, audio, or point cloud data.

2. The method according to claim 1, wherein, The plurality of tags include modal tag pairs, each modal tag pair including a modal data start tag and a modal data end tag indicating the same data modality, wherein at least one tag belonging to the corresponding data modality is included between the modal data start tag and the modal data end tag, and the method further includes: The data modality to which the current tag belongs is determined based on the start tag of the last modality data in the current response data sequence.

3. The method according to claim 2, wherein, Determining the data modality to which the current tag belongs based on the start tag of the last modality data in the current response data sequence includes: In response to determining that the start tag of the last modality data in the current response data sequence indicates the first data modality, the current tag is determined to belong to the first data modality; or In response to determining that the last modality data start tag in the current response data sequence indicates the second data modality, it is determined that the current tag belongs to the second data modality.

4. The method according to claim 3, further comprising: After generating the result tag sequence, a modal data end tag indicating the second data modality and a modal data start tag indicating the first data modality are generated sequentially.

5. The method according to any one of claims 2-4, wherein, The modal data start label and the modal data end label are both markers of the first data modality, and the method further includes: In response to determining that the current response data sequence is empty, it is determined that the current tag belongs to the first data modality.

6. The method according to any one of claims 1-4, wherein, The multimodal model denoises the initial labeled sequence based on the query data sequence and the current response data sequence to generate the resulting labeled sequence, including: The multimodal model performs a preset number of back-diffusion operations on the initial labeled sequence based on the query data sequence and the current response data sequence to generate the result labeled sequence, wherein each back-diffusion operation includes: The query data sequence, the current response data sequence, the current tag sequence, and the identifier of the current backdiffusion round are input into the multimodal model so that the multimodal model generates a denoised tag sequence. The current label sequence of the first inverse diffusion operation is the initial label sequence, the current label sequence of the second and subsequent inverse diffusion operations is the denoised label sequence generated by the previous inverse diffusion operation, and the denoised label sequence generated by the last inverse diffusion operation is the result label sequence.

7. The method according to claim 6, wherein, The preset quantity and / or the preset number of times are the markers of the first data mode output by the multimodal model.

8. The method according to any one of claims 1-4, further comprising: Obtain an initial query data sequence input by the user, wherein the initial query data sequence includes at least one initial data segment, and each initial data segment corresponds to a data modality; and For each of the at least one initial data segments, a modal data start label indicating the data modality of the initial data segment is added before the initial data segment, and a modal data end label indicating the data modality of the initial data segment is added after the initial data segment to obtain the query data sequence.

9. The method according to any one of claims 1-4, wherein, The multimodal model is a converter model that only includes the decoder.

10. The method according to any one of claims 1-4, wherein, The initial label sequence is a random noise sequence.

11. A training method for a multimodal model, comprising: Obtain a sample data sequence, wherein the sample data sequence includes at least one data segment, and each data segment in the at least one data segment corresponds to a data modality; The sample data sequence is input into the multimodal model to obtain multiple labels in the predicted data sequence output sequentially by the multimodal model, wherein the current label among the multiple labels is generated through the following operation: In response to determining that the current label belongs to a first data modality, the sample data sequence and the current predicted data sequence are input into the multimodal model so that the multimodal model generates the current label based on the sample data sequence and the current predicted data sequence, wherein the unit data values ​​of the first data modality are discrete, and wherein the first data modality includes at least one of natural language text, tables, code, SMILE molecular formulas, or protein sequences; or In response to determining that the current label belongs to a second data modality, the sample data sequence and the current predicted data sequence are input into the multimodal model so that the multimodal model denoises the initial label sequence based on the sample data sequence and the current predicted data sequence to generate a result label sequence, wherein the unit data values ​​of the second data modality are continuous, the initial label sequence includes a preset number of initial labels, and the result label sequence includes the preset number of labels starting from the current label, wherein the second data modality includes at least one of image, video, audio, or point cloud data; and The parameters of the multimodal model are adjusted based on the difference between the predicted data sequence and the target data sequence corresponding to the sample data sequence.

12. The method according to claim 11, wherein, The sample data sequence includes at least two of the following: The first sample data sequence includes only data fragments of the first data modality; A second sample data sequence that includes only data fragments of the second data modality; or The third sample data sequence includes data segments from the first data modality and data segments from the second data modality, arranged in an alternating pattern.

13. The method according to claim 12, wherein, The method is performed by a plurality of electronic devices, any one of which is configured to train the multimodal model using a sequence of sample data of the same type, wherein the loss value of the multimodal model is the sum of the local loss values ​​of the plurality of electronic devices.

14. The method according to any one of claims 11-13, wherein, The sample data sequence includes only data segments of the first data modality, and the target data sequence is the sample data sequence itself.

15. The method according to any one of claims 11-13, wherein, The sample data sequence includes at least a data segment of the second data modality, and the method further includes: Obtain a first data sequence, wherein the first data sequence includes at least a first data segment of the second data modality; Multiple noise addition operations are performed on the first data segment to obtain multiple second data segments corresponding to each of the multiple noise addition operations, thereby obtaining multiple second data sequences corresponding to each of the multiple noise addition operations. Wherein, the sample data sequence is any one of the plurality of second data sequences, and the target data sequence is the first data sequence.

16. The method according to claim 15, wherein, The initial label sequence is obtained by segmenting the second data segment in the sample data sequence.

17. A multimodal data generation apparatus, comprising: The acquisition module is configured to acquire a query data sequence, wherein the query data sequence includes at least one data segment, and each data segment in the at least one data segment corresponds to a data modality; and The output module is configured to input the query data sequence into a multimodal model to obtain multiple tags in the response data sequence sequentially output by the multimodal model, wherein the current tag among the multiple tags is generated through the following operation: In response to determining that the current tag belongs to a first data modality, the query data sequence and the current response data sequence are input into the multimodal model so that the multimodal model generates the current tag based on the query data sequence and the current response data sequence, wherein the unit data values ​​of the first data modality are discrete, and wherein the first data modality includes at least one of natural language text, tables, code, SMILE molecular formulas, or protein sequences; or In response to determining that the current tag belongs to the second data modality, the query data sequence and the current response data sequence are input into the multimodal model so that the multimodal model denoises the initial tag sequence based on the query data sequence and the current response data sequence to generate a result tag sequence, wherein the unit data values ​​of the second data modality are continuous, the initial tag sequence includes a preset number of initial tags, and the result tag sequence includes the preset number of tags starting from the current tag, wherein the second data modality includes at least one of image, video, audio, or point cloud data.

18. A training device for a multimodal model, comprising: The acquisition module is configured to acquire a sample data sequence, wherein the sample data sequence includes at least one data segment, and each data segment in the at least one data segment corresponds to a data modality; The output module is configured to input the sample data sequence into the multimodal model to obtain multiple labels in the prediction data sequence sequentially output by the multimodal model, wherein the current label among the multiple labels is generated through the following operation: In response to determining that the current label belongs to a first data modality, the sample data sequence and the current predicted data sequence are input into the multimodal model so that the multimodal model generates the current label based on the sample data sequence and the current predicted data sequence, wherein the unit data values ​​of the first data modality are discrete, and wherein the first data modality includes at least one of natural language text, tables, code, SMILE molecular formulas, or protein sequences; or In response to determining that the current label belongs to a second data modality, the sample data sequence and the current predicted data sequence are input into the multimodal model so that the multimodal model denoises the initial label sequence based on the sample data sequence and the current predicted data sequence to generate a result label sequence, wherein the unit data values ​​of the second data modality are continuous, the initial label sequence includes a preset number of initial labels, and the result label sequence includes the preset number of labels starting from the current label, wherein the second data modality includes at least one of image, video, audio, or point cloud data; and The adjustment module is configured to adjust the parameters of the multimodal model based on the difference between the predicted data sequence and the target data sequence corresponding to the sample data sequence.

19. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-16.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-16.

21. A computer program product comprising computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of any one of claims 1-16.

Images