Multi-modal data generation method, model training method and related products

Abstract

The invention discloses a multi-modal data generation method, a model training method and a related product. The method comprises the steps that a target cue word is obtained, the target cue word comprises description for generating target multi-modal data, the target multi-modal data comprises first data of a first modal and second data of a second modal, and the first modal is different from the second modal; the target cue word is input into a target generation model, first intermediate data generated by the target generation model are obtained, and the first intermediate data comprise embedding vectors of the first data and the second data; and determining second data in the target multi-modal data based on the embedded vector. Based on the multi-modal data generation method, the efficiency of obtaining the single-modal data in the multi-modal data can be improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a multimodal data generation method, a model training method and related products. Background Art

[0002] Multimodal data refers to data that includes two or more different modalities, where the modalities of the data include: text, image, audio, and video. Obviously, multimodal data carries more comprehensive and rich information than single-modal data, so the application of multimodal data is becoming more and more widespread. Therefore, how to obtain single-modal data in multimodal data is an urgent problem to be solved. Summary of the invention

[0003] The present application provides a multimodal data generation method, a model training method and related products to improve the efficiency of obtaining single-modal data in multimodal data, wherein the related products include a multimodal data generation device, a training device, an electronic device, a computer-readable storage medium and a computer program product.

[0004] In a first aspect, a method for generating multimodal data is provided, the method comprising: Acquire a target prompt word, the target prompt word including a description of generating target multimodal data, the target multimodal data including first data of a first modality and second data of a second modality, the first modality being different from the second modality; Inputting the target prompt word into a target generation model to obtain first intermediate data generated by the target generation model, wherein the first intermediate data includes embedding vectors of the first data and the second data; Based on the embedding vector, the second data in the target multimodal data is determined.

[0005] In combination with any implementation manner of the present application, after determining the second data in the target multimodal data based on the embedding vector, the method further includes: The target multimodal data is obtained by replacing the embedding vector of the second data in the first intermediate data with the second data.

[0006] In combination with any embodiment of the present application, before obtaining the target multimodal data by replacing the embedding vector of the second data in the first intermediate data with the second data, the method further includes: determining a source of the second data; The step of replacing the embedding vector of the second data in the first intermediate data with the second data to obtain the target multimodal data includes: The target multimodal data is obtained by replacing the embedding vector of the second data in the first intermediate data with the second data and adding the source of the second data to the first intermediate data.

[0007] In combination with any embodiment of the present application, the second data includes two or more third data, and the two or more third data are all related to the same content in the first intermediate data; The step of replacing the embedding vector of the second data in the first intermediate data with the second data to obtain the target multimodal data includes: The target multimodal data is obtained by replacing the embedding vector of the second data in the first intermediate data with the second data, and adding a description of the difference between the different third data in the first intermediate data.

[0008] In combination with any implementation manner of the present application, before determining the second data in the target multimodal data based on the embedding vector, the method further includes: Acquire a candidate data set, wherein the modality of data in the candidate data set is all the second modality; The determining, based on the embedding vector, the second data in the target multimodal data comprises: The second data is determined based on data in the candidate data set having a feature vector matching the embedding vector of the second data.

[0009] In combination with any implementation manner of the present application, the step of obtaining a target prompt word includes: Receiving a generation request from a target object, wherein the generation request is used to request generation of the target multimodal data; generating the target prompt word based on the generation request; The determining, based on the embedding vector, the second data in the target multimodal data comprises: Determine that second modality data matching the embedding vector of the second data is matching data, and the modality of the second modality data is the second modality; The style of the matching data is converted into a style that matches the features of the target object to obtain the second data.

[0010] In combination with any embodiment of the present application, the first intermediate data further includes an identifier for determining the embedding vector from the first intermediate data, and before determining the second data in the target multimodal data based on the embedding vector, the method further includes: The embedding vector is determined from the first intermediate data based on the identification.

[0011] In a second aspect, a method for generating multimodal data is provided, the method comprising: Obtain a target prompt word, wherein the target prompt word includes a description of generating target multimodal data, wherein the target multimodal data includes first data of a first modality and second data of a second modality, wherein the first modality is text and the second modality is an image Inputting the target prompt word into a target generation model to obtain first intermediate data generated by the target generation model, wherein the first intermediate data includes embedding vectors of the first data and the second data; Based on the embedding vector, the second data in the target multimodal data is determined.

[0012] In a third aspect, a model training method is provided, the method comprising: Acquire training multimodal data, where the training multimodal data includes first training data of a first modality and second training data of a second modality; Based on the training multimodal data, obtaining training prompt words, wherein the training prompt words include a description of generating the training multimodal data; Inputting the training prompt word into a first generation model to obtain second intermediate data generated by the first generation model, wherein the second intermediate data includes first prediction data of the first modality and a first training embedding vector of second prediction data of the second modality; determining a first loss of the first generative model based on a first difference between the first training embedding vector and an embedding vector of second training data in the training multimodal data, the first loss being positively correlated with the first difference; Based on the first loss, the parameters of the first generation model are updated to obtain a target generation model.

[0013] In combination with any embodiment of the present application, the method further includes: Based on the training multimodal data, acquiring training input data and training target data, wherein the training input data or the training target data is at least a part of the training multimodal data, and the training target data includes third training data of the first modality and fourth training data of the second modality; Inputting the training input data into a second generative model so that the second generative model generates third intermediate data, wherein the third intermediate data includes a second training embedding vector of third predicted data of the first modality and fourth predicted data of the second modality; Determining a second loss of the second generative model based on a second difference between the second training embedding vector and the embedding vector of the fourth training data in the training target data, wherein the second loss is positively correlated with the second difference; Based on the second loss, update the parameters of the second generation model to obtain the first generation model.

[0014] In combination with any embodiment of the present application, the method further includes: Extracting an embedding vector of second training data of a second modality in the training multimodal data using an encoder of the second modality; and / or The encoder of the second modality is used to extract the embedding vector of the fourth training data of the second modality in the training target data.

[0015] In a fourth aspect, a multimodal data generating device is provided, the device comprising: an acquisition unit, configured to acquire a target prompt word, wherein the target prompt word includes a description of generating target multimodal data, wherein the target multimodal data includes first data of a first modality and second data of a second modality, wherein the first modality is different from the second modality; an obtaining unit, configured to input the target prompt word into a target generation model to obtain first intermediate data generated by the target generation model, wherein the first intermediate data includes embedding vectors of the first data and the second data; A determining unit is used to determine the second data in the target multimodal data based on the embedding vector.

[0016] In combination with any embodiment of the present application, the multimodal data generating device further includes: A replacement unit is used to obtain the target multimodal data by replacing the embedding vector of the second data in the first intermediate data with the second data.

[0017] In combination with any implementation manner of the present application, the determining unit is further used to determine a source of the second data; The replacement unit is further used to obtain the target multimodal data by replacing the embedding vector of the second data in the first intermediate data with the second data, and adding the source of the second data to the first intermediate data.

[0018] In combination with any embodiment of the present application, the second data includes two or more third data, and the two or more third data are all related to the same content in the first intermediate data; The replacement unit is also used to obtain the target multimodal data by replacing the embedding vector of the second data in the first intermediate data with the second data, and adding a description of the difference between the different third data in the first intermediate data.

[0019] In combination with any embodiment of the present application, the acquisition unit is further used to acquire a candidate data set, and the modality of the data in the candidate data set is all the second modality; The determining unit is used to determine the second data based on data in the candidate data set having an embedded vector matching a feature vector of the second data.

[0020] In combination with any implementation manner of the present application, the acquisition unit is used to: Receiving a generation request from a target object, wherein the generation request is used to request generation of the target multimodal data; generating the target prompt word based on the generation request; The determining unit is used for: Determine that second modality data matching the embedding vector of the second data is matching data, and the modality of the second modality data is the second modality; The style of the matching data is converted into a style that matches the features of the target object to obtain the second data.

[0021] In combination with any embodiment of the present application, the first intermediate data also includes an identifier for determining the embedding vector from the first intermediate data, and the determination unit is further used to determine the embedding vector from the first intermediate data based on the identifier.

[0022] In a fifth aspect, a multimodal data generating device is provided, the device comprising: an acquisition unit, configured to acquire a target prompt word, wherein the target prompt word includes a description of generating target multimodal data, wherein the target multimodal data includes first data of a first modality and second data of a second modality, wherein the first modality is an image and the second modality is a text; an obtaining unit, configured to input the target prompt word into a target generation model to obtain first intermediate data generated by the target generation model, wherein the first intermediate data includes embedding vectors of the first data and the second data; A determining unit is used to determine the second data in the target multimodal data based on the embedding vector.

[0023] In a sixth aspect, a model training device is provided, the device comprising: An acquisition unit, configured to acquire training multimodal data, wherein the training multimodal data includes first training data of a first modality and second training data of a second modality; The acquisition unit is further used to acquire a training prompt word based on the training multimodal data, wherein the training prompt word includes a description of generating the training multimodal data; an obtaining unit, configured to input the training prompt word into a first generation model to obtain second intermediate data generated by the first generation model, wherein the second intermediate data includes first prediction data of the first modality and a first training embedding vector of second prediction data of the second modality; a determining unit, configured to determine a first loss of the first generative model based on a first difference between the first training embedding vector and an embedding vector of second training data in the training multimodal data, wherein the first loss is positively correlated with the first difference; An updating unit is used to update the parameters of the first generation model based on the first loss to obtain a target generation model.

[0024] In combination with any embodiment of the present application, the acquisition unit is further used to acquire training input data and training target data based on the training multimodal data, where the training input data or the training target data is at least a part of the training multimodal data, and the training target data includes third training data of the first modality and fourth training data of the second modality; The obtaining unit is further configured to input the training input data into a second generation model so that the second generation model generates third intermediate data, wherein the third intermediate data includes third prediction data of the first modality and a second training embedding vector of fourth prediction data of the second modality; The determining unit is further configured to determine a second loss of the second generation model based on a second difference between the second training embedding vector and the embedding vector of the fourth training data in the training target data, wherein the second loss is positively correlated with the second difference; The updating unit is further used to update the parameters of the second generation model based on the second loss to obtain the first generation model.

[0025] In combination with any implementation manner of the present application, the acquisition unit is further used to: Extracting an embedding vector of second training data of a second modality in the training multimodal data using an encoder of the second modality; and / or The encoder of the second modality is used to extract the embedding vector of the fourth training data of the second modality in the training target data.

[0026] In the seventh aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device executes the above-mentioned first aspect and any embodiment thereof, the electronic device either executes the above-mentioned second aspect and any embodiment thereof, or the electronic device either executes the above-mentioned third aspect and any embodiment thereof.

[0027] In an eighth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the first aspect and any embodiment thereof, or the electronic device executes the second aspect and any embodiment thereof, or the electronic device executes the third aspect and any embodiment thereof.

[0028] In the ninth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the first aspect and any embodiment thereof, or the processor is caused to execute the second aspect and any embodiment thereof, or the processor is caused to execute the third aspect and any embodiment thereof.

[0029] In the tenth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to execute the above-mentioned first aspect and any embodiment thereof, or the computer is caused to execute the above-mentioned second aspect and any embodiment thereof, or the computer is caused to execute the above-mentioned third aspect and any embodiment thereof.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0031] In the present application, the target prompt word includes a description of generating target multimodal data, and the target multimodal data includes second data of modality. After acquiring the target prompt word, the generating device inputs the target prompt word into the target generation model, and can obtain the first intermediate data generated by the target generation model, wherein the first intermediate data includes the embedding vectors of the first data and the second data. Finally, based on the embedding vector of the second data, the second data in the target multimodal data can be determined. Since the generating device determines the second data in the target multimodal data based on the embedding vector of the second data in the first intermediate data generated by the target generation model, and the maximum number of embedding vectors of the second data that can be generated by the target generation model is more than the number of second data that can be processed by the target generation model, the generating device obtains the second data in the target multimodal data in this way, which can improve the efficiency of obtaining the second data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0033] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0034] Figure 1 A schematic diagram of a process for generating multimodal data provided in an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of another multimodal data generation method provided in an embodiment of the present application; Figure 3 A flowchart of another multimodal data generation method provided in an embodiment of the present application; Figure 4 A schematic diagram of first intermediate data including an embedded vector of second data provided in an embodiment of the present application; Figure 5 A schematic diagram of target multimodal data provided in an embodiment of the present application; Figure 6 A flowchart of another multimodal data generation method provided in an embodiment of the present application; Figure 7 A flowchart of another multimodal data generation method provided in an embodiment of the present application; Figure 8 A flowchart of a model training method provided in an embodiment of the present application; Fig. 9 A schematic diagram of a flow chart for calculating contrastive learning loss provided in an embodiment of the present application; Fig.10 A schematic diagram of the structure of a multimodal data generating device provided in an embodiment of the present application; Fig.11 A schematic diagram of the structure of another multimodal data generating device provided in an embodiment of the present application; Fig.12 A schematic diagram of the structure of a model training device provided in an embodiment of the present application; Fig.13 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0037] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] Multimodal data refers to data that includes two or more different modes, where the modes of data include: text, image, audio, and video. Obviously, multimodal data carries more comprehensive and rich information than single-modal data, so the application of multimodal data is becoming more and more extensive. In one possible application scenario, the copy includes two or more different modal data, and the content of the copy is richer and more attractive. For example, the copy can display the corresponding content in a graphic way through text and images. In another possible application scenario, in the search scenario, the search results returned to the object can be multimodal data, thereby improving the search experience of the object. For example, the search term entered by the object is: What is the snow scene like? The search results can include text that introduces the snow scene and images and videos used to reflect the snow scene, so that the object can feel the snow scene more intuitively through the search results. In another possible implementation, when the subject has a conversation with a digital person or a virtual character, the content of the conversation between the digital person or the virtual character and the subject may include multimodal data, which can increase the subject's interest in the conversation, wherein the digital person is a digital character image close to the human image created by digital technology. They can simulate virtual entities with human characteristics and behaviors through computer software. Digital people rely on display devices to exist, have human appearance and behavior, can recognize the external environment, and communicate and interact with humans. Virtual characters are character images created and simulated by means of computer technology, artificial intelligence, etc. For example, when the subject talks to the digital person, he mentions: "It rained for many days, and it finally cleared up today", and the digital person can respond: "Yes, the sky on a sunny day is particularly beautiful, image A", where image A includes the sky on a sunny day.

[0039] In current technology, a language model processes single-modal data to determine single-modal data in multimodal data, and then generates multimodal data based on single-modal data. The language model is a model with natural language processing (NLP) capabilities. For example, please refer to Figure 1 , Figure 1 A schematic diagram of a process for generating multimodal data provided in an embodiment of the present application. Figure 1As shown in the figure, the prompt of the language model includes the following content: Please describe the snow scene, and insert relevant images in appropriate places, replaced by the image label [image x]. The images that can be referenced are as follows: image 1, image 2...image N. The multimodal data to be generated at this time includes text and images. After the prompt word is output to the language model, the language model generates a text describing the snow scene based on the prompt word, and leaves a position for inserting images related to the snow scene in the text. For example, the text is: Snow scene refers to the natural scenery when it snows or after snowing [image 1]. According to the form of snow, the snow scene can be divided into drifting snow [image 2], accumulated snow [image N]... Then replace [image 1] in the text with image 1, replace [image 2] in the text with image 2, and replace [image N] in the text with image N, and a multimodal data including two modalities of text and image can be obtained.

[0040] In this approach, when a language model processes single-modal data, it needs to convert the single-modal data into characters, and then process the converted characters to process the single-modal data. Since the number of characters that a language model can process is limited (for example, the maximum number of characters that a language model can process at one time is 16,000), and the number of characters converted from single-modal data is usually large, the number of single-modal data that a language model can process is limited, which leads to the low efficiency of this approach in obtaining single-modal data in multimodal data. Taking single-modal data as an image as an example, an image needs to be represented by 256 characters, that is, converting an image into characters can obtain 256 characters. If the maximum number of characters that a language model can process at one time is 2500, then the number of images that the language model can process is less than 10.

[0041] Based on this, the embodiment of the present application provides a multimodal data generation method to improve the efficiency of obtaining single-mode data in multimodal data. The execution subject of the multimodal data generation method in the embodiment of the present application is a multimodal data generation device (hereinafter referred to as a generation device), wherein the generation device can be any electronic device that can execute the multimodal data generation method. Optionally, the generation device is one of the following: a computer, a server.

[0042] The generating device can generate data of any modality in the target multimodal data through this method. Specifically, in this method, the generating device can obtain first intermediate data generated by the target generation model by inputting the target prompt words including the description of generating the target multimodal data into the target generation model, wherein the target multimodal data includes first data of the first modality and second data of the second modality, and the first intermediate data includes the embedding vectors of the first data and the second data.

[0043] In an embodiment of the present application, the generative model (including the above-mentioned target generative model and the first generative model and the second generative model to be mentioned below) has the ability to generate data of the first modality. In one possible implementation, the first modality is text, and the generative model is a language model, for example, the generative model is a generative pretrained transformer (GPT) 4, a llama. In another possible implementation, the first modality is audio, and the generative model is a model for generating audio, for example, the generative model is a hidden Markov model (HMM), or a Gaussian mixture model (GMM). In another possible implementation, the first modality is an image, and the generative model is a model for generating images, for example, the generative model is a decoder of a generative adversarial network (GAN), or a stable diffusion model.

[0044] In the embodiment of the present application, the embedding vectors include information about the data. For example, the embedding vector of the text includes information about the text, and the embedding vector of the image includes information about the image. After generating the embedding vector of the second data, the generating device may determine the second data in the target multimodal data based on the embedding vector of the second data. Since the generating device determines the second data in the target multimodal data based on the embedding vector of the second data generated by the target generation model, and the maximum number of embedding vectors that can be generated by the target generation model is greater than the number of second data that can be processed by the target generation model, the generating device obtains the second data in the target multimodal data in this way, which can improve the efficiency of obtaining the second data.

[0045] For example, see Figure 2 , Figure 2 A schematic diagram of another multimodal data generation method provided in an embodiment of the present application. Figure 2 In the example, the target prompt word is: Please describe the snow scene with pictures and texts, wherein the target prompt word includes a description of generating target multimodal data including images and texts, and the second data in the target multimodal data is an image. Figure 2As shown, after the target prompt word is input into the target generation model, the first intermediate data output by the target generation model is: snow scene refers to the natural scenery when it snows or after snowing [embedded vector 1 of image 1], and according to the form of snow, the snow scene can be divided into drifting snow [embedded vector 2 of image 2], accumulated snow [embedded vector N of image N]..., wherein the embedding vector 1 of image 1, the embedding vector 2 of image 2, and the embedding vector N of image N are all embedded vectors. After generating the embedding vector 1 of image 1, the embedding vector 2 of image 2, and the embedding vector N of image N, image 1 can be determined based on the embedding vector 1 of image 1, image 2 can be determined based on the embedding vector 2 of image 2, and image N can be determined based on the embedding vector N of image N, wherein image 1, image 2, and image N are all second data in the target multimodal data.

[0046] By Figure 1 and Figure 2 By comparison, Figure 1 The language model in needs to process the data of the second modality (i.e., image 1, image 2, ..., image N), while the target generation model does not need to process the data of the second modality. Figure 1 The language model in does not generate an embedding vector for the second modality, but Figure 2 The first intermediate data generated by the target generation model in includes the embedding vector of the data of the second modality (ie, the embedding vector of the second data). And Figure 1 The data of the second modality in the generated multimodal data is the data of the second modality input to the language model, and Figure 2 The second data in is determined based on the embedding vector of the second data generated by the target generation model. Figure 2 The multimodal data generation method shown does not need to process the data of the second modality, and can generate an embedding vector of the data of the second modality, and determine the second data in the target multimodal data based on the embedding vector of the data of the second modality, thereby improving the efficiency of obtaining the second data.

[0047] The following will describe in detail the multimodal data generation method provided by the embodiment of the present application. Figure 3 , Figure 3 A flowchart of another multimodal data generation method provided in an embodiment of the present application.

[0048] 301. Obtain a target prompt word, wherein the target prompt word includes a description of generating target multimodal data, the target multimodal data includes first data of a first modality and second data of a second modality, and the first modality is different from the second modality.

[0049] In an embodiment of the present application, the target prompt word can be used to guide the language model to perform the task in the target prompt word. Specifically, the target prompt word includes a description of generating target multimodal data, then the task in the target prompt word includes generating target multimodal data, then the target prompt word can be used to guide the language model to generate target multimodal data. The description of generating target multimodal data can be a text indicating the generation of target multimodal data. For example, the target prompt word includes the following text: Please generate a copy with both text and images. For another example, the target prompt word includes the following text: Please generate a multimedia data with both audio and images.

[0050] In a possible implementation, the target prompt word also includes requirements that the target multimodal data needs to meet. Accordingly, the target prompt word is used to guide the language model to generate target multimodal data that meets the requirements. For example, Figure 2 The target prompt words in include: Please describe the snow scene with pictures and texts, where describing the snow scene is the requirement that the target multimodal data needs to meet, that is, the target prompt words expect the target generation model to generate target multimodal data that describes the snow scene through text and images. For another example, the target prompt words include: Please introduce the highest mountain in the world with pictures and texts, where introducing the highest mountain is the requirement that the target multimodal data needs to meet.

[0051] In an embodiment of the present application, the target multimodal data includes first data of a first modality and second data of a second modality, wherein the first modality is different from the second modality. That is to say, the target multimodal data includes at least two different modal data. Optionally, both the first modality and the second modality are non-text, wherein the non-text is a modality other than text. In one possible implementation, the first modality is text, the second modality is an image, and correspondingly, the first data is text, the second data is an image, and the target multimodal data includes text and images. In another possible implementation, the first modality is audio, the second modality is video, and correspondingly, the first data is audio, the second data is video, and the target multimodal data includes audio and video.

[0052] In an implementation method of obtaining a target prompt word, a generating device receives a target prompt word input by a user through an input component, wherein the input component includes: a mouse, a keyboard, a touch screen, a touch pad, and an audio input device.

[0053] In another implementation method of obtaining the target prompt word, the generating device receives the target prompt word sent by the user through a terminal, wherein the terminal includes: a mobile phone, a computer, a tablet computer, and a smart wearable device.

[0054] 302. Input the target prompt word into the target generation model to obtain first intermediate data generated by the target generation model, wherein the first intermediate data includes an embedding vector of the first data and the second data.

[0055] In the embodiment of the present application, the target generation model is a model with NLP capabilities. Optionally, the target generation model includes a large language model (LLM). For example, the target large language model is one of the following: intelligent chat model (chatgpt), llama.

[0056] The generating device inputs the target prompt word into the target generating model, so that the target generating model can execute the task in the target prompt word, specifically, can enable the target generating model to generate target multimodal data.

[0057] In one possible implementation, the target generation model encodes the target prompt word to obtain a hidden state sequence, wherein the hidden state sequence includes information about the result of executing the task in the target prompt word. For example, if the task in the target prompt word is to generate audio and images related to sunny days, then the hidden state sequence includes information related to sunny days. Since the execution result of the task in the target prompt word includes the second data, and the information of the second data matches the information of the execution result of the task in the target prompt word, the target generation model can determine the embedding vector of the second data based on the hidden state sequence.

[0058] For example, the target prompt words include: Please describe the snow scene with both text and pictures. At this time, the modalities of the target multimodal data include text and images, wherein the image is the second data. After the generating device inputs the target prompt words into the target generating model, the generating device uses the target generating model to generate text (i.e., the first data) describing the snow scene based on the target prompt words, and uses the target generating model's ability to generate embedding vectors to generate an image embedding vector describing the snow scene. At this time, the image embedding vector is the embedding vector of the second data, and the image embedding vector carries information about the image describing the snow scene. The target generating model then outputs the first intermediate data including the text describing the snow scene and the embedding vector describing the image of the snow scene (see for details). Figure 2 The first intermediate data in ).

[0059] For another example, the target prompt words include: Please use songs and videos to describe the snow scene. In this case, the modalities of the target multimodal data include audio and video, and the second data, among which, audio is the second data. After the generating device inputs the target prompt words into the target generating model, the generating device uses the target generating model to generate a video describing the snow scene (i.e., the first data) based on the target prompt words, and uses the target generating model's ability to generate embedding vectors to generate an audio embedding vector for the audio describing the snow scene. In this case, the audio embedding vector is the embedding vector of the second data. The target generating model then outputs the first intermediate data including the embedding vectors of the video describing the snow scene and the audio describing the snow scene.

[0060] Optionally, when the modality of the target multimodal data includes text, the information carried by the embedding vector of the second data matches the first data. For example, the target prompt word includes: please describe the snow scene with pictures and text, and the first intermediate data may include: snow scene refers to the natural scenery when it snows or after snowing, and the embedding vector of the image describing the snow scene, wherein the information carried by the embedding vector of the image matches "snow scene refers to the natural scenery when it snows or after snowing".

[0061] For example, in Figure 2 In the example, the first intermediate data includes: snow scene refers to the natural scenery when it snows or after it snows [embedded vector 1 of image 1], and according to the shape of snow, the snow scene can be divided into drifting snow [embedded vector 2 of image 2], accumulated snow [embedded vector N of image N]... At this time, the first data includes: snow scene refers to the natural scenery when it snows or after it snows, and according to the shape of snow, the snow scene can be divided into drifting snow, the information carried by [embedded vector 1 of image 1] matches "snow scene refers to the natural scenery when it snows or after it snows", the information carried by [embedded vector 2 of image 2] matches "snow scene can be divided into drifting snow", and the information carried by [embedded vector 3 of image 3] matches "accumulated snow".

[0062] In another possible implementation, after the generating device inputs the target prompt word into the target generation model, the target generation model generates text matching the target prompt word, and then generates an embedding vector of the second data based on the text matching the target prompt word. For example, the target prompt word includes "please describe the snow scene with text and audio", and the modality of the target multimodal data includes text and audio, wherein the audio is the second data. After the target prompt word is input into the target generation model, the target generation model generates text describing the snow scene. Since the audio information is describing the snow scene, and the generated text is describing the snow scene, the target generation model can generate an embedding vector of the audio based on the generated text. For another example, the target prompt word includes "please describe the snow scene with images and audio", and the modality of the target multimodal data includes images and audio, wherein the audio is the second data. After the target prompt word is input into the target generation model, the target generation model generates text describing the snow scene. Since the audio information is describing the snow scene, and the generated text is describing the snow scene, the target generation model can generate an embedding vector of the audio based on the generated text. Optionally, an embedding vector of the text matching the target prompt word is used as the embedding vector of the second data.

[0063] Optionally, the target prompt word also includes requirements that the target multimodal data needs to meet. In this case, the generating device performs the following steps during the execution of step 302: inputting the target prompt word into the language model to obtain first intermediate data generated by the language model based on the requirements. For example, Figure 2The target prompt words include: please describe the snow scene with pictures and texts, wherein describing the snow scene is the requirement that the target multimodal data needs to meet. Correspondingly, the first intermediate data generated by the target generation model describes the snow scene. Specifically, the text in the first intermediate data describes the snow scene. The embedding vector 1 of image 1, the embedding vector 2 of image 2, and the embedding vector 3 of image 3 in the first intermediate data are all embedding vectors of images describing the snow scene.

[0064] 303. Determine second data in the target multimodal data based on the embedding vector.

[0065] Since the embedding vector of the second data carries information of the second data, the generating device may determine the second data based on the embedding vector of the second data. In one possible implementation, the generating device generates the second data based on the embedding vector of the second data. Optionally, the generating device processes the embedding vector of the second data using a deep learning model to generate the second data. For example, when the second data is an image or a video, the embedding vector of the second data is decoded using a GAN decoder to obtain the image or video.

[0066] As an optional implementation, the first intermediate data also includes an identifier for determining the embedding vector of the second data from the first intermediate data. Based on the identifier of the embedding vector of the second data, the embedding vector of the second data can be determined from the first intermediate data. In one possible implementation, in the first intermediate data, the identifier of the embedding vector of the second data is the character before the embedding vector of the second data. For example, Figure 4 A schematic diagram of first intermediate data including an embedded vector of second data provided in an embodiment of the present application. Figure 4 As shown, the first intermediate data includes: snow scene refers to the natural scenery when it snows or after snowing [image] [embedded vector 1 of image 1], according to the form of snow, the snow scene can be divided into drifting snow [image] [embedded vector 2 of image 2], accumulated snow [image] [embedded vector N of image N]..., where [image] is the identifier of the embedded vector, and the embedded vector 1 of image 1, the embedded vector 2 of image 2, and the embedded vector N of image N are all embedded vectors. The characters before the embedded vector of each image (i.e., the embedded vector of the second data) are the identifiers of the embedded vector of the second data.

[0067] In another possible implementation, the identifier of the embedding vector of the second data carries the position information of the embedding vector of the second data, and the generating device can determine the embedding vector of the second data from the first intermediate data based on the identifier of the embedding vector of the second data. Optionally, the information carried by the identifier of the embedding vector of the second data is used to indicate the number of characters after the identifier of the embedding vector of the second data, for example, if the information carried by the identifier of the embedding vector of the second data includes 2 characters, then based on the identifier of the embedding vector of the second data, it can be determined that in the first intermediate data, the second character after the identifier of the embedding vector of the second data is the embedding vector of the second data. Optionally, the information carried by the identifier of the embedding vector of the second data is used to indicate the number of characters before the identifier of the embedding vector of the second data, for example, if the information carried by the identifier of the embedding vector of the second data includes 3 characters, then based on the identifier of the embedding vector of the second data, it can be determined that in the first intermediate data, the third character before the identifier of the embedding vector of the second data is the embedding vector of the second data. Optionally, the information carried by the identifier of the embedding vector of the second data is used to indicate which characters in the first intermediate data are the embedding vectors of the second data. For example, the information carried by the identifier of the embedding vector of the second data includes that the 7th character and the 20th character in the first intermediate data are both the embedding vectors of the second data. Then, based on the identifier of the embedding vector of the second data, it can be determined that the 7th character and the 20th character in the first intermediate data are both the embedding vectors of the second data.

[0068] In this embodiment, before executing step 303, the generating device further determines the embedding vector of the second data from the first intermediate data based on the identifier of the embedding vector of the second data, thereby quickly and accurately determining the embedding vector of the second data from the first intermediate data. Optionally, the generating device first determines the identifier of the embedding vector of the second data from the first intermediate data, and then determines the embedding vector of the second data from the first intermediate data based on the identifier of the embedding vector of the second data. For example, the identifier of the embedding vector of the second data is Figure 4 , then determining the [image] from the first intermediate data can determine the identity of the embedding vector of the second data.

[0069] In an embodiment of the present application, the target prompt word includes a description of generating target multimodal data, and the target multimodal data includes second data of modality. After acquiring the target prompt word, the generating device inputs the target prompt word into the target generation model, and can obtain the first intermediate data generated by the target generation model, wherein the first intermediate data includes the embedding vectors of the first data and the second data. Finally, based on the embedding vector of the second data, the second data in the target multimodal data can be determined. Since the generating device determines the second data in the target multimodal data based on the embedding vector of the second data in the first intermediate data generated by the target generation model, and the maximum value of the number of embedding vectors of the second data that can be generated by the target generation model is more than the number of second data that can be processed by the target generation model, the generating device obtains the second data in the target multimodal data in this way, which can improve the efficiency of obtaining the second data.

[0070] As an optional implementation, after determining the second data in the target multimodal data based on the embedding vector, the generating device further performs the following steps: 4001. Obtain the target multimodal data by replacing the embedding vector of the second data in the first intermediate data with the second data.

[0071] As mentioned above, since the target generation model cannot generate the second data, the second data is represented by the embedding vector of the second data in the first intermediate data, but the modality of the embedding vector is not the second modality, and the target multimodal data includes the second data whose modality is the second modality. Therefore, the generation device obtains the target multimodal data by executing step 4001.

[0072] For example, Figure 5 A schematic diagram of target multimodal data provided in an embodiment of the present application, Figure 5 The target multimodal data shown is obtained by using Figure 2 Image 1, Image 2, Image N in are replaced respectively Figure 2 The first intermediate data in [embedded vector 1 of image 1], [embedded vector 2 of image 2], and [embedded vector N of image N] are obtained. Figure 5 As shown, the target multimodal data includes both text and images, and both text and images are related to the task in the target cue word (i.e., describing the snow scene).

[0073] Optionally, when the first intermediate data includes an identifier of an embedding vector of second data, the generating device obtains the target multimodal data by replacing the embedding vector of the second data and the identifier of the embedding vector of the second data in the first intermediate data with the second data.

[0074] As an optional implementation manner, before executing step 4001, the generating device further executes the following steps: 5001. Determine the source of the second data.

[0075] After executing step 5001, the generating device performs the following steps in the process of executing step 4001: 5002. Obtain target multimodal data by replacing an embedding vector of second data in the first intermediate data with the second data and adding a source of the second data to the first intermediate data.

[0076] In this embodiment, the source of the second data includes: the generation method of the second data when the second data is generated, and the source of the second data when the second data is not generated. For example, the second data is obtained by the generating device using GAN to process the embedding vector of the second data, then the source of the second data includes generating using GAN. For another example, the second data is determined by the generating device from document W based on the embedding vector of the second data, then the source of the second data includes document W.

[0077] After determining the source of the second data, the generating device obtains the target multimodal data by adding the source of the second data to the first intermediate data. The source of the second data can be reflected in the target multimodal data, thereby better protecting the legitimacy of the intellectual property rights and copyright of the second data, and increasing the viewing and attractiveness of the target multimodal data.

[0078] In one possible implementation, the generating device first replaces the embedding vector of the second data in the first intermediate data with the second data, and then adds the source of the second data to the first intermediate data to obtain the target multimodal data. In another possible implementation, the generating device first adds the source of the second data to the first intermediate data, and then replaces the embedding vector of the second data in the first intermediate data with the second data to obtain the target multimodal data.

[0079] As an optional implementation, the second data includes two or more third data, wherein the two or more third data are all related to the same content in the first intermediate data. That is, the same content in the first intermediate data can be illustrated by two or more third data, wherein any two different third data in the two or more third data are different, that is, different third data carry different information. For example, the first intermediate data includes the following text: Snow scene refers to the natural scenery when it snows or after snowing. For this text, the image describing the snowing and the image describing the snowing are both related to it, so the two or more third data may include the image describing the snowing and the image describing the snowing. For another example, the first intermediate data includes the following text: Snow scene refers to the natural scenery when it snows or after snowing. For this text, the image of A when it snows and the image of B when it snows are both related to it, so the two or more third data may include the image of A when it snows and the image of B when it snows.

[0080] In the target multimodal data, the third data can be used as a representation of other contents to make the other contents better understood. Therefore, when there are two or more different third data related to the same content in the target multimodal data, the two or more different third data can be used to better illustrate the related contents, thereby improving the richness and understandability of the contents of the target multimodal data.

[0081] In this implementation manner, the generating device performs the following steps during the execution of step 4001: 6001. Obtain target multimodal data by replacing an embedding vector of second data in first intermediate data with second data and adding a description of a difference between different third data to the first intermediate data.

[0082] Although the third data are all related to the same content, different third data express different information. Therefore, in order to make the information expressed by different third data better understood, the generating device adds a description of the difference between different third data in the first intermediate data. For example, the target multimodal data includes the following content: As shown in Image 1 and Image 2, the snow scene refers to the natural scenery during or after snowing, wherein Image 1 represents the scene during snowing, and Image 2 represents the scene after snowing.

[0083] In one possible implementation, the generating device first replaces the embedding vector of the second data in the first intermediate data with the second data, and then adds a description of the difference between different third data to the first intermediate data, to obtain the target multimodal data. In another possible implementation, the generating device first adds a description of the difference between different third data to the first intermediate data, and then replaces the embedding vector of the second data in the first intermediate data with the second data, to obtain the target multimodal data.

[0084] As an optional embodiment, when the generating device determines the source of the second data and the second data includes two or more third data, the generating device obtains the target multimodal data by replacing the embedding vector of the second data in the first intermediate data with the second data, adding the source of the second data to the first intermediate data, and adding a description of the difference between different third data to the first intermediate data.

[0085] As an optional implementation manner, before executing step 303, the generating device further executes the following steps: 7001. Obtain a candidate data set, wherein the modality of the data in the candidate data set is all the second modality.

[0086] After executing step 7001, the generating device performs the following steps in the process of executing step 303: 7002. Determine second data based on data in the candidate data set that has a feature vector that matches an embedding vector of the second data.

[0087] In an embodiment of the present application, the candidate data set is a set of data of modality . The feature vector of the data in the candidate data set carries information about the data in the candidate data set. Optionally, the feature vector of the data in the candidate data set is an embedding vector of the data in the candidate data set. If the feature vector of the data in the candidate data set matches the embedding vector of the second data, it means that the information of the data in the candidate data set matches the information carried by the embedding vector of the second data, and the data in the candidate data set can be used as the second data. Therefore, after acquiring the candidate data set, the generating device determines the data having the feature vector matching the embedding vector of the second data from the candidate data set to obtain the second data.

[0088] In a possible implementation, the candidate data set also includes a feature vector of the data in the candidate data set, for example, if the candidate data set is an image, the data in the candidate data set is an image, and the candidate data set also includes a feature vector of the image. Optionally, in the candidate data set, there is an index between the data and the feature vector, for example, the candidate data set includes image 1 and image 2, wherein there is an index between image 1 and the feature vector of image 1, and there is an index between image 2 and the feature vector of image 2.

[0089] Optionally, the feature vector of the data in the candidate data set is obtained by encoding the data in the candidate data set using an encoder of the second modality, wherein the encoder of the second modality is used to encode the data of the second modality. For example, the data in the candidate data set is an image, and the encoder of the second modality is an image encoder. The image in the candidate data set is encoded using an image encoder to obtain a feature vector of the image in the candidate data set, wherein the image encoder includes: an image encoder of a contrastive language–image pre-training (CLIP) model, an encoder of a variational auto-encoder (VAE), and a convolutional neural network.

[0090] As an optional implementation, the target generation model and the encoder of the second modality are trained together through an end-to-end training method. Optionally, a training prompt word is obtained, wherein the training prompt word is generated based on the training multimodal data, the training multimodal data includes the second training data of the second modality, and the training prompt word includes a description of generating the training multimodal data. The training prompt word is input into the first generation model to generate second intermediate data, wherein the second intermediate data includes the first predicted data of the first modality and the first predicted data of the second modality. The encoder of the second modality is used to extract the embedding vector of the second training data in the training multimodal data. Based on the first difference between the embedding vector of the second training data and the first training embedding vector, the training loss is determined, wherein the first difference is negatively correlated with the training loss. Based on the training loss, the parameters of the target generation model and the parameters of the encoder of the second modality are updated until the training loss converges and the training is completed. By training the target generation model and the encoder of the second modality in an end-to-end training manner, the embedding vector of the second data generated by the target generation model can be aligned with the embedding vector of the second modality data extracted by the encoder of the second modality, which is facilitating the subsequent determination of whether the embedding vector of the second data generated by the target generation model matches the embedding vector of the second modality data extracted by the encoder of the second modality.

[0091] Optionally, in the case where the candidate data set includes data and an embedding vector of the data, the generating device calculates the similarity between the embedding vector of the second data and the embedding vector of the data in the candidate data set, and determines the second data based on the data corresponding to the largest m similarities. Optionally, any one of the data corresponding to the largest m similarities is selected as the second data, or the data corresponding to the largest m similarities is selected as the second data.

[0092] In this embodiment, the modality of the data in the candidate data set is all the second modality. Therefore, after acquiring the candidate data set, the generating device determines the second data based on the data in the candidate data set having a feature vector that matches the embedded vector of the second data, thereby determining the second data based on the embedded vector of the second data.

[0093] See also Figure 6 , Figure 6 A flowchart of another multimodal data generation method provided in an embodiment of the present application. Figure 6 In the example, the target prompt word is: Please describe the snow scene with pictures and texts, wherein the target prompt word includes a description of generating target multimodal data including images and texts, and the second data in the target multimodal data is an image. Figure 6 As shown, after the target prompt word is input into the target generation model, the first intermediate data output by the target generation model is: snow scene refers to the natural scenery when it snows or after snowing [image] [embedded vector 1 of image 1], according to the form of snow, the snow scene can be divided into drifting snow [image] [embedded vector 2 of image 2], accumulated snow [image] [embedded vector N of image N]..., where [image] is the identifier of the embedded vector, and the embedded vector 1 of image 1, the embedded vector 2 of image 2, and the embedded vector N of image N are all embedded vectors. After the first intermediate data is generated, the embedded vector 1 of image 1, the embedded vector 2 of image 2, and the embedded vector N of image N can be determined from the first intermediate data based on [image] (i.e., the identifier of the embedded vector), and then the embedded vector retrieval can be performed based on the embedded vector 1 of image 1, the embedded vector 2 of image 2, and the embedded vector N of image N.

[0094] like Figure 6 As shown, the object of embedding vector retrieval is a candidate data set, wherein the candidate data set is obtained by encoding the images in the candidate data set using an image encoder, that is, the candidate data set includes images and embedding vectors of the images obtained by encoding. Embedding vector retrieval includes: using embedding vector 1 of image 1 to retrieve the candidate data set, and obtaining an image with an embedding vector matching embedding vector 1 as image 1, using embedding vector 2 of image 2 to retrieve the candidate data set, and obtaining an image with an embedding vector matching embedding vector 2 as image 2, and using embedding vector 3 of image 3 to retrieve the candidate data set, and obtaining an image with an embedding vector matching embedding vector 3 as image 3. After obtaining image 1, image 2, and image 3 through embedding vector retrieval, the target multimodal data can be obtained by replacing [image][embedded vector 1 of image 1] in the first intermediate data with image 1, replacing [image][embedded vector 2 of image 2] with image 2, and replacing [image][embedded vector 3 of image 3] with image 3, wherein, as Figure 6As shown, the target multimodal data includes: snow scene refers to a natural scenery image 1 when it is snowing or after snowing. According to the shape of snow, the snow scene can be divided into a falling snow image 2, an accumulated snow image N...

[0095] In current technology, another commonly used method for generating multimodal data is to generate data in multimodal data in a generative way, wherein the generative way can be realized through generative artificial intelligence technology. For example, the prompt words of the language model include the following: Please describe the snow scene, and insert relevant images in appropriate places, replaced by the image label [image x]. After the prompt word is output to the language model, the language model generates a text describing the snow scene based on the prompt word, and leaves a position for inserting images related to the snow scene in the text, such as the text: Snow scene refers to the natural scenery when it snows or after snowing [image 1]. According to the form of snow, the snow scene can be divided into drifting snow [image 2] and accumulated snow [image N]. Then, image 1 matching "natural scenery when it snows or after snowing" is generated through a diffusion model, image 2 matching "drifting snow" is generated through a diffusion model, and image 3 matching "accumulated snow" is generated through a diffusion model. Finally, by replacing [Image 1] in the text with Image 1, [Image 2] in the text with Image 2, and [Image N] in the text with Image N, a text including both text and image modes can be obtained.

[0096] In this approach, the data in the multimodal data is generated in a generative manner, so the authenticity of the data is poor, which in turn leads to poor authenticity of the data in the multimodal data obtained by this approach. For example, the authenticity of the image generated by the diffusion model is poor, which in turn leads to poor authenticity of the image in the multimodal data.

[0097] Based on this, the embodiment of the present application also provides an optional implementation, specifically, the data in the candidate data set is real data. For example, when the data in the candidate data set is an image, the image in the candidate data set may be a photograph obtained by shooting. For another example, when the data in the candidate data set is a video, the video in the candidate data set may be a video obtained by shooting. For another example, when the data in the candidate data set is audio, the audio in the candidate data set may be audio obtained by collecting actual sounds. In this way, the second data determined by the generating device are all real data based on the data in the candidate data set having an embedded vector that matches the embedded vector of the second data, thereby improving the authenticity of the second data in the target multimodal data.

[0098] As an optional implementation, the generating device obtains the target prompt word by executing the following steps: 8001. Receive a generation request from a target object.

[0099] In an embodiment of the present application, a generation request is used to request the generation of target multimodal data. In one possible implementation, the target object inputs a request to search based on the target search term to the generation device, and the generation device uses the request to search based on the target search term as the generation request. In another possible implementation, the generation device can run a digital person or virtual character that has a conversation with the target object, and when it is determined based on the conversation content of the target object that the content of the conversation with the target object includes multimodal data, the generation request is generated based on the content of the conversation with the target object. For example, when the target object is talking to the digital person or virtual character, he mentioned: "It rained for many days, and it finally cleared up today." The generation device determines that the digital person or virtual character can respond: "Yes, the sky on a sunny day is particularly beautiful.", and gives an image including a sunny day. At this time, the content of the conversation with the target object includes both text (i.e. "Yes, the sky on a sunny day is particularly beautiful.") and an image (i.e. an image of a sunny day), that is, the content of the conversation includes multimodal data. Therefore, the content of the conversation between the generation device and the target object generates the generation request. In another possible implementation, the target object explicitly expects the generating device to generate multimodal data, and then inputs a generation request to the generating device, and accordingly, the generating device receives the generation request from the target object. For example, the target object expects the generating device to generate a text with both pictures and text, and then inputs a generation request to the generating device.

[0100] 8002. Generate a target prompt word based on the generation request.

[0101] Since the target generation model performs the task of generating target multimodal data under the guidance of the target prompt word, the generation device needs to generate the target prompt word based on the generation request. In one possible implementation, the generation request includes the requirements that the target multimodal data needs to meet, and the generation device generates the target prompt word based on the requirements that the target multimodal data needs to meet.

[0102] After generating the target prompt word, the generating device performs the following steps in the process of executing step 303: 8003. Determine data that matches the embedding vector of the second data as matching data.

[0103] The modality of the data is , and the information of the data matches the information carried by the embedding vector of the second data. In one possible implementation, the generating device generates matching data based on the embedding vector of the second data. Optionally, the generating device uses a deep learning model to process the embedding vector of the second data to generate matching data. In another possible implementation, the generating device uses the embedding vector of the second data to perform an embedding vector search on the candidate data set, and obtains data having an embedding vector that matches the embedding vector of the second data as matching data.

[0104] 8004. Convert the style of the matching data into a style that matches the features of the target object to obtain second data.

[0105] In the embodiment of the present application, the characteristics of the target object include: the hobbies of the target object, the age of the target object, the occupation of the target object, and the gender of the target object. Since different objects like different styles of data, the styles of data suitable for different objects are also different. For example, an object who likes sports may prefer sports-style data and is more suitable for sports-style data. Therefore, the generating device obtains the second data whose style matches the characteristics of the target object by executing step 8004, which can increase the probability that the target object likes the second data and the matching degree between the second data and the target object.

[0106] In one possible implementation, the generating device inputs the matching data and the features of the target object into a style conversion model, so that the style conversion model converts the style of the matching data into a style that matches the features of the target object, thereby obtaining the second data, wherein the style conversion model is used to convert the style of the data into a style that matches the style of the features of the object. For example, if the matching data is an image, and the style that matches the features of the target object is motion, then the style conversion model can convert the style of the image into a motion style, and accordingly, the second data includes an image with a motion style.

[0107] See also Figure 7 , Figure 7 A flowchart of another multimodal data generation method provided in an embodiment of the present application. Figure 7 In the multimodal data generation method shown, the first modality in the target multimodal data is text, and the second modality in the target multimodal data is an image.

[0108] 701. Obtain a target prompt word, wherein the target prompt word includes a description of generating target multimodal data, the target multimodal data includes first data of a first modality and second data of a second modality, the first modality is text, and the second modality is an image.

[0109] The implementation of this step can refer to the description of step 301, which will not be repeated here.

[0110] 702. Input the target prompt word into the target generation model to obtain first intermediate data generated by the target generation model, where the first intermediate data includes an embedding vector of the first data and the second data.

[0111] The generating device inputs the target prompt word into the target generating model, so that the target generating model can perform tasks in the target prompt word, specifically, can enable the target generating model to generate target multimodal data.

[0112] Optionally, the target generation model is a target language model. Since the target language model can generate text but cannot generate images, after the generation device inputs the target prompt word into the target language model, the target language model generates the first data. As for the second data in the target multimodal data, the target language model cannot generate it directly. However, the target language model has the ability to generate an embedding vector, so the target language model generates an embedding vector of the second data, which is used to characterize the second data to be generated. Therefore, the first intermediate data generated by the target language model includes the embedding vector of the second data, wherein the embedding vector of the second data carries information of the second data.

[0113] The implementation of this step can refer to the description of step 302, which will not be repeated here.

[0114] 703. Determine second data in the target multimodal data based on the embedding vector of the second data.

[0115] The implementation of this step can refer to the description of step 303, which will not be repeated here.

[0116] In an embodiment of the present application, the target prompt word includes a description of generating target multimodal data, and the target multimodal data includes first data whose modality is text and second data whose modality is image, that is, the first data is text and the second data is image. After acquiring the target prompt word, the generating device inputs the target prompt word into the target generation model, and the first intermediate data generated by the target generation model can be obtained, wherein the first intermediate data includes the embedding vectors of the first data and the second data. Finally, based on the embedding vector of the second data, the second data in the target multimodal data can be determined. Since the generating device determines the second data in the target multimodal data based on the embedding vector of the second data in the first intermediate data generated by the target generation model, and the maximum value of the number of embedding vectors of the second data that can be generated by the target generation model is more than the number of second data that can be processed by the target generation model, the generating device obtains the second data in the target multimodal data in this way, which can improve the efficiency of obtaining the second data. Thus, the efficiency of obtaining multimodal data including images and texts can be improved.

[0117] The embodiment of the present application also provides a model training method, through which the target generation model described above can be trained. The execution subject of the model training method in the embodiment of the present application is a model training device (hereinafter referred to as a training device), wherein the training device can be any electronic device that can execute the model training method. Optionally, the training device is one of the following: a computer, a server.

[0118] See also Figure 8 , Figure 8 A flowchart of a model training method provided in an embodiment of the present application.

[0119] 801. Acquire training multimodal data, where the training multimodal data includes first training data of a first modality and second training data of a second modality.

[0120] 802. Acquire training prompt words based on the training multimodal data, wherein the training prompt words include a description of generating the training multimodal data.

[0121] In an embodiment of the present application, the training prompt words can be used to guide the generation model to perform the tasks in the training prompt words. Specifically, the training prompt words include a description of generating training multimodal data, then the tasks in the training prompt words include generating training multimodal data, then the training prompt words can be used to guide the generation model to generate training multimodal data. The description of generating training multimodal data can be text that indicates the generation of training multimodal data. For example, the training prompt words include the following text: Please generate a copy with both text and images.

[0122] The training prompt words are generated based on the training multimodal data. Optionally, the training prompt words are generated based on the content of the training multimodal data. For example, if the training multimodal data includes text describing a snow scene and images related to the snow scene, then the training prompt words may include "please describe the snow scene with both text and images".

[0123] In an embodiment of the present application, the training multimodal data includes first training data of a first modality and second training data of a second modality. In one possible implementation, the first modality is text and the second modality is an image, and accordingly, the training multimodal data includes text and images. In another possible implementation, the first modality is an image and the second modality is a video, and accordingly, the training multimodal data includes an image and a video.

[0124] In an implementation method of obtaining a training prompt word, a training device receives a training prompt word input by a user through an input component.

[0125] In yet another implementation of acquiring the training prompt word, the training device receives the training prompt word sent by the user through the terminal.

[0126] 803. Input the training prompt word into the first generation model to obtain second intermediate data generated by the first generation model, wherein the second intermediate data includes a first training embedding vector of first prediction data of the first modality and second prediction data of the second modality.

[0127] In an embodiment of the present application, the first generation model is any model capable of generating embedding vectors of data of the first modality and data of the second modality. Optionally, the first generation model includes LLM. The second intermediate data generated by the first generation model includes the first predicted data of the first modality and the first training embedding vector of the second predicted data of the second modality, wherein the first training embedding vector is used to represent the second predicted data to be generated, and the first training embedding vector carries information of the second predicted data.

[0128] 804. Determine a first loss of the first generative model based on a first difference between the first training embedding vector and the embedding vector of second training data in the training multimodal data, wherein the first loss is positively correlated with the first difference.

[0129] Since the first training embedding vector is generated by the first generative model based on the training prompt words, and the training prompt words are used to instruct the first generative model to generate training multimodal data, the second training data in the training multimodal data can be used as supervisory information for the first training embedding vector. Specifically, the greater the difference between the first training embedding vector and the embedding vector of the second training data, the lower the accuracy of the embedding vector of the second modality data generated by the first generative model based on the training prompt words. Conversely, the smaller the difference between the first training embedding vector and the embedding vector of the second training data, the higher the accuracy of the embedding vector of the second modality data generated by the first generative model based on the training prompt words. Therefore, in the case where the difference between the first training embedding vector and the embedding vector of the second training data in the training multimodal data is the first difference, the training device determines the first loss of the first generative model based on the first difference, wherein the first loss is positively correlated with the first difference.

[0130] Optionally, the training device determines second prediction data in the second intermediate data based on the first training embedding vector, and then determines the first difference based on the difference between the second prediction data and the second training data in the training multimodal data.

[0131] As an optional implementation, the training device obtains the first loss by determining the contrastive learning loss between the first training embedding vector and the embedding vector of the second training data. Fig. 9 , Fig. 9 A schematic diagram of a flow chart for calculating contrastive learning loss provided in an embodiment of the present application. Fig. 9 As shown, the training prompt words are: Please describe the snow scene with pictures and texts, wherein the training prompt words include generating a description of training multimodal data including images and texts, and the second modality of the training multimodal data is an image. Fig. 9As shown, after the training prompt words are input into the first generation model, the second intermediate data output by the first generation model is: snow scene refers to the natural scenery when it snows or after snowing [image] [embedded vector 1 of image 1], according to the form of snow, the snow scene can be divided into drifting snow [image] [embedded vector 2 of image 2], accumulated snow [image] [embedded vector N of image N]..., where [image] is the identifier of the training embedding vector, and the embedding vector 1 of image 1, the embedding vector 2 of image 2, and the embedding vector N of image N are all first training embedding vectors. After the second intermediate data is generated, the following first training embedding vectors can be determined from the second intermediate data based on [image] (i.e., the identifier of the first training embedding vector): embedding vector 1 of image 1, embedding vector 2 of image 2, and embedding vector N of image N.

[0132] like Fig. 9 The training multimodal data includes: snow scene refers to a natural scenery image 1 when snowing or after snowing, and according to the form of snow, the snow scene can be divided into a falling snow image 2, an accumulated snow image 3..., wherein image 1, image 2, and image N are all second training data. After encoding image 1, image 2, and image N using an image encoder, an embedding vector of the second training data can be obtained.

[0133] After determining the embedding vector 1 of image 1, the embedding vector 2 of image 2, the embedding vector N of image N, and the embedding vector of the second training data, the comparative learning loss of the embedding vector 1 of image 1, the embedding vector 2 of image 2, the embedding vector N of image N, and the embedding vector of the second training data can be calculated.

[0134] As an optional embodiment, the training device determines a third difference between the first prediction data and the first training data. Based on the first difference and the third difference, a first loss is determined, wherein the first loss is positively correlated with both the first difference and the third difference. For example, if the first modality is text and the second modality is an image, then the first difference is the difference between the embedding vector of the image generated by the first generative model and the embedding vector of the image in the training multimodal data, and the third difference is the difference between the text generated by the first generative model and the text in the training multimodal data.

[0135] 805. Based on the first loss, update the parameters of the first generation model to obtain a target generation model.

[0136] The training device updates the parameters of the first generation model based on the first loss until the first loss converges to obtain the target generation model.

[0137] In an embodiment of the present application, after obtaining the training multimodal data, the model training device obtains training prompt words based on the training multimodal data, wherein the training prompt words include a description of generating the training multimodal data, and the training multimodal data include first training data of the first modality and second training data of the second modality. The training device then inputs the training prompt words into the first generation model to obtain the second intermediate data generated by the first generation model, wherein the second intermediate data includes the first training embedding vector of the first predicted data of the first modality and the second predicted data of the second modality. Then, based on the first difference between the first training embedding vector and the embedding vector of the second training data, the first loss of the first generation model is determined, wherein the first loss is positively correlated with the first difference. Finally, based on the first loss, the parameters of the first generation model are updated to obtain the target generation model, thereby enabling the target generation model to generate multimodal data including data of the first modality and an embedding vector for representing data of the second modality based on the description of generating multimodal data in the prompt words.

[0138] As an optional implementation, the training device obtains the embedding vector of the second training data by performing the following steps: extracting the embedding vector of the second training data of the second modality in the training multimodal data using the encoder of the second modality. In this way, the embedding vector of the data of the second modality generated by the first generation model can be aligned with the embedding vector of the data of the second modality extracted by the encoder of the second modality, while determining the second loss based on the first difference and updating the parameters of the first generation model based on the first loss.

[0139] As an optional implementation, the training device obtains the first generation model by performing the following steps: 9001. Based on the training multimodal data, obtain training input data and training target data including the multimodal data.

[0140] In an embodiment of the present application, the training input data or the training target data is at least a part of the training multimodal data. In one possible implementation, the training multimodal data includes training input data and training target data, and the training target data is the context of the training input data. Optionally, the training multimodal data includes images and texts, and the training input data and the training target data are the context in the training multimodal data. For example, a position is selected from a news article, and the news is divided into two parts, the upper text and the lower text, wherein the upper text is the training input data and the lower text is the training target data. Optionally, the corpus used to obtain the training input data and the training target data is downloaded from the Internet, for example, a document is downloaded from the Internet.

[0141] In another possible implementation, the training target data is training multimodal data, and the training input data is obtained by adding a mask to the training multimodal data. For example, the training input data is obtained by adding a mask to text in the training multimodal data. For another example, the training input data is obtained by adding a mask to an image in the training multimodal data.

[0142] The training target data includes third training data of the first modality and fourth training data of the second modality. For example, when the first modality is text and the second modality is image, the training target data includes text and image. For another example, when the first modality is audio and the second modality is video, the training target data includes audio and video.

[0143] 9002. Input the training input data into the second generation model so that the second generation model generates third intermediate data.

[0144] In an embodiment of the present application, the third intermediate data includes a second training embedding vector of third prediction data of the first modality and fourth prediction data of the second modality. In one possible implementation, the training target data is the context of the training input data. The training device inputs the training input data into the second generation model so that the second generation model predicts the context of the training input data, thereby generating the third intermediate data. In another possible implementation, the training target data is training multimodal data, and the training input data is obtained by adding a mask to the training multimodal data. The training device inputs the training input data into the second generation model so that the second generation model removes the mask from the training input data, thereby generating the third intermediate data.

[0145] 9003. Determine a second loss of the second generation model based on a second difference between the second training embedding vector and the embedding vector of fourth training data in the training target data, wherein the second loss is positively correlated with the second difference.

[0146] In the embodiment of the present application, the fourth training data in the training target data can be used as the GT of the second training embedding vector. Therefore, the greater the second difference between the embedding vector of the fourth training data in the training target data and the second training embedding vector, the lower the accuracy of the second training embedding vector in the third intermediate data generated by the second generation model, which also means that the embedding vector of the second modality data generated by the second generation model has a lower accuracy. Therefore, the training device determines the second loss of the second generation model according to the principle that the second loss is positively correlated with the second difference.

[0147] In one possible implementation, the model training device determines fourth prediction data of the second modality in the third intermediate data based on the second training embedding vector, and determines a second difference based on a difference between the fourth prediction data and fourth training data in the training target data.

[0148] As an optional implementation, the training device obtains the embedding vector of the fourth training data by performing the following steps: extracting the embedding vector of the fourth training data of the second modality in the training target data using the encoder of the second modality. In this way, the embedding vector of the data of the second modality generated by the second generation model can be aligned with the embedding vector of the data of the second modality extracted by the encoder of the second modality, while determining the second loss based on the second difference and updating the parameters of the second generation model based on the second loss.

[0149] 9004. Based on the second loss, update the parameters of the second generation model to obtain the first generation model.

[0150] The training device updates the parameters of the second generation model based on the second loss until the second loss converges to obtain the first generation model.

[0151] In this embodiment, after the training device obtains the training input data and the training target data based on the training multimodal data, the training input data is input into the second generation model so that the second generation model generates third intermediate data, wherein the third intermediate data includes a second training embedding vector of the fourth prediction data of the second modality. Based on the second difference between the second training embedding vector and the embedding vector of the fourth training data in the training target data, the second loss of the second generation model is determined, and the second loss is positively correlated with the second difference. Based on the second loss, the parameters of the second generation model are updated to obtain the first generation model. In this way, the first generation model can be equipped with the ability to generate multimodal data including the embedding vectors of the data of the first modality and the data of the second modality.

[0152] In the model training method provided in the embodiment of the present application, steps 9001 to 9004 are the pre-training stage, and steps 801 to 805 are the supervised fine-tuning stage, that is, the model training method includes a pre-training stage and a supervised fine-tuning stage. Since the training data used in the pre-training stage are training input data and training target data, and the number of available training input data and the number of training target data are large, for example, a document including multimodal data can be downloaded from the Internet, and the training input data and training target data can be obtained based on the document. Therefore, the amount of training data in the pre-training stage is sufficient, which can improve the training effect of the pre-training stage, that is, the first generation model obtained through the pre-training stage can have a strong ability to generate multimodal data including the embedding vector of the data of the first modality and the data of the second modality. In this way, in the supervised fine-tuning stage, the first generation model is trained by a small amount of training prompt words to obtain a target generation model, so that the target generation model can generate a description of the multimodal data generated in the prompt words, and generate multimodal data including the data of the first modality and the embedding vector for representing the data of the second modality.

[0153] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0154] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0155] See also Fig.10 , Fig.10 A schematic diagram of the structure of a multimodal data generating device provided in an embodiment of the present application. The multimodal data generating device 1 comprises: an acquisition unit 11, an obtaining unit 12, a determination unit 13, and optionally, the multimodal data generating device 1 further comprises: a replacement unit 14, wherein: An acquisition unit 11 is used to acquire a target prompt word, wherein the target prompt word includes a description of generating target multimodal data, wherein the target multimodal data includes first data of a first modality and second data of a second modality, wherein the first modality is different from the second modality; An obtaining unit 12 is used to input the target prompt word into a target generation model to obtain first intermediate data generated by the target generation model, where the first intermediate data includes embedding vectors of the first data and the second data; A determining unit 13 is used to determine the second data in the target multimodal data based on the embedding vector.

[0156] In combination with any embodiment of the present application, the multimodal data generating device 1 further includes: The replacement unit 14 is configured to obtain the target multimodal data by replacing the embedding vector of the second data in the first intermediate data with the second data.

[0157] In combination with any embodiment of the present application, the determining unit 13 is further used to determine the source of the second data; The replacement unit 14 is further configured to obtain the target multimodal data by replacing the embedding vector of the second data in the first intermediate data with the second data, and by adding the source of the second data to the first intermediate data.

[0158] In combination with any embodiment of the present application, the second data includes two or more third data, and the two or more third data are all related to the same content in the first intermediate data; The replacement unit 14 is further configured to obtain the target multimodal data by replacing the embedding vector of the second data in the first intermediate data with the second data, and by adding a description of the difference between the different third data in the first intermediate data.

[0159] In combination with any embodiment of the present application, the acquisition unit 11 is further used to acquire a candidate data set, and the modality of the data in the candidate data set is the second modality; The determining unit 13 is used to determine the second data based on data in the candidate data set having an embedded vector that matches a feature vector of the second data.

[0160] In combination with any embodiment of the present application, the acquisition unit 11 is used to: Receiving a generation request from a target object, wherein the generation request is used to request generation of the target multimodal data; generating the target prompt word based on the generation request; The determining unit 13 is used to: Determine that second modality data matching the embedding vector of the second data is matching data, and the modality of the second modality data is the second modality; The style of the matching data is converted into a style that matches the features of the target object to obtain the second data.

[0161] In combination with any embodiment of the present application, the first intermediate data also includes an identifier for determining the embedding vector from the first intermediate data, and the determination unit 13 is further used to determine the embedding vector from the first intermediate data based on the identifier.

[0162] In an embodiment of the present application, the target prompt word includes a description of generating target multimodal data, and the target multimodal data includes second data of modality. After acquiring the target prompt word, the generating device inputs the target prompt word into the target generation model, and can obtain the first intermediate data generated by the target generation model, wherein the first intermediate data includes the embedding vectors of the first data and the second data. Finally, based on the embedding vector of the second data, the second data in the target multimodal data can be determined. Since the generating device determines the second data in the target multimodal data based on the embedding vector of the second data in the first intermediate data generated by the target generation model, and the maximum value of the number of embedding vectors of the second data that can be generated by the target generation model is more than the number of second data that can be processed by the target generation model, the generating device obtains the second data in the target multimodal data in this way, which can improve the efficiency of obtaining the second data.

[0163] See also Fig.11 , Fig.11A schematic diagram of the structure of another multimodal data generating device provided in an embodiment of the present application. The multimodal data generating device 2 comprises: an acquisition unit 21, an obtaining unit 22, and a determination unit 23, wherein: An acquisition unit 21 is used to acquire a target prompt word, wherein the target prompt word includes a description of generating target multimodal data, wherein the target multimodal data includes first data of a first modality and second data of a second modality, wherein the first modality is an image and the second modality is a text; An obtaining unit 22, configured to input the target prompt word into a target generation model to obtain first intermediate data generated by the target generation model, wherein the first intermediate data includes embedding vectors of the first data and the second data; A determining unit 23 is used to determine the second data in the target multimodal data based on the embedding vector.

[0164] In an embodiment of the present application, the target prompt word includes a description of generating target multimodal data, and the target multimodal data includes first data whose modality is text and second data whose modality is image, that is, the first data is text and the second data is image. After acquiring the target prompt word, the generating device inputs the target prompt word into the target generation model, and the first intermediate data generated by the target generation model can be obtained, wherein the first intermediate data includes the embedding vectors of the first data and the second data. Finally, based on the embedding vector of the second data, the second data in the target multimodal data can be determined. Since the generating device determines the second data in the target multimodal data based on the embedding vector of the second data in the first intermediate data generated by the target generation model, and the maximum value of the number of embedding vectors of the second data that can be generated by the target generation model is more than the number of second data that can be processed by the target generation model, the generating device obtains the second data in the target multimodal data in this way, which can improve the efficiency of obtaining the second data. Thus, the efficiency of obtaining multimodal data including images and texts can be improved.

[0165] See also Fig.12 , Fig.12 A schematic diagram of the structure of a model training device provided in an embodiment of the present application. The model training device 3 includes: an acquisition unit 31, an acquisition unit 32, a determination unit 33, and an update unit 34, wherein: An acquiring unit 31 is used to acquire training multimodal data, where the training multimodal data includes first training data of a first modality and second training data of a second modality; The acquisition unit 31 is further used to acquire a training prompt word based on the training multimodal data, wherein the training prompt word includes a description of generating the training multimodal data; an obtaining unit 32, configured to input the training prompt word into a first generation model to obtain second intermediate data generated by the first generation model, wherein the second intermediate data includes first prediction data of the first modality and a first training embedding vector of second prediction data of the second modality; A determining unit 33, configured to determine a first loss of the first generative model based on a first difference between the first training embedding vector and an embedding vector of second training data in the training multimodal data, wherein the first loss is positively correlated with the first difference; An updating unit 34 is used to update the parameters of the first generation model based on the first loss to obtain a target generation model.

[0166] In combination with any embodiment of the present application, the acquisition unit 31 is further used to acquire training input data and training target data based on the training multimodal data, where the training input data or the training target data is at least a part of the training multimodal data, and the training target data includes third training data of the first modality and fourth training data of the second modality; The obtaining unit 32 is further configured to input the training input data into a second generation model so that the second generation model generates third intermediate data, wherein the third intermediate data includes a second training embedding vector of third prediction data of the first modality and fourth prediction data of the second modality; The determining unit 33 is further configured to determine a second loss of the second generation model based on a second difference between the second training embedding vector and the embedding vector of the fourth training data in the training target data, wherein the second loss is positively correlated with the second difference; The updating unit 34 is further configured to update the parameters of the second generation model based on the second loss to obtain the first generation model.

[0167] In combination with any embodiment of the present application, the acquisition unit 31 is further used for: Extracting an embedding vector of second training data of a second modality in the training multimodal data using an encoder of the second modality; and / or The encoder of the second modality is used to extract the embedding vector of the fourth training data of the second modality in the training target data.

[0168] In an embodiment of the present application, after obtaining the training multimodal data, the model training device obtains training prompt words based on the training multimodal data, wherein the training prompt words include a description of generating the training multimodal data, and the training multimodal data include first training data of the first modality and second training data of the second modality. The training device then inputs the training prompt words into the first generation model to obtain the second intermediate data generated by the first generation model, wherein the second intermediate data includes the first training embedding vector of the first predicted data of the first modality and the second predicted data of the second modality. Then, based on the first difference between the first training embedding vector and the embedding vector of the second training data, the first loss of the first generation model is determined, wherein the first loss is positively correlated with the first difference. Finally, based on the first loss, the parameters of the first generation model are updated to obtain the target generation model, thereby enabling the target generation model to generate multimodal data including data of the first modality and an embedding vector for representing data of the second modality based on the description of generating multimodal data in the prompt words.

[0169] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0170] Fig.13 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 4 includes a processor 41 and a memory 42. Optionally, the electronic device 4 also includes an input device 43 and an output device 44. The processor 41, the memory 42, the input device 43 and the output device 44 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0171] The processor 41 may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Optionally, the processor 41 may be a processor group consisting of multiple CPUs, and the multiple processors are coupled to each other through one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0172] The memory 42 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0173] The input device 43 is used to input data and / or signals, and the output device 44 is used to output data and / or signals. The input device 43 and the output device 44 can be independent devices or an integrated device.

[0174] It can be understood that in the embodiment of the present application, the memory 42 can be used not only to store relevant instructions, but also to store relevant data. The embodiment of the present application does not limit the specific data stored in the memory.

[0175] Understandably, Fig.13 Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0176] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0177] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0178] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0181] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0182] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A method for generating multimodal data, characterized in that: The method comprises: Acquire a target prompt word, the target prompt word including a description of generating target multimodal data, the target multimodal data including first data of a first modality and second data of a second modality, the first modality being different from the second modality; Inputting the target prompt word into a target generation model to obtain first intermediate data generated by the target generation model, wherein the first intermediate data includes embedding vectors of the first data and the second data; Based on the embedding vector, the second data in the target multimodal data is determined.

2. The method according to claim 1, characterized in that After determining the second data in the target multimodal data based on the embedding vector, the method further includes: The target multimodal data is obtained by replacing the embedding vector of the second data in the first intermediate data with the second data.

3. The method according to claim 2, characterized in that Before obtaining the target multimodal data by replacing the embedding vector of the second data in the first intermediate data with the second data, the method further includes: determining a source of the second data; The step of replacing the embedding vector of the second data in the first intermediate data with the second data to obtain the target multimodal data includes: The target multimodal data is obtained by replacing the embedding vector of the second data in the first intermediate data with the second data and adding the source of the second data to the first intermediate data.

4. The method according to claim 2, characterized in that: The second data includes two or more third data, and the two or more third data are all related to the same content in the first intermediate data; The step of replacing the embedding vector of the second data in the first intermediate data with the second data to obtain the target multimodal data includes: The target multimodal data is obtained by replacing the embedding vector of the second data in the first intermediate data with the second data, and adding a description of the difference between the different third data in the first intermediate data.

5. The method according to any one of claims 1 to 4, characterized in that: Before determining the second data in the target multimodal data based on the embedding vector, the method further includes: Acquire a candidate data set, wherein the modality of data in the candidate data set is all the second modality; The determining, based on the embedding vector, the second data in the target multimodal data comprises: The second data is determined based on data in the candidate data set having a feature vector matching the embedding vector of the second data.

6. The method according to any one of claims 1 to 4, characterized in that The step of obtaining the target prompt word includes: Receiving a generation request from a target object, wherein the generation request is used to request generation of the target multimodal data; generating the target prompt word based on the generation request; The determining, based on the embedding vector, the second data in the target multimodal data comprises: Determine that second modality data matching the embedding vector of the second data is matching data, and the modality of the second modality data is the second modality; The style of the matching data is converted into a style that matches the features of the target object to obtain the second data.

7. The method according to any one of claims 1 to 4, characterized in that The first intermediate data further includes an identifier for determining the embedding vector from the first intermediate data. Before determining the second data in the target multimodal data based on the embedding vector, the method further includes: The embedding vector is determined from the first intermediate data based on the identification.

8. A method for generating multimodal data, characterized in that: The method comprises: Acquire a target prompt word, wherein the target prompt word includes a description of generating target multimodal data, wherein the target multimodal data includes first data of a first modality and second data of a second modality, wherein the first modality is text and the second modality is an image; Inputting the target prompt word into a target generation model to obtain first intermediate data generated by the target generation model, wherein the first intermediate data includes embedding vectors of the first data and the second data; Based on the embedding vector, the second data in the target multimodal data is determined.

9. A model training method, characterized in that: The method comprises: Acquire training multimodal data, where the training multimodal data includes first training data of a first modality and second training data of a second modality; Based on the training multimodal data, obtaining training prompt words, wherein the training prompt words include a description of generating the training multimodal data; Inputting the training prompt word into a first generation model to obtain second intermediate data generated by the first generation model, wherein the second intermediate data includes first prediction data of the first modality and a first training embedding vector of second prediction data of the second modality; determining a first loss of the first generative model based on a first difference between the first training embedding vector and an embedding vector of second training data in the training multimodal data, the first loss being positively correlated with the first difference; Based on the first loss, the parameters of the first generation model are updated to obtain a target generation model.

10. The method according to claim 9, characterized in that The method further comprises: Based on the training multimodal data, acquiring training input data and training target data, wherein the training input data or the training target data is at least a part of the training multimodal data, and the training target data includes third training data of the first modality and fourth training data of the second modality; Inputting the training input data into a second generative model so that the second generative model generates third intermediate data, wherein the third intermediate data includes a second training embedding vector of third predicted data of the first modality and fourth predicted data of the second modality; Determining a second loss of the second generative model based on a second difference between the second training embedding vector and the embedding vector of the fourth training data in the training target data, wherein the second loss is positively correlated with the second difference; Based on the second loss, update the parameters of the second generation model to obtain the first generation model.

11. The method according to claim 10, characterized in that The method further comprises: Extracting an embedding vector of second training data of a second modality in the training multimodal data using an encoder of the second modality; and / or The encoder of the second modality is used to extract the embedding vector of the fourth training data of the second modality in the training target data.

12. A multimodal data generating device, characterized in that: The multimodal data generating device comprises: an acquisition unit, configured to acquire a target prompt word, wherein the target prompt word includes a description of generating target multimodal data, wherein the target multimodal data includes first data of a first modality and second data of a second modality, wherein the first modality is an image and the second modality is a text; an obtaining unit, configured to input the target prompt word into a target generation model to obtain first intermediate data generated by the target generation model, wherein the first intermediate data includes embedding vectors of the first data and the second data; A determining unit is used to determine the second data in the target multimodal data based on the embedding vector.

13. A multimodal data generating device, characterized in that: The multimodal data generating device comprises: an acquisition unit, configured to acquire a target prompt word, wherein the target prompt word includes a description of generating target multimodal data, wherein the target multimodal data includes first data of a first modality and second data of a second modality, wherein the first modality is text and the second modality is an image; an obtaining unit, configured to input the target prompt word into a target generation model to obtain first intermediate data generated by the target generation model, wherein the first intermediate data includes embedding vectors of the first data and the second data; A determining unit is used to determine the second data in the target multimodal data based on the embedding vector.

14. A model training device, characterized in that: The model training device comprises: An acquisition unit, configured to acquire a training prompt word and training multimodal data, wherein the training prompt word includes a description of generating the training multimodal data, the training multimodal data includes third data of a modality , and the training multimodal data includes a true value of the third data; an obtaining unit, configured to input the training prompt word into a first generation model, and obtain second intermediate data generated by the first generation model, wherein the second intermediate data includes a training embedding vector, and the training embedding vector includes an embedding vector of the third data; a determining unit, configured to determine a first loss of the first generation model based on a first difference between the training embedding vector and the true value, wherein the first loss is positively correlated with the first difference; An updating unit is used to update the parameters of the first generation model based on the first loss to obtain a target generation model.

15. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions. When the processor executes the computer instructions, the electronic device executes the method as claimed in any one of claims 1 to 7, or the electronic device executes the method as claimed in claim 8, or the electronic device executes the method as claimed in any one of claims 9 to 11.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method described in any one of claims 1 to 7, or the processor is caused to execute the method described in claim 8, or the processor is caused to execute the method described in any one of claims 9 to 11.

17. A computer program product, characterized in that The computer program product includes a computer program or instructions; when the computer program or instructions are run on a computer, the computer is enabled to execute the method described in any one of claims 1 to 7, or the computer is enabled to execute the method described in claim 8, or the computer is enabled to execute the method described in claim 9.

Images