Multi-modal large model training method and device and electronic equipment

Abstract

The invention provides a multi-modal large model training method and device and electronic equipment, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, natural language processing, computer vision, voice technology, large models and the like. According to the specific implementation scheme, first training data and second training data are obtained; obtaining an initial multi-modal large model; the multi-mode large model comprises a backbone network and coding and decoding networks corresponding to all non-text modes; each coding and decoding network is combined with the same multi-mode word list to carry out coding and decoding processing; performing joint training processing on each coding and decoding network and the multi-mode word list in combination with data in each non-text mode in the first training data; performing training processing on the backbone network in combination with multi-modal sample reference data and sample generation data under the target task in the second training data; wherein each coding and decoding network is combined with the same multi-mode word list to carry out coding and decoding processing, so that the model training difficulty and the model training cost are reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning, natural language processing, computer vision, speech technology, and large models, and in particular to a training method, device, and electronic device for a multimodal large model. Background Art

[0002] Current multimodal large models, such as video generation models, include an encoding network, a backbone network, and a decoding network. Among them, there are three encoding networks, namely, encoding networks for video, text, and audio modes.

[0003] In the above video generation model, the encoding networks of different modes use different modal vocabularies and need to be trained separately using different modal data. This means that during the training process of the video generation model, the vocabularies of different modalities need to be trained separately, which increases the difficulty and cost of model training. Summary of the invention

[0004] The present invention provides a method, device and electronic device for training a multimodal large model.

[0005] According to one aspect of the present disclosure, a training method for a multimodal large model is provided, the method comprising: obtaining first training data and second training data; the first training data comprises data under each non-text modality; the second training data comprises multimodal sample reference data and sample generation data under a target task; obtaining an initial multimodal large model; the multimodal large model comprises a backbone network and a codec network corresponding to each non-text modality; each of the codec networks performs codec processing in combination with the same multimodal vocabulary; in combination with the data under each non-text modality, each of the codec networks and the multimodal vocabulary is jointly trained; when the training of each of the codec networks and the multimodal vocabulary is completed, the backbone network is trained in combination with the multimodal sample reference data and sample generation data under the target task.

[0006] According to another aspect of the present disclosure, a method for processing a target task is provided, the method comprising: obtaining the target task; the target task comprises data under at least two modalities; obtaining a multimodal large model; the multimodal large model is determined based on the training method of the multimodal large model as described above; inputting the data under at least two modalities into the multimodal large model, and obtaining generated data output by the multimodal large model.

[0007] According to another aspect of the present disclosure, a training device for a multimodal large model is provided, the device comprising: a first acquisition module, used to acquire first training data and second training data; the first training data comprises data under each non-text modality; the second training data comprises multimodal sample reference data and sample generation data under a target task; a second acquisition module, used to acquire an initial multimodal large model; the multimodal large model comprises a backbone network and codec networks corresponding to each non-text modality; each of the codec networks performs codec processing in combination with the same multimodal vocabulary; a first training processing module, used to perform joint training processing on each of the codec networks and the multimodal vocabulary in combination with data under each non-text modality; a second training processing module, used to train the backbone network in combination with the multimodal sample reference data and sample generation data under the target task when the training of each of the codec networks and the multimodal vocabulary is completed.

[0008] According to another aspect of the present disclosure, a device for processing a target task is provided, the device comprising: a first acquisition module, used to acquire the target task; the target task comprises data under at least two modalities; a second acquisition module, used to acquire a multimodal large model; the multimodal large model is determined based on the training method of the multimodal large model as described above; a third acquisition module, used to input the data under the at least two modalities into the multimodal large model, and acquire generated data output by the multimodal large model.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the multimodal large model proposed above in the present disclosure; or, execute the processing method of the target task proposed above in the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the training method of the multimodal large model proposed above in the present disclosure; or, execute the processing method of the target task proposed above in the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the method for training a multimodal large model proposed above in the present disclosure; or, implements the steps of the method for processing the target task proposed above in the present disclosure.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0017] Figure 4 This is a schematic diagram of the training of each encoding and decoding network in a multimodal large model;

[0018] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0020] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0021] Figure 8 It is a block diagram of an electronic device used to implement the training method of a multimodal large model or the processing method of a target task of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] Current multimodal large models, such as video generation models, include an encoding network, a backbone network, and a decoding network. Among them, there are three encoding networks, namely, encoding networks for video, text, and audio modes.

[0024] In the above video generation model, the encoding networks of different modes use different modal vocabularies and need to be trained separately using different modal data. This means that during the training process of the video generation model, the vocabularies of different modalities need to be trained separately, which increases the difficulty and cost of model training.

[0025] In order to solve the above problems, the present invention provides a method, device and electronic device for training a multimodal large model.

[0026] Figure 1 It is a schematic diagram according to the first embodiment of the present disclosure. It should be noted that the training method of the multimodal large model of the embodiment of the present disclosure can be applied to the training device of the multimodal large model, and the device can be configured in an electronic device so that the electronic device can perform the training function of the multimodal large model. In the following embodiments, the execution subject is an electronic device as an example for explanation.

[0027] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0028] The training device for the multimodal large model may also be software in an electronic device, such as training software for the multimodal large model, etc. In the following embodiments, the training device for the multimodal large model is an electronic device as an example for description.

[0029] like Figure 1 As shown, the training method of the multimodal large model may include the following steps:

[0030] Step 101, obtaining first training data and second training data; the first training data includes data in various non-text modes; the second training data includes multimodal sample reference data and sample generation data under the target task.

[0031] In the disclosed embodiment, the non-text modality includes at least one of the following: audio modality, silent video modality, and image modality. Correspondingly, the data in the non-text modality is, for example, audio, silent video, image, etc. Among them, for audio and video, it can be split into two types of data in non-text modality. For example, audio and video can be split into data in audio modality and data in video modality.

[0032] Among them, the setting of multiple non-text modes enables the multimodal large model to flexibly process data under different text modes, thereby expanding the use scenarios of the multimodal large model and improving the processing efficiency of the multimodal large model.

[0033] In the embodiments of the present disclosure, the target task may include at least one of the following: an image generation task, a video generation task, an audio generation task, a text generation task, and a multimodal understanding task.

[0034] The multimodal sample reference data in each target task may include, for example, sample reference data in at least one of the following modes: text mode, image mode, video mode, and audio mode. The multimodal understanding task refers to a task of understanding the input multimodal data to output the understood text. When the understanding process is performed, the input multimodal data may include data in at least one of the following modes: text mode, image mode, video mode, and audio mode.

[0035] Among them, the setting of multimodal sample reference data and sample generation data under multiple target tasks enables the multimodal large model to be combined with training data to train models suitable for different target tasks, thereby expanding the task application scenarios of the multimodal large model and further improving the processing efficiency of the multimodal large model.

[0036] Step 102, obtaining an initial multimodal large model; the multimodal large model includes a backbone network and encoding and decoding networks corresponding to each non-text modality; each encoding and decoding network performs encoding and decoding processing in combination with the same multimodal vocabulary.

[0037] In the embodiments of the present disclosure, there may be multiple codec networks, for example, a one-dimensional codec network, a two-dimensional codec network, a three-dimensional codec network, etc. Among them, taking the non-text modality including audio modality, silent video modality, and image modality as an example, the codec network corresponding to the audio modality may be a one-dimensional codec network; the codec network corresponding to the image modality may be a two-dimensional codec network; and the codec network corresponding to the video modality may be a three-dimensional codec network.

[0038] Each codec network uses the same multimodal vocabulary. The multimodal vocabulary may include multiple integer identifiers and vectors corresponding to each integer identifier. A vector in the multimodal vocabulary may represent an image, audio, and video at the same time. In other words, a vector in the multimodal vocabulary may be a vector of an image block in an image, or a vector of an audio segment in an audio, or a vector of a video block in a video.

[0039] Among them, different encoding and decoding networks are used to process data in different non-text modes, so as to extract features in the data in the non-text mode as much as possible and improve the accuracy of the extracted features.

[0040] In the embodiments of the present disclosure, taking a two-dimensional codec network as an example, a method for processing an image in an image modality in combination with a coding network in the two-dimensional codec network includes: inputting the image into the coding network in the two-dimensional codec network to obtain two-dimensional image features; performing one-dimensional deformation processing on the two-dimensional image features to obtain a deformed one-dimensional feature vector; and mapping the features in the one-dimensional feature vector in combination with a multimodal vocabulary to obtain an integer sequence corresponding to the image.

[0041] Among them, taking a one-dimensional codec network as an example, the method of processing audio in an audio modality in combination with the coding network in the one-dimensional codec network includes: inputting the audio into the coding network in the one-dimensional codec network to obtain a one-dimensional audio feature; wherein the one-dimensional audio feature is a one-dimensional feature vector; mapping the features in the one-dimensional feature vector in combination with a multimodal vocabulary to obtain an integer sequence corresponding to the audio.

[0042] Among them, taking the three-dimensional codec network as an example, the method of processing the video under the video modality in combination with the coding network in the three-dimensional codec network includes: inputting the video into the coding network in the three-dimensional codec network to obtain the three-dimensional video features; performing one-dimensional deformation processing on the three-dimensional video features to obtain the deformed one-dimensional feature vector; combining the multimodal vocabulary to map the features in the one-dimensional feature vector to obtain the integer sequence corresponding to the video.

[0043] Among them, after the encoding network in each dimensional encoding and decoding network encodes the data, it can perform one-dimensional deformation processing on the encoded features, so that the data under multiple modes can all be determined to obtain one-dimensional feature vectors, so that feature mapping processing can be realized through the same multimodal vocabulary, thereby realizing the interaction between the features of each modality, thereby further improving the feature extraction accuracy of the encoding network in each dimensional encoding and decoding network.

[0044] Step 103 , combining the data in each non-text mode, and jointly training each encoding and decoding network and the multimodal vocabulary.

[0045] In the disclosed embodiment, the electronic device can combine the data in each non-text mode and each codec network to determine the total loss function value of each codec network; then, combined with the total loss function value, the parameters of each codec network and the multimodal vocabulary are adjusted and processed simultaneously to achieve joint training.

[0046] Step 104, when the training of each codec network and the multimodal vocabulary is completed, the backbone network is trained in combination with the multimodal sample reference data and sample generation data under the target task.

[0047] In the embodiment of the present disclosure, the backbone network may be, for example, an autoregressive model, which is a time series analysis model that uses a linear combination of past values ​​of the same variable plus a random error term to predict the future value of the variable.

[0048] Among them, the electronic device can combine multimodal sample reference data, the encoding network corresponding to each non-text modality and the backbone network to determine the integer sequence or integer sequence combination corresponding to the predicted generated data; determine the modality to which the sample generated data belongs; combine the sample generated data and the encoding network in the encoding and decoding network corresponding to the modality to determine the integer sequence or integer sequence combination corresponding to the sample generated data; determine the value of the loss function based on the integer sequence or integer sequence combination corresponding to the predicted generated data and the integer sequence or integer sequence combination corresponding to the sample generated data; and then adjust the parameters of the backbone network to achieve training.

[0049] The training method of the multimodal large model of the embodiment of the present disclosure is through obtaining first training data and second training data; the first training data includes data under each non-text mode; the second training data includes multimodal sample reference data and sample generation data under the target task; obtaining an initial multimodal large model; the multimodal large model includes a backbone network and a codec network corresponding to each non-text mode; each codec network performs codec processing in combination with the same multimodal vocabulary; in combination with the data under each non-text mode, each codec network and the multimodal vocabulary are jointly trained; when the training of each codec network and the multimodal vocabulary is completed, the backbone network is trained in combination with the multimodal sample reference data and sample generation data under the target task; wherein, each codec network performs codec processing in combination with the same multimodal vocabulary, and the joint training processing can avoid the separate training processing of the vocabulary of different modes during the training process, thereby reducing the difficulty and cost of model training.

[0050] In order to further improve the training speed of the codec network corresponding to each non-text modality, the loss function value of each codec network can be determined separately by combining the data under each non-text modality; then, the loss function values ​​are combined to adjust each codec network and multimodal vocabulary. Figure 2 As shown, Figure 2 is a schematic diagram according to a second embodiment of the present disclosure, Figure 2 The illustrated embodiment may include the following steps:

[0051] Step 201, obtaining first training data and second training data; the first training data includes data in various non-text modes; the second training data includes multimodal sample reference data and sample generation data under the target task.

[0052] Step 202, obtaining an initial multimodal large model; the multimodal large model includes a backbone network and encoding and decoding networks corresponding to each non-text modality; each encoding and decoding network performs encoding and decoding processing in combination with the same multimodal vocabulary.

[0053] Step 203 , combining the data in each non-text mode and the codec network corresponding to each non-text mode, to determine the loss function value of the codec network corresponding to each non-text mode.

[0054] In the embodiment of the present disclosure, the process of the electronic device executing step 203 may, for example, be to determine the predicted data corresponding to each data by combining the data under each non-text modality and the codec network corresponding to each non-text modality; for each non-text modality, determine the true or false discrimination result by combining the data under the non-text modality, the predicted data corresponding to the data, and the discrimination network in the codec network corresponding to the non-text modality; determine the loss function value of the codec network corresponding to the non-text modality by combining at least one of the true or false discrimination result, the difference between the data and the predicted data, and the difference in features between the data and the predicted data.

[0055] In an embodiment of the present disclosure, when there are no at least two candidate data with an associated relationship in the data under each non-text modality, the process for the electronic device to determine the predicted data corresponding to each data may be, for example, inputting the data under the non-text modality into the codec network corresponding to the non-text modality for each non-text modality in turn, and obtaining the predicted data corresponding to the data under the non-text modality.

[0056] Specifically, for each non-text mode, the codec network corresponding to the non-text mode may include an encoding network and a decoding network; the electronic device may combine the data in the non-text mode and the encoding network to determine the integer sequence corresponding to the data; and combine the integer sequence and the decoding network to determine the predicted data corresponding to the data.

[0057] Specifically, the electronic device may input the data in the non-text mode into the encoding network to obtain output data features; when the data features are multi-dimensional feature vectors, the data features may be subjected to one-dimensional deformation processing to obtain a one-dimensional feature vector; and the features in the one-dimensional feature vector may be mapped in combination with the multi-modal vocabulary to obtain an integer sequence corresponding to the data;

[0058] Specifically, the electronic device can combine the multimodal vocabulary and the integer sequence corresponding to the data to determine the predicted data features corresponding to the data; and then input the predicted data features corresponding to the data into the decoding network to obtain the predicted data corresponding to the data output by the decoding network.

[0059] It should be noted that the combination of at least two candidate data with an associated relationship may include: audio and video. The setting of audio and video and the image to be audio enables the multimodal large model to extract the interactive features between at least two modalities, thereby improving the accuracy of the extracted features.

[0060] Among them, when there are no at least two candidate data with a correlation relationship in the data under each non-text modality, there is no correlation relationship between the data under each non-text modality, and the data under each non-text modality can be processed separately to obtain predicted data, so that the data under each non-text modality can be processed in parallel, thereby improving data processing efficiency.

[0061] In an embodiment of the present disclosure, when there are at least two candidate data with an associated relationship in the data under each non-text modality, for the at least two candidate data, the electronic device can input the at least two candidate data into the encoding network in the encoding and decoding network corresponding to the modality to obtain data features corresponding to each of the at least two candidate data; perform one-dimensional deformation processing and bitwise addition processing on at least two of the data features to obtain a processed one-dimensional feature vector; map the features in the processed one-dimensional feature vector in combination with a multimodal vocabulary to obtain a processed integer sequence; and determine the predicted data corresponding to the at least two candidate data in combination with the processed integer sequence and the decoding network in the encoding and decoding network corresponding to the modality to which the at least two candidate data belong.

[0062] Specifically, the electronic device can determine the predicted data features in combination with the multimodal vocabulary and the processed integer sequence; input the predicted data features into the decoding network in the encoding and decoding network corresponding to the modalities to which at least two candidate data belong, and obtain the predicted data corresponding to the at least two candidate data.

[0063] Among them, after the data features corresponding to at least two candidate data with an associated relationship are processed by bitwise addition, and then feature mapping processing is performed in combination with a multimodal vocabulary, feature interaction between at least two candidate data can be achieved, avoiding the separation between at least two candidate data, thereby further improving the training efficiency of the codec network.

[0064] The encoding and decoding networks corresponding to each non-text mode may also include a discriminant network. The discriminant network combines the data in the non-text mode to discriminate the truth or falsity of the predicted data corresponding to the data. The truth or falsity discrimination result is that the predicted data is true; or the predicted data is false. Combined with the truth or falsity discrimination result, the difference between the data and the predicted data in the truth or falsity discrimination can be determined.

[0065] The difference between the data and the predicted data can be determined based on the similarity between the data and the predicted data. The difference in features between the data and the predicted data can be at least one of the following: the difference between target information in the data and target information in the predicted data; the difference between the description content in the data and the description content in the predicted data; the difference between the Fourier transform result of the data and the Fourier transform result of the predicted data, etc., which can be set according to actual needs.

[0066] Among them, combining the differences in various aspects between the data and the predicted data under the non-text mode to comprehensively determine the loss function value of the codec network corresponding to the non-text mode can further improve the accuracy of the determined loss function value, thereby further improving the training accuracy of each codec network.

[0067] Step 204 , according to the loss function values ​​of the codec networks corresponding to the non-text modalities, each codec network and the multimodal vocabulary are adjusted to achieve joint training.

[0068] In the embodiment of the present disclosure, the process of the electronic device executing step 204 may, for example, be to add the loss function values ​​of the codec networks corresponding to each non-text modality to obtain a total loss function value; according to the total loss function value, adjust the parameters of each codec network, and adjust the vector corresponding to the integer identifier in the multimodal vocabulary to achieve joint training.

[0069] Among them, according to the total loss function value, the parameters of each codec network are adjusted, and the vectors corresponding to the integer identifiers in the multimodal vocabulary are adjusted, so that the vectors corresponding to the integer identifiers in the multimodal vocabulary can reflect the characteristics of each non-text modality, thereby further improving the accuracy of the trained codec network.

[0070] Step 205, when the training of each codec network and the multimodal vocabulary is completed, the backbone network is trained in combination with the multimodal sample reference data and sample generation data under the target task.

[0071] It should be noted that the details of steps 201 to 202 and step 205 can be found in Figure 1 Steps 101 to 102 and step 104 in the illustrated embodiment will not be described in detail herein.

[0072] The training method of the multimodal large model of the embodiment of the present disclosure is through obtaining first training data and second training data; the first training data includes data under each non-text mode; the second training data includes multimodal sample reference data and sample generation data under the target task; obtaining an initial multimodal large model; the multimodal large model includes a backbone network and a codec network corresponding to each non-text mode; each codec network performs codec processing in combination with the same multimodal vocabulary; combining the data under each non-text mode and the codec network corresponding to each non-text mode, determining the loss function of the codec network corresponding to each non-text mode Value; according to the loss function value of the codec network corresponding to each non-text modality, each codec network and the multimodal vocabulary are adjusted to achieve joint training; when the training of each codec network and the multimodal vocabulary is completed, the backbone network is trained in combination with the multimodal sample reference data and sample generation data under the target task; wherein, the loss function value of each codec network is determined separately; and then the loss function values ​​are combined to adjust each codec network and the multimodal vocabulary, which can further improve the training speed and training accuracy of the codec network corresponding to each non-text modality.

[0073] In order to further improve the training accuracy of the backbone network, the electronic device can combine the multimodal sample reference data, the codec network corresponding to each non-text modality and the backbone network to determine the predicted generated data, and then determine the loss function value of the backbone network for adjustment processing. Figure 3 As shown, Figure 3 is a schematic diagram according to a third embodiment of the present disclosure, Figure 3 The illustrated embodiment may include the following steps:

[0074] Step 301, obtaining first training data and second training data; the first training data includes data in various non-text modes; the second training data includes multimodal sample reference data and sample generation data under the target task.

[0075] Step 302, obtaining an initial multimodal large model; the multimodal large model includes a backbone network and encoding and decoding networks corresponding to each non-text modality; each encoding and decoding network performs encoding and decoding processing in combination with the same multimodal vocabulary.

[0076] Step 303 , combining the data in each non-text mode, and jointly training each encoding and decoding network and the multimodal vocabulary.

[0077] Step 304, combining the multimodal sample reference data, the codec network corresponding to each non-text modality, and the backbone network to determine the predicted generated data.

[0078] In an embodiment of the present disclosure, multimodal sample reference data may include data in at least two of the following modalities: audio modality, silent video modality, image modality, and text modality; sample generation data may include data in at least one of the following modalities: audio modality, silent video modality, image modality, and text modality.

[0079] Among them, the flexible setting of multimodal sample reference data and data of various modalities in sample generation data enables the trained multimodal large model to be applicable to a variety of target tasks, expanding the applicable scenarios of the multimodal large model.

[0080] In the embodiment of the present disclosure, the process of the electronic device executing step 304 may, for example, be to determine a multimodal integer sequence combination in combination with the multimodal sample reference data and the encoding network in the encoding and decoding network corresponding to each non-text modality; distinguish and process the integer sequences under different modalities in the multimodal integer sequence combination by modality labels; input the multimodal integer combination into the backbone network to obtain a predicted integer sequence or a predicted integer sequence combination output by the backbone network; input the predicted integer sequence or the predicted integer sequence in the predicted integer sequence combination into the decoding network in the encoding and decoding network corresponding to the modality to obtain predicted generated data.

[0081] Among them, the multimodal sample reference data may include sample reference data under each candidate non-text mode and sample text data under text mode. Correspondingly, the process of the electronic device determining the multimodal integer sequence combination may be, for example, for each candidate non-text mode, combining the sample reference data under the candidate non-text mode and the encoding network in the encoding and decoding network corresponding to the candidate non-text mode, to determine the integer sequence under the candidate non-text mode; the integer sequence under each candidate non-text mode and the integer sequence corresponding to the sample text data are spliced ​​according to the modality label to obtain a multimodal integer sequence combination.

[0082] It should be noted that the electronic device may be provided with a text vocabulary for the text, and the text vocabulary includes integer identifiers and corresponding words. The electronic device may perform word segmentation on the sample text data to obtain each word; query the text vocabulary according to each word, obtain the integer identifier corresponding to each word, and then combine to obtain the integer sequence corresponding to the sample text data.

[0083] Among them, when the target task is an image generation task, a video generation task, an audio generation task, or a text generation task, the predicted generation data output by the multimodal large model is data under a single mode. Correspondingly, the backbone network can output a predicted integer sequence. When the target task is a multi-output task, such as a picture and text generation task, an audio and video generation task, etc., the predicted generation data output by the multimodal large model is data under multiple modes. Correspondingly, the backbone network can output a set of predicted integer sequences.

[0084] Among them, a modality tag may be set for the predicted integer sequence output by the backbone network. A modality tag may be set for each predicted integer sequence in the set of predicted integer sequences output by the backbone network. Combined with the modality tag, it can be determined which decoding network corresponding to the non-text modality is used to decode the predicted integer sequence, so that the decoding process can be carried out in a targeted manner.

[0085] Among them, the modality marking of the integer sequence under each candidate non-text modality and the word sequence corresponding to the sample text data enables the backbone network to distinguish integer sequences of different modalities for learning processing, thereby further improving the training speed and training accuracy of the backbone network.

[0086] In an embodiment of the present disclosure, when there are at least two candidate non-text modal sample reference data with an associated relationship in the multimodal sample reference data, for example, the multimodal sample reference data includes audio and video data, the electronic device can determine the data features of the at least two candidate non-text modal sample reference data in combination with the coding network in the coding and decoding network under each modality; when the data feature is a multidimensional feature vector, perform one-dimensional deformation processing on the data feature to obtain a one-dimensional feature vector; perform bitwise addition processing on at least two one-dimensional feature vectors to obtain a processed one-dimensional feature vector; map the features in the processed one-dimensional feature vector in combination with the multimodal vocabulary to obtain an integer sequence; and perform combined modal labeling processing on the integer sequence. Among them, the combined modal labeling, for example, is an audio and video labeling, etc.

[0087] In an embodiment of the present disclosure, when there are sample generation data under at least two candidate non-text modalities with an associated relationship in the sample generation data, the backbone network may also generate an integer sequence of a combined modality for at least two candidate non-text modalities with an associated relationship when outputting a predicted integer sequence combination; the integer sequence may be respectively input into a decoding network in the encoding and decoding network corresponding to the at least two candidate non-text modalities to obtain predicted generation data under the at least two candidate non-text modalities.

[0088] Among them, the generation process of the integer sequence of the combined modality enables the multimodal large model to achieve the synchronous generation process of the predicted generated data under at least two candidate non-text modalities.

[0089] Step 305, combining the sample generated data and the predicted generated data to determine the loss function value of the backbone network.

[0090] In an embodiment of the present disclosure, the electronic device may determine the loss function value of the backbone network by combining the sample generation data, the prediction generation data, and the loss function of the multimodal large model.

[0091] Step 306, adjusting the parameters of the backbone network in combination with the loss function value to implement training.

[0092] It should be noted that the details of steps 301 to 303 can be found in Figure 1 Steps 101 to 103 in the illustrated embodiment will not be described in detail herein.

[0093] The training method of the multimodal large model of the embodiment of the present disclosure is through obtaining first training data and second training data; the first training data includes data under each non-text mode; the second training data includes multimodal sample reference data and sample generation data under the target task; obtaining an initial multimodal large model; the multimodal large model includes a backbone network and a codec network corresponding to each non-text mode; each codec network performs codec processing in combination with the same multimodal vocabulary; in combination with the data under each non-text mode, each codec network and the multimodal vocabulary are jointly trained; in combination with the multimodal sample reference data, the codec network corresponding to each non-text mode and the backbone network, predictive generation data is determined; in combination with the sample generation data and the predictive generation data, the loss function value of the backbone network is determined; in combination with the loss function value, the backbone network is parameter adjusted to achieve training; wherein, in combination with the multimodal sample reference data, the codec network corresponding to each non-text mode and the backbone network, predictive generation data is determined, and then the loss function value of the backbone network is determined for adjustment processing, which can improve the training speed and training accuracy of the backbone network.

[0094] The following examples are used to illustrate this. Figure 4 The following is a schematic diagram of the training of each encoding and decoding network in a multimodal large model. Figure 4The following steps may be included. Step 401: Input the audio into the coding network in 1D-CNN (the coding and decoding network corresponding to the audio modality) to obtain the output audio features; input the video into the coding network in 3D-CNN (the coding and decoding network corresponding to the video modality) to obtain the output video features; input the image into the coding network in 2D-CNN (the coding and decoding network corresponding to the image modality) to obtain the output image features. Step 402: Perform one-dimensional deformation processing on the image features and the video features respectively to obtain the one-dimensional feature vector corresponding to the image and the one-dimensional feature vector corresponding to the video. In the case where the audio is the dubbing corresponding to the video, that is, the audio and video are combined to obtain the audio video, the audio features (one-dimensional feature vector) corresponding to the audio and the one-dimensional feature vector corresponding to the video are summed (bitwise addition) to obtain the processed one-dimensional feature vector. Step 403: Map the one-dimensional feature vector corresponding to the image and the processed one-dimensional feature vector in combination with the codebook (multimodal vocabulary) to obtain the integer sequence corresponding to the image and the processed integer sequence; the processed integer sequence can be used as the integer sequence corresponding to the audio and the integer sequence corresponding to the video respectively. Step 404, combining the integer sequence corresponding to the audio, the integer sequence corresponding to the image, the integer sequence corresponding to the video, the decoding network of 1D-CNN, the decoding network of 2D-CNN, and the decoding network of 3D-CNN, determines to obtain predicted audio, predicted image, and predicted video for joint training processing of 1D-CNN, 2D-CNN, and 3D-CNN.

[0095] Figure 5 1 is a schematic diagram according to the fourth embodiment of the present disclosure. It should be noted that the target task processing method of the embodiment of the present disclosure can be applied to a target task processing device, which can be configured in an electronic device so that the electronic device can perform the target task processing function. The following embodiments are described by taking the execution subject as an electronic device as an example.

[0096] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0097] The target task processing device may also be software in an electronic device, such as target task processing software, etc. In the following embodiments, the target task processing device is an electronic device as an example for description.

[0098] like Figure 5 As shown, the method for processing the target task may include the following steps:

[0099] Step 501, obtaining a target task; the target task includes data under at least two modes.

[0100] In the embodiment of the present disclosure, the target task may include at least one of the following: an image generation task, a video generation task, an audio generation task, a text generation task, and a multimodal understanding task. Among them, the data under at least two modes included in the target task are the data required to perform the target task.

[0101] Among them, taking the target task as an image generation task as an example, it is assumed that the image generation task is to perform image generation processing based on images and texts. Correspondingly, the image generation task can include data in the image mode and data in the text mode. Taking the target task as a video generation task as an example, it is assumed that the video generation task is to perform video generation processing based on images and audios. Correspondingly, the video generation task can include data in the image mode and data in the audio mode.

[0102] Among them, the setting of each target task enables the multimodal large model to be used to process the data under each target task, thereby expanding the task application scenarios of the multimodal large model.

[0103] Step 502, obtaining a multimodal large model; the multimodal large model is based on Figures 1 to 3 The training method of the multimodal large model in any of the embodiments is determined.

[0104] In the disclosed embodiment, the multimodal large model includes a backbone network and encoding and decoding networks corresponding to each non-text modality; each encoding and decoding network performs encoding and decoding processing in conjunction with the same multimodal vocabulary.

[0105] There may be multiple codec networks, for example, a one-dimensional codec network, a two-dimensional codec network, a three-dimensional codec network, etc. The one-dimensional codec network is the codec network corresponding to the audio modality; the two-dimensional codec network is the codec network corresponding to the image modality; and the three-dimensional codec network is the codec network corresponding to the video modality.

[0106] Step 503: input data under at least two modes into the multimodal large model to obtain generated data output by the multimodal large model.

[0107] In an embodiment of the present disclosure, taking at least two modalities, namely, image modality, audio modality, video modality and text modality, as an example, the process of the electronic device executing step 503 may be, for example, inputting a two-dimensional codec network in combination with an image in image modality to determine an integer sequence corresponding to the image; inputting a one-dimensional codec network in combination with an audio in audio modality to determine an integer sequence corresponding to the audio; inputting a three-dimensional codec network in combination with a video in video modality to determine an integer sequence corresponding to the video; performing word segmentation processing on the text in text modality, and then determining an integer sequence corresponding to the text in combination with integer identifiers corresponding to each word in a text vocabulary; performing modality marking and splicing processing on the integer sequence corresponding to the image, the integer sequence corresponding to the audio, the integer sequence corresponding to the video, and the integer sequence corresponding to the text to obtain a processed integer sequence combination; inputting the processed integer sequence combination into a backbone network to obtain a predicted integer sequence or a predicted integer sequence combination output by the backbone network; and determining to generate data according to the predicted integer sequence or the predicted integer sequence combination and the decoding networks in each codec network.

[0108] In the disclosed embodiment, data in at least two modes in the target task may have an association relationship. Assume that the target task includes data in audio mode, data in video mode, and data in image mode. Among them, the data in audio mode and the data in video mode have an association relationship. The electronic device can input the data in audio mode into the coding network in the codec network corresponding to the audio mode to obtain the output audio data features; wherein the audio data features are one-dimensional feature vectors; input the data in video mode into the coding network in the codec network corresponding to the video mode to obtain the output video data features; perform one-dimensional deformation processing on the video data features to obtain a one-dimensional feature vector; perform bitwise addition processing on the two one-dimensional feature vectors, and then perform feature mapping processing in combination with the multimodal vocabulary to obtain an integer sequence; perform combined modality labeling processing on the integer sequence, that is, perform audio and video labeling processing.

[0109] In the embodiment of the present disclosure, when the target task is to generate data under at least two non-text modalities with an associated relationship, the backbone network can generate an integer sequence of a combined modality for at least two candidate non-text modalities with an associated relationship when outputting the predicted integer sequence combination; the integer sequence can be respectively input into the decoding network in the encoding and decoding network corresponding to the at least two candidate non-text modalities to obtain the predicted generated data under the at least two candidate non-text modalities. Thereby, the synchronous generation processing of the predicted generated data under at least two non-text modalities with an associated relationship is realized.

[0110] The method for processing the target task of the embodiment of the present disclosure is as follows: obtaining the target task; the target task includes data under at least two modes; obtaining a multi-modal large model; the multi-modal large model is based on Figures 1 to 3 The training method of the multimodal large model in any embodiment is determined; data under at least two modalities are input into the multimodal large model to obtain generated data output by the multimodal large model; wherein each encoding and decoding network in the multimodal large model is combined with the same multimodal vocabulary for encoding and decoding processing, thereby realizing unified modeling of data under multiple modalities, and improving the accuracy of the determined generated data.

[0111] In order to implement the above embodiment, the present disclosure also provides a training device for a multi-modal large model. Figure 6 As shown, Figure 6 6 is a schematic diagram according to the fifth embodiment of the present disclosure. The multimodal large model training device 60 may include: a first acquisition module 601 , a second acquisition module 602 , a first training processing module 603 and a second training processing module 604 .

[0112] Among them, the first acquisition module 601 is used to acquire the first training data and the second training data; the first training data includes data under each non-text mode; the second training data includes multimodal sample reference data and sample generation data under the target task; the second acquisition module 602 is used to acquire the initial multimodal large model; the multimodal large model includes a backbone network and a codec network corresponding to each non-text mode; each of the codec networks performs codec processing in combination with the same multimodal vocabulary; the first training processing module 603 is used to perform joint training processing on each of the codec networks and the multimodal vocabulary in combination with the data under each non-text mode; the second training processing module 604 is used to train the backbone network in combination with the multimodal sample reference data and sample generation data under the target task when the training of each of the codec networks and the multimodal vocabulary is completed.

[0113] As a possible implementation method of the embodiment of the present disclosure, the first training processing module 603 includes a first determination unit and a first adjustment processing unit; the first determination unit is used to determine the loss function value of the codec network corresponding to each non-text modality in combination with the data under each non-text modality and the codec network corresponding to each non-text modality; the first adjustment processing unit is used to adjust each of the codec networks and the multimodal vocabulary according to the loss function value of the codec network corresponding to each non-text modality to achieve joint training.

[0114] As a possible implementation method of the embodiment of the present disclosure, the first determination unit includes a first determination subunit, a second determination subunit and a third determination subunit; the first determination subunit is used to determine the prediction data corresponding to each data in combination with the data under each non-text modality and the codec network corresponding to each non-text modality; the second determination subunit is used to determine the true or false discrimination result for each non-text modality in combination with the data under the non-text modality, the prediction data corresponding to the data and the discrimination network in the codec network corresponding to the non-text modality; the third determination subunit is used to determine the loss function value of the codec network corresponding to the non-text modality in combination with at least one of the true or false discrimination result, the difference between the data and the prediction data, and the difference in features between the data and the prediction data.

[0115] As a possible implementation method of the embodiment of the present disclosure, the first determination subunit is specifically used to determine whether there are at least two candidate data with an associated relationship in the data under each non-text modality; the modalities to which at least two of the candidate data belong are different; in the case that there are not at least two candidate data in the data under each non-text modality, inputting the data under the non-text modality into the codec network corresponding to the non-text modality for each non-text modality in turn, and obtaining predicted data corresponding to the data under the non-text modality.

[0116] As a possible implementation method of the embodiment of the present disclosure, the first determination subunit is also used to, when there are at least two candidate data in the data under each non-text modality, input the at least two candidate data into the encoding network in the encoding and decoding network corresponding to the modality to obtain data features corresponding to the at least two candidate data respectively; perform one-dimensional deformation processing and bit-wise addition processing on the at least two data features to obtain a processed one-dimensional feature vector; map the features in the processed one-dimensional feature vector in combination with the multimodal vocabulary to obtain a processed integer sequence; and determine the predicted data corresponding to the at least two candidate data in combination with the processed integer sequence and the decoding network in the encoding and decoding network corresponding to the modality to which the at least two candidate data belong.

[0117] As a possible implementation method of the embodiment of the present disclosure, the first adjustment processing unit is specifically used to add the loss function values ​​of the codec networks corresponding to each non-text modality to obtain a total loss function value; according to the total loss function value, adjust the parameters of each codec network, and adjust the vector corresponding to the integer identifier in the multimodal vocabulary to achieve joint training.

[0118] As a possible implementation method of an embodiment of the present disclosure, the non-text modality includes at least one of the following: an audio modality, a silent video modality, and an image modality; the codec network corresponding to the audio modality is a one-dimensional codec network; the codec network corresponding to the silent video modality is a three-dimensional codec network; the codec network corresponding to the image modality is a two-dimensional codec network.

[0119] As a possible implementation method of an embodiment of the present disclosure, a method for processing an image in an image modality in combination with the coding network in the two-dimensional coding and decoding network includes: inputting the image into the coding network in the two-dimensional coding and decoding network to obtain two-dimensional image features; performing one-dimensional deformation processing on the two-dimensional image features to obtain a deformed one-dimensional feature vector; and mapping the features in the one-dimensional feature vector in combination with the multimodal vocabulary to obtain an integer sequence corresponding to the image.

[0120] As a possible implementation manner of the embodiment of the present disclosure, the combination of at least two candidate data having an associated relationship includes at least one of the following: video with audio.

[0121] As a possible implementation method of an embodiment of the present disclosure, the second training processing module 604 includes a second determination unit, a third determination unit and a second adjustment processing unit; the second determination unit is used to determine the predicted generation data in combination with the multimodal sample reference data, the codec network corresponding to each non-text modality and the backbone network; the third determination unit is used to determine the loss function value of the backbone network in combination with the sample generation data and the predicted generation data; the second adjustment processing unit is used to perform parameter adjustment processing on the backbone network in combination with the loss function value to achieve training.

[0122] As a possible implementation manner of an embodiment of the present disclosure, the second determination unit is specifically used to determine a multimodal integer sequence combination in combination with the multimodal sample reference data and an encoding network in a codec network corresponding to each non-text modality; in the multimodal integer sequence combination, integer sequences under different modalities are distinguished by modality labels; the multimodal integer sequence combination is input into the backbone network to obtain a predicted integer sequence or a predicted integer sequence combination output by the backbone network; the predicted integer sequence or a predicted integer sequence in the predicted integer sequence combination is input into a decoding network in the codec network corresponding to the modality to obtain the predicted generated data.

[0123] As a possible implementation method of an embodiment of the present disclosure, the multimodal sample reference data includes sample reference data under each candidate non-text modality and sample text data under a text modality; the second determination unit is specifically further used to, for each candidate non-text modality, determine the integer sequence under the candidate non-text modality in combination with the sample reference data under the candidate non-text modality and the encoding network in the encoding and decoding network corresponding to the candidate non-text modality; the integer sequences under each candidate non-text modality and the integer sequences corresponding to the sample text data are concatenated according to the modality mark to obtain the multimodal integer sequence combination.

[0124] As a possible implementation method of an embodiment of the present disclosure, the multimodal sample reference data includes data in at least two of the following modalities: audio modality, silent video modality, image modality, and text modality; the sample generation data includes data in at least one of the following modalities: audio modality, silent video modality, image modality, and text modality.

[0125] As a possible implementation manner of an embodiment of the present disclosure, the target task includes at least one of the following: an image generation task, a video generation task, an audio generation task, a text generation task, and a multimodal understanding task.

[0126] The training device of the multimodal large model of the embodiment of the present disclosure obtains first training data and second training data; the first training data includes data under each non-text mode; the second training data includes multimodal sample reference data and sample generation data under the target task; obtains an initial multimodal large model; the multimodal large model includes a backbone network and a codec network corresponding to each non-text mode; each codec network performs codec processing in combination with the same multimodal vocabulary; in combination with the data under each non-text mode, each codec network and the multimodal vocabulary are jointly trained; when the training of each codec network and the multimodal vocabulary is completed, the backbone network is trained in combination with the multimodal sample reference data and sample generation data under the target task; wherein, each codec network performs codec processing in combination with the same multimodal vocabulary, and the joint training processing can avoid the separate training processing of the vocabulary of different modes during the training process, thereby reducing the difficulty and cost of model training.

[0127] In order to implement the above embodiment, the present disclosure also provides a target task processing device. Figure 7 As shown, Figure 7 70 is a schematic diagram according to the sixth embodiment of the present disclosure. The target task processing device 70 may include: a first acquisition module 701 , a second acquisition module 702 and a third acquisition module 703 .

[0128] The first acquisition module 701 is used to acquire a target task; the target task includes data under at least two modes; the second acquisition module 702 is used to acquire a multi-modal large model; the multi-modal large model is based on Figures 1 to 3 The training method of the multimodal large model in any embodiment is determined; the third acquisition module 703 is used to input the data under the at least two modalities into the multimodal large model to obtain the generated data output by the multimodal large model.

[0129] As a possible implementation manner of an embodiment of the present disclosure, the target task includes at least one of the following: an image generation task, a video generation task, an audio generation task, a text generation task, and a multimodal understanding task.

[0130] The target task processing device of the embodiment of the present disclosure obtains the target task; the target task includes data under at least two modes; obtains a multi-modal large model; the multi-modal large model is based on Figures 1 to 3 The training method of the multimodal large model in any embodiment is determined; data under at least two modalities are input into the multimodal large model to obtain generated data output by the multimodal large model; wherein each encoding and decoding network in the multimodal large model is combined with the same multimodal vocabulary for encoding and decoding processing, thereby realizing unified modeling of data under multiple modalities, and improving the accuracy of the determined generated data.

[0131] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with the relevant laws and regulations, and do not violate public order and good morals.

[0132] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0133] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0134] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0135] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0136] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a training method for a multimodal large model or a processing method for a target task. For example, in some embodiments, the training method for a multimodal large model or a processing method for a target task may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or a communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the training method for a multimodal large model or the processing method for a target task described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute a training method for a multimodal large model or a processing method for a target task in any other appropriate manner (for example, by means of firmware).

[0137] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0139] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0141] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0142] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0143] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0144] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a multimodal large model, the method comprising: Acquire first training data and second training data; The first training data includes data in various non-text modes; The second training data includes multimodal sample reference data and sample generation data under the target task; Acquire an initial multimodal large model; the multimodal large model includes a backbone network and encoding and decoding networks corresponding to each non-text mode; each encoding and decoding network performs encoding and decoding processing in combination with the same multimodal vocabulary; Combined with the data in each non-text mode, the encoding and decoding networks and the multimodal vocabulary are jointly trained; When the training of each of the codec networks and the multimodal vocabulary is completed, the backbone network is trained in combination with the multimodal sample reference data and sample generation data under the target task.

2. The method according to claim 1, wherein: The combining of the data in each non-text mode to jointly train each of the encoding and decoding networks and the multimodal vocabulary includes: Combining the data under each non-text mode and the codec network corresponding to each non-text mode, determining the loss function value of the codec network corresponding to each non-text mode; According to the loss function value of the codec network corresponding to each non-text modality, each codec network and the multimodal vocabulary are adjusted to achieve joint training.

3. The method according to claim 2, wherein: The combining the data in each non-text mode and the codec network corresponding to each non-text mode to determine the loss function value of the codec network corresponding to each non-text mode includes: Determine prediction data corresponding to each of the data by combining the data in each non-text mode and the codec network corresponding to each non-text mode; For each non-text mode, combining the data in the non-text mode, the predicted data corresponding to the data, and the discriminant network in the encoding and decoding network corresponding to the non-text mode, to determine a true or false discrimination result; The loss function value of the codec network corresponding to the non-text modality is determined based on at least one of the true and false discrimination result, the difference between the data and the predicted data, and the difference in features between the data and the predicted data.

4. The method according to claim 3, wherein: The combining the data in each non-text mode and the codec network corresponding to each non-text mode to determine the prediction data corresponding to each of the data includes: Determine whether there are at least two candidate data with an associated relationship in the data in each non-text mode; at least two of the candidate data belong to different modes; In the case that there are not at least two of the candidate data in the data under each non-text mode, the data under the non-text mode is input into the codec network corresponding to the non-text mode for each non-text mode in turn to obtain the predicted data corresponding to the data under the non-text mode.

5. The method according to claim 4, wherein: The combining the data in each non-text mode and the codec network corresponding to each non-text mode to determine the prediction data corresponding to each of the data also includes: When there are at least two candidate data in the data under each non-text mode, the at least two candidate data are respectively input into the encoding network in the encoding and decoding network corresponding to the corresponding mode to obtain data features corresponding to each of the at least two candidate data; Performing one-dimensional deformation processing and bitwise addition processing on at least two of the data features to obtain a processed one-dimensional feature vector; Mapping the features in the processed one-dimensional feature vector in combination with the multimodal vocabulary to obtain a processed integer sequence; Determine predicted data corresponding to at least two of the candidate data by combining the processed integer sequence and a decoding network in an encoding and decoding network corresponding to a modality to which at least two of the candidate data belong.

6. The method according to claim 2, wherein: The method of adjusting each of the codec networks and the multimodal vocabulary according to the loss function values ​​of the codec networks corresponding to each non-text modality to achieve joint training includes: The loss function values ​​of the encoding and decoding networks corresponding to each non-text modality are summed up to obtain the total loss function value; According to the total loss function value, the parameters of each encoding and decoding network are adjusted, and the vectors corresponding to the integer identifiers in the multimodal vocabulary are adjusted to achieve joint training.

7. The method according to claim 1, wherein: The non-text modality includes at least one of the following: an audio modality, a silent video modality, and an image modality; The codec network corresponding to the audio modality is a one-dimensional codec network; The codec network corresponding to the silent video modality is a three-dimensional codec network; the codec network corresponding to the image modality is a two-dimensional codec network.

8. The method according to claim 7, wherein: The method of processing the image in the image modality in combination with the encoding network in the two-dimensional encoding and decoding network includes: Inputting the image into the encoding network in the two-dimensional encoding and decoding network to obtain two-dimensional image features; Performing one-dimensional deformation processing on the two-dimensional image feature to obtain a deformed one-dimensional feature vector; The features in the one-dimensional feature vector are mapped in combination with the multimodal vocabulary to obtain an integer sequence corresponding to the image.

9. The method according to claim 4 or 5, wherein: The combination of at least two candidate data having an associated relationship includes: video with audio.

10. The method according to claim 1, wherein: The training process of the backbone network is performed by combining the multimodal sample reference data and the sample generation data under the target task, including: Determine prediction generation data by combining the multimodal sample reference data, the codec network corresponding to each non-text modality, and the backbone network; Determine the loss function value of the backbone network by combining the sample generation data and the prediction generation data; The backbone network is subjected to parameter adjustment processing in combination with the loss function value to achieve training.

11. The method according to claim 10, wherein: The step of combining the multimodal sample reference data, the codec networks corresponding to each non-text modality, and the backbone network to determine the predicted generated data includes: Determine a multimodal integer sequence combination by combining the multimodal sample reference data and the encoding network in the encoding and decoding network corresponding to each non-text modality; in the multimodal integer sequence combination, distinguish and process integer sequences under different modalities by modality marking; Inputting the multimodal integer combination into the backbone network to obtain a predicted integer sequence or a predicted integer sequence combination output by the backbone network; The predicted integer sequence or the predicted integer sequence in the combination of predicted integer sequences is input into a decoding network in the encoding and decoding network corresponding to the modality to obtain the predicted generated data.

12. The method according to claim 11, wherein: The multimodal sample reference data includes sample reference data under each candidate non-text modality and sample text data under text modality; the step of combining the multimodal sample reference data and the encoding network in the encoding and decoding network corresponding to each non-text modality to determine the multimodal integer sequence combination includes: For each candidate non-text modality, combining sample reference data under the candidate non-text modality and a coding network in a coding and decoding network corresponding to the candidate non-text modality, determining an integer sequence under the candidate non-text modality; The integer sequences under each candidate non-text modality and the integer sequences corresponding to the sample text data are concatenated according to the modality labels to obtain the multi-modal integer sequence combination.

13. The method according to claim 1 or 10, wherein: The multimodal sample reference data includes data in at least two of the following modes: audio mode, silent video mode, image mode, and text mode; The sample generated data includes data in at least one of the following modes: audio mode, silent video mode, image mode, and text mode.

14. The method according to claim 1, wherein: The target task includes at least one of the following: an image generation task, a video generation task, an audio generation task, a text generation task, and a multimodal understanding task.

15. A method for processing a target task, the method comprising: Get the target task; The target task includes data under at least two modalities; Get a multimodal large model; The multimodal large model is determined based on the training method of the multimodal large model according to any one of claims 1 to 14; The data under the at least two modes are input into the multimodal large model to obtain generated data output by the multimodal large model.

16. The method according to claim 15, wherein: The target task includes at least one of the following: an image generation task, a video generation task, an audio generation task, a text generation task, and a multimodal understanding task.

17. A training device for a multimodal large model, the device comprising: A first acquisition module, used to acquire first training data and second training data; The first training data includes data in various non-text modes; The second training data includes multimodal sample reference data and sample generation data under the target task; The second acquisition module is used to acquire an initial multimodal large model; the multimodal large model includes a backbone network and encoding and decoding networks corresponding to each non-text modality; each encoding and decoding network performs encoding and decoding processing in combination with the same multimodal vocabulary; A first training processing module, used for combining data in each non-text mode to perform joint training processing on each of the encoding and decoding networks and the multimodal vocabulary; The second training processing module is used to train the backbone network in combination with the multimodal sample reference data and sample generation data under the target task when the training of each of the codec networks and the multimodal vocabulary is completed.

18. The device according to claim 17, wherein: The first training processing module includes a first determination unit and a first adjustment processing unit; The first determining unit is used to determine the loss function value of the codec network corresponding to each non-text modality by combining the data in each non-text modality and the codec network corresponding to each non-text modality; The first adjustment processing unit is used to adjust each of the codec networks and the multimodal vocabulary according to the loss function value of the codec network corresponding to each non-text modality to achieve joint training.

19. The device according to claim 18, wherein: The first determining unit includes a first determining subunit, a second determining subunit and a third determining subunit; The first determination subunit is used to determine the prediction data corresponding to each of the data by combining the data in each non-text mode and the codec network corresponding to each non-text mode; The second determination subunit is used to determine the true or false discrimination result for each non-text mode by combining the data in the non-text mode, the predicted data corresponding to the data, and the discrimination network in the encoding and decoding network corresponding to the non-text mode; The third determination subunit is used to determine the loss function value of the codec network corresponding to the non-text modality based on at least one of the true and false discrimination result, the difference between the data and the predicted data, and the difference in features between the data and the predicted data.

20. The device according to claim 19, wherein The first determining subunit is specifically configured to: Determine whether there are at least two candidate data with an associated relationship in the data in each non-text mode; at least two of the candidate data belong to different modes; In the case that there are not at least two of the candidate data in the data under each non-text mode, the data under the non-text mode is input into the codec network corresponding to the non-text mode for each non-text mode in turn to obtain the predicted data corresponding to the data under the non-text mode.

21. The device according to claim 20, wherein: The first determining subunit is further configured to: When there are at least two candidate data in the data under each non-text mode, the at least two candidate data are respectively input into the encoding network in the encoding and decoding network corresponding to the corresponding mode to obtain data features corresponding to each of the at least two candidate data; Performing one-dimensional deformation processing and bitwise addition processing on at least two of the data features to obtain a processed one-dimensional feature vector; Mapping the features in the processed one-dimensional feature vector in combination with the multimodal vocabulary to obtain a processed integer sequence; Determine predicted data corresponding to at least two of the candidate data by combining the processed integer sequence and a decoding network in an encoding and decoding network corresponding to a modality to which at least two of the candidate data belong.

22. The device according to claim 18, wherein The first adjustment processing unit is specifically configured to: The loss function values ​​of the encoding and decoding networks corresponding to each non-text modality are summed up to obtain the total loss function value; According to the total loss function value, the parameters of each encoding and decoding network are adjusted, and the vectors corresponding to the integer identifiers in the multimodal vocabulary are adjusted to achieve joint training.

23. The device according to claim 17, wherein: The non-text modality includes at least one of the following: an audio modality, a silent video modality, and an image modality; The codec network corresponding to the audio modality is a one-dimensional codec network; The codec network corresponding to the silent video modality is a three-dimensional codec network; the codec network corresponding to the image modality is a two-dimensional codec network.

24. The device according to claim 23, wherein: The method of processing the image in the image modality in combination with the encoding network in the two-dimensional encoding and decoding network includes: Inputting the image into the encoding network in the two-dimensional encoding and decoding network to obtain two-dimensional image features; Performing one-dimensional deformation processing on the two-dimensional image feature to obtain a deformed one-dimensional feature vector; The features in the one-dimensional feature vector are mapped in combination with the multimodal vocabulary to obtain an integer sequence corresponding to the image.

25. The device according to claim 20 or 21, wherein: The combination of at least two candidate data having an associated relationship includes: video with audio.

26. The device according to claim 17, wherein: The second training processing module includes a second determining unit, a third determining unit, and a second adjustment processing unit; The second determination unit is used to determine the predicted generated data in combination with the multimodal sample reference data, the codec network corresponding to each non-text modality, and the backbone network; The third determination unit is used to determine the loss function value of the backbone network in combination with the sample generation data and the prediction generation data; The second adjustment processing unit is used to perform parameter adjustment processing on the backbone network in combination with the loss function value to achieve training.

27. The device according to claim 26, wherein: The second determining unit is specifically configured to: Determine a multimodal integer sequence combination by combining the multimodal sample reference data and the encoding network in the encoding and decoding network corresponding to each non-text modality; in the multimodal integer sequence combination, distinguish and process integer sequences under different modalities by modality marking; Inputting the multimodal integer combination into the backbone network to obtain a predicted integer sequence or a predicted integer sequence combination output by the backbone network; The predicted integer sequence or the predicted integer sequence in the combination of predicted integer sequences is input into a decoding network in the encoding and decoding network corresponding to the modality to obtain the predicted generated data.

28. The device according to claim 27, wherein The multimodal sample reference data includes sample reference data in each candidate non-text mode and sample text data in a text mode; the second determining unit is further configured to: For each candidate non-text modality, combining sample reference data under the candidate non-text modality and a coding network in a coding and decoding network corresponding to the candidate non-text modality, determining an integer sequence under the candidate non-text modality; The integer sequences under each candidate non-text modality and the integer sequences corresponding to the sample text data are concatenated according to the modality labels to obtain the multi-modal integer sequence combination.

29. The device according to claim 17 or 26, wherein: The multimodal sample reference data includes data in at least two of the following modes: audio mode, silent video mode, image mode, and text mode; The sample generated data includes data in at least one of the following modes: audio mode, silent video mode, image mode, and text mode.

30. The apparatus of claim 17, wherein: The target task includes at least one of the following: an image generation task, a video generation task, an audio generation task, a text generation task, and a multimodal understanding task.

31. A target task processing device, the device comprising: A first acquisition module is used to acquire a target task; The target task includes data under at least two modalities; The second acquisition module is used to acquire a multimodal large model; The multimodal large model is determined based on the training method of the multimodal large model according to any one of claims 1 to 14; The third acquisition module is used to input the data under the at least two modes into the multimodal large model to obtain the generated data output by the multimodal large model.

32. The device according to claim 31, wherein The target task includes at least one of the following: an image generation task, a video generation task, an audio generation task, a text generation task, and a multimodal understanding task.

33. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 14; Alternatively, the method according to any one of claims 15 to 16 is performed.

34. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14; or to execute the method according to any one of claims 15 to 16.

35. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 14; or implements the method according to any one of claims 15 to 16.

Images