Multi-modal large model training method and device and electronic equipment

Abstract

The invention provides a multi-modal large model training method and device and electronic equipment, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, natural language processing, computer vision, voice technology, large models and the like. According to the specific implementation scheme, training data and an initial multi-modal large model are obtained; splicing the sample data block sequences in the training data under at least two modals to obtain a sample spliced sequence; in combination with the sample splicing sequence, training the multi-modal large model; wherein the multi-modal large model is trained in combination with sample data block sequences in at least two modals, so that the multi-modal large model obtained through training can be suitable for various multi-modal understanding tasks and multi-modal generation tasks, and the application scene of the multi-modal large model is expanded; and only the multi-modal large model is used, so that cascade errors and the like caused by cascade of multiple networks can be avoided.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning, natural language processing, computer vision, speech technology, and large models, and in particular to a training method, device, and electronic device for a multimodal large model. Background Art

[0002] The current large multimodal model, when processing multimodal understanding tasks, needs to connect the understanding network of the corresponding modality before the multimodal large model; when processing multimodal generation tasks, it needs to connect the generation network of the corresponding modality after the multimodal large model.

[0003] In the above scheme, the cascade between multiple networks is prone to cascade errors; and when the connected networks are fixed, it can only be applied to specific multimodal understanding tasks or multimodal generation tasks, and the applicable scenarios are limited. Summary of the invention

[0004] The present invention provides a method, device and electronic device for training a multimodal large model.

[0005] According to one aspect of the present disclosure, a method for training a multimodal large model is provided, the method comprising: acquiring training data and an initial multimodal large model; the training data comprising: a sequence of sample data blocks corresponding to sample data under at least two modalities; splicing the sequence of sample data blocks under the at least two modalities to obtain a sample splicing sequence; and training the multimodal large model in combination with the sample splicing sequence.

[0006] According to another aspect of the present disclosure, a method for processing a multimodal task is provided, the method comprising: obtaining a multimodal task; the multimodal task comprising data in at least one modality; data in a text modality in the at least one modality, used to indicate the need to generate data in a target modality; obtaining a multimodal large model; the multimodal large model is determined based on the training method of the multimodal large model as described above; and determining the generated data in the target modality in combination with the data in the at least one modality and the multimodal large model.

[0007] According to another aspect of the present disclosure, a training device for a multimodal large model is provided, the device comprising: an acquisition module, used to acquire training data and an initial multimodal large model; the training data comprises: a sequence of sample data blocks corresponding to sample data under at least two modalities; a splicing processing module, used to splice the sequence of sample data blocks under the at least two modalities to obtain a sample splicing sequence; and a training module, used to train the multimodal large model in combination with the sample splicing sequence.

[0008] According to another aspect of the present disclosure, a device for processing a multimodal task is provided, the device comprising: a first acquisition module, used to acquire a multimodal task; the multimodal task comprises data in at least one modality; data in a text modality in the at least one modality is used to indicate that data in a target modality needs to be generated; a second acquisition module, used to acquire a multimodal large model; the multimodal large model is determined based on the training method of the multimodal large model as described above; a determination module, used to determine the generated data in the target modality in combination with the data in the at least one modality and the multimodal large model.

[0009] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the multimodal large model proposed above in the present disclosure; or, execute the processing method of the multimodal task proposed above in the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the training method of the multimodal large model proposed above in the present disclosure; or, execute the processing method of the multimodal task proposed above in the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the method for training a multimodal large model proposed above in the present disclosure; or, implements the steps of the method for processing a multimodal task proposed above in the present disclosure.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0020] Figure 7 It is a block diagram of an electronic device used to implement the training method of a multimodal large model or the processing method of a multimodal task of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] The current large multimodal model, when processing multimodal understanding tasks, needs to connect the understanding network of the corresponding modality before the multimodal large model; when processing multimodal generation tasks, it needs to connect the generation network of the corresponding modality after the multimodal large model.

[0023] In the above scheme, the cascade between multiple networks is prone to cascade errors; and when the connected networks are fixed, it can only be applied to specific multimodal understanding tasks or multimodal generation tasks, and the applicable scenarios are limited.

[0024] In order to solve the above problems, the present invention provides a method, device and electronic device for training a multimodal large model.

[0025] Figure 1 It is a schematic diagram according to the first embodiment of the present disclosure. It should be noted that the training method of the multimodal large model of the embodiment of the present disclosure can be applied to the training device of the multimodal large model, and the device can be configured in an electronic device so that the electronic device can perform the training function of the multimodal large model. In the following embodiments, the execution subject is an electronic device as an example for explanation.

[0026] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0027] The training device for the multimodal large model may also be software in an electronic device, such as training software for the multimodal large model, etc. In the following embodiments, the training device for the multimodal large model is an electronic device as an example for description.

[0028] like Figure 1 As shown, the training method of the multimodal large model may include the following steps:

[0029] Step 101, obtaining training data and an initial multimodal large model; the training data includes: a sequence of sample data blocks corresponding to sample data under at least two modalities.

[0030] In the embodiment of the present disclosure, the modalities in the training data may include at least two of the following: text, image, audio, and video. The number of sample data block sequences under each modality in the training data is one or more. The sample data block sequence may be obtained by segmenting the sample data.

[0031] The number of training data may be at least one. For example, one training data may include M texts, N images, K videos, L audios, etc. The value of M may be an integer greater than or equal to 0; the value of N may be an integer greater than or equal to 0; the value of K may be an integer greater than or equal to 0; and the value of L may be an integer greater than or equal to 0. The values ​​of M, N, K, and L cannot be 0 at the same time.

[0032] Among them, M texts, N images, K videos, and L audios can be combined in any order. And the data of each modality can be interspersed. For example, a text + an image + a video + a text + an audio, etc.

[0033] Among them, the modalities in the training data are at least two of text, images, audio and video, and the number is one or more, so that the multimodal large model can be combined with any number of sample data block sequences in any modality for training processing, so that it can fully learn the features in the data under each modality, so that the trained multimodal large model can be applicable to a variety of multimodal understanding tasks and multimodal generation tasks, expanding the applicable scenarios of the multimodal large model.

[0034] Step 102: splicing sample data block sequences under at least two modes to obtain a sample splicing sequence.

[0035] In an embodiment of the present disclosure, in one example, an electronic device may perform splicing processing on sample data block sequences under at least two modalities according to the arrangement order of the sample data block sequences under at least two modalities in training data to obtain a sample splicing sequence for training a multimodal large model.

[0036] In another example, in order to expand the amount of training data so that the multimodal large model can learn the interaction characteristics between data under multiple modalities, the electronic device can shuffle the arrangement order of the sample data block sequences under at least two modalities in the training data; splice the sample data block sequences under at least two modalities according to the shuffled order to obtain a shuffled sample splicing sequence for training the multimodal large model.

[0037] In the disclosed embodiment, in order to ensure that the multimodal large model can distinguish the sample data block sequences under each mode in the sample splicing sequence, so as to extract features in a targeted manner and improve the accuracy of feature extraction, the electronic device can perform head and tail modality marking processing on the sample data block sequence to obtain a processed sample data block sequence; and perform splicing processing on each processed sample data block sequence to obtain a sample splicing sequence.

[0038] The processing of marking the first and last modalities of the sample data chunk sequence refers to adding a modal marker before the first sample data chunk in the sample data chunk sequence and adding a modal marker after the last sample data chunk in the sample data chunk sequence. Sample data chunk sequences under different modalities may be marked with different modal markers.

[0039] Step 103, training the multimodal large model in combination with the sample splicing sequence.

[0040] In the disclosed embodiment, the electronic device may also perform the following process: obtaining a sample data block sequence corresponding to sample data in a single modality; and training a multimodal large model in combination with the sample data block sequence in a single modality. The single modality may be any one of text, image, video, and audio.

[0041] The training method of the multimodal large model of the embodiment of the present disclosure obtains training data and an initial multimodal large model; the training data includes: a sequence of sample data blocks corresponding to sample data under at least two modalities; the sample data block sequences under at least two modalities are spliced ​​to obtain a sample splicing sequence; the multimodal large model is trained in combination with the sample data block sequences under at least two modalities; wherein, the multimodal large model is trained in combination with the sample data block sequences under at least two modalities, so that the trained multimodal large model can be applicable to a variety of multimodal understanding tasks and multimodal generation tasks, thereby expanding the applicable scenarios of the multimodal large model; and only using the multimodal large model can avoid cascading errors caused by multiple network cascades, etc.

[0042] In order to further improve the accuracy of obtaining the sample data block sequence under at least two modalities in the training data, the electronic device can obtain the sample data block sequence by segmentation, horizontal arrangement or mapping of the sample data under at least two modalities. Figure 2 As shown, Figure 2 is a schematic diagram according to a second embodiment of the present disclosure, Figure 2 The illustrated embodiment may include the following steps:

[0043] Step 201, obtaining an initial multi-modal large model.

[0044] Step 202: Obtain sample data in at least two modes.

[0045] Step 203 : segmenting and horizontally arranging the sample data in the non-text mode among the at least two modes to obtain a sequence of sample data blocks in the non-text mode.

[0046] In the embodiment of the present disclosure, it is assumed that at least two modalities include image modality, video modality and audio modality. For the image modality, the process of the electronic device determining the sample data block sequence in the image modality may be, for example, performing two-dimensional segmentation processing on the image in the image modality to obtain multiple image blocks; performing horizontal arrangement processing on the multiple image blocks from left to right and from top to bottom according to the image dimension to obtain an image block sequence; and determining the image block sequence as a sample data block sequence in the image modality.

[0047] For the video modality, the process of the electronic device determining the sample data block sequence in the video modality may, for example, be: performing three-dimensional segmentation processing on the video in the video modality to obtain multiple video blocks (wherein the video blocks are obtained by combining image blocks at the same position in multiple consecutive frames of images); horizontally arranging the multiple video blocks from front to back in the time dimension and from left to right and from top to bottom in the image dimension to obtain a video block sequence; and determining the video block sequence as a sample data block sequence in the video modality.

[0048] For the audio modality, when the audio in the audio modality is audio in the time domain, in one example, the process of the electronic device determining the sample data block sequence in the audio modality may be, for example, dividing the audio according to time periods to obtain multiple audio segments; arranging the multiple audio segments in chronological order to obtain an audio segment sequence; and determining the audio segment sequence as a sample data block sequence in the audio modality.

[0049] For the audio modality, when the audio in the audio modality is audio in the time domain, in another example, the process of the electronic device determining the sample data block sequence in the audio modality can be, for example, performing frequency domain conversion processing on the audio to obtain a spectrogram; dividing the spectrogram according to frequency bands to obtain multiple audio frequency bands; arranging the multiple audio frequency bands in order of frequency bands to obtain an audio frequency band sequence; and determining the audio frequency band sequence as a sample data block sequence in the audio modality.

[0050] Among them, the spectrum diagram can reflect the distribution of audio in frequency and has a large amount of information, so that the multimodal large model can extract more features, thereby enabling the multimodal large model to learn more features and improve the accuracy of the trained multimodal large model.

[0051] Step 204: when the at least two modalities include a text modality, segment and map the sample data in the text modality in combination with the text vocabulary to obtain a sequence of sample data blocks in the text modality.

[0052] In the embodiment of the present disclosure, the text vocabulary may include words and integer identifiers corresponding to the words. Correspondingly, the process of the electronic device executing step 204 may be, for example, to perform word segmentation processing on the sample data in the text mode in combination with the words in the text vocabulary to obtain a word sequence; query the text vocabulary according to the words in the word sequence to obtain the integer identifiers corresponding to the words in the text vocabulary; determine the sequence composed of the integer identifiers corresponding to each word in the word sequence as the sample data block sequence in the text mode.

[0053] Among them, combining each word in the text vocabulary and the corresponding integer identifier to determine the sequence of sample data blocks under the text modality can reduce the amount of data that needs to be processed by the multimodal large model and improve the processing efficiency of the multimodal large model.

[0054] Step 205 , performing splicing processing on the sample data block sequences under at least two modes to obtain a sample splicing sequence.

[0055] In the disclosed embodiment, in order to further reduce the amount of data processing of the multimodal large model, before step 205, the electronic device may also perform the following process: combining a dimensionality change strategy to perform dimensionality reduction processing on a sequence of sample data blocks in a non-text mode in at least two modes.

[0056] Among them, the dimension change strategy includes at least one of the following: a dimension change strategy based on a variational autoencoder, and a dimension change strategy based on matrix transformation.

[0057] The variational autoencoder may include an encoder and a decoder. The encoder may map high-dimensional input data to a low-dimensional latent space to achieve dimensionality reduction. The decoder may map the low-dimensional latent space to high-dimensional output data to achieve dimensionality increase. The electronic device may combine the encoding network in the variational autoencoder to perform dimensionality reduction processing on each sample data block in the sample data block sequence under the non-text modality.

[0058] Among them, the dimension changing strategy based on matrix transformation, for example, performs deformation processing based on a deformation matrix, that is, reshapes a matrix into a matrix with different dimensions, but keeps the total number of elements unchanged.

[0059] Step 206, training the multimodal large model in combination with the sample concatenation sequence.

[0060] It should be noted that the details of step 205 to step 206 can be found in Figure 1 Steps 102 to 103 in the illustrated embodiment will not be described in detail herein.

[0061] The training method of the multimodal large model of the embodiment of the present disclosure comprises the following steps: obtaining an initial multimodal large model; obtaining sample data under at least two modalities; segmenting and horizontally arranging the sample data under the non-text modality in the at least two modalities to obtain a sequence of sample data blocks under the non-text modality; when the at least two modalities include a text modality, segmenting and mapping the sample data under the text modality in combination with a text vocabulary to obtain a sequence of sample data blocks under the text modality; splicing the sample data block sequences under the at least two modalities to obtain a sample splicing sequence; and training the multimodal large model in combination with the sample splicing sequence; wherein, by segmenting, horizontally arranging or mapping the sample data under the at least two modalities to obtain a sample data block sequence, the accuracy of obtaining the sample data block sequence under the at least two modalities in the training data is further improved, thereby further improving the accuracy of the trained multimodal large model.

[0062] In order to further improve the training speed and training accuracy of the multimodal large model, the electronic device can input the sample splicing sequence into the multimodal large model to obtain the predicted splicing sequence; and then determine the loss function value to perform training processing. Figure 3 As shown, Figure 3 is a schematic diagram according to a third embodiment of the present disclosure, Figure 3 The illustrated embodiment may include the following steps:

[0063] Step 301, obtaining training data and an initial multimodal large model; the training data includes: a sequence of sample data blocks corresponding to sample data under at least two modalities.

[0064] Step 302: splice sample data block sequences under at least two modes to obtain a sample splicing sequence.

[0065] Step 303: input the sample splicing sequence into the multimodal large model to obtain a predicted splicing sequence.

[0066] In the disclosed embodiment, the multimodal large model adopts a unidirectional attention mechanism. It is assumed that the sample splicing sequence is obtained by splicing the sample data block sequence under the first modality, the sample data block sequence under the second modality, and the sample data block sequence under the third modality. Correspondingly, the multimodal large model processes the sample splicing sequence to obtain the predicted splicing sequence in the following manner: for the last sample data block in the sample data block sequence under the first modality, the first predicted data block under the second modality is predicted based on the last sample data block under the first modality; for the last sample data block in the sample data block sequence under the second modality, the first predicted data block under the third modality is predicted based on the last sample data block under the second modality.

[0067] For the i-th sample data block in the sample data block sequence under the first mode, if the i-th sample data block is not the last sample data block, the i-th sample data block can be combined to predict the i+1-th prediction data block in the prediction data block sequence under the first mode. For the i-th sample data block in the sample data block sequence under the second mode, if the i-th sample data block is not the last sample data block, the i-th sample data block can be combined to predict the i+1-th prediction data block in the prediction data block sequence under the second mode. For the i-th sample data block in the sample data block sequence under the third mode, if the i-th sample data block is not the last sample data block, the i-th sample data block can be combined to predict the i+1-th prediction data block in the prediction data block sequence under the third mode.

[0068] Among them, the adoption of the unidirectional attention mechanism enables the multimodal large model to predict the first predicted data block in the predicted data block sequence under the next modality based on the last sample data block in the sample data block sequence under the previous modality in the sample splicing sequence, so that the multimodal large model can learn the interactive features between data under multiple modalities and realize cross-modal feature transfer.

[0069] Step 304, determining the loss function value of the multimodal large model according to the sample splicing sequence and the predicted splicing sequence.

[0070] In the embodiment of the present disclosure, the sample data block sequence in the sample splicing sequence corresponds to the prediction data block sequence in the prediction splicing sequence. Correspondingly, the process of the electronic device performing step 304 may be, for example, determining a sub-loss function value for each sample data block sequence in the sample splicing sequence according to the sample data block sequence and the prediction data block sequence corresponding to the sample data block sequence in the prediction splicing sequence; and determining the total loss function value according to at least two sub-loss function values.

[0071] The sample data blocks in the sample data block sequence correspond one-to-one to the prediction data blocks in the prediction data block sequence corresponding to the sample data block sequence. Correspondingly, the process of the electronic device determining the value of the sub-loss function may be, for example, determining, for each sample data block in the sample data block sequence, a difference value between the sample data block and the prediction data block corresponding to the sample data block; and determining the value of the sub-loss function according to the difference value corresponding to each sample data block in the sample data block sequence.

[0072] Among them, the determination of the difference value between the sample data block and the prediction data block, and the determination of the sub-loss function value between the sample data block sequence and the prediction data block sequence can improve the accuracy of the determined total loss function value and further improve the training accuracy of the multimodal large model.

[0073] Step 305, adjusting parameters of the multimodal large model according to the loss function value to implement training.

[0074] It should be noted that the details of step 301 to step 302 can be found in Figure 1 Steps 101 to 102 in the illustrated embodiment will not be described in detail herein.

[0075] The training method of the multimodal large model of the embodiment of the present disclosure is through obtaining training data and an initial multimodal large model; the training data includes: a sequence of sample data blocks corresponding to sample data under at least two modalities; the sample data block sequences under at least two modalities are spliced ​​to obtain a sample spliced ​​sequence; the sample spliced ​​sequence is input into the multimodal large model to obtain a predicted spliced ​​sequence; the loss function value of the multimodal large model is determined according to the sample spliced ​​sequence and the predicted spliced ​​sequence; the multimodal large model is parameter-adjusted according to the loss function value to achieve training; wherein, the loss function value is determined in combination with the sample spliced ​​sequence and the predicted spliced ​​sequence for training processing, which can further improve the training speed and training accuracy of the multimodal large model.

[0076] Figure 4 4 is a schematic diagram according to the fourth embodiment of the present disclosure. It should be noted that the method for processing a multimodal task in the embodiment of the present disclosure can be applied to a device for processing a multimodal task, and the device can be configured in an electronic device so that the electronic device can perform the processing function of the multimodal task. The following embodiments are described by taking the execution subject as an electronic device as an example.

[0077] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0078] The multimodal task processing device may also be software in an electronic device, such as multimodal task processing software, etc. In the following embodiments, the multimodal task processing device is an electronic device as an example for description.

[0079] like Figure 4 As shown, the processing method of the multimodal task may include the following steps:

[0080] Step 401, obtaining a multimodal task; the multimodal task includes data in at least one modality; data in a text modality in at least one modality is used to indicate that data in a target modality needs to be generated.

[0081] In the disclosed embodiment, the data in the Chinese mode can be used to describe the multimodal task, or to describe the target mode to which the data to be generated by the multimodal task belongs. For example, the data in the Chinese mode can include the name of the multimodal task, the target mode, etc.

[0082] Step 402, obtaining a multimodal large model; the multimodal large model is based on Figures 1 to 3The training method of the multimodal large model in any of the embodiments is determined.

[0083] Step 403, combining data under at least one modality and the multimodal large model to determine generated data under the target modality.

[0084] In the embodiment of the present disclosure, the process of the electronic device executing step 403 may, for example, be to segment and map the data in the text mode in combination with the text vocabulary to obtain a data block sequence in the text mode; when at least one mode includes a non-text mode, segment and horizontally arrange the data in the non-text mode to obtain a data block sequence in the non-text mode; splice the data block sequence in at least one mode to obtain a spliced ​​sequence; input the spliced ​​sequence into a multimodal large model to obtain a predicted sequence; and determine the generated data in the target mode based on the predicted data block sequence in the target mode in the predicted sequence.

[0085] In which, when at least one modality includes an audio modality, the data block sequence under the audio modality can be determined in combination with the audio in the time domain; or, the audio in the time domain can be transformed into the frequency domain to obtain the audio in the frequency domain, and then the data block sequence under the audio modality can be determined in combination with the audio in the frequency domain.

[0086] Among them, the segmentation, horizontal arrangement or mapping processing of the data under at least one modality can be carried out in units of data blocks in the data, so as to obtain the accuracy of the acquired data block sequence, and enable the multimodal large model to be trained and processed in units of data blocks, thereby further improving the accuracy of the trained multimodal large model.

[0087] In the disclosed embodiment, in order to further reduce the amount of data processing of the multimodal large model, before splicing the data block sequence under at least one modality to obtain the spliced ​​sequence, the electronic device may also perform the following process: when at least one modality includes a non-text modality, the data block sequence under the non-text modality is reduced in dimension by combining the variable dimension strategy. Correspondingly, after inputting the spliced ​​sequence into the multimodal large model and obtaining the predicted sequence, the electronic device may also perform the following process: when the target modality includes the target non-text modality, the data block sequence under the target non-text modality in the predicted sequence is increased in dimension by combining the variable dimension strategy.

[0088] Among them, the dimension change strategy may include at least one of the following: a dimension change strategy based on a variational autoencoder, and a dimension change strategy based on matrix transformation.

[0089] The method for processing a multimodal task in an embodiment of the present disclosure comprises: obtaining a multimodal task; the multimodal task comprises data in at least one mode; data in a text mode in at least one mode is used to indicate that data in a target mode needs to be generated; obtaining a multimodal large model; the multimodal large model is based on Figures 1 to 3 The training method of the multimodal large model in any of the embodiments is determined; combining the data under at least one modality and the multimodal large model, the generated data under the target modality is determined; wherein the multimodal large model is trained based on a sequence of sample data blocks corresponding to sample data under at least two modalities, and can be applied to processing a variety of multimodal understanding tasks and multimodal generation tasks, thereby improving the processing efficiency of multimodal tasks.

[0090] In order to implement the above embodiment, the present disclosure also provides a training device for a multi-modal large model. Figure 5 As shown, Figure 5 5 is a schematic diagram according to the fifth embodiment of the present disclosure. The multimodal large model training device 50 may include: an acquisition module 501 , a splicing processing module 502 and a training module 503 .

[0091] Among them, the acquisition module 501 is used to obtain training data and an initial multimodal large model; the training data includes: a sample data block sequence corresponding to the sample data under at least two modalities; the splicing processing module 502 is used to splice the sample data block sequence under the at least two modalities to obtain a sample splicing sequence; the training module 503 is used to train the multimodal large model in combination with the sample splicing sequence.

[0092] As a possible implementation of an embodiment of the present disclosure, the acquisition module 501 includes a first acquisition unit, a first processing unit, and a second processing unit; the first acquisition unit is used to acquire sample data under the at least two modalities; the first processing unit is used to segment and horizontally arrange the sample data under the non-text modality among the at least two modalities to obtain a sequence of sample data blocks under the non-text modality; the second processing unit is used to segment and map the sample data under the text modality in combination with a text vocabulary when the at least two modalities include a text modality to obtain a sequence of sample data blocks under the text modality.

[0093] As a possible implementation manner of an embodiment of the present disclosure, the at least two modes include an audio mode; the first processing unit is specifically used to, with respect to the audio data in the audio mode, perform frequency domain transform processing on the audio data to obtain frequency domain data; perform frequency band segmentation processing and horizontal arrangement processing on the frequency domain data to obtain a sample data block sequence in the audio mode.

[0094] As a possible implementation manner of the embodiment of the present disclosure, the at least two modalities include a video modality and an image modality; the first processing unit is specifically configured to, with respect to video data in the video modality, perform three-dimensional segmentation processing on the video data and perform horizontal arrangement processing according to a time dimension and an image dimension to obtain a sequence of sample data blocks in the video modality; with respect to image data in the image modality, perform two-dimensional segmentation processing on the image data and perform horizontal arrangement processing according to an image dimension to obtain a sequence of sample data blocks in the image modality.

[0095] As a possible implementation manner of an embodiment of the present disclosure, the text vocabulary includes words and integer identifiers corresponding to the words; the second processing unit is specifically used to perform word segmentation processing on the sample data in the text modality in combination with the words in the text vocabulary to obtain a word sequence; query the text vocabulary according to the words in the word sequence to obtain the integer identifiers corresponding to the words in the text vocabulary; and determine the sequence composed of the integer identifiers corresponding to each word in the word sequence as the sample data block sequence in the text modality.

[0096] As a possible implementation manner of the embodiment of the present disclosure, the splicing processing module 502 is specifically configured to, for each sample data chunk sequence, perform head and tail modality marking processing on the sample data chunk sequence to obtain a processed sample data chunk sequence; and perform splicing processing on each of the processed sample data chunk sequences to obtain a sample splicing sequence.

[0097] As a possible implementation of the embodiment of the present disclosure, the device also includes: a scrambling processing module, which is used to scramble the splicing order of each sample data block sequence in the sample splicing sequence to obtain a scrambled sample splicing sequence; the training module 503 is also used to train the multimodal large model in combination with the scrambled sample splicing sequence.

[0098] As a possible implementation of the embodiment of the present disclosure, the device further includes: a dimension change processing module, which is used to perform dimension reduction processing on the sample data block sequence in the non-text mode of the at least two modes in combination with the dimension change strategy.

[0099] As a possible implementation method of an embodiment of the present disclosure, the dimension change strategy includes at least one of the following: a dimension change strategy based on a variational autoencoder, and a dimension change strategy based on matrix transformation.

[0100] As a possible implementation method of an embodiment of the present disclosure, the training module 503 includes: a second acquisition unit, a determination unit and an adjustment processing unit; the second acquisition unit is used to input the sample splicing sequence into the multimodal large model to obtain a predicted splicing sequence; the determination unit is used to determine the loss function value of the multimodal large model according to the sample splicing sequence and the predicted splicing sequence; the adjustment processing unit is used to perform parameter adjustment processing on the multimodal large model according to the loss function value to achieve training.

[0101] As a possible implementation of an embodiment of the present disclosure, the multimodal large model adopts a unidirectional attention mechanism; the sample splicing sequence is obtained by splicing a sample data block sequence under the first modality, a sample data block sequence under the second modality, and a sample data block sequence under the third modality; the multimodal large model processes the sample splicing sequence to obtain the predicted splicing sequence in the following manner: for the last sample data block in the sample data block sequence under the first modality, according to the last sample data block under the first modality, predicting the first predicted data block under the second modality; for the last sample data block in the sample data block sequence under the second modality, according to the last sample data block under the second modality, predicting the first predicted data block under the third modality.

[0102] As a possible implementation manner of the embodiment of the present disclosure, a sample data block sequence in the sample splicing sequence corresponds one-to-one to a prediction data block sequence in the prediction splicing sequence; the determination unit is specifically configured to determine, for each sample data block sequence in the sample splicing sequence, a sub-loss function value according to the sample data block sequence and a prediction data block sequence corresponding to the sample data block sequence in the prediction splicing sequence; and determine a total loss function value according to at least two of the sub-loss function values.

[0103] As a possible implementation manner of the embodiment of the present disclosure, the sample data blocks in the sample data block sequence correspond one-to-one to the prediction data blocks in the prediction data block sequence corresponding to the sample data block sequence; the determining unit is further configured to determine, for each sample data block in the sample data block sequence, a difference value between the sample data block and the prediction data block corresponding to the sample data block; and determine the sub-loss function value according to the difference value corresponding to each sample data block in the sample data block sequence.

[0104] As a possible implementation manner of an embodiment of the present disclosure, the modalities in the training data include at least two of the following: text, image, audio, and video; and the number of sample data block sequences under each modality in the training data is one or more.

[0105] The training device of the multimodal large model of the embodiment of the present disclosure obtains training data and an initial multimodal large model; the training data includes: a sequence of sample data blocks corresponding to sample data under at least two modalities; the sample data block sequences under at least two modalities are spliced ​​to obtain a sample splicing sequence; the multimodal large model is trained in combination with the sample data block sequences under at least two modalities; wherein, the multimodal large model is trained in combination with the sample data block sequences under at least two modalities, so that the trained multimodal large model can be applicable to a variety of multimodal understanding tasks and multimodal generation tasks, thereby expanding the applicable scenarios of the multimodal large model; and only using the multimodal large model can avoid cascading errors caused by multiple network cascades, etc.

[0106] In order to implement the above embodiment, the present disclosure also provides a multi-modal task processing device. Figure 6 As shown, Figure 6 6 is a schematic diagram according to the sixth embodiment of the present disclosure. The multimodal task processing device 60 may include: a first acquisition module 601 , a second acquisition module 602 and a determination module 603 .

[0107] Among them, the first acquisition module 601 is used to acquire a multimodal task; the multimodal task includes data in at least one mode; the data in the text mode in the at least one mode is used to indicate that data in the target mode needs to be generated; the second acquisition module 602 is used to acquire a multimodal large model; the multimodal large model is based on Figures 1 to 3 The training method of the multimodal large model in any of the embodiments is determined; a determination module 603 is used to determine the generated data under the target modality by combining the data under the at least one modality and the multimodal large model.

[0108] As a possible implementation of an embodiment of the present disclosure, the determination module 603 is specifically used to segment and map the data in the text modality in combination with a text vocabulary to obtain a data block sequence in the text modality; when at least one of the modalities includes a non-text modality, segment and horizontally arrange the data in the non-text modality to obtain a data block sequence in the non-text modality; splice the data block sequence in the at least one modality to obtain a spliced ​​sequence; input the spliced ​​sequence into the multimodal large model to obtain a predicted sequence; and determine the generated data in the target modality based on the predicted data block sequence in the target modality in the predicted sequence.

[0109] As a possible implementation of an embodiment of the present disclosure, the device also includes: a variable dimension processing module, which is used to, when the at least one modality includes a non-text modality, combine the variable dimension strategy to perform dimensionality reduction processing on the data block sequence under the non-text modality; the variable dimension processing module is also used to, when the target modality includes a target non-text modality, combine the variable dimension strategy to perform dimensionality increase processing on the data block sequence under the target non-text modality in the prediction sequence.

[0110] The processing device of the multimodal task of the embodiment of the present disclosure obtains the multimodal task; the multimodal task includes data in at least one mode; the data in the text mode of at least one mode is used to indicate that data in the target mode needs to be generated; obtains a multimodal large model; the multimodal large model is based on Figures 1 to 3 The training method of the multimodal large model in any of the embodiments is determined; combining the data under at least one modality and the multimodal large model, the generated data under the target modality is determined; wherein the multimodal large model is trained based on a sequence of sample data blocks corresponding to sample data under at least two modalities, and can be applied to processing a variety of multimodal understanding tasks and multimodal generation tasks, thereby improving the processing efficiency of multimodal tasks.

[0111] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with the relevant laws and regulations, and do not violate public order and good morals.

[0112] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0113] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0114] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0115] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a training method for a multimodal large model or a processing method for a multimodal task. For example, in some embodiments, the training method for a multimodal large model or the processing method for a multimodal task may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the training method for the multimodal large model or the processing method for the multimodal task described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute a training method for a multimodal large model or a processing method for a multimodal task in any other appropriate manner (for example, by means of firmware).

[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0119] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a multimodal large model, the method comprising: Obtain training data and an initial multimodal large model; The training data includes: a sequence of sample data blocks corresponding to sample data under at least two modalities; Performing splicing processing on the sample data block sequences under the at least two modes to obtain a sample splicing sequence; The multimodal large model is trained in combination with the sample splicing sequence.

2. The method according to claim 1, wherein: Get training data, including: Acquire sample data under the at least two modes; Performing segmentation and horizontal arrangement processing on the sample data in the non-text mode of the at least two modes to obtain a sequence of sample data blocks in the non-text mode; In the case where the at least two modalities include a text modality, the sample data in the text modality is segmented and mapped in combination with a text vocabulary to obtain a sequence of sample data blocks in the text modality.

3. The method according to claim 2, wherein: The at least two modes include an audio mode; the segmentation and horizontal arrangement of the sample data in the non-text mode of the at least two modes to obtain a sequence of sample data blocks in the non-text mode includes: For the audio data in the audio mode, performing frequency domain transformation processing on the audio data to obtain frequency domain data; The frequency domain data is subjected to frequency band segmentation processing and horizontal arrangement processing to obtain a sample data block sequence under the audio mode.

4. The method according to claim 2, wherein: The at least two modalities include a video modality and an image modality; the segmentation and horizontal arrangement of the sample data in the non-text modality of the at least two modalities to obtain a sample data block sequence in the non-text modality includes: For the video data in the video mode, three-dimensional segmentation processing is performed on the video data and horizontal arrangement processing is performed according to the time dimension and the image dimension to obtain a sample data block sequence in the video mode; For the image data in the image modality, two-dimensional segmentation processing is performed on the image data and horizontal arrangement processing is performed according to the image dimension to obtain a sample data block sequence in the image modality.

5. The method according to claim 2, wherein: The text vocabulary includes words and integer identifiers corresponding to the words; the combining text vocabulary performs segmentation and mapping processing on the sample data in the text modality to obtain a sequence of sample data blocks in the text modality, including: Performing word segmentation processing on the sample data in the text modality in combination with the words in the text vocabulary to obtain a word sequence; Query the text vocabulary according to the words in the word sequence to obtain an integer identifier corresponding to the word in the text vocabulary; A sequence consisting of integer identifiers corresponding to each word in the word sequence is determined as a sample data block sequence in the text mode.

6. The method according to claim 1, wherein: The step of performing splicing processing on the sample data block sequences under the at least two modes to obtain a sample splicing sequence includes: For each sample data chunk sequence, performing head and tail modality marking processing on the sample data chunk sequence to obtain a processed sample data chunk sequence; The processed sample data block sequences are spliced ​​to obtain a sample splicing sequence.

7. The method according to claim 1 or 6, wherein: The method further comprises: Performing a shuffling process on the splicing order of each sample data block sequence in the sample splicing sequence to obtain a shuffled sample splicing sequence; The multimodal large model is trained in combination with the shuffled sample concatenation sequence.

8. The method according to claim 1, wherein: Before splicing the sample data block sequences under the at least two modalities to obtain the sample splicing sequence, the method further includes: performing dimensionality reduction processing on the sample data block sequence under the non-text modality of the at least two modalities in combination with a dimensionality change strategy.

9. The method according to claim 8, wherein: The dimension change strategy includes at least one of the following: a dimension change strategy based on a variational autoencoder, and a dimension change strategy based on matrix transformation.

10. The method according to claim 1, wherein: The step of training the multimodal large model in combination with the sample splicing sequence includes: Inputting the sample splicing sequence into the multimodal large model to obtain a predicted splicing sequence; Determining a loss function value of the multimodal large model according to the sample splicing sequence and the predicted splicing sequence; The parameters of the multimodal large model are adjusted according to the loss function value to achieve training.

11. The method according to claim 10, wherein: The multimodal large model adopts a unidirectional attention mechanism; the sample splicing sequence is obtained by splicing a sample data block sequence under the first modality, a sample data block sequence under the second modality, and a sample data block sequence under the third modality; The method in which the multimodal large model processes the sample splicing sequence to obtain the predicted splicing sequence includes: For a last sample data block in the sequence of sample data blocks in the first mode, predicting a first prediction data block in the second mode according to the last sample data block in the first mode; For the last sample data block in the sequence of sample data blocks under the second mode, a first prediction data block under the third mode is predicted based on the last sample data block under the second mode.

12. The method according to claim 10, wherein: The sample data block sequence in the sample splicing sequence corresponds one-to-one to the prediction data block sequence in the prediction splicing sequence; and determining the loss function value of the multimodal large model according to the sample splicing sequence and the prediction splicing sequence includes: For each sample data block sequence in the sample splicing sequence, determine a sub-loss function value according to the sample data block sequence and a prediction data block sequence corresponding to the sample data block sequence in the prediction splicing sequence; Determine a total loss function value based on at least two of the sub-loss function values.

13. The method according to claim 12, wherein: The sample data blocks in the sample data block sequence correspond one-to-one to the prediction data blocks in the prediction data block sequence corresponding to the sample data block sequence; and determining a sub-loss function value according to the sample data block sequence and the prediction data block sequence corresponding to the sample data block sequence in the prediction concatenated sequence, comprising: For each specimen data chunk in the sequence of specimen data chunks, determining a difference value between the specimen data chunk and a prediction data chunk corresponding to the specimen data chunk; The sub-loss function value is determined according to the difference value corresponding to each sample data block in the sample data block sequence.

14. The method according to claim 1, wherein: The modalities in the training data include at least two of the following: text, image, audio, and video; The number of sample data block sequences under each modality in the training data is one or more.

15. A method for processing a multimodal task, the method comprising: Acquire multimodal tasks; The multimodal task includes data in at least one modality; The data in the text mode in the at least one mode is used to indicate that data in the target mode needs to be generated; Acquire a multimodal large model; the multimodal large model is determined based on the training method of the multimodal large model according to any one of claims 1 to 14; The generated data under the target modality is determined by combining the data under the at least one modality and the multimodal large model.

16. The method according to claim 15, wherein: The combining the data under the at least one modality and the multimodal large model to determine the generated data under the target modality includes: Segmenting and mapping the data in the text mode in combination with the text vocabulary to obtain a data block sequence in the text mode; In the case where the at least one modality includes a non-text modality, segmenting and horizontally arranging the data in the non-text modality to obtain a data block sequence in the non-text modality; Performing splicing processing on the data block sequence under the at least one mode to obtain a spliced ​​sequence; Inputting the spliced ​​sequence into the multimodal macromodel to obtain a predicted sequence; Determine generated data under the target modality according to a prediction data block sequence under the target modality in the prediction sequence.

17. The method according to claim 16, wherein: Before splicing the data block sequence under the at least one modality to obtain the spliced ​​sequence, the method further includes: when the at least one modality includes a non-text modality, performing dimensionality reduction processing on the data block sequence under the non-text modality in combination with a dimensionality change strategy; Correspondingly, after inputting the spliced ​​sequence into the multimodal large model to obtain the predicted sequence, the method further includes: when the target modality includes a target non-text modality, performing dimensionality increase processing on the data block sequence under the target non-text modality in the predicted sequence in combination with the dimensionality change strategy.

18. A training device for a multimodal large model, the device comprising: The acquisition module is used to obtain training data and the initial multimodal large model; The training data includes: a sequence of sample data blocks corresponding to sample data under at least two modalities; A splicing processing module, used for splicing the sample data block sequences under the at least two modes to obtain a sample splicing sequence; A training module is used to train the multimodal large model in combination with the sample splicing sequence.

19. The device according to claim 18, wherein: The acquisition module includes a first acquisition unit, a first processing unit and a second processing unit; The first acquisition unit is used to acquire sample data under the at least two modes; The first processing unit is used to perform segmentation processing and horizontal arrangement processing on the sample data in the non-text mode of the at least two modes to obtain a sequence of sample data blocks in the non-text mode; The second processing unit is used to, when the at least two modalities include a text modality, segment and map the sample data in the text modality in combination with a text vocabulary to obtain a sequence of sample data blocks in the text modality.

20. The device according to claim 19, wherein The at least two modes include an audio mode; the first processing unit is specifically configured to: For the audio data in the audio mode, performing frequency domain transformation processing on the audio data to obtain frequency domain data; The frequency domain data is subjected to frequency band segmentation processing and horizontal arrangement processing to obtain a sample data block sequence under the audio mode.

21. The device according to claim 19, wherein The at least two modalities include a video modality and an image modality; the first processing unit is specifically configured to: For the video data in the video mode, three-dimensional segmentation processing is performed on the video data and horizontal arrangement processing is performed according to the time dimension and the image dimension to obtain a sample data block sequence in the video mode; For the image data in the image modality, two-dimensional segmentation processing is performed on the image data and horizontal arrangement processing is performed according to the image dimension to obtain a sample data block sequence in the image modality.

22. The device according to claim 19, wherein The text vocabulary includes words and integer identifiers corresponding to the words; the second processing unit is specifically configured to: Performing word segmentation processing on the sample data in the text modality in combination with the words in the text vocabulary to obtain a word sequence; Query the text vocabulary according to the words in the word sequence to obtain an integer identifier corresponding to the word in the text vocabulary; A sequence consisting of integer identifiers corresponding to each word in the word sequence is determined as a sample data block sequence in the text mode.

23. The device according to claim 18, wherein The splicing processing module is specifically used for: For each sample data chunk sequence, performing head and tail modality marking processing on the sample data chunk sequence to obtain a processed sample data chunk sequence; The processed sample data block sequences are spliced ​​to obtain a sample splicing sequence.

24. The device according to claim 18 or 23, wherein: The device further comprises: a shuffling processing module, configured to shuffle the splicing order of each sample data block sequence in the sample splicing sequence to obtain a shuffled sample splicing sequence; The training module is also used to train the multimodal large model in combination with the shuffled sample splicing sequence.

25. The apparatus of claim 18, wherein: The device further comprises: a dimension change processing module, which is used to perform dimension reduction processing on the sample data block sequence in the non-text mode of the at least two modes in combination with the dimension change strategy.

26. The device according to claim 25, wherein The dimension change strategy includes at least one of the following: a dimension change strategy based on a variational autoencoder, and a dimension change strategy based on matrix transformation.

27. The device according to claim 18, wherein The training module includes: a second acquisition unit, a determination unit and an adjustment processing unit; The second acquisition unit is used to input the sample splicing sequence into the multimodal large model to obtain a predicted splicing sequence; The determining unit is used to determine the loss function value of the multimodal large model according to the sample splicing sequence and the predicted splicing sequence; The adjustment processing unit is used to perform parameter adjustment processing on the multimodal large model according to the loss function value to achieve training.

28. The device according to claim 27, wherein The multimodal large model adopts a unidirectional attention mechanism; the sample splicing sequence is obtained by splicing a sample data block sequence under the first modality, a sample data block sequence under the second modality, and a sample data block sequence under the third modality; The method in which the multimodal large model processes the sample splicing sequence to obtain the predicted splicing sequence includes: For a last sample data block in the sequence of sample data blocks in the first mode, predicting a first prediction data block in the second mode according to the last sample data block in the first mode; For the last sample data block in the sequence of sample data blocks under the second mode, a first prediction data block under the third mode is predicted based on the last sample data block under the second mode.

29. The device according to claim 27, wherein: The sample data block sequence in the sample splicing sequence corresponds one-to-one to the prediction data block sequence in the prediction splicing sequence; the determining unit is specifically configured to: For each sample data block sequence in the sample splicing sequence, determine a sub-loss function value according to the sample data block sequence and a prediction data block sequence corresponding to the sample data block sequence in the prediction splicing sequence; Determine a total loss function value based on at least two of the sub-loss function values.

30. The device according to claim 29, wherein: The sample data blocks in the sample data block sequence correspond one-to-one to the prediction data blocks in the prediction data block sequence corresponding to the sample data block sequence; the determining unit is further configured to: For each specimen data chunk in the sequence of specimen data chunks, determining a difference value between the specimen data chunk and a prediction data chunk corresponding to the specimen data chunk; The sub-loss function value is determined according to the difference value corresponding to each sample data block in the sample data block sequence.

31. The apparatus of claim 18, wherein: The modalities in the training data include at least two of the following: text, image, audio, and video; The number of sample data block sequences under each modality in the training data is one or more.

32. A multimodal task processing device, the device comprising: A first acquisition module, used for acquiring multimodal tasks; The multimodal task includes data in at least one modality; The data in the text mode in the at least one mode is used to indicate that data in the target mode needs to be generated; A second acquisition module is used to acquire a multimodal large model; the multimodal large model is determined based on the training method of the multimodal large model according to any one of claims 1 to 14; A determination module is used to determine the generated data under the target modality by combining the data under the at least one modality and the multimodal large model.

33. The device according to claim 32, wherein: The determination module is specifically used to: Segmenting and mapping the data in the text mode in combination with the text vocabulary to obtain a data block sequence in the text mode; In the case where the at least one modality includes a non-text modality, segmenting and horizontally arranging the data in the non-text modality to obtain a data block sequence in the non-text modality; Performing splicing processing on the data block sequence under the at least one mode to obtain a spliced ​​sequence; Inputting the spliced ​​sequence into the multimodal macromodel to obtain a predicted sequence; Determine generated data under the target modality according to a prediction data block sequence under the target modality in the prediction sequence.

34. The device according to claim 33, wherein: The device further comprises: a dimension change processing module, which is used to perform dimension reduction processing on the data block sequence in the non-text mode in combination with the dimension change strategy when the at least one mode includes a non-text mode; The dimension changing processing module is also used to perform dimension increasing processing on the data block sequence under the target non-text mode in the prediction sequence in combination with the dimension changing strategy when the target mode includes a target non-text mode.

35. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 14; Alternatively, the method according to any one of claims 15 to 17 is performed.

36. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14; or, to execute the method according to any one of claims 15 to 17.

37. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14; or, implements the method according to any one of claims 15 to 17.

Images