Text-audio-visual multi-modal pre-training model method and device, electronic equipment and medium

Abstract

The application relates to the technical field of computers and provides a kind of picture-text-audio multi-modal pre-training model method, device, electronic equipment and medium, wherein the method comprises: obtaining multi-modal information of training samples;Based on the semantic similarity between the text modal feature and other modal features in the multi-modal information, the multi-modal information is grouped and aligned, the modal grouping is obtained and the grouping alignment loss is determined;Based on one of the picture, the sound and the picture-sound modal in each modal grouping, the text of the random mask is reconstructed, and the text reconstruction loss is determined based on each group of reconstructed text and each group of sample text;Based on the grouping alignment loss and the text reconstruction loss, the parameters of the model are iterated to obtain a multi-modal model. The picture-text-audio multi-modal pre-training model method, device, electronic equipment and medium provided by the application can improve the downstream task performance and generalization performance of the multi-modal model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and medium for pre-training a multimodal model of graphics, text, and audio. Background Technology

[0002] Multimodal pre-training is an interdisciplinary field encompassing multiple research areas and involving various modalities of data. This task aims to train a general multimodal model using large-scale datasets to achieve the understanding and generation of information across various modalities, such as text-video retrieval, audio-video text reconstruction, and visual question answering.

[0003] In the current field of multimodal pre-training, common methods and frameworks use only two modalities of data for learning, such as visual and language modalities, including but not limited to image-text and video-text data. However, videos in real-world scenarios often also contain audio data, which provides rich semantic information to the video data, either related to it or as a supplement. Common methods and frameworks ignore the usable information in this audio data, limiting model performance improvement. Some researchers have proposed using video, text, and audio data for multimodal pre-training; however, in currently available public datasets, the data correlation between these three modalities is weak, leading to difficulties in models learning multimodal information fusion and alignment. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and medium for pre-training a multimodal model with text, image, and audio, in order to address the limitation of performance improvement in existing multimodal models.

[0005] This invention provides a method for pre-training a multimodal image-text-audio model, comprising:

[0006] Obtain multimodal information of training samples, wherein the multimodal information includes visual, audio and text, and the visual information includes video and / or images;

[0007] Based on the semantic similarity between text modal features and other modal features in the multimodal information, the multimodal information is grouped and aligned to obtain each modal group and determine the group alignment loss; the other modal features include one of visual modal features, audio modal features, and audio-visual combined modal features;

[0008] Based on other modal features in each modality group, text reconstruction is performed on the text with a random mask to obtain the reconstructed text of each group, and the text reconstruction loss is determined based on the reconstructed text of each group and the sample text of each group.

[0009] Based on the group alignment loss and the text reconstruction loss, the model is iterated to obtain a multimodal model.

[0010] According to the image-text-audio multimodal pre-training model method provided by the present invention, each modal grouping includes at least one of text-visual modal grouping, text-audio modal grouping, and text-audio-visual combined modal grouping, and the determination of group alignment loss includes:

[0011] Based on the semantic similarity between the text modal features and the visual modal features, the first group alignment loss is determined;

[0012] Based on the semantic similarity between the text modal features and the audio modal features, the second group alignment loss is determined;

[0013] Based on the semantic similarity between the text modal features and the audiovisual combined modal features, the third group alignment loss is determined;

[0014] The group alignment loss is determined based on at least one of the first group alignment loss, the second group alignment loss, and the third group alignment loss.

[0015] According to the image-text-audio multimodal pre-training model method provided by the present invention, the step of determining the semantic similarity between text modal features and other modal features in the multimodal information includes:

[0016] Based on the weights of the local features of each modality in the multimodal information, a weighted average of the local features of each modality is performed to obtain the global features of each modality.

[0017] Based on the global features of each modality, the semantic similarity between text modal features and other modal features in the multimodal information is determined.

[0018] According to the multimodal pre-training model method for text, image, and audio provided by the present invention, the step of reconstructing text from randomly masked text based on other modal features in each modal group to obtain reconstructed text for each group includes:

[0019] The text modal features and other modal features in each modal group are fused across modal features to obtain the fused features of each modal group;

[0020] Based on the fusion features of each modality group, a text masking modeling method is applied to reconstruct the text of each modality group using a random mask, thereby obtaining the reconstructed text of each group.

[0021] According to the image-text-audio multimodal pre-training model method provided by the present invention, the step of determining the text reconstruction loss based on the reconstructed text of each group and the sample text of each group includes:

[0022] The first text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-visual modality grouping.

[0023] The second text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-audio modality grouping.

[0024] The third text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-audio-visual combined modality grouping.

[0025] The text reconstruction loss is determined based on at least one of the first text reconstruction loss, the second text reconstruction loss, and the third text reconstruction loss.

[0026] According to the image-text-audio multimodal pre-training model method provided by the present invention, the step of grouping and aligning the multimodal information based on the semantic similarity between the text modal features and other modal features in the multimodal information further includes:

[0027] Based on each single-modal encoder, feature extraction is performed on the multimodal information to obtain visual modal features, audio modal features, and text modal features;

[0028] The visual modal features and the audio modal features are fused to obtain the audio-visual combined modal features.

[0029] According to the multimodal pre-training model method for text, image, and audio provided by the present invention, the multimodal model is used for at least one of text-visual retrieval, text-audio retrieval, text-audio-visual retrieval, visual text annotation, audio text annotation, audio-visual text annotation, visual question answering, and audio-visual question answering.

[0030] The present invention also provides a device for pre-training a multimodal image-text-audio model, comprising:

[0031] An information acquisition unit is used to acquire multimodal information of training samples, wherein the multimodal information includes visual, audio and text, and the visual information includes video and / or images;

[0032] The grouping alignment unit is used to perform feature grouping and alignment of the multimodal information based on the semantic similarity between text modal features and other modal features in the multimodal information, to obtain each modal group and determine the grouping alignment loss; the other modal features include one of visual modal features, audio modal features, and audio-visual combined modal features;

[0033] The text reconstruction unit is used to reconstruct text from the randomly masked text based on other modal features in each modality group, to obtain reconstructed text in each group, and to determine the text reconstruction loss based on the reconstructed text in each group and the sample text in each group.

[0034] The parameter iteration unit is used to perform parameter iteration on the model based on the group alignment loss and the text reconstruction loss to obtain a multimodal model.

[0035] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the graph-text-audio multimodal pre-training model method as described above.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the graph-text-audio multimodal pre-training model method as described above.

[0037] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the graph-text-audio multimodal pre-training model method as described above.

[0038] The present invention provides a method, apparatus, electronic device, and medium for pre-training a multimodal model of image, text, and audio. It makes full use of the strong correlation between visual, audio, and text multimodal information, designs a multimodal understanding and generation pre-training method, and iterates the parameters of the model based on group alignment loss and text reconstruction loss. This enables the trained multimodal model to learn the alignment of visual, audio, and text data more accurately, enhances the cross-modal understanding and generation capabilities of the multimodal model, and improves the downstream task performance and generalization performance of the multimodal model. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is one of the flowcharts illustrating the image-text-audio multimodal pre-training model method provided by the present invention;

[0041] Figure 2 This is a schematic diagram of the multimodal grouping alignment learning task provided by the present invention;

[0042] Figure 3 This is a schematic diagram of the multimodal grouped text reconstruction task provided by the present invention;

[0043] Figure 4 This is a schematic diagram of the modeling of the multimodal model provided by the present invention;

[0044] Figure 5This is a schematic diagram of the structure of the image-text-audio multimodal pre-training model device provided by the present invention;

[0045] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] Multimodal pre-training aims to train a general multimodal model using large-scale datasets, thereby enabling the understanding and generation of information across various modalities. In the multimodal field, various cross-modal applications have been extensively studied, with visual-language tasks currently representing the mainstream research area.

[0048] However, current methods and frameworks mainly focus on cross-modal applications between two modalities, which makes it difficult to achieve the goal of building a general multimodal model in multimodal pre-training. In addition, some scholars have proposed using video, text and audio data for multimodal pre-training; however, in currently available public datasets, the data correlation between the above three modalities is weak, which makes it difficult for the model to learn the fusion and alignment of multimodal information.

[0049] To address the aforementioned shortcomings and improve the performance of multimodal models, this invention provides a pre-training method for a vision-audio-text full-perception model from the perspective of pre-training strongly correlated vision, audio, and text multimodal models.

[0050] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is one of the flowcharts illustrating the image-text-audio multimodal pre-training model method provided by the present invention. The execution entity for each step in this method can be an image-text-audio multimodal pre-training model device. This device can be implemented through software and / or hardware, and can be integrated into an electronic device. The electronic device can be a terminal device (such as a smartphone, personal computer, etc.), a server (such as a local server or cloud server, or a server cluster, etc.), a processor, or a chip, etc. Figure 1 As shown, the method may include the following steps:

[0051] Step 110: Obtain multimodal information of the training samples. The multimodal information includes visual, audio, and text information. Visual information includes video and / or images.

[0052] Specifically, multimodal information is strongly correlated data encompassing visual, audio, and textual modalities. It includes information from these three modalities, where visual information can include video and / or images. These three modalities are strongly correlated. For example, a set of trimodal information might contain visual elements such as a puppy, a sofa, and a living room; audio elements include the corresponding barking of the puppy and the sound of a police siren in the background; and text elements describing the combined visual and audio information: "In the living room, a black dog is lying on the sofa barking, with the sound of a police siren in the background."

[0053] Step 120: Based on the semantic similarity between text modal features and other modal features in multimodal information, perform feature grouping and alignment of multimodal information to obtain each modal group and determine the grouping alignment loss; other modal features include one of visual modal features, audio modal features and audio-visual combined modal features.

[0054] Specifically, in order to train a general multimodal model and realize the understanding and generation of information across various modalities, multimodal group alignment (MGA) task learning and multimodal group text reconstruction (MGC) task learning can be performed on the model to be trained.

[0055] For the learning of multimodal group alignment (MGA) tasks, considering that the data correlation between the three modalities in the existing public datasets is weak, which makes it difficult for the model to learn the alignment of multimodal information, this embodiment of the invention performs feature group alignment of multimodal information based on the semantic similarity between text modal features and other modal features in multimodal information.

[0056] It should be noted that, in order to achieve feature group alignment, features can be extracted in advance for each modality to obtain each modality feature. The modality features here may include text modality features, visual modality features, audio modality features, and audio-visual combined modality features.

[0057] Preferably, feature group alignment can be achieved through a multimodal group alignment module of the model to be trained. Each modality's features are input into the multimodal group alignment module, mapping the features of the visual, audio, and text modalities to a common space. A contrastive learning method is then used to establish fine-grained feature alignment between different modalities within the visual-text, audio-text, and audio-visual-text modal groups.

[0058] Group alignment loss can be used to measure the similarity between text modal features and other modal features. During model training, modal features that match the semantics of text modal features will be grouped together, while modal features that do not match the semantics of text modal features will be kept apart.

[0059] Understandably, within each modality group, the greater the semantic similarity between text modality features and other modality features, the greater the group alignment loss; conversely, the smaller the semantic similarity between text modality features and other modality features, the smaller the group alignment loss.

[0060] Step 130: Based on the other modal features in each modal group, perform text reconstruction on the text of the random mask to obtain the reconstructed text of each group, and determine the text reconstruction loss based on the reconstructed text of each group and the sample text of each group.

[0061] Specifically, for the Multimodal Grouped Text Reconstruction (MGC) task, based on the obtained modal groups, text reconstruction can be performed separately for each modal group according to the different modal features within each modal group. Here, text reconstruction can be the restoration and reconstruction of the masked target text to obtain the text sequence prediction output under multimodal conditions, that is, to obtain the reconstructed text of each group.

[0062] Preferably, text reconstruction can be achieved through a cross-modal decoder of the model to be trained. The features of each modality are input into the cross-modal decoder, which then performs text reconstruction for each modality group separately, resulting in the reconstructed text for each group.

[0063] Based on this, the text reconstruction loss is determined according to the differences between each group of reconstructed text and each group of sample text. The text reconstruction loss here can be negative log-likelihood loss, cross-entropy loss, etc., and the embodiments of the present invention do not specifically limit it.

[0064] Step 140: Based on the group alignment loss and text reconstruction loss, perform parameter iteration on the model to obtain a multimodal model.

[0065] Specifically, based on steps 120 and 130, in order to improve the multimodal model's ability to understand and generate multimodal information, group alignment loss and text reconstruction loss can be used simultaneously as the target loss of the model to be trained, and the parameters of the model to be trained can be iterated to obtain the multimodal model.

[0066] In one embodiment, the loss function of the multimodal model can be expressed as follows:

[0067] L=αL MGA +L MGC

[0068] Where α is the weighting coefficient, L MGA L represents the group alignment loss. MGC This indicates the text reconstruction loss.

[0069] The method provided in this invention fully utilizes the strong correlation between visual, audio, and textual multimodal information to design a multimodal understanding and generation pre-training method. Based on group alignment loss and text reconstruction loss, the model's parameters are iterated, enabling the trained multimodal model to learn the alignment methods of visual, audio, and textual data more accurately. This enhances the multimodal model's cross-modal understanding and generation capabilities, and improves the downstream task performance and generalization performance of the multimodal model.

[0070] Based on the above embodiments, each modal grouping includes at least one of text-visual modal grouping, text-audio modal grouping, and text-audio-visual combined modal grouping. Step 120, determining the grouping alignment loss, specifically includes:

[0071] The first group alignment loss is determined based on the semantic similarity between text modal features and visual modal features;

[0072] The second group alignment loss is determined based on the semantic similarity between text modal features and audio modal features;

[0073] The third group alignment loss is determined based on the semantic similarity between text modal features and audiovisual combined modal features;

[0074] The group alignment loss is determined based on at least one of the first group alignment loss, the second group alignment loss, and the third group alignment loss.

[0075] Specifically, each modal group may include at least one of the following: text-visual modal group, text-audio modal group, and text-audio-visual combined modal group.

[0076] Among them, the text-visual modality grouping consists of visual and text information in multimodal data, where visual information may include video and / or images; the text-audio modality grouping consists of audio and text information in multimodal data; and the text-audio-visual combined modality grouping consists of audio-visual combined information and text information in multimodal data.

[0077] It should be noted that before calculating the semantic similarity between text modal features and visual modal features, each modal feature can be mapped to a common space, and the semantic similarity can be calculated using a contrastive learning method.

[0078] The group alignment loss for each group is determined based on the feature similarity between different modalities within each group. The first group alignment loss is determined based on the semantic similarity between text modal features and visual modal features; the second group alignment loss is determined based on the semantic similarity between text modal features and audio modal features; and the third group alignment loss is determined based on the semantic similarity between text modal features and audio-visual combined modal features. This method can bring content-matching features closer together while distancing content-mismatching features from each other.

[0079] The alignment loss for each group can be expressed as follows:

[0080]

[0081] Where B is the batch size during the learning process, T is the set of text modal features within the batch, X is the set of visual modal features, audio modal features, or audio-visual combined modal features within the batch, s(·,·) is the similarity function, and τ is the temperature coefficient.

[0082] Based on the obtained group alignment loss, the group alignment loss can be determined based on at least one of the first group alignment loss, the second group alignment loss, and the third group alignment loss.

[0083] In one embodiment, Figure 2 This is a schematic diagram of the multimodal group alignment learning task provided by the present invention, such as... Figure 2 As shown, the group alignment loss of each group is determined based on the feature similarity between different modalities within each group. The group alignment losses are summed to obtain the task loss of the multimodal group alignment learning task.

[0084] Furthermore, the first group alignment loss, the second group alignment loss, and the third group alignment loss can be weighted and averaged to obtain the final group alignment loss. The group alignment loss can be expressed as follows:

[0085]

[0086] In the formula, L MGA The expression represents the group alignment loss, where T is the text modality, AV is the audio-visual combined modality, V is the visual modality, A is the audio modality, and L is the group alignment loss. MGA(T-V) L represents the alignment loss of the first group. MGA(T-A) L represents the alignment loss of the second group. MGA(T-AV) This represents the alignment loss of the third group.

[0087] The method provided in this invention determines the group alignment loss of a multimodal model by using the group alignment loss of each modality group, enabling the multimodal model to learn the alignment methods of visual, audio, and text data more accurately.

[0088] Furthermore, through multimodal grouping alignment learning, visual, audio, and textual modal data are mapped to a common space, while the data from the three modalities are finely aligned to enable cross-modal fusion, understanding, and generation of multimodal models.

[0089] Based on any of the above embodiments, the steps for determining the semantic similarity between text modal features and other modal features in multimodal information include:

[0090] Based on the weights of the local features of each modality in the multimodal information, the local features of each modality are weighted and averaged to obtain the global features of each modality.

[0091] Based on the global features of each modality, the semantic similarity between text modal features and other modal features in multimodal information is determined.

[0092] Specifically, the similarity between different modalities in each modality group can be achieved based on the global features of each modality. The global features of each modality can be obtained by weighted averaging of the local features of its respective modality. Therefore, the calculation of its weighted feature similarity can be defined as follows:

[0093]

[0094] In the formula, e t It is a local feature of the text within a batch, e av It is a local feature of the sound-visual combination within a batch, e x It refers to local visual features, local audio features, or combined audio-visual features within a batch. θ The weight is The fully connected layer.

[0095] Based on any of the above embodiments, in step 130, text reconstruction is performed on the text of the random mask based on other modal features in each modal group to obtain the reconstructed text of each group, including:

[0096] The text modal features and other modal features in each modal group are fused across modal features to obtain the fused features of each modal group;

[0097] Based on the fusion features of each modality group, a text masking modeling method is applied to reconstruct the text of each modality group using a random mask, thereby obtaining the reconstructed text of each group.

[0098] Specifically, in order to improve the understanding and generation capabilities of multimodal models, the text modal features and other modal features in each modal group can be fused across modal features to obtain the fused features of each modal group.

[0099] Preferably, cross-modal feature fusion can be achieved through a cross-modal decoder. The cross-modal decoder is used to perform multimodal fusion and text generation under multimodal conditions by utilizing an attention mechanism to integrate input visual features, audio features, and text features. It should be noted that the cross-modal decoder can be implemented using currently mature cross-modal decoder structures, and this embodiment of the invention does not impose specific limitations on this. For example, the cross-modal decoder can use a pre-trained BERT model, employing self-attention for feature transformation within each modality and cross-attention for cross-modal fusion and understanding between different modalities, and then outputting the target text sequence.

[0100] In the three modal groups of visual-text, audio-text, and audio-visual-text, the cross-attention layer of the cross-modal decoder is used to perform cross-modal fusion between different modalities in the same modal group, realize information query and interaction between multiple modalities, and obtain cross-modal fusion features.

[0101] Based on this, a text masking modeling method is applied to reconstruct the text for each modality group, resulting in the reconstructed text for each group. Specifically, for each modality group, the cross-attention layer of the cross-modal decoder is used to restore and reconstruct the masked target text, obtaining the text sequence prediction output under multimodal conditions, i.e., the reconstructed text.

[0102] The multimodal grouped text reconstruction task uses a causal text masking modeling method. It randomly replaces components of the input text sequence with [MASK] labels with a 60% probability, and inputs them along with other modal information into a cross-modal fusion decoder for masked text reconstruction.

[0103] It should be noted that the above mask probability can be any value greater than 0% and less than or equal to 100%. Selecting 60% is only one feasible solution, and the embodiments of the present invention do not specifically limit it.

[0104] The cross-modal decoder uses a causal attention masking strategy to prevent information leakage and ensure consistency with the autoregressive generation process. Specifically, the causal attention masking strategy completely masks the text sequence following the part to be generated, thus preventing the model from acquiring information about the ungenerated sequence.

[0105] Based on any of the above embodiments, step 130, which determines the text reconstruction loss based on each group of reconstructed text and each group of sample text, includes:

[0106] The first text reconstruction loss is determined based on the difference between the reconstructed text and the sample text, which are grouped according to text-visual modalities.

[0107] The second text reconstruction loss is determined based on the difference between the reconstructed text and the sample text, which are grouped by text-audio modalities.

[0108] The third text reconstruction loss is determined based on the difference between the reconstructed text and the sample text, which are grouped according to the text-audio-visual combined modality.

[0109] The text reconstruction loss is determined based on at least one of the first text reconstruction loss, the second text reconstruction loss, and the third text reconstruction loss.

[0110] Specifically, in the multimodal grouping text reconstruction task based on the modal grouping strategy, the objective function of text masking modeling is determined by the masked text sequence and the text sequence under multimodal conditions obtained by the cross-modal decoder.

[0111] For the predicted output of text sequences under multimodal conditions, i.e. reconstructing the text, a negative log-likelihood loss is calculated with the true label, so that the model learns the expected output of the true label, which is the sample text.

[0112] In one embodiment, the negative log-likelihood loss L MGC(T-X) It can be defined as:

[0113]

[0114] Where D is the training batch, T m It is the currently masked text portion, T <m It is all the text preceding the currently masked text portion, F x It is visual modal information, audio modal information, or audio-visual combined modal information.

[0115] Using negative log-likelihood loss can measure the difference between the model's predicted text output and the actual expected text output under multimodal conditions, thereby enabling the model to gradually learn to output the correct prediction results.

[0116] Understandably, the first text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-visual modality group; the second text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-audio modality group; and the third text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-audio-visual combined modality group.

[0117] Based on this, the text reconstruction loss can be determined based on at least one of the first text reconstruction loss, the second text reconstruction loss, and the third text reconstruction loss.

[0118] In one embodiment, Figure 3 This is a schematic diagram of the multimodal grouped text reconstruction task provided by the present invention, such as... Figure 3 As shown, for visual-text grouping, audio-text grouping, and audio-visual-text grouping, the cross-attention layer of the cross-modal decoder is used to restore and reconstruct the masked target text, obtaining the text sequence prediction output under multimodal conditions. The negative log-likelihood loss is calculated with the ground truth label, enabling the model to learn the expected output of the ground truth label. The text reconstruction loss of each modal group is summed to obtain the task loss of the multimodal grouping text reconstruction task.

[0119] Furthermore, the text reconstruction loss can be obtained by weighted averaging the first, second, and third text reconstruction losses. The text reconstruction loss can be defined as:

[0120]

[0121] In the formula, L MGC Let T be the text reconstruction loss, AV be the audio-visual combined loss, V be the visual loss, A be the audio loss, and L be the text reconstruction loss. MGC(T-V) L represents the loss for reconstructing the first text. MGC(T-A) L represents the loss for reconstructing the second text. MGC(T-AV) This represents the loss in the reconstruction of the third text.

[0122] The method provided in this invention determines the text reconstruction loss of a multimodal model by using the text reconstruction loss of each modality group, thereby enhancing the cross-modal understanding and generation capabilities of the multimodal model and further improving the downstream task performance and generalization performance of the multimodal model.

[0123] Based on any of the above embodiments, the method further includes the following steps before step 120:

[0124] Based on each single-modal encoder, feature extraction is performed on the multimodal information to obtain visual modal features, audio modal features, and text modal features;

[0125] Visual modal features and audio modal features are fused to obtain audio-visual combined modal features.

[0126] Specifically, in order to achieve the multimodal grouping alignment task and the multimodal grouping text reconstruction task, feature extraction can be performed on the multimodal information of the training samples beforehand to obtain visual modal features, audio modal features and text modal features.

[0127] Feature extraction can be achieved through individual unimodal encoders within a multimodal model. The unimodal encoders of a multimodal pre-trained model can include a visual encoder, an audio encoder, and a text encoder. The visual encoder encodes features from the input video and / or image data, outputting global and local visual features. The audio encoder encodes features from the input audio data, outputting global and local audio features. The text encoder encodes features from the input text data, outputting global and local text features. Here, the visual encoder, audio encoder, and text encoder are independent of each other and do not perform cross-modal data fusion.

[0128] It should be noted that the visual encoder, audio encoder, and text encoder can all be implemented using mature feature encoder structures for their respective modalities, and this embodiment of the invention does not impose any specific limitations on them.

[0129] Furthermore, the visual encoder can use the CLIP model's visual encoder to sparsely sample complete visual frames, input them into the patch embedding layer for image patch encoding to obtain patch features, and then input them into the temporal window attention module of the VideoSwin Transformer for inter-frame feature fusion to obtain global and local features of the visual data. The audio encoder can use the Audio Spectrogram Transformer pre-trained on AudioSet to sample audio waveform segments and then extract features from them to obtain global and local features of the audio data. The text encoder can use the BERT model to perform word segmentation, word embedding, and positional encoding on the original sentence, and then input it into the BERT Transformer encoder layer for text feature extraction to obtain global and local features of the text data.

[0130] After obtaining the visual modality features, audio modality features, and text modality features, the visual modality features and audio modality features can be further fused to obtain the audio-visual combined modality features. Specifically, the visual modality features and audio modality features can be concatenated in the sequence dimension to obtain the audio-visual combined modality features.

[0131] Based on any of the above embodiments, the multimodal model is used for at least one of text-visual retrieval, text-audio retrieval, text-audiovisual retrieval, visual text annotation, audio text annotation, audiovisual text annotation, visual question answering, and audiovisual question answering.

[0132] Specifically, the multimodal pre-trained model trained based on the above method can realize the understanding and generation of information across various modalities, and can be used for at least one of text-visual retrieval, text-audio retrieval, text-audiovisual retrieval, visual text annotation, audio text annotation, audiovisual text annotation, visual question answering, and audiovisual question answering.

[0133] For text-visual retrieval, text-audio retrieval, and text-audio-visual retrieval tasks, this embodiment of the invention uses a multimodal grouping alignment objective function as the training objective. Multimodal data is input into the single-modal encoder of the multimodal model to obtain the features of each single modality. Then, the similarity between modal features is obtained by calculating the weighted feature similarity. Finally, this similarity is sorted to obtain the final retrieval result.

[0134] For visual text annotation, audio text annotation, and audiovisual text annotation tasks, this embodiment of the invention uses the objective function of multimodal grouped text reconstruction as the training objective. The unimodal features obtained by the unimodal encoder of the multimodal model are input into the cross-modal decoder of the multimodal model, and text generation is performed in an autoregressive manner. Specifically, “[CLS][MASK]” is input into the multimodal model to obtain the first text prediction output [TK1]. This prediction output is concatenated to obtain “[CLS][TK1][MASK]”, and then input into the multimodal model to obtain the next text prediction output, and so on, until the model generates the text end symbol [SEP].

[0135] For visual question answering and audio-visual question answering tasks, this embodiment of the invention treats them as text generation tasks. Therefore, compared to using predefined high-frequency words as answer candidates, this embodiment of the invention can use the entire model vocabulary to predict question answers, resulting in better performance in answering open-ended questions. During training, similar to text reconstruction tasks, the objective function of a multimodal grouped text reconstruction task is used as the training objective.

[0136] Preferably, for visual question answering and audiovisual question answering tasks, the training objective is to simultaneously input the question sequence and answer sequence into the cross-modal decoder of the multimodal model for answer prediction learning. Unlike text reconstruction tasks, this training objective only masks the answer sequence using causal masking attention, while not masking the questions and using bidirectional self-attention.

[0137] Based on any of the above embodiments, in order to verify the effectiveness of the visual, text, and audio multimodal dataset construction and image-text-audio multimodal pre-training model methods provided by the present invention, the embodiments of the present invention were tested and compared on video-text, audio-text, and image-text datasets.

[0138] It should be noted that the embodiments of the present invention have constructed multiple model versions according to the method provided by the present invention, including... VALOR B VALOR L and VALOR L +DSL was used, and the manually labeled VALOR-1M and VALOR-32K datasets (collectively referred to as the VALOR datasets) were used for training and testing. All models used BERT-base as the text encoder / cross-modal decoder and AST as the audio encoder. and VALOR B VideoSwin-base is used as the video encoder, while VALOR... L and VALOR L +DSL uses CLIP-large as the video encoder. Furthermore, VALOR was obtained by pre-training using only WebVid-2.5M and CC3M. B VALOR L and VALOR L +DSL additionally incorporates the audio dataset and the VALOR dataset for pre-training. VALOR L +DSL refers to the use of bidirectional softmax post-processing during evaluation.

[0139] Specifically, in a series of video-text datasets, the embodiments of the present invention tested and compared existing methods with the methods proposed in the embodiments of the present invention on three tasks: video-text retrieval, video text annotation, and open video question answering.

[0140] For video-text retrieval tasks, the embodiments of this invention were tested on the VALOR, MSRVTT, DiDeMo, ActivityNet, LSMDC, and VATEX datasets, and the results are shown in Table 1. Compared with all other models in Group A, the VALOR constructed by the method provided in these embodiments of this invention... B The model shows significant improvements in metrics on the VALOR-32K, MSRVTT, DiDeMo, and LSMDC datasets. On the ActivityNet and VATEX datasets, VALOR... B Even with only 6.5M pre-trained data, it outperformed all other models in Group B, demonstrating the high quality and effectiveness of the VALOR dataset. Furthermore, compared to other models in Group B, the VALOR constructed using the method provided in this embodiment of the invention... LThe model achieves the best performance among all known models on the MSRVTT, DiDeMo, ActivityNet, LSMDC, and VATEX datasets, thanks to the application of trimodal data alignment (video, audio, and text) and the strong correlation and fine-grained alignment between modal data in the VALOR dataset.

[0141] Table 1. Performance Comparison of Different Models for Video-Text Retrieval Tasks

[0142]

[0143] For video text annotation tasks, this invention was tested on the VALOR, MSVD, MSRVTT, and VATEX datasets, and the results are shown in Table 2. For group A, the VALOR dataset constructed by the method provided in this invention... B The model achieved optimal performance on all four datasets tested. For group B, this embodiment of the invention primarily utilizes the VALOR constructed using the method provided in this embodiment. L A detailed comparison and analysis of the model and the GIT model are conducted. Specifically, as shown in Table 2, the VALOR constructed by the method provided in this embodiment of the invention... L The model significantly outperforms the GIT model and GIT in most metrics. L Model. Additionally, on the VATEX dataset, VALOR... L The model even outperformed the GIT2 model with only 11.6% of the parameters, 0.26% of the data, and lower image resolution. These results demonstrate that learning text generation under audio and video conditions is more efficient than increasing the number of model parameters and data.

[0144] Table 2 Comparison of performance of different models for video text annotation tasks

[0145]

[0146] For open video question answering tasks, the embodiments of this invention were tested on the MSRVTT-QA, MSVD-QA, FGIF-FrameQA, ActivityNet-QA, and MUSIC-AVQA datasets, and the results are shown in Table 3. As can be seen from Table 3, the model constructed by the method provided in the embodiments of this invention achieved optimal performance in each comparison. It is worth noting that in group B, FrozenBiLM uses VALOR... L Same video encoder and more than VALOR L A more powerful decoder, while Flamingo, CoCa, and GIT2 models use far more VALOR. L The number of parameters and the amount of training data. Under these conditions, VALOR LThe model still achieves the best performance metrics across all open video question answering tasks.

[0147] Table 3 Performance Comparison of Different Models for Open Video Question Answering Task

[0148]

[0149] Specifically, in a series of audio-text datasets, the embodiments of the present invention tested and compared existing methods with the methods proposed in the embodiments of the present invention on two tasks: audio-text retrieval and audio-text annotation.

[0150] For audio-text retrieval and audio-text annotation tasks, this invention was tested on the ClothoV1 and AudioCaps datasets, and the results are shown in Tables 4 and 5. On the ClothoV1 and AudioCaps datasets, the model constructed by the method provided in this invention performs best among all known methods. The Nagarani method is pre-trained on its proposed VideoCC3M trimodal dataset. In contrast, the VALOR dataset has more explicit and stronger audio-text data correlations, thus showing a significant improvement in performance on both audio-text datasets compared to the Nagarani method. Furthermore, unlike other methods, thanks to the modality grouping strategy, the method provided in this invention only requires multiple fine-tunings of the same pre-trained model to complete multiple downstream tasks, resulting in lower computational overhead compared to pre-training multiple models.

[0151] Table 4. Performance Comparison of Different Models in Audio-Text Retrieval Task

[0152]

[0153] Table 5. Performance Comparison of Different Models for Audio Text Annotation Task

[0154]

[0155] Specifically, in a series of image-text datasets, the embodiments of the present invention tested and compared existing methods with the methods proposed in the embodiments of the present invention on three tasks: image-text retrieval, image-text annotation, and image question answering.

[0156] For image-text retrieval, image-text annotation, and image question answering tasks, this embodiment of the invention was tested on the COCO-Retrieval dataset, and the results are shown in Table 6. As can be seen from Table 6, VALOR... L On the COCO retrieval task, it performs comparably to FILIP; on the COCO text annotation task, it surpasses GIT and is similar in performance to the GIT2 model; and on the VQAv2 task, it outperforms GIT, which is of similar size.L Model.

[0157] Table 6. Performance Comparison of Different Models in Image-Text Retrieval, Image Text Annotation, and Image Question Answering Tasks

[0158]

[0159] Based on any of the above embodiments Figure 4 This is a schematic diagram of the multimodal model provided by the present invention, such as... Figure 4 As shown, a method for pre-training a multimodal image-text-audio model is provided, including:

[0160] S1, Obtain multimodal information of training samples, including visual, audio and text information, with visual information including video and / or images.

[0161] S2, based on each single-modal encoder (visual encoder, text encoder and audio encoder), extract features from the multimodal information to obtain visual modal features, audio modal features and text modal features; fuse the visual modal features and audio modal features to obtain audio-visual combined modal features.

[0162] S3. Based on the semantic similarity between text modal features and other modal features in multimodal information, feature grouping and alignment of multimodal information are performed to obtain each modal group and determine the grouping alignment loss.

[0163] S4. Based on the cross-modal decoder, the text modal features and other modal features in each modal group are fused to obtain the fused features of each modal group. Based on the fused features of each modal group, the text masking modeling method is applied to reconstruct the text for each modal group to obtain the reconstructed text of each group. The text reconstruction loss is determined based on the reconstructed text of each group and the sample text of each group.

[0164] S5. Based on the group alignment loss and text reconstruction loss, the model parameters are iterated to obtain a multimodal model.

[0165] The following describes the image-text-audio multimodal pre-training model device provided by the present invention. The image-text-audio multimodal pre-training model device described below can be referred to in correspondence with the image-text-audio multimodal pre-training model method described above.

[0166] Figure 5 This is a schematic diagram of the structure of the image-text-audio multimodal pre-training model device provided by the present invention, as shown below. Figure 5 As shown, the image-text-audio multimodal pre-training model device includes an information acquisition unit 510, a grouping and alignment unit 520, a text reconstruction unit 530, and a parameter iteration unit 540, wherein:

[0167] The information acquisition unit 510 is used to acquire multimodal information of training samples, the multimodal information including visual, audio and text, the visual including video and / or images;

[0168] The grouping alignment unit 520 is used to perform feature grouping and alignment of the multimodal information based on the semantic similarity between text modal features and other modal features in the multimodal information, to obtain each modal group and determine the grouping alignment loss; the other modal features include one of visual modal features, audio modal features, and audio-visual combined modal features;

[0169] The text reconstruction unit 530 is used to reconstruct text from the text of the random mask based on other modal features in each modal group, to obtain each group of reconstructed text, and to determine the text reconstruction loss based on each group of reconstructed text and each group of sample text.

[0170] The parameter iteration unit 540 is used to perform parameter iteration on the model based on the group alignment loss and the text reconstruction loss to obtain a multimodal model.

[0171] The image-text-audio multimodal pre-training model device provided in this invention fully utilizes the strong correlation between visual, audio, and text multimodal information, designs a multimodal understanding and generation pre-training method, and iterates the model parameters based on group alignment loss and text reconstruction loss, so that the trained multimodal model can more accurately learn the alignment methods of visual, audio, and text data, enhance the cross-modal understanding and generation capabilities of the multimodal model, and improve the downstream task performance and generalization performance of the multimodal model.

[0172] Based on the above embodiments, the group alignment unit is specifically used for:

[0173] Based on the semantic similarity between the text modal features and the visual modal features, the first group alignment loss is determined;

[0174] Based on the semantic similarity between the text modal features and the audio modal features, the second group alignment loss is determined;

[0175] Based on the semantic similarity between the text modal features and the audiovisual combined modal features, the third group alignment loss is determined;

[0176] The group alignment loss is determined based on at least one of the first group alignment loss, the second group alignment loss, and the third group alignment loss.

[0177] Based on the above embodiments, a similarity determination unit is further included, used for:

[0178] Based on the weights of the local features of each modality in the multimodal information, a weighted average of the local features of each modality is performed to obtain the global features of each modality.

[0179] Based on the global features of each modality, the semantic similarity between text modal features and other modal features in the multimodal information is determined.

[0180] Based on the above embodiments, the text reconstruction unit is specifically used for:

[0181] The text modal features and other modal features in each modal group are fused across modal features to obtain the fused features of each modal group;

[0182] Based on the fusion features of each modality group, a text masking modeling method is applied to reconstruct the text of each modality group using a random mask, thereby obtaining the reconstructed text of each group.

[0183] Based on the above embodiments, the text reconstruction unit is further specifically used for:

[0184] The first text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-visual modality grouping.

[0185] The second text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-audio modality grouping.

[0186] The third text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-audio-visual combined modality grouping.

[0187] The text reconstruction loss is determined based on at least one of the first text reconstruction loss, the second text reconstruction loss, and the third text reconstruction loss.

[0188] Based on the above embodiments, a feature extraction unit is also included, for:

[0189] Based on each single-modal encoder, feature extraction is performed on the multimodal information to obtain visual modal features, audio modal features, and text modal features;

[0190] The visual modal features and the audio modal features are fused to obtain the audio-visual combined modal features.

[0191] Based on the above embodiments, the multimodal model is used for at least one of text-visual retrieval, text-audio retrieval, text-audiovisual retrieval, visual text annotation, audio text annotation, audiovisual text annotation, visual question answering, and audiovisual question answering.

[0192] Figure 6An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a graph-text-audio multimodal pre-training model method, which includes:

[0193] Obtain multimodal information of training samples, wherein the multimodal information includes visual, audio and text, and the visual information includes video and / or images;

[0194] Based on the semantic similarity between text modal features and other modal features in the multimodal information, the multimodal information is grouped and aligned to obtain each modal group and determine the group alignment loss; the other modal features include one of visual modal features, audio modal features, and audio-visual combined modal features;

[0195] Based on other modal features in each modality group, text reconstruction is performed on the text with a random mask to obtain the reconstructed text of each group, and the text reconstruction loss is determined based on the reconstructed text of each group and the sample text of each group.

[0196] Based on the group alignment loss and the text reconstruction loss, the model is iterated to obtain a multimodal model.

[0197] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the graph-text-audio multimodal pre-training model method provided by the above methods, the method comprising:

[0199] Obtain multimodal information of training samples, wherein the multimodal information includes visual, audio and text, and the visual information includes video and / or images;

[0200] Based on the semantic similarity between text modal features and other modal features in the multimodal information, the multimodal information is grouped and aligned to obtain each modal group and determine the group alignment loss; the other modal features include one of visual modal features, audio modal features, and audio-visual combined modal features;

[0201] Based on other modal features in each modality group, text reconstruction is performed on the text with a random mask to obtain the reconstructed text of each group, and the text reconstruction loss is determined based on the reconstructed text of each group and the sample text of each group.

[0202] Based on the group alignment loss and the text reconstruction loss, the model is iterated to obtain a multimodal model.

[0203] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the graph-text-audio multimodal pre-training model method provided by the methods described above, the method comprising:

[0204] Obtain multimodal information of training samples, wherein the multimodal information includes visual, audio and text, and the visual information includes video and / or images;

[0205] Based on the semantic similarity between text modal features and other modal features in the multimodal information, the multimodal information is grouped and aligned to obtain each modal group and determine the group alignment loss; the other modal features include one of visual modal features, audio modal features, and audio-visual combined modal features;

[0206] Based on other modal features in each modality group, text reconstruction is performed on the text with a random mask to obtain the reconstructed text of each group, and the text reconstruction loss is determined based on the reconstructed text of each group and the sample text of each group.

[0207] Based on the group alignment loss and the text reconstruction loss, the model is iterated to obtain a multimodal model.

[0208] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0209] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for pre-training a multimodal image-text-audio model, characterized in that, include: Obtain multimodal information of training samples, wherein the multimodal information includes visual, audio and text, and the visual information includes video and / or images; Based on the semantic similarity between text modal features and other modal features in the multimodal information, the multimodal information is grouped and aligned to obtain each modal group and determine the group alignment loss. The other modal features include one of visual modal features, audio modal features, and audio-visual combined modal features. Based on other modal features in each modality group, text reconstruction is performed on the text with a random mask to obtain the reconstructed text of each group, and the text reconstruction loss is determined based on the reconstructed text of each group and the sample text of each group. Based on the group alignment loss and the text reconstruction loss, the model is iterated to obtain a multimodal model; The modal groups include at least one of text-visual modal groups, text-audio modal groups, and text-audio-visual combined modal groups. Determining the group alignment loss includes: Based on the semantic similarity between the text modal features and the visual modal features, the first group alignment loss is determined; Based on the semantic similarity between the text modal features and the audio modal features, the second group alignment loss is determined; Based on the semantic similarity between the text modal features and the audiovisual combined modal features, the third group alignment loss is determined; The group alignment loss is determined based on at least one of the first group alignment loss, the second group alignment loss, and the third group alignment loss; The text reconstruction is performed on the randomly masked text based on other modal features in each modal group, resulting in reconstructed text for each group, including: The text modal features and other modal features in each modal group are fused across modal features to obtain the fused features of each modal group; Based on the fusion features of each modality group, a text masking modeling method is applied to reconstruct the text of each modality group using a random mask, thereby obtaining the reconstructed text of each group.

2. The method for pre-training a multimodal image-text-audio pre-training model according to claim 1, characterized in that, The steps for determining the semantic similarity between text modal features and other modal features in the multimodal information include: Based on the weights of the local features of each modality in the multimodal information, a weighted average of the local features of each modality is performed to obtain the global features of each modality. Based on the global features of each modality, the semantic similarity between text modal features and other modal features in the multimodal information is determined.

3. The method for pre-training a multimodal image-text-audio pre-training model according to claim 1, characterized in that, The determination of text reconstruction loss based on each group of reconstructed text and each group of sample text includes: The first text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-visual modality grouping. The second text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-audio modality grouping. The third text reconstruction loss is determined based on the difference between the reconstructed text and the sample text in the text-audio-visual combined modality grouping. The text reconstruction loss is determined based on at least one of the first text reconstruction loss, the second text reconstruction loss, and the third text reconstruction loss.

4. The method for pre-training a multimodal image-text-audio pre-training model according to any one of claims 1-3, characterized in that, The step of grouping and aligning the multimodal information based on the semantic similarity between text modal features and other modal features in the multimodal information, before which the following steps are also included: Based on each single-modal encoder, feature extraction is performed on the multimodal information to obtain visual modal features, audio modal features, and text modal features; The visual modal features and the audio modal features are fused to obtain the audio-visual combined modal features.

5. The method for pre-training a multimodal image-text-audio pre-training model according to any one of claims 1-3, characterized in that, The multimodal model is used for at least one of text-visual retrieval, text-audio retrieval, text-audiovisual retrieval, visual text annotation, audio text annotation, audiovisual text annotation, visual question answering, and audiovisual question answering.

6. A pre-training device for a multimodal image-text-audio model, characterized in that, include: An information acquisition unit is used to acquire multimodal information of training samples, wherein the multimodal information includes visual, audio and text, and the visual information includes video and / or images; The group alignment unit is used to perform feature group alignment on the multimodal information based on the semantic similarity between text modal features and other modal features in the multimodal information, to obtain each modal group and determine the group alignment loss. The other modal features include one of visual modal features, audio modal features and audio-visual combined modal features. The text reconstruction unit is used to reconstruct text from the randomly masked text based on other modal features in each modality group, to obtain reconstructed text in each group, and to determine the text reconstruction loss based on the reconstructed text in each group and the sample text in each group. The parameter iteration unit is used to perform parameter iteration on the model based on the group alignment loss and the text reconstruction loss to obtain a multimodal model; The group alignment unit is specifically used for: Based on the semantic similarity between the text modal features and the visual modal features, the first group alignment loss is determined; Based on the semantic similarity between the text modal features and the audio modal features, the second group alignment loss is determined; Based on the semantic similarity between the text modal features and the audiovisual combined modal features, the third group alignment loss is determined; The group alignment loss is determined based on at least one of the first group alignment loss, the second group alignment loss, and the third group alignment loss; The text reconstruction unit is specifically used for: The text modal features and other modal features in each modal group are fused across modal features to obtain the fused features of each modal group; Based on the fusion features of each modality group, a text masking modeling method is applied to reconstruct the text of each modality group using a random mask, thereby obtaining the reconstructed text of each group.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the text-to-speech multimodal pre-training model method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the text-to-speech multimodal pre-training model method as described in any one of claims 1 to 5.

Images