Processing method and device of multi-mode encoder, computer equipment and storage medium

Abstract

The invention relates to the technical field of computers, and provides a processing method and device of a multi-mode encoder, computer equipment and a storage medium. The method comprises the steps that a mask sample pair to be processed is acquired, the mask sample pair comprises a text sample and an audio sample which are associated with each other, and at least one of the text sample and the audio sample is subjected to mask processing; based on a multi-modal encoder, generating text coding features of the text samples and audio coding features of the audio samples, the text coding features fusing spectrum linear features of the audio samples, and the audio coding features fusing word linear features of the text samples; and predicting mask information subjected to mask processing according to the text coding features and the audio coding features, and correcting the multi-mode encoder based on the accuracy of the mask information.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a processing method, device, computer equipment and storage medium of a multimodal encoder. Background Art

[0002] Multimodal text, music learning and pre-training have strong application demands in search and recommendation interaction scenarios. However, the current mainstream multimodal fusion learning methods are often completed through the dual-tower model in the form of contrastive learning. Taking the transformer model (Transformers) based on the self-attention mechanism as an example, its unified network architecture can seamlessly handle multiple modalities, but the end task format needs to be manually converted according to a specific architecture, and the parameters are usually not effectively shared between modalities, making it difficult to achieve effective fusion of multimodal features. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide a processing method, apparatus, computer device and storage medium for a multimodal encoder to solve the problem that multimodal features are difficult to effectively fuse.

[0004] In a first aspect, an embodiment of the present disclosure provides a processing method for a multimodal encoder, comprising: obtaining a mask sample pair to be processed, the mask sample pair comprising an associated text sample and an audio sample, at least one of the text sample and the audio sample being subjected to mask processing; generating text coding features of the text sample based on the multimodal encoder, and generating audio coding features of the audio sample, wherein the text coding features incorporate the spectral linear features of the audio sample, and the audio coding features incorporate the word linear features of the text sample; predicting mask information that has been masked based on the text coding features and the audio coding features, and correcting the multimodal encoder based on the accuracy of the mask information.

[0005] The processing method of the multimodal encoder provided in the embodiment of the present disclosure takes audio and text as fused modalities, and performs masking on the audio-text sample pairs. During the encoding process of the multimodal encoder, the spectral linear features of the audio samples are fused into the text encoding features, and the word linear features of the text samples are fused into the audio encoding features. In this way, the sharing of audio features and text features can be achieved through one training task, and specific encoding and cross-modal fusion for each modality can be effectively realized.

[0006] In a second aspect, an embodiment of the present disclosure provides a processing device for a multimodal encoder, comprising: a sample acquisition module, used to acquire mask sample pairs to be processed, the mask sample pairs comprising associated text samples and audio samples, at least one of the text samples and the audio samples being subjected to mask processing; a coding feature generation module, used to generate text coding features of text samples based on the multimodal encoder, and to generate audio coding features of audio samples, wherein the text coding features incorporate the spectral linear features of the audio samples, and the audio coding features incorporate the word linear features of the text samples; a correction module, used to predict mask information that has been masked based on the text coding features and / or the audio coding features, and to correct the multimodal encoder based on the accuracy of the mask information.

[0007] In a third aspect, an embodiment of the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the processing method of the multimodal encoder of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the processing method of a multimodal encoder according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 is a flowchart of a processing method of a multimodal encoder according to an embodiment of the present disclosure;

[0011] Figure 2 is another flowchart of a processing method of a multimodal encoder according to an embodiment of the present disclosure;

[0012] Figure 3 is another flowchart of a processing method of a multimodal encoder according to an embodiment of the present disclosure;

[0013] Figure 4 is another flowchart of a processing method of a multimodal encoder according to an embodiment of the present disclosure;

[0014] Figure 5is another flowchart of a processing method of a multimodal encoder according to an embodiment of the present disclosure;

[0015] Figure 6 is another flowchart of a processing method of a multimodal encoder according to an embodiment of the present disclosure;

[0016] Figure 7 is a structural block diagram of a processing device of a multimodal encoder according to an embodiment of the present disclosure;

[0017] Figure 8 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0019] Multimodal text-based music learning and pre-training have strong application demands in search and recommendation interaction scenarios. However, the current mainstream multimodal fusion learning methods are often completed through the dual-tower model and in the form of contrastive learning.

[0020] The success of Transformers has been transferred from language to music and multimodal problems. The unification of network architectures allows us to handle multiple modalities seamlessly. For music-language modeling, there are many ways to apply Transformer models due to the different nature of downstream tasks. For example, dual encoder architectures are used for efficient retrieval, encoder-decoder networks are used for generation tasks, and fused encoder architectures are used for music-text encoding. However, most base models must be manually converted to the end task format according to a specific architecture. In addition, parameters are usually not effectively shared between modalities.

[0021] Pre-training tasks based on masked data modeling have been successfully applied to various modalities such as text, music, and music-text pairs. Current music-language based models often multi-task other pre-training objectives (such as music-text matching), making them unfriendly to extension and inefficient.

[0022] Based on this, this technical solution realizes the sharing of audio features and text features through a training task, and can effectively fuse audio and text features.

[0023] Specifically, a multi-way transformer model is used here for general modeling, that is, a unified architecture is used for various downstream tasks, and the modular network also comprehensively considers modality-specific encoding and cross-modal fusion.

[0024] Only one pre-training task, masking followed by prediction, is used to train a common multimodal base model. Text and music are processed in the same way, with no fundamental modeling differences. Therefore, audio-text pairs are used as "parallel sentences" to learn alignment between modalities, demonstrating the superiority of generative pre-training with a unified framework.

[0025] The model size is expanded to billions of parameters, and the scale of pre-training data is expanded. Specifically, the model consists of a 40-layer Multiway Transformer with a hidden size of 1408, an intermediate size of 6144, and 16 attention heads. All layers contain music experts and language experts. Music-language experts are also used in the first three Mutliway Transformer layers. The self-attention module is shared between different modalities and pre-trained on unimodal and multimodal data respectively. The training process contains audio samples, text samples, and audio-text sample pairs. As a result, the generalization quality of the base model is improved and it can be transferred to various downstream tasks.

[0026] According to an embodiment of the present disclosure, an embodiment of a processing method for a multimodal encoder is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] In this embodiment, a processing method of a multimodal encoder is provided, which can be used in computer devices, such as notebook computers, computers, etc. Figure 1 is a flowchart of a processing method of a multimodal encoder according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:

[0028] Step S101, obtaining a mask sample pair to be processed.

[0029] The masked sample pair includes an associated text sample and an audio sample, and at least one of the text sample and the audio sample is subjected to masking processing.

[0030] The text sample is a pre-set description of the audio, for example, "a beautiful piano accompaniment". The audio sample is a variety of audio clips selected from the authorized audio library. By associating the text sample with the audio sample, the corresponding text-audio sample pair can be obtained.

[0031] The masked sample pair to be processed is a sample pair consisting of a text sample and an audio sample that have been subjected to masking processing. Specifically, the masked sample pair to be processed can be obtained by performing partial masking processing on the text sample and / or the audio sample in the associated text-audio sample pair.

[0032] Step S102: Generate text encoding features of the text sample based on the multimodal encoder, and generate audio encoding features of the audio sample.

[0033] Among them, the text encoding features are integrated with the spectral linear features of the audio samples, and the audio encoding features are integrated with the word linear features of the text samples.

[0034] The multimodal encoder is a multi-layer Transformer model, each layer of which includes a self-attention layer and a multi-layer perceptron (MLP), which can encode the input text samples and audio samples and output the corresponding encoding features.

[0035] Specifically, the text sample is input into the multimodal encoder for encoding, and the spectral linear features of the audio are fused in the text encoding process through the self-attention layer and the MLP layer to generate the corresponding generated text encoding features. At the same time, the audio sample is input into the multimodal encoder for encoding, and the word linear features of the text are fused in the audio encoding process through the self-attention layer and the MLP layer to generate the corresponding audio encoding features.

[0036] Step S103: predicting mask information to be masked according to the text encoding features and the audio encoding features, and calibrating the multimodal encoder based on the accuracy of the mask information.

[0037] As mentioned above, the spectral linear features of the audio samples are integrated into the text encoding features. The content of the audio samples can be better understood through text encoding, so that the masked audio samples can be predicted to determine the masked audio mask information in the audio samples.

[0038] Similarly, since the audio coding features incorporate the word linear features of the text samples, the masked text samples can be predicted through the audio coding features to determine the masked text information in the text samples.

[0039] After predicting the masked information, the predicted mask information is compared with the real mask information to determine the prediction accuracy of the mask information. Then, the multimodal encoder can be corrected by the prediction accuracy so that the multimodal encoder can output more accurate text encoding features and audio encoding features.

[0040] The processing method of the multimodal encoder provided in this embodiment takes audio and text as fused modalities, and performs masking on the audio-text sample pairs. During the encoding process of the multimodal encoder, the spectral linear features of the audio samples are fused into the text encoding features, and the word linear features of the text samples are fused into the audio encoding features. In this way, the sharing of audio features and text features can be achieved through one training task, and specific encoding and cross-modal fusion for each modality can be effectively achieved.

[0041] In this embodiment, a processing method of a multimodal encoder is provided, which can be used in computer devices, such as notebook computers, computers, etc. Figure 2 is a flowchart of a processing method of a multimodal encoder according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:

[0042] Step S201, obtaining a mask sample pair to be processed. The mask sample pair includes an associated text sample and an audio sample, and at least one of the text sample and the audio sample is subjected to mask processing. For detailed description, please refer to the relevant description corresponding to the above embodiment, which will not be repeated here.

[0043] Step S202: Generate text encoding features of the text sample based on the multimodal encoder, and generate audio encoding features of the audio sample.

[0044] The text encoding feature incorporates the spectral linear features of the audio sample, and the audio encoding feature incorporates the word linear features of the text sample. Specifically, the spectral linear features and word linear features are generated in the following way:

[0045] Step a1, generating a word vector sequence corresponding to the text sample, and generating a spectrum slice sequence corresponding to the audio sample.

[0046] The word vector sequence includes multiple word vectors, and the spectrum slice sequence includes multiple spectrum slices.

[0047] The multimodal encoder performs contextual content analysis on the text sample input thereto to implement word segmentation processing for the text sample, and converts the text sample into a word vector sequence. Specifically, the step of generating a word vector sequence corresponding to the text sample may include:

[0048] Step a11, perform word segmentation on the text sample to obtain a phrase sequence. Here, a word segmenter can be used to perform word segmentation on the input text sample to obtain multiple phrases corresponding to the text sample. Then, the multiple phrases are converted into a phrase sequence consisting of an ID.

[0049] Step a12, generating word vectors for each word group in the word group sequence to obtain a word vector sequence containing each word vector. Here, the word vector corresponding to each word group is determined by word embedding lookup, thereby obtaining a word vector sequence corresponding to the word group sequence.

[0050] The multimodal encoder performs spectrum analysis on the audio samples input thereto, and converts the audio samples into a spectrum slice sequence in combination with the spectrum information. Specifically, the step of generating a spectrum slice sequence corresponding to the audio samples may include:

[0051] Step a13, generate spectrum information of the audio sample, and slice the spectrum information to generate a spectrum slice sequence containing each spectrum slice. Different audio samples have different audio parameters, and their corresponding spectra are also different. Here, the audio parameters of the audio sample can be analyzed to convert the audio sample into a music spectrum, that is, the spectrum information of the audio sample. Then, the spectrum information is sliced ​​to obtain a spectrum slice sequence consisting of multiple spectrum slices.

[0052] In the above implementation, by performing word segmentation processing on the text sample to construct a word vector sequence, it is convenient to extract the linear features of the text sample through the word vector sequence. At the same time, by performing slicing processing on the spectrum information of the audio sample to construct a spectrum slice sequence, it is convenient to extract the linear features of the audio sample through the spectrum slice sequence.

[0053] Step a2: for any word vector in the word vector sequence, perform a linear transformation on the word vector to obtain the word linear features of the word vector, wherein the word linear features include word query features, word index features and word content features.

[0054] Through the attention mechanism of the multimodal encoder, any word vector in the word vector sequence is linearly transformed by combining the word query feature (Q), the word index feature (K) and the word content feature (V) to obtain the word linear feature corresponding to the word vector.

[0055] Step a3: for any spectrum slice in the spectrum slice sequence, perform a linear transformation on the spectrum slice to obtain the spectrum linear feature of the spectrum slice.

[0056] The spectrum linear features include spectrum query features, spectrum index features and spectrum content features.

[0057] Similarly, through the attention mechanism of the multimodal encoder, any spectrum slice in the spectrum slice sequence is linearly transformed in combination with the spectrum query feature (Q), the spectrum index feature (K) and the spectrum content feature (V) to obtain the spectrum linear feature corresponding to the spectrum slice.

[0058] In the above implementation, the word linear features and the spectrum linear features are determined by a unified linear transformation method, which facilitates the subsequent fusion of the word linear features and the spectrum linear features.

[0059] Specifically, the above step S202 may include:

[0060] Step S2021, for any word vector in the word vector sequence represented by the text sample, obtain the word query feature in the word linear feature of the word vector.

[0061] As mentioned above, word linear features include word query features, word index features, and word content features. Here, the self-attention layer of the multimodal encoder can extract the word query features corresponding to the word vector by parsing the word linear features.

[0062] Step S2022, calculating the similarity between the word query feature and the word index feature in each word linear feature and the spectrum index feature in each spectrum linear feature.

[0063] The dimensions of the word query feature, word index feature and spectrum index feature are consistent and are all represented in vector form. Specifically, the word query feature is dot-multiplied with each word index feature and each spectrum index feature, that is, each word index feature corresponding to the text sample and each spectrum index feature corresponding to the audio sample are taken as K, and the word query feature Q is dot-multiplied with each K to obtain the dot product result, which is used to measure the similarity between the word query feature and the word index feature in each word linear feature and the spectrum index feature in each spectrum linear feature.

[0064] Step S2023, generating word attention features of the word vector according to the similarity, the word content features in each word linear feature, and the spectrum content features in each spectrum linear feature.

[0065] In order to ensure the stability of the linear transformation, the similarity obtained above is scaled and normalized to obtain the processed similarity. Then, the processed similarity is multiplied with each word content feature and each spectrum content feature, that is, each word content feature corresponding to the text sample and each spectrum content feature corresponding to the audio sample are used as V, and the processed similarity is multiplied with each K to obtain the product result, which is determined as the word attention feature of the word vector.

[0066] Step S2024, generating text encoding features of the text sample based on the word attention features of each word vector in the word vector sequence.

[0067] The word attention features of each word vector obtained are input into the multi-layer perceptron of the multimodal encoder for text encoding processing to generate corresponding text encoding features. Since the word attention features incorporate the spectral content features and the spectral index features, the text encoding features output by the multimodal encoder incorporate the spectral content features and the spectral index features in the spectral linear features of the audio samples.

[0068] Step S2025: for any spectrum slice in the spectrum slice sequence represented by the audio sample, obtain a spectrum query feature in the spectrum linear feature of the spectrum slice.

[0069] As mentioned above, the spectrum linear features include spectrum query features, spectrum index features, and spectrum content features. Here, the self-attention layer of the multimodal encoder can extract the spectrum query features corresponding to the spectrum slice by parsing the spectrum linear features.

[0070] Step S2026, calculating the similarity between the spectrum query feature and the spectrum index features in each spectrum linear feature and the word index features in each word linear feature.

[0071] The calculation method here is similar to the similarity between the word query feature and the word index feature in each word linear feature and the spectrum index feature in each spectrum linear feature.

[0072] That is, take each spectral index feature and each word index feature as K, take the spectral query feature as Q, perform dot product of the spectral query feature Q with each K respectively, and obtain the dot product result, which is used as the similarity between the spectral query feature and the spectral index features in each spectral linear feature and the word index features in each word linear feature.

[0073] Step S2027, generating a spectrum attention feature of the spectrum slice according to the similarity, the spectrum content feature in each spectrum linear feature, and the word content feature in each word linear feature.

[0074] This is similar to the way word attention features are generated.

[0075] That is, each spectrum content feature and each word content feature is used as V, and the similarity obtained in the above step S2026 is scaled and normalized to obtain a processed similarity. The processed similarity is multiplied with each K to obtain a product result, which is determined as the spectrum attention feature.

[0076] Step S2028: Generate audio coding features of the audio sample according to the spectrum attention features of each spectrum slice in the spectrum slice sequence.

[0077] The obtained spectral attention features of each spectrum slice are input into the multi-layer perceptron of the multimodal encoder for spectral encoding processing to generate corresponding audio coding features. Since the spectral attention features incorporate word content features and word index features, the spectral coding features output by the multimodal encoder incorporate word content features and word index features in the word linear features of the text sample.

[0078] Step S203, predicting mask information to be masked according to the text encoding features and the audio encoding features, and calibrating the multimodal encoder based on the accuracy of the mask information. For detailed description, please refer to the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0079] The processing method of the multimodal encoder provided in this embodiment encodes the audio sample and the text sample as a whole through a multimodal encoder with a unified architecture. When the audio sample and the text sample are simultaneously input into the multimodal encoder, the multimodal encoder can fuse the linear features of the audio modality into the text encoding features to generate text encoding features, and can fuse the linear features of the text modality into the audio encoding to generate audio encoding features. Thus, the feature fusion capability of the multimodal encoder is realized, and the multimodal encoder's ability to understand its input content is improved.

[0080] In this embodiment, a processing method of a multimodal encoder is provided, which can be used in computer devices, such as notebook computers, computers, etc. Figure 3 is a flowchart of a processing method of a multimodal encoder according to an embodiment of the present disclosure, such as Figure 3 As shown, the process includes the following steps:

[0081] Step S301, obtaining a mask sample pair to be processed. The mask sample pair includes an associated text sample and an audio sample, and at least one of the text sample and the audio sample is masked. For a detailed description, please refer to the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0082] Step S302, based on the multimodal encoder, generates text coding features of the text sample and generates audio coding features of the audio sample. The text coding features incorporate the spectral linear features of the audio sample, and the audio coding features incorporate the word linear features of the text sample. For detailed description, please refer to the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0083] Step S303: predict the mask information to be masked according to the text encoding features and the audio encoding features, and calibrate the multimodal encoder based on the accuracy of the mask information. For detailed description, please refer to the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0084] Step S304, obtaining text description information, and generating encoding features of the text description information through the multimodal encoder that has completed the correction.

[0085] The text description information is the text input to the multimodal encoder, and the audio information to be matched is represented by the text description information. Specifically, the text description information can be input by the user or selected from the authorized text library. The method for obtaining the text description information is not specifically limited here.

[0086] The text description information is used as the input of the multimodal encoder, which can extract and encode the text expression vector of the text description information and output the encoding features for the text description information.

[0087] Step S305: input the encoding features of the text description information into an audio decoder, so as to generate audio information matching the text description information through the audio decoder.

[0088] The audio decoder is a pre-trained model for outputting matching audio according to the text description. The multimodal encoder uses the encoding features of the text description information it outputs as the input of the audio decoder, and the audio decoder predicts the music code matching the text description information to obtain the audio information.

[0089] In some optional embodiments, the audio decoder may be trained in the following manner:

[0090] Step b1, obtaining a training sample pair, wherein the training sample pair includes a text description sample and an audio mask sample matching the text description sample, and the audio mask sample includes at least part of known audio information.

[0091] Step b2: Processing the known audio information in the text description sample and the audio mask sample by an audio decoder to predict the masked audio information in the audio mask sample.

[0092] Step b3: calibrating the audio decoder based on the predicted accuracy of the masked audio information.

[0093] The audio information is represented by the music vocoder. When training the audio decoder, the audio decoder predicts the masked music vocoder through the text features corresponding to the text description samples and the known music in the audio mask samples.

[0094] Then, the predicted music code is compared with the real music code to determine the prediction accuracy of the music decoder, and the parameters of the audio decoder are reversely adjusted according to the prediction accuracy to correct the audio decoder so that the audio decoder can accurately infer the audio information corresponding to the text description information.

[0095] The processing method of the multimodal encoder provided in this embodiment determines the matching audio information by inputting text description information, thereby realizing inference of the correct audio information based on the text description information.

[0096] In some optional embodiments, after obtaining the multi-modal encoder that has completed the correction, as shown in FIG. Figure 4 As shown, the above method may also include:

[0097] Step S306, obtaining text search information and an audio file set.

[0098] The text retrieval information representation is a retrieval description for audio information. Specifically, a multimodal encoder is deployed in the computer device, which can respond to the text input operation triggered by the user and obtain the file retrieval information corresponding to the text input operation. The audio file collection is a collection of various audio files, which can be obtained from an authorized audio library.

[0099] Step S307: Generate retrieval coding features of text retrieval information and generate audio coding features of each audio file in the audio file set through the corrected multimodal encoder.

[0100] The text retrieval information and the audio file set are input into the multimodal encoder, and the multimodal encoder can extract the text expression vector of the text retrieval information to generate the retrieval coding feature of the text retrieval information. At the same time, the audio expression vector of each audio file in the audio file set is extracted to generate the audio coding feature of each audio file.

[0101] Step S308, searching the audio coding features of each audio file for a target audio coding feature that matches the search coding feature, and taking the audio file represented by the target audio coding feature as the audio file retrieved by the text search information.

[0102] The similarity between the audio coding feature of each audio file and the retrieval coding feature is calculated respectively, and the audio coding feature with the highest similarity is determined. The audio coding feature with the highest similarity is determined as the target audio coding feature that matches the retrieval coding feature. The audio file represented by the target audio coding feature is the audio file to be retrieved by the text retrieval information.

[0103] The processing method of the multimodal encoder provided in this embodiment generates encoding features of text and audio respectively through the multimodal encoder, and then calculates the similarity between the encoding features to match the audio file with the text retrieval information, thereby realizing audio retrieval based on language understanding.

[0104] In some optional embodiments, after obtaining the multi-modal encoder that has completed the correction, as shown in FIG. Figure 5 As shown, the above method may also include:

[0105] Step S309: Obtain text description information, and a first candidate audio file and a second candidate audio file corresponding to the text description information.

[0106] The text description information is the text input to the multimodal encoder, and the audio information to be matched is represented by the text description information. The first candidate audio file and the second candidate audio file are both audio files corresponding to the text description information. The text description information and the first candidate audio file are constructed as a sample pair, and the text description information and the second candidate audio file are constructed as another sample pair.

[0107] Step S310: Generate a first fusion coding feature of the text description information and the first candidate audio file, and generate a second fusion coding feature of the text description information and the second candidate audio file by completing the corrected multimodal encoder.

[0108] The sample pair constructed by the text description information and the first candidate audio file is input into the multimodal encoder, and the text description information and the first candidate audio file are taken as a whole, thereby obtaining the first fusion coding feature of the sample pair. Similarly, the sample pair constructed by the text description information and the first candidate audio file is input into the multimodal encoder, and the corresponding second fusion coding feature can be obtained.

[0109] Step S311, classify the first fused coding feature and the second fused coding feature, and output the classification result, wherein the classification result represents the audio file that better matches the text description information among the first candidate audio file and the second candidate audio file.

[0110] The first fused coding feature and the second fused coding feature are classified by a classifier to determine an audio file that has a higher matching degree with the text description information.

[0111] The processing method of the multimodal encoder provided in this embodiment treats the text and audio files as a whole, and the encoding features output by the multimodal encoder integrate the features of the text and audio, rather than a single text feature or audio feature. Therefore, classification prediction is performed in combination with the integrated features, realizing the compatibility detection between music and text.

[0112] In some optional embodiments, after obtaining the multi-modal encoder that has completed the correction, as shown in FIG. Figure 6 As shown, the above method may also include:

[0113] Step S312: Acquire an audio file, and generate audio coding features of the audio file by using the multimodal encoder that completes the correction.

[0114] The audio file may be input by the user, randomly selected from a local authorized audio library, or downloaded from the cloud. The method for obtaining the audio file is not specifically limited here.

[0115] The acquired audio file is used as the input of the multimodal encoder, and the multimodal encoder can extract and encode the audio expression vector of the audio file to output audio coding features for the audio file.

[0116] Step S313: input the audio coding features of the audio file into the subtitle generation model, so that the subtitle generation model outputs subtitle information that matches the audio coding features of the audio file.

[0117] The subtitle generation model is a pre-trained model used to output matching subtitles based on audio files. The multimodal encoder uses the audio encoding features it outputs as input to the subtitle generation model, and the subtitle generation model predicts the text that matches the audio file to obtain the corresponding subtitle information.

[0118] In some optional embodiments, the subtitle generation model can be trained in the following manner:

[0119] Step c1, obtaining a subtitle training sample pair, wherein the subtitle training sample pair includes an audio file sample and a subtitle mask sample, and part of the subtitle content in the subtitle mask sample is masked;

[0120] Step c2, determining the known subtitle content before the masked subtitle content in the subtitle mask sample, and processing the audio file sample and the known subtitle content through the subtitle generation model to restore the masked subtitle content in the subtitle mask sample;

[0121] Step c3: calibrating the subtitle generation model based on the accuracy of the restored subtitle content.

[0122] When training the subtitle generation model, the masked subtitle content is predicted through the audio features corresponding to the audio file samples and the known subtitle content in the subtitle mask samples (for example, for left-hand subtitles, the known subtitle content is the content before the mask). Then, the predicted subtitle content is compared with the real subtitle content in the masked subtitle content to determine the prediction accuracy of the subtitle content. The parameters of the subtitle generation model are reversely adjusted based on the prediction accuracy to correct the subtitle generation model so that the subtitle generation model can accurately infer the subtitle content corresponding to the audio file.

[0123] In some optional embodiments, the subtitle mask sample also includes a subtitle end mark after mask processing, and the above method may further include:

[0124] Step d1, processing the known subtitle contents in the audio file sample and the subtitle mask sample through the subtitle generation model to predict the position of the subtitle end mark in the subtitle mask sample.

[0125] Step d2, based on the accuracy of the predicted position of the subtitle end mark, calibrate the subtitle generation model.

[0126] Subtitle content usually has a special end marker, which can be masked during the training of the subtitle generation model so that the subtitle generation model can predict the end position of the subtitle content through the known subtitle content in the audio file sample and the subtitle mask sample.

[0127] By comparing the predicted subtitle ending position with the actual ending position, the prediction accuracy of the subtitle ending position is determined, and the parameters of the subtitle generation model are reversely adjusted according to the prediction accuracy of the subtitle ending position to correct the subtitle generation model so that the subtitle generation model can infer the correct ending position of the subtitle content.

[0128] The processing method of the multimodal encoder provided in this embodiment outputs the audio coding features corresponding to the audio file through the multimodal encoder, and then determines the corresponding subtitle information in combination with the trained subtitle generation model, thereby realizing the generation of text information based on the audio file.

[0129] In the present embodiment, a processing device of a multimodal encoder is also provided, and the device is used to implement the above-mentioned embodiment and preferred implementation mode, and the descriptions that have been made are not repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0130] This embodiment provides a processing device for a multi-modal encoder, such as Figure 7 As shown, including:

[0131] The sample acquisition module 401 is used to acquire a masked sample pair to be processed, wherein the masked sample pair includes an associated text sample and an audio sample, and at least one of the text sample and the audio sample is subjected to masking processing.

[0132] The coding feature generation module 402 is used to generate text coding features of the text sample based on a multimodal encoder, and to generate audio coding features of the audio sample, wherein the text coding features incorporate the spectral linear features of the audio sample, and the audio coding features incorporate the word linear features of the text sample.

[0133] The correction module 403 is used to predict the mask information to be masked according to the text encoding feature and / or the audio encoding feature, and correct the multimodal encoder based on the accuracy of the mask information.

[0134] In some optional embodiments, the coding feature generation module 402 may include:

[0135] The sequence generation unit is used to generate a word vector sequence corresponding to the text sample and a spectrum slice sequence corresponding to the audio sample, wherein the word vector sequence includes a plurality of word vectors and the spectrum slice sequence includes a plurality of spectrum slices.

[0136] The first linear transformation unit is used to perform a linear transformation on any word vector in the word vector sequence to obtain a word linear feature of the word vector, wherein the word linear feature includes a word query feature, a word index feature, and a word content feature.

[0137] The second linear transformation unit is used to perform a linear transformation on any spectrum slice in the spectrum slice sequence to obtain spectrum linear features of the spectrum slice, wherein the spectrum linear features include spectrum query features, spectrum index features and spectrum content features.

[0138] In some optional embodiments, the sequence generation unit may include:

[0139] The word segmentation subunit is used to perform word segmentation on the text sample to obtain a phrase sequence.

[0140] The vector generation subunit is used to generate a word vector for each word in the word sequence to obtain a word vector sequence containing each word vector.

[0141] The slicing subunit is used to generate spectrum information of the audio sample and slice the spectrum information to generate a spectrum slice sequence including each spectrum slice.

[0142] In some optional embodiments, the coding feature generation module 402 may include:

[0143] The first feature acquisition unit is used to acquire a word query feature in a word linear feature of any word vector in a word vector sequence represented by a text sample.

[0144] The first similarity determination unit is used to calculate the similarity between the word query feature and the word index feature in each word linear feature and the spectrum index feature in each spectrum linear feature.

[0145] The first attention feature generating unit is used to generate a word attention feature of a word vector according to the similarity, the word content feature in each word linear feature, and the spectrum content feature in each spectrum linear feature.

[0146] The first encoding generation unit is used to generate text encoding features of the text sample according to the word attention features of each word vector in the word vector sequence.

[0147] In some optional embodiments, the coding feature generation module 402 may further include:

[0148] The second feature acquisition unit acquires, for any spectrum slice in the spectrum slice sequence represented by the audio sample, a spectrum query feature in the spectrum linear feature of the spectrum slice.

[0149] The second similarity determination unit is used to calculate the similarity between the spectrum query feature and the spectrum index features in each spectrum linear feature and the word index features in each word linear feature.

[0150] The second attention feature generating unit is used to generate the spectrum attention feature of the spectrum slice according to the similarity, the spectrum content feature in each spectrum linear feature and the word content feature in each word linear feature.

[0151] The second coding generation unit is used to generate audio coding features of the audio sample according to the spectrum attention features of each spectrum slice in the spectrum slice sequence.

[0152] In some optional embodiments, the processing device of the multimodal encoder may further include:

[0153] The text information acquisition module is used to acquire text description information and generate encoding features of the text description information by completing the corrected multimodal encoder.

[0154] The audio decoding module is used to input the encoding features of the text description information into the audio decoder so as to generate audio information matching the text description information through the audio decoder.

[0155] In some optional embodiments, the audio decoding module includes:

[0156] The audio decoding training unit is used to train the audio decoder. Specifically, the audio decoding training unit is used to: obtain a training sample pair, wherein the training sample pair includes a text description sample and an audio mask sample matching the text description sample, and the audio mask sample includes at least part of known audio information; process the known audio information in the text description sample and the audio mask sample through the audio decoder to predict the masked audio information in the audio mask sample; and calibrate the audio decoder based on the accuracy of the predicted masked audio information.

[0157] In some optional embodiments, the processing device of the multimodal encoder may further include:

[0158] The collection acquisition module is used to obtain text retrieval information and audio file collections.

[0159] The feature generation module is used to generate retrieval coding features of text retrieval information through a multimodal encoder that completes the correction, and to generate audio coding features of each audio file in an audio file collection.

[0160] The retrieval module is used to search for target audio coding features that match the retrieval coding features in the audio coding features of each audio file, and use the audio file represented by the target audio coding features as the audio file retrieved by the text retrieval information.

[0161] In some optional embodiments, the processing device of the multimodal encoder may further include:

[0162] The text and candidate audio acquisition module is used to acquire text description information, and a first candidate audio file and a second candidate audio file corresponding to the text description information.

[0163] The fusion coding module is used to generate a first fusion coding feature of the text description information and the first candidate audio file and to generate a second fusion coding feature of the text description information and the second candidate audio file by completing the corrected multimodal encoder.

[0164] The classification module is used to classify the first fused coding feature and the second fused coding feature and output a classification result, wherein the classification result represents the audio file that better matches the text description information between the first candidate audio file and the second candidate audio file.

[0165] In some optional embodiments, the processing device of the multimodal encoder may further include:

[0166] The audio acquisition module is used to acquire an audio file and generate audio coding features of the audio file by completing the corrected multi-modal encoder.

[0167] The subtitle generation module is used to input the audio coding features of the audio file into the subtitle generation model so as to output subtitle information matching the audio coding features of the audio file through the subtitle generation model.

[0168] In some optional embodiments, the subtitle generation module may include:

[0169] The subtitle generation model training unit is used to train the subtitle generation model. Specifically, the subtitle generation model training unit is used to: obtain a subtitle training sample pair, the subtitle training sample pair includes an audio file sample and a subtitle mask sample, and part of the subtitle content in the subtitle mask sample is masked; determine the known subtitle content before the masked subtitle content in the subtitle mask sample, and process the audio file sample and the known subtitle content through the subtitle generation model to restore the masked subtitle content in the subtitle mask sample; and calibrate the subtitle generation model based on the accuracy of the restored subtitle content.

[0170] In some optional implementation modes, the above-mentioned subtitle generation model training unit can also be used to: process known subtitle content in audio file samples and subtitle mask samples through the subtitle generation model to predict the position of the subtitle end marker in the subtitle mask sample; and calibrate the subtitle generation model based on the accuracy of the predicted position of the subtitle end marker.

[0171] The further functional description of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0172] The processing device of the multimodal encoder in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0173] The processing device of the multimodal encoder provided in this embodiment fuses the spectral linear features of audio samples into the text encoding features, and fuses the word linear features of text samples into the audio encoding features during the encoding processing of the multimodal encoder. Thus, the sharing of audio features and text features can be achieved through one training task, and specific encoding and cross-modal fusion for each modality can be effectively achieved.

[0174] The present disclosure also provides a computer device having the above Figure 7 The processing device of the multimodal encoder is shown.

[0175] See also Figure 8 , Figure 8 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0176] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0177] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0178] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0179] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0180] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0181] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0182] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0183] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0184] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A processing method for a multimodal encoder, characterized in that: The method comprises: Obtaining a mask sample pair to be processed, wherein the mask sample pair includes an associated text sample and an audio sample, and at least one of the text sample and the audio sample is subjected to mask processing; Generate text encoding features of the text sample based on a multimodal encoder, and generate audio encoding features of the audio sample, wherein the text encoding features incorporate the spectral linear features of the audio sample, and the audio encoding features incorporate the word linear features of the text sample; According to the text encoding features and the audio encoding features, mask information that has been masked is predicted, and the multimodal encoder is corrected based on the accuracy of the mask information.

2. The method according to claim 1, characterized in that The spectrum linear feature and the word linear feature are generated in the following manner: Generate a word vector sequence corresponding to the text sample, and generate a spectrum slice sequence corresponding to the audio sample, wherein the word vector sequence includes a plurality of word vectors, and the spectrum slice sequence includes a plurality of spectrum slices; For any word vector in the word vector sequence, linear transformation is performed on the word vector to obtain word linear features of the word vector, wherein the word linear features include word query features, word index features, and word content features; For any spectrum slice in the spectrum slice sequence, a linear transformation is performed on the spectrum slice to obtain spectrum linear features of the spectrum slice, where the spectrum linear features include spectrum query features, spectrum index features, and spectrum content features.

3. The method according to claim 2, characterized in that Generating a word vector sequence corresponding to the text sample and generating a spectrum slice sequence corresponding to the audio sample includes: Performing word segmentation processing on the text sample to obtain a phrase sequence; Generate a word vector for each word group in the word group sequence to obtain a word vector sequence containing each word vector; Generate frequency spectrum information of the audio sample, and slice the frequency spectrum information to generate a frequency spectrum slice sequence including each frequency spectrum slice.

4. The method according to any one of claims 1 to 3, characterized in that: The text encoding features for generating the text sample include: For any word vector in the word vector sequence represented by the text sample, obtain a word query feature in the word linear feature of the word vector; Calculating the similarity between the word query feature and the word index feature in each word linear feature and the spectrum index feature in each spectrum linear feature; Generate a word attention feature of the word vector according to the similarity, the word content feature in each word linear feature, and the spectrum content feature in each spectrum linear feature; Generate text encoding features of the text sample based on the word attention features of each word vector in the word vector sequence.

5. The method according to any one of claims 1 to 3, characterized in that: Generating the audio coding feature of the audio sample includes: For any spectrum slice in the spectrum slice sequence represented by the audio sample, obtaining a spectrum query feature in the spectrum linear feature of the spectrum slice; Calculating the similarity between the spectrum query feature and the spectrum index features in each spectrum linear feature and the word index features in each word linear feature; Generate a spectrum attention feature of the spectrum slice according to the similarity, the spectrum content feature in each spectrum linear feature, and the word content feature in each word linear feature; The audio coding features of the audio samples are generated according to the spectrum attention features of each spectrum slice in the spectrum slice sequence.

6. The method according to claim 1, characterized in that After obtaining the multimodal encoder that has completed the correction, the method further includes: Acquire text description information, and generate encoding features of the text description information through the multimodal encoder that completes the correction; The encoded features of the text description information are input into an audio decoder so as to generate audio information matching the text description information through the audio decoder.

7. The method according to claim 6, characterized in that The audio decoder is trained in the following manner: Acquire a training sample pair, wherein the training sample pair includes a text description sample and an audio mask sample matching the text description sample, wherein the audio mask sample includes at least part of known audio information; Processing the known audio information in the text description sample and the audio mask sample by an audio decoder to predict the masked audio information in the audio mask sample; The audio decoder is corrected based on the predicted accuracy of the masked audio information.

8. The method according to claim 1, characterized in that After obtaining the multimodal encoder that has completed the correction, the method further includes: Get text retrieval information and audio file collections; Generate the retrieval coding features of the text retrieval information and generate the audio coding features of each audio file in the audio file set by the multimodal encoder that completes the correction; Among the audio coding features of each audio file, a target audio coding feature matching the retrieval coding feature is searched, and the audio file represented by the target audio coding feature is used as the audio file retrieved by the text retrieval information.

9. The method according to claim 1, characterized in that: After obtaining the multimodal encoder that has completed the correction, the method further includes: Acquire text description information, and a first candidate audio file and a second candidate audio file corresponding to the text description information; Generate, by means of the multimodal encoder that completes the correction, a first fusion coding feature of the text description information and the first candidate audio file, and generate a second fusion coding feature of the text description information and the second candidate audio file; The first fused coding feature and the second fused coding feature are classified, and a classification result is output, wherein the classification result represents an audio file that better matches the text description information among the first candidate audio file and the second candidate audio file.

10. The method according to claim 1, characterized in that After obtaining the multimodal encoder that has completed the correction, the method further includes: Acquire an audio file, and generate audio coding features of the audio file through the multimodal encoder that completes the correction; The audio coding features of the audio file are input into a subtitle generation model, so that the subtitle generation model outputs subtitle information that matches the audio coding features of the audio file.

11. The method according to claim 10, characterized in that The subtitle generation model is trained in the following way: Acquire a subtitle training sample pair, wherein the subtitle training sample pair includes an audio file sample and a subtitle mask sample, and a portion of the subtitle content in the subtitle mask sample has been masked; Determining known subtitle content before the masked subtitle content in the subtitle mask sample, and processing the audio file sample and the known subtitle content through a subtitle generation model to restore the masked subtitle content in the subtitle mask sample; The subtitle generation model is corrected based on the accuracy of the restored subtitle content.

12. The method according to claim 11, characterized in that The subtitle mask sample also includes a subtitle end mark after mask processing; the method also includes: Processing the audio file sample and known subtitle content in the subtitle mask sample by a subtitle generation model to predict the position of a subtitle end mark in the subtitle mask sample; The subtitle generation model is corrected based on the accuracy of the predicted position of the subtitle end marker.

13. A processing device for a multimodal encoder, characterized in that: The device comprises: A sample acquisition module, used for acquiring a mask sample pair to be processed, wherein the mask sample pair includes an associated text sample and an audio sample, and at least one of the text sample and the audio sample is subjected to mask processing; A coding feature generation module, used to generate text coding features of the text sample and generate audio coding features of the audio sample based on a multimodal encoder, wherein the text coding features incorporate the spectral linear features of the audio sample, and the audio coding features incorporate the word linear features of the text sample; A correction module is used to predict mask information that has been masked according to the text encoding feature and / or the audio encoding feature, and to correct the multimodal encoder based on the accuracy of the mask information.

14. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images