Multimodal training data construction method, device and multimodal large model construction method

Abstract

The present application provides a method and device for constructing multimodal training data and a method for constructing a multimodal large model. The multimodal training data construction method includes: obtaining multiple speech sequences and text sequences corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by extracting features from speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data; for each speech sequence, the corresponding text primitives are sequentially inserted between the feature primitives or audio primitives included therein to obtain multimodal training data; all multimodal training data are collected to form a training data set. In this way, a mapping relationship between short-term speech and text data can be established, thereby enhancing the model's ability to understand and process cross-modal information, thereby improving the effect of real-time speech recognition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for constructing multimodal training data and a method for constructing a multimodal large model. Background Art

[0002] Real-time speech recognition technology is a special case of speech recognition technology that provides instant feedback, meaning it outputs the corresponding text content simultaneously with the voice input (allowing for a small delay). This technology is suitable for scenarios such as meeting minutes where the spoken content needs to be displayed immediately. In other words, as the speaker speaks, the spoken content can be displayed immediately.

[0003] Currently, there are two approaches to achieving real-time speech recognition. One is an end-to-end deep learning model based on Continuous Temporal Classification (CTC). The CTC model can output recognized text during speech input. During the training phase, the CTC model relies on speech-text "data pairs" for training. During the inference phase, the CTC model can generate the entire text sequence in one go by learning the overall mapping relationship between audio and text sequences. However, this mapping method is relatively simple and does not fully utilize the complex structure and grammatical rules of language, nor does it take into account the continuity and randomness of language. As a result, the real-time speech recognition performance of the CTC model is relatively mediocre.

[0004] Another approach is to use a Large Language Model (LLM). During the training phase, LLMs rely on very large amounts of text data and possess large model parameters, enabling them to fully utilize language knowledge. However, during the inference phase, LLMs generate text units one by one for each piece of input speech data. Therefore, they can only be used for speech recognition, or in other words, for real-time speech recognition with significant latency. Summary of the Invention

[0005] The embodiments of the present application provide a multimodal training data construction method, device and multimodal large model construction method to solve the problem of poor real-time speech recognition effect of traditional models.

[0006] In a first aspect, an embodiment of the present application provides a method for constructing multimodal training data, which is applied to a multimodal large model for real-time speech recognition; the method includes: obtaining multiple speech sequences and a text sequence corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by feature extraction of speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data; for each speech sequence, the corresponding text primitives are inserted in sequence between the feature primitives or audio primitives included therein to obtain multimodal training data; all multimodal training data are collected to form a training data set.

[0007] In one implementable manner, speech data and speech sequences are arranged in frames, and each frame of speech data corresponds to at least one frame of feature primitives or audio primitives; multimodal training data includes training input samples and training output samples; for each speech sequence, the corresponding text primitives are inserted in sequence between the feature primitives or audio primitives included therein, and the steps of obtaining multimodal training data include: for each text primitive corresponding to the speech sequence, determining the first frame sequence number of the pronunciation end position in the speech sequence; determining a delay parameter, the delay parameter is D frames, D≥0; for each text primitive corresponding to the speech sequence, determining its insertion position in the speech sequence based on its corresponding first frame sequence number and delay parameter; wherein the frame sequence number of the insertion position is the second frame sequence number, and the second frame sequence number is equal to the sum of the first frame sequence number and the delay parameter; adding a start mark at the starting position of the speech sequence, and inserting each text primitive into its corresponding insertion position to form a training input sample; and determining a training output sample based on the training input sample.

[0008] In one achievable manner, the step of inserting each text primitive after its corresponding insertion position includes: determining whether the second frame sequence number corresponding to the target text primitive is greater than the total number of frames of the speech sequence, the target text primitive being the last text primitive in the text sequence; if it is not greater than the total number of frames of the speech sequence, inserting each text primitive after its corresponding insertion position; if it is greater than the total number of frames of the speech sequence, adding N blank frames at the end of the speech sequence, and inserting each text primitive after its corresponding insertion position; wherein N is equal to the difference between the second frame sequence number corresponding to the target text primitive and the total number of frames.

[0009] In one achievable manner, the step of determining the training output samples based on the training input samples includes: offsetting the training input samples by one frame to form the training output samples, so that the second frame of the training output samples is aligned with the first frame of the training input samples.

[0010] In a second aspect, an embodiment of the present application provides a method for constructing a multimodal large model for real-time speech recognition, the method comprising: obtaining a pre-trained language model; wherein the pre-trained language model is constructed based on a decoder-only transformer architecture; training the pre-trained language model using multimodal training data to obtain a multimodal large model for real-time speech recognition; wherein the multimodal training data is constructed by the multimodal training data construction method provided by the aforementioned first aspect and its various implementation methods.

[0011] In one achievable manner, the step of training a pre-trained language model using multimodal training data includes: obtaining a training speech sequence, which is a speech sequence composed of audio primitives obtained by segmenting speech data in the multimodal training data construction method provided by the first aspect and its various implementations; obtaining a first vocabulary and a first embedding matrix; wherein the first vocabulary is a collection of text primitives, and the step of segmenting the text data corresponding to the speech data to obtain text primitives is performed based on the first vocabulary; each row of the first embedding matrix corresponds to each text primitive in the first vocabulary, and each column corresponds to the dimension of the embedding vector, and the first embedding matrix is ​​used to map the text primitives to the embedding vector; each audio primitive in the training speech sequence is added to the first vocabulary of the pre-trained language model to obtain a second vocabulary, which is a collection of audio primitives and text primitives; and, expanding the rows of the first embedding matrix to obtain a second embedding matrix, so that each row of the second embedding matrix corresponds to each audio primitive or text primitive in the second vocabulary; and training the pre-trained language model using the second vocabulary, the second embedding matrix and the multimodal training data to obtain a multimodal large model.

[0012] In one achievable manner, the step of training the pre-trained language model using multimodal training data also includes: obtaining a training speech sequence; constructing a third vocabulary, which is a set of audio primitives in the training speech sequence; and constructing a third embedding matrix, wherein each row of the third embedding matrix corresponds to each audio primitive in the third vocabulary, each column corresponds to the dimension of the embedding vector, and the second embedding matrix is ​​used to map the audio primitives to the embedding vector; obtaining a first vocabulary and a first embedding matrix; and training the pre-trained language model using the first vocabulary, the third vocabulary, the first embedding matrix, the third embedding matrix, and the multimodal training data to obtain a large multimodal model.

[0013] In a third aspect, an embodiment of the present application further provides a method for constructing a multimodal large model for real-time speech recognition, wherein the multimodal large model is obtained by training the multimodal large model construction method for real-time speech recognition provided by the aforementioned second aspect and its various implementation methods; the method comprises: inputting the real-time speech data to be recognized into the multimodal large model, using the multimodal large model to segment the real-time speech data according to a preset length, and obtaining a speech sequence to be recognized including audio primitives to be recognized; wherein the length of the real-time speech data gradually increases with time, and the real-time speech data and the speech sequence to be recognized are gradually increased. It is arranged in frames, and each frame of real-time speech data corresponds to at least one frame of audio primitives to be recognized; a multimodal large model is used to predict the current frame to obtain a predicted data frame, and the current frame is one of the frames in the speech sequence to be recognized; if the predicted data frame is a text frame, the multimodal large model is used to output the text frame to achieve real-time speech recognition; and, the text frame is inserted after the current frame to form a new speech sequence to be recognized, and the multimodal large model is continued to be used to predict the next frame of the current frame; if the predicted data frame is an audio frame, the multimodal large model is continued to be used to predict the next frame of the current frame.

[0014] In a fourth aspect, an embodiment of the present application also provides a multimodal training data construction device, which is applied to a multimodal large model for real-time speech recognition; the device includes: a first acquisition module, used to obtain multiple speech sequences and text sequences corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by feature extraction of speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data; a data construction module, used to, for each speech sequence, sequentially insert the corresponding text primitives between the feature primitives or audio primitives included therein to obtain multimodal training data; a collection module, used to collect all multimodal training data to form a training data set.

[0015] In a fifth aspect, an embodiment of the present application also provides a multimodal large model construction device for real-time speech recognition, the device comprising: a second acquisition module for obtaining a pre-trained language model; wherein the pre-trained language model is constructed based on a decoder-only transformer architecture; a training module for training the pre-trained language model using multimodal training data to obtain a multimodal large model for real-time speech recognition; wherein the multimodal training data is constructed based on the multimodal training data construction method provided in the aforementioned first aspect and its various implementation methods.

[0016] In the sixth aspect, the embodiment of the present application also provides a multimodal large model construction device for real-time speech recognition, the multimodal large model is obtained by training the multimodal large model construction method for real-time speech recognition provided by the aforementioned second aspect and its various implementation methods; the device includes: a segmentation module for inputting the real-time speech data to be recognized into the multimodal large model, using the multimodal large model to segment the real-time speech data according to a preset length, and obtaining a speech sequence to be recognized including audio primitives to be recognized; wherein the length of the real-time speech data gradually increases with time, the real-time speech data and the speech sequence to be recognized are arranged in frames, and each frame The real-time speech data corresponds to at least one frame of audio primitives to be recognized; the prediction output module is used to use the multimodal large model to predict the current frame to obtain a predicted data frame, and the current frame is one of the frames in the speech sequence to be recognized; if the predicted data frame is a text frame, the prediction output module is also used to output the text frame using the multimodal large model to achieve real-time speech recognition; and, insert the text frame after the current frame to form a new speech sequence to be recognized, and continue to use the multimodal large model to predict the next frame of the current frame; if the predicted data frame is an audio frame, the prediction output module is also used to continue to use the multimodal large model to predict the next frame of the current frame.

[0017] From the above content, it can be seen that the embodiment of the present application provides a multimodal training data construction method, device and multimodal large model construction method, which includes: obtaining multiple speech sequences and text sequences corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by feature extraction of speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data; for each speech sequence, the corresponding text primitives are inserted in order between the feature primitives or audio primitives included therein to obtain multimodal training data; all multimodal training data are collected to form a training data set. In this way, a mapping relationship between short-term speech and text data can be established, thereby enhancing the model's understanding and processing capabilities of cross-modal information, thereby improving the effect of real-time speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A first flow chart of the multimodal training data construction method provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of segmenting voice data and text data provided in an embodiment of the present application;

[0020] Figure 3A second flow chart of the multimodal training data construction method provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of a process for inserting text primitives into a speech sequence according to an embodiment of the present application;

[0022] Figure 5 A decomposition diagram of inserting text primitives into a speech sequence provided in an embodiment of the present application;

[0023] Figure 6 A flowchart of a first method for constructing a multimodal large model for real-time speech recognition provided in an embodiment of the present application;

[0024] Figure 7 A flowchart of a second method for constructing a large multimodal model for real-time speech recognition provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of the multimodal large model prediction or inference process provided in an embodiment of the present application;

[0026] Figure 9 A schematic diagram of the structure of a multimodal training data construction device provided in an embodiment of the present application;

[0027] Figure 10 A schematic diagram of the structure of a first multimodal large model building device for real-time speech recognition provided in an embodiment of the present application;

[0028] Figure 11 A schematic structural diagram of a second multimodal large model building device for real-time speech recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0030] Before introducing the technical solutions of the embodiments of the present application, an exemplary introduction to the terms involved in the embodiments of the present application is first given.

[0031] 1. Multimodal Data: A data set that contains two or more different types of information.

[0032] 2. Tokenization: Also known as tokenization, this refers to the process of breaking text into smaller units, known as tokens. In natural language processing, tokenization is a common preprocessing step. Tokenization can divide raw text into an ordered sequence of tokens based on preset rules or algorithms. Token sequences can then serve as input data for subsequent deep learning model training and inference. Common text tokenization methods include word segmentation based on spaces, segmentation based on punctuation, and segmentation by characters or subwords. The goal of tokenization is to convert text into a form that machines can understand and process.

[0033] In the application of large models, text data (also known as corpus) can usually be divided into multiple small fragments, namely text tokens. A text token can represent the smallest semantic character unit in the text, such as a single Chinese character or word in Chinese. After deduplication of all possible text tokens, a token list, namely the vocabulary, can be obtained. Afterwards, a unique identifier (such as an integer ID) can be assigned to each text token in the vocabulary, which can be used as an embedding index of the large model embedding matrix. When processing text, the large model can first divide the text into text tokens and determine their corresponding indices, and then the indices can be used to find the corresponding embedding vectors in the embedding matrix. For example, if the character "cat" appears in the text, the index of "cat" (such as 0) can be found through the vocabulary, and then the embedding vector of "cat" can be found in the 0th row of the embedding matrix.

[0034] With technological advancements, the applications and implementations of tokenization are constantly evolving, providing powerful support for a wide range of AI tasks. Tokens are no longer exclusively text snippets, but have gradually expanded to encompass other types of data representations, such as image patches or audio clips. This expansion enables large models to process not only traditional text data but also multimodal inputs, further enhancing their adaptability and application scope.

[0035] Speech recognition technology converts speech signals into text or commands. Speech recognition technology can identify the content expressed by the speaker by analyzing and processing human voice input. Real-time speech recognition technology is a special case of speech recognition technology that can provide instant feedback, that is, output the corresponding text content simultaneously with the voice input (allowing for a small delay). This technology is suitable for scenarios such as meeting minutes where the content of the speech needs to be displayed immediately. In other words, as the speaker speaks, the content of the speech can be displayed immediately. The corresponding technology is offline speech recognition technology, which generates complete text for the entire input speech at once, such as meeting recordings and call recordings.

[0036] Currently, there are two approaches to achieving real-time speech recognition. One is an end-to-end deep learning model based on Continuous Temporal Classification (CTC). The CTC model can output recognized text during speech input. During the training phase, the CTC model relies on speech-text "data pairs" for training. During the inference phase, the CTC model can generate the entire text sequence in one go by learning the overall mapping relationship between audio and text sequences. However, this mapping method is relatively simple and does not fully utilize the complex structure and grammatical rules of language, nor does it take into account the continuity and randomness of language. As a result, the real-time speech recognition performance of the CTC model is relatively mediocre.

[0037] Another approach is to use a large language model (LLM). During the training phase, the LLM relies on very large amounts of text data and has large model parameters, allowing it to fully utilize language knowledge. However, during the inference phase, the LLM generates text units one by one for each piece of input speech data and outputs them uniformly. Therefore, it can only be used to implement speech recognition, not real-time speech recognition. In other words, it can only be used to implement real-time speech recognition with a large delay.

[0038] The embodiments of the present application can provide a multimodal training data construction method, a multimodal large model construction method and a corresponding device. The method provided by the embodiments of the present application can not only utilize the powerful language modeling capabilities of the large model, but also realize real-time speech recognition function, which can solve the aforementioned problems, improve recognition accuracy and ensure recognition effect.

[0039] Before introducing the specific method provided in the embodiments of the present application, some exemplary explanations of text tokenization and speech tokenization are required.

[0040] As mentioned above, text tokenization breaks down a text string into smaller, processable units (called text tokens), typically words, subwords, or characters. For example, for the sentence "Natural language processing is an important field of artificial intelligence," the result of subword segmentation is: ["nature", "language", "processing", "is", "artificial", "intelligence", "an", "important", "field", "."], where "nature" is a single text token. The result of character segmentation is: ["self", "natural", "language", "processing", "is", "artificial", "intelligence", "an", "important", "field", "."], where "self" and "natural" are two separate text tokens.

[0041] Speech tokenization is the process of decomposing audio signals into smaller, processable units. This allows the continuous audio signal to be converted into a discrete, recognizable sequence of symbols. The smaller, processable units are typically phonemes, syllables, or acoustic features. Acoustic features, for example, are Mel-Frequency Cepstral Coefficients (MFCCs) or Filter Bank (FBank) features. These phonemes, syllables, or acoustic features can then be mapped to feature tokens using a vocabulary, where the vocabulary includes a mapping relationship between feature tokens and embedded representations of phonemes, syllables, or acoustic features. Alternatively, a more direct approach is to segment the audio signal into short frames, each of which represents an audio signal over a period of time. For example, each frame is 10ms or 20ms, and each frame of audio signal can be called a speech token.

[0042] In order to achieve speech recognition, the token arrangement of the existing text-speech multimodal model is [speech feature / speech token sequence][text feature / text token sequence]. It can be understood that here only the "speech feature" is equivalent to the aforementioned "feature token".

[0043] The method provided in the embodiment of the present application can modify the existing token arrangement to: [short-time speech feature / short-time speech token sequence][text feature / token][short-time speech feature / short-time speech token sequence][text feature / token], that is, the text token is interspersed in the speech feature sequence or the speech token sequence. Specifically, the text token can be inserted at the position where it is expected to appear (for example, shortly after the text token appears completely in the speech). The specific method of insertion will be described in detail below and will not be repeated here.

[0044] For example, if "*" represents a speech feature or speech token and "+" represents a text token, then for a speech recognition task, the existing token arrangement is "****************+++++++", and the modified arrangement is "***+**+*+***+**+**+***+".

[0045] It's important to emphasize that, for ease of description and to avoid ambiguity, we will refer to text tokens as text primitives, speech tokens derived from frame segmentation as audio primitives, and feature tokens derived from acoustic feature mapping as feature primitives. It's understood that these name changes do not limit the actual processing steps of this solution.

[0046] It can be understood that the life cycle of a model includes three stages, namely, the dataset construction stage, the training stage, and the inference stage. The embodiment of this application will introduce the multimodal large model for real-time speech recognition in stages.

[0047] The following is an introduction to the dataset construction stage.

[0048] Figure 1 This is a first flowchart of the multimodal training data construction method provided in an embodiment of the present application.

[0049] like Figure 1 As shown, the multimodal training data construction method provided in the embodiment of the present application includes the following steps S101-S104.

[0050] S101: Acquire voice data.

[0051] Among them, voice data is specifically an audio signal, which contains certain voice content, such as words, phrases or sentences, and can be used as a basic data source for subsequent processing. In practical applications, for example, the sound emitted by a speaker randomly or with reference to a corpus can be captured in real time by a recording device such as a microphone, and then converted into an audio signal in a digital format through sampling and quantization to obtain voice data. For another example, the voice data can be a public voice dataset, specifically Common Voice, LibriSpeech and other datasets, which are not specifically limited in this embodiment of the present application. For another example, voice data obtained in a real business scenario can be the sound recorded by a user using a recording device in a meeting or interview. This embodiment of the present application does not specifically limit this.

[0052] In the embodiments of the present application, the voice data is arranged in frames. For example, if the sampling rate is 44.1kHz, then there are 44,100 samples per second. When the length of the audio frame is 10ms, then each audio frame contains 441 samples (44,100*0.01). The voice data may be in different formats such as WAV and MP3, which are not specifically limited in the embodiments of the present application.

[0053] S102: Transcribing the voice data into text to obtain text data.

[0054] The text transcription step can be performed manually and proofread to ensure the accuracy of the text data. Alternatively, the text data can be generated by transcription based on an automatic speech recognition (ASR) system, which can then be manually proofread to ensure the accuracy of the text data.

[0055] In some implementations, when voice data is captured in real time by a recording device, if the speaker typically pronounces based on existing corpus, that is, if this step is performed based on existing corpus, then the text data is known, and therefore there is no need to perform a text transcription step.

[0056] It is understandable that when a public speech dataset is used, the text data corresponding to the speech dataset can usually be directly obtained.

[0057] S103: Segment the text data, and record the data sequence formed after segmentation as a text sequence. That is, the text sequence is composed of text primitives obtained by segmenting the text data.

[0058] The specific tokenization (segmentation) method may be space-based tokenization, punctuation-based tokenization, subword tokenization, character-level tokenization, or sentence-level tokenization, which is not specifically limited in the embodiments of the present application.

[0059] S104: Segment the speech data, and record the data sequence formed after segmentation as a speech sequence. That is, the speech sequence is composed of audio primitives obtained by segmenting the speech data.

[0060] Specifically, the step of segmenting the speech data may include dividing the speech into a number of short time frames, for example, each frame is 10 ms or 20 ms. Figure 2 The results of segmenting speech data and text data are exemplarily shown.

[0061] It is worth noting that the voice data and voice sequence are arranged in frames, and each frame of voice data corresponds to at least one frame of audio primitives. If the audio frame length of the voice data is equal to the target frame length for segmentation, then each frame of voice data corresponds to one frame of audio primitives. If the audio frame length of the voice data is less than the target frame length for segmentation, then each frame of voice data corresponds to multiple frames of audio primitives. This embodiment of the present application does not specifically limit this.

[0062] Before segmenting the speech data, a preprocessing step may be performed first. The preprocessing may include denoising and normalization, etc., which is not specifically limited in the embodiment of the present application.

[0063] In some implementations, the speech sequence may also be determined based on the following step S105 .

[0064] S105: Extract features from the speech data to obtain a speech sequence. That is, the speech sequence is composed of feature primitives obtained by extracting features from the speech data.

[0065] This step can be divided into two stages: first, extracting embeddings from an existing speech representation model, and then further discretizing these embeddings into a limited set of feature primitives based on a vocabulary. Speech data and speech sequences are arranged in frames, and each frame of speech data corresponds to at least one frame of feature primitives.

[0066] Specifically, after preprocessing, the speech data can be subjected to feature extraction, such as Mel Spectrogram and Mel-Frequency Cepstral Coefficients (MFCC). The audio features are then fed into a speech representation model to generate an embedding. The embedding vector is then mapped to the closest feature primitive using a vocabulary. The vocabulary contains mappings between feature primitives and embedded representations of acoustic features.

[0067] Figure 3 This is a second flowchart of the multimodal training data construction method provided in an embodiment of the present application.

[0068] like Figure 3 As shown, the multimodal training data construction method provided in the embodiment of the present application is applied to a multimodal large model for real-time speech recognition, and can specifically include the following steps S200-S400.

[0069] S200: Acquire multiple voice sequences and a text sequence corresponding to each voice sequence.

[0070] The speech sequence is composed of feature primitives obtained by extracting features from speech data, and can be specifically determined based on step S105.

[0071] Alternatively, the speech sequence is composed of audio primitives obtained by segmenting the speech data, which can be specifically determined based on step S104.

[0072] The embodiment of the present application does not specifically limit the number of speech sequences. For example, dozens, hundreds or even more speech sequences can be obtained to enhance the learning and generalization capabilities of the model.

[0073] For example, in this embodiment of the present application, "*" is used to represent a feature primitive or audio primitive. For example, for a speech sequence with the text content "I love you, my motherland", the corresponding speech sequence may be "***************", with a total of 15 frames. It will be understood that the speech sequence here is merely an example, and "*" does not represent an actual feature primitive or audio primitive.

[0074] The text sequence is composed of text primitives obtained by segmenting the text data corresponding to the speech data. Each text primitive is a character, word, or sentence in the text data. The text sequence may be determined by step S103. Different tokenization (segmentation) methods may result in different contents contained in a text primitive.

[0075] Exemplarily, the embodiment of the present application performs the text tokenization step in a character-by-character segmentation manner. Then, for the text data: "I love your motherland", the tokenization result is [I, love, you, motherland, country], "I" is a text primitive, "love" is a text primitive, "you" is a text primitive, "ancestor" is a text primitive, and accordingly, "country" is also a text primitive.

[0076] S300: For each speech sequence, insert the corresponding text primitives into the feature primitives or audio primitives included therein in order to obtain multimodal training data.

[0077] The multimodal training data includes training input samples and training output samples. The training input samples can also be referred to as input or training input, and the training output samples can also be referred to as target or training output. The specific steps for inserting text primitives into speech sequences will be described in detail below and will not be repeated here.

[0078] S400: Collect all multimodal training data to form a training dataset D.

[0079] The present embodiment may also collect a certain amount of the above multimodal training data to form a training dataset. The amount of multimodal training data in the training dataset can be designed based on actual conditions and is not specifically limited in the present embodiment. This training dataset D can be used to train a large multimodal model, so that the large multimodal model can achieve real-time speech recognition with high recognition accuracy and good recognition effect.

[0080] In summary, based on the method provided in the embodiment of the present application, a multimodal training data that integrates text and voice information can be obtained, and in the multimodal training data, the text information and the voice information are intertwined, rather than independent of each other. Then, based on the data training model, a mapping relationship between short-term speech (such as 10ms frame data) and text data can be established to replace the mapping relationship between long speech (such as a sentence) and text data, thereby enhancing the model's understanding and processing capabilities of cross-modal information. Moreover, based on the data training model, the model can be made to predict the next output based on the previous output (whether text or voice), thereby enhancing the model's understanding of language continuity and achieving autoregression.

[0081] From the above content, it can be seen that the embodiment of the present application provides a method for constructing multimodal training data, which includes: obtaining multiple speech sequences and text sequences corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by feature extraction of speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data; for each speech sequence, the corresponding text primitives are inserted in order between the feature primitives or audio primitives included therein to obtain multimodal training data; all multimodal training data are collected to form a training data set. In this way, a mapping relationship between short-term speech and text data can be established, thereby enhancing the model's understanding and processing capabilities of cross-modal information.

[0082] Figure 4 A schematic diagram of the process of inserting text primitives into a speech sequence provided in an embodiment of the present application.

[0083] Figure 5 A decomposition diagram of inserting text primitives into a speech sequence provided in an embodiment of the present application.

[0084] like Figure 4 and Figure 5 As shown, step S300 may include the following steps S301-S305.

[0085] S301: For each text primitive corresponding to the speech sequence, determine the first frame sequence number of the pronunciation end position in the speech sequence.

[0086] Since the speech sequence is arranged in frames, each frame of audio primitives or feature primitives will have a unique frame sequence number. The starting value of the frame sequence number can be set to 1, that is, starting from the 1st frame and continuing to the Mth frame, where M is the total number of frames.

[0087] Determining the pronunciation end position may refer to, for each text primitive, such as for the text primitive "我", determining the timestamp (t_end) in the speech data when its pronunciation ends, and then determining the audio primitive frame or feature primitive frame corresponding to the pronunciation end position based on the timestamp. The pronunciation end position may also be referred to as the right boundary or end boundary of the audio primitive or feature primitive corresponding to the text primitive. This step may be implemented based on an algorithm or with the help of manual proofreading, and the embodiments of this application do not make specific limitations on this.

[0088] Exemplarily, after marking the pronunciation end position, the text data is as follows: [我3, 爱5, 你7, 祖11, 国14], where "我3" means that the pronunciation of "我" ends at the 3rd frame, that is, the first frame sequence number B corresponding to the pronunciation end position of "我" is 3.

[0089] S302: Determine the delay parameter, and the delay parameter is D frames.

[0090] Among them, D should be a non-negative integer, D≥0. The delay parameter can be set according to actual needs. D is, for example, equal to 0, 3, 5 or 7, and the embodiments of this application do not make specific limitations on this. Setting the delay parameter can help the model better understand the fluency of the language and the natural rhythm of the speech.

[0091] This is because, for Chinese, the same pronunciation can correspond to multiple Chinese characters. If the predicted Chinese characters are directly output without setting a delay, the continuity between contexts will be ignored. And in the process of speech recognition, setting the delay parameter can help the model identify the appropriate context in the speech input and then correctly decode it into the corresponding Chinese characters. For example, when the speaker says "我爱你祖", the model can predict and output the character "你" to avoid prediction errors.

[0092] S303: For each text primitive corresponding to the speech sequence, based on its corresponding first frame sequence number and the delay parameter, determine its insertion position in the speech sequence; where the frame sequence number of the insertion position is the second frame sequence number.

[0093] It can be understood that the frame sequence number (the second frame sequence number) of the insertion position is the sum of the first frame sequence number and the delay parameter, that is, the second frame sequence number T = B + D, where B represents the first frame sequence number and D represents the value of the delay parameter.

[0094] Exemplarily, if the delay parameter is 3 frames, that is, D = 3, then the insertion position can be expressed as: [我3 + 3, 爱5 + 3, 你7 + 3, 祖11 + 3, 国14 + 3], that is, [我6, 爱8, 你10, 祖14, 国17], that is, the second frame sequence number T of the insertion position of "我" is 6.

[0095] Step S303 may specifically include the following steps S3031 - S3033.

[0096] S3031: Determine whether the second frame sequence number corresponding to the target text primitive is greater than the total number of frames in the speech sequence, and the target text primitive is the last text primitive in the text sequence.

[0097] It is understandable that when the delay parameter is 0 frames, i.e., D=0, then the second frame sequence number corresponding to the insertion position of the target text primitive (the last text primitive in the text sequence) must not exceed the total number of frames in the speech sequence. When the delay parameter is not 0, i.e., D>0, the second frame sequence number corresponding to the insertion position of the target text primitive (the last text primitive in the text sequence) may exceed the total number of frames in the speech sequence, in which case the speech sequence needs to be extended.

[0098] S3032: If the number of text primitives is not greater than the total number of frames of the speech sequence, insert each text primitive after its corresponding insertion position.

[0099] It can be understood that when the second frame sequence number corresponding to the insertion position is 6, it means that the text element should be inserted after the 6th frame of the speech sequence. Correspondingly, when the second frame sequence number corresponding to the insertion position is 14, it means that the text element should be inserted after the 14th frame of the speech sequence.

[0100] For example, when the delay parameter is 0 frames, that is, D=0, the second frame sequence number corresponding to the target text primitive is 14, which does not exceed the total number of frames 14. Therefore, each text primitive can be directly inserted after its corresponding insertion position, and the insertion result is: "***I**love**you****mother***country*".

[0101] S3033: If it is greater than the total number of frames of the speech sequence, N blank frames are added to the end of the speech sequence, and each text primitive is inserted after its corresponding insertion position.

[0102] N is the difference between the second frame sequence number corresponding to the target text primitive and the total number of frames. Blank frames, also known as silent frames, silent frames, or speechless frames, are frames that contain no valid speech signals or only background noise. Blank frames appear as regions with low or near-zero amplitude in the speech waveform, serving as placeholders.

[0103] For example, when the delay parameter is 3 frames, that is, D>0, the second frame sequence number corresponding to the text primitive is 17, which exceeds the total number of frames 15. Therefore, N blank frames can be added to the end of the speech sequence to ensure that the speech sequence is large enough to accommodate all text primitives, where N = 17-15 = 2. The result of the insertion is: "****** I** love** you **** motherland*--country", where "-" represents a blank frame.

[0104] It can be understood that the delay parameter is 3 frames, which means that it is expected that the corresponding output is "I" when the voice sequence is input to the 6th frame, and so on, which will not be elaborated here.

[0105] S304: Add a start marker at the start position of the speech sequence, and insert each text primitive into its corresponding insertion position to form a training input sample.

[0106] The start tag can be " ", used to indicate the beginning of the model training input sample. In some implementations, the start marker can also be " <bos>", which is not specifically limited in the embodiments of this application.

[0107] Exemplarily, the training input sample can be "b******I love you****Motherland".

[0108] S305: Determine the training output sample based on the training input sample.

[0109] Step S305 can specifically include the following step S3051: Shift the training input sample by one frame to form the training output sample, so that the second frame of the training output sample is aligned with the first frame of the training input sample.

[0110] Among them, shifting the training input sample by one frame specifically means that, relative to the training input sample, the training input sample is shifted one unit to the left.

[0111] Exemplarily, the training output sample can be "******I love you****Motherland".

[0112] It can be understood that, in order to ensure that the number of frames of the training input sample and the training output sample is the same, the last frame of the training input sample can be removed, that is, the training input sample can be "b******I love you****Mother".

[0113] Then, the training input sample and the training output sample can be regarded as a multi-modal training data. A certain number of the above multi-modal training data sets can form a training data set D. This training data set D can be used to train the model so that the model can learn the mapping relationship between the feature primitives or audio primitives and the text primitives, thereby achieving accurate understanding and processing of the speech signal.

[0114] It can be understood that for each speech sequence, steps S301 - S305 need to be executed.

[0115] Next, the model training stage will be introduced.

[0116] Figure 6 It is a schematic flowchart of the first multi-modal large model construction method for real-time speech recognition provided by the embodiments of this application.

[0117] As Figure 6 shown, the multi-modal large model construction method provided by the embodiments of this application can include the following steps S501 - S502.

[0118] S501: Obtain a pre-trained language model.

[0119] The pre-trained language model is built based on a decoder-only transformer architecture. Thus, the embodiments of the present application can use a pre-trained LLM model as a basis and retrain it on a training dataset D to obtain the final model. For example, the pre-trained language model is, for example, an LLAMA3 pre-trained model, i.e., a base LLM. This shows that the embodiments of the present application can reuse pre-trained language models, shortening the model building process.

[0120] S502: Using multimodal training data to train a pre-trained language model to obtain a large multimodal model for real-time speech recognition; wherein the multimodal training data is constructed based on the aforementioned multimodal training data construction method.

[0121] The embodiment of the present application can provide the following two training methods.

[0122] Specifically, in the first training method, step S502 may include the following steps S5021-S5025.

[0123] S5021: Obtain a training speech sequence. The training speech sequence is a speech sequence composed of audio primitives obtained by segmenting speech data in the aforementioned multimodal training data construction method.

[0124] It is understandable that this step can obtain all training speech sequences corresponding to the training data set D, rather than a training speech sequence corresponding to a multimodal training data set.

[0125] S5022: Obtain a first vocabulary and a first embedding matrix; wherein the first vocabulary is a set of text primitives, and the step of segmenting the text data corresponding to the speech data to obtain text primitives is performed based on the first vocabulary; each row of the first embedding matrix corresponds to each text primitive in the first vocabulary, and each column corresponds to the dimension of the embedding vector, and the first embedding matrix is ​​used to map the text primitives to the embedding vectors.

[0126] For example, the first vocabulary may include 30,000 text primitives, so the index ID of the text primitives can range from 0 to 29,999. Furthermore, the number of rows in the first embedding matrix is ​​30,000, and the number of columns can be designed based on the actual required embedding vector dimension. If the index ID of the text primitive "I" is 3, the index ID of the text primitive "love" is 5, the index ID of the text primitive "you" is 7, the index ID of the text primitive "ancestor" is 9, and the index ID of the text primitive "country" is 11, then the text sequence [I, love, you, ancestor, country] can be converted to the index ID sequence [3, 5, 7, 9, 11]. Afterwards, the corresponding embedding vectors can be extracted from the first embedding matrix. For example, for the index ID sequence [3, 5, 7, 9, 11], the model can extract the 3rd, 5th, 7th, 9th, and 11th rows from the first embedding matrix to obtain the embedding vectors of these five text primitives.

[0127] S5023: Add each audio primitive in the training speech sequence to the first vocabulary of the pre-trained language model to obtain a second vocabulary, where the second vocabulary is a collection of audio primitives and text primitives.

[0128] For example, if there are 10,000 audio primitives in the training speech sequence, then the 10,000 audio primitives can be added after the 30,000 text primitives, so that the first vocabulary is expanded to include 40,000 primitives. It is understandable that the index ID corresponding to the audio primitives can range from 30,000 to 39,999.

[0129] S5024: Expand the rows of the first embedding matrix to obtain a second embedding matrix, so that each row of the second embedding matrix corresponds to each audio primitive or text primitive in the second vocabulary.

[0130] It is understandable that this step can expand the rows of the first embedded matrix from 30,000 rows to 40,000 rows. In addition, the embodiment of the present application does not limit the columns of the first embedded matrix, and the columns can be expanded based on actual needs.

[0131] This yields a second embedding matrix that includes not only the embedding vectors for the original text primitives but also the embedding vectors for the newly added audio primitives. Each row in the second embedding matrix corresponds to a single audio primitive or text primitive in the second vocabulary, ensuring that the model accurately maps audio primitives or text primitives to embedding vectors during training.

[0132] S5025: Train the pre-trained language model using the second vocabulary, the second embedding matrix, and the multimodal training data to obtain a large multimodal model.

[0133] In some implementations, audio primitives other than the audio primitives in the training speech sequence may be added to the second vocabulary to expand the coverage of the audio primitives in the second vocabulary.

[0134] It is understandable that the training method provided in the embodiment of the present application can use audio primitives as expansion tokens. In this way, the multimodal large model not only inherits the text processing capabilities of the pre-trained language model, but also adds voice processing capabilities, realizing real-time conversion from voice signals to text output.

[0135] In the second training method, step S502 may include the following steps S5026-S5030.

[0136] S5026: Obtain a training speech sequence.

[0137] It can be understood that the training speech sequence is a speech sequence composed of audio primitives obtained by segmenting speech data in the aforementioned multimodal training data construction method.

[0138] S5027: Construct a third vocabulary, where the third vocabulary is a set of audio primitives in the training speech sequence.

[0139] For example, the training speech sequence involves 10,000 audio primitives, and the third vocabulary may be a set of the 10,000 audio primitives, and the index ID of the audio primitives ranges from 0 to 9999.

[0140] S5028: Construct a third embedding matrix, where each row of the third embedding matrix corresponds to each audio primitive in the third vocabulary, and each column corresponds to the dimension of the embedding vector. The second embedding matrix is ​​used to map the audio primitive to the embedding vector.

[0141] Exemplarily, the number of rows of the third embedding matrix can be 10,000, corresponding to each audio primitive, and the number of columns can be designed based on the actual required embedding vector dimension, which is not specifically limited in the embodiment of the present application.

[0142] S5029: Obtain a first vocabulary and a first embedding matrix.

[0143] It can be understood that the first vocabulary and the first embedding matrix are the same as the first vocabulary and the first embedding matrix in step S5022.

[0144] S5030: Train the pre-trained language model using the first vocabulary, the third vocabulary, the first embedding matrix, the third embedding matrix, and the multimodal training data to obtain a large multimodal model.

[0145] It is worth noting that the training steps of step S5025 and step S5030 can be the same as the pre-training, continued training and / or fine-tuning method of LLM, and the embodiment of the present application does not make specific limitations on this.

[0146] In some implementations, audio primitives other than the audio primitives in the training speech sequence may be added to the third vocabulary to expand the coverage of the audio primitives by the third vocabulary.

[0147] It should be noted that, in actual applications, one of the training methods may be selected for training, for example, using steps S5021-S5025 for training, or using steps S5026-S5030 for training.

[0148] The following is an introduction to the model inference stage.

[0149] Figure 7 A flowchart of a second method for constructing a multimodal large model for real-time speech recognition provided in an embodiment of the present application.

[0150] Figure 8 A schematic diagram of the multimodal large model prediction or inference process provided in an embodiment of the present application.

[0151] like Figure 7 and Figure 8 As shown, the multimodal large model construction method for real-time speech recognition provided by the embodiment of the present application may include the following steps S600-S900. The multimodal large model is trained based on the first multimodal large model construction method for real-time speech recognition.

[0152] S600: Input the real-time speech data to be recognized into the multimodal large model, and use the multimodal large model to segment the real-time speech data according to a preset length to obtain a speech sequence to be recognized including audio primitives to be recognized.

[0153] The real-time voice data is dynamic, and the length of the real-time voice data gradually increases over time. For example, the real-time voice data is the voice data generated during a call, or the voice data generated during an online or offline meeting. The real-time voice data and the voice sequence to be recognized are arranged in frames, and each frame of real-time voice data corresponds to at least one frame of audio primitives to be recognized. After segmentation, the audio primitives to be recognized can be audio segments with a duration of 10ms, 20ms, or 30ms, which is not specifically limited in the embodiments of the present application.

[0154] It can be understood that the step of segmenting the real-time voice data according to the preset length using the multimodal large model is the step of tokenizing the real-time voice data.

[0155] S700: Using the multimodal large model to predict the current frame, to obtain a predicted data frame, where the current frame is one of the frames in the speech sequence to be recognized.

[0156] It is understood that in the embodiment of the present application, the multimodal large model can start prediction from the first frame of the speech sequence to be recognized. In this step, the multimodal large model can make predictions based on the current frame and the data of each frame before the current frame. The predicted data frame can be a text frame including text information or an audio frame including audio information.

[0157] S800: If the predicted data frame is a text frame, use the multimodal large model to output the text frame to achieve real-time speech recognition; and insert the text frame after the current frame to form a new speech sequence to be recognized, and continue to use the multimodal large model to predict the next frame of the current frame.

[0158] If the predicted data frame is a text frame, the multimodal large model can immediately convert it into readable text information and output it. This process marks the realization of real-time speech recognition, and users can instantly see the results of the speech being converted into text.

[0159] like Figure 8 As shown, in an embodiment of the present application, if the predicted data frame output by the previous frame is a text frame (text primitive), then the current input comes from the output of the previous frame. This step not only ensures the continuity of the recognition process, but also provides the model with the latest contextual information, enabling it to more accurately predict the next speech content and achieve autoregression. Subsequently, the model will continue its prediction work and turn to the next frame of the current frame for analysis. It is worth noting that at this time, the "next frame of the current frame" is the text frame (predicted data frame) that has just been inserted.

[0160] S900: If the predicted data frame is an audio frame, continue to use the multimodal large model to predict the next frame of the current frame.

[0161] It is understandable that when the predicted data frame is an audio frame, the model can keep processing the real-time speech and continue to predict the next frame.

[0162] It is worth noting that if Figure 8 As shown, the input data and the inference output are aligned in the time dimension. For example, for time point A, the content on its left can be used to represent the form of the inference data and input at the current time point A. Furthermore, for each frame of data input, the multimodal large model will form an output (inference output). Specifically, after training with multimodal training data, the multimodal large model can output the text frame "I" when the input data is the first 7 frames, and the text frame "I" is located after the 6th frame of the inference output. Then, along the time direction, at the next time point (or time step), the text frame "I" appears after the 7th frame of the input data, forming a new speech sequence to be recognized.

[0163] It's understandable that a large multimodal model can output two types of inference output: one for internal use and the other for external use. The format of the internal output is the same as the training output samples described above. For example, if the seventh frame of the speech sequence is output, "I" will be output. In the internal output sequence, "I" appears after the sixth frame. The external output is the final result presented by the model to the user or application system, namely the text result of real-time speech recognition.

[0164] It should also be noted that when the embodiment of the present application adopts the aforementioned first training method to train the model, the index ID of the predicted data frame can be determined by retrieving the first vocabulary, and based on the specific data of the index ID, it is determined whether the predicted data frame is a text frame or an audio frame. This is because the index IDs of the text primitives (text frames) and the audio primitives (audio frames) do not overlap. When the embodiment of the present application adopts the aforementioned second training method to train the model, other methods can be used to determine whether the predicted data frame is a text frame or an audio frame. For example, the method of performing feature extraction on the predicted data frame determines whether it is a text frame or an audio frame.

[0165] From the above content, it can be seen that the embodiment of the present application provides a method for constructing a multimodal large model for real-time speech recognition. Based on this method, the embodiment of the present application can add the inference output to the model's input sequence when it is a text frame (text token / text primitive), so that the text information and voice information of the input sequence are intertwined. That is, the embodiment of the present application provides a new arrangement method for text frames and audio frames in the model input data, which can achieve autoregression and improve the prediction effect. At the same time, the output mode of the model also changes. The model can generate output when a short speech is input. In this way, the inference output can be output word by word (or token by token) instead of the entire sentence, thereby improving the effect of real-time speech recognition.

[0166] Figure 9 A schematic diagram of the structure of a multimodal training data construction device provided in an embodiment of the present application.

[0167] like Figure 9 As shown, the embodiment of the present application provides a multimodal training data construction device, which is applied to a multimodal large model for real-time speech recognition; the device includes:

[0168] A first acquisition module 1001 is configured to acquire a plurality of speech sequences and a text sequence corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by extracting features from speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, each text primitive being a character, word, or sentence in the text data;

[0169] A data construction module 1002 is configured to sequentially insert the corresponding text primitives between the feature primitives or audio primitives included in each speech sequence to obtain multimodal training data;

[0170] The collection module 1003 is used to collect all multimodal training data to form a training data set.

[0171] In some implementations, speech data and speech sequences are arranged in frames, and each frame of speech data corresponds to at least one frame of feature primitives or audio primitives; the multimodal training data includes training input samples and training output samples; for each speech sequence, the data construction module 1002 is specifically used to: for each text primitive corresponding to the speech sequence, determine the first frame sequence number of its pronunciation end position in the speech sequence; determine the delay parameter, the delay parameter is D frames, D≥0; for each text primitive corresponding to the speech sequence, determine its insertion position in the speech sequence based on its corresponding first frame sequence number and delay parameter; wherein the frame sequence number of the insertion position is the second frame sequence number, and the second frame sequence number is equal to the sum of the first frame sequence number and the delay parameter; add a start mark at the starting position of the speech sequence, and insert each text primitive into its corresponding insertion position to form a training input sample; determine the training output sample based on the training input sample.

[0172] In some implementations, the data construction module 1002 is further used to: determine whether the second frame sequence number corresponding to the target text primitive is greater than the total number of frames of the speech sequence, the target text primitive being the last text primitive in the text sequence; if it is not greater than the total number of frames of the speech sequence, insert each text primitive after its corresponding insertion position; if it is greater than the total number of frames of the speech sequence, add N blank frames at the end of the speech sequence, and insert each text primitive after its corresponding insertion position; wherein N is equal to the difference between the second frame sequence number corresponding to the target text primitive and the total number of frames.

[0173] In some implementations, the data construction module 1002 is further configured to: offset the training input samples by one frame to form training output samples, so that the second frame of the training output samples is aligned with the first frame of the training input samples.

[0174] Figure 10 A schematic structural diagram of the first multimodal large model building device for real-time speech recognition provided in an embodiment of the present application.

[0175] like Figure 10 As shown, the embodiment of the present application provides a multimodal large model construction device for real-time speech recognition, the device comprising:

[0176] The second acquisition module 2001 is used to acquire a pre-trained language model; wherein the pre-trained language model is constructed based on a decoder-only transformer architecture;

[0177] The training module 2002 is used to train the pre-trained language model using multimodal training data to obtain a large multimodal model for real-time speech recognition; wherein the multimodal training data is constructed based on the aforementioned multimodal training data construction method.

[0178] In one implementation, the training module 2002 is specifically used to: obtain a training speech sequence, which is a speech sequence composed of audio primitives obtained by segmenting speech data in the aforementioned multimodal training data construction method; obtain a first vocabulary and a first embedding matrix; wherein the first vocabulary is a collection of text primitives, and the step of segmenting the text data corresponding to the speech data to obtain text primitives is performed based on the first vocabulary; each row of the first embedding matrix corresponds to each text primitive in the first vocabulary, and each column corresponds to the dimension of the embedding vector, and the first embedding matrix is ​​used to map the text primitives to the embedding vector; each audio primitive in the training speech sequence is added to the first vocabulary of the pre-trained language model to obtain a second vocabulary, which is a collection of audio primitives and text primitives; and, expand the rows of the first embedding matrix to obtain a second embedding matrix, so that each row of the second embedding matrix corresponds to each audio primitive or text primitive in the second vocabulary; and train the pre-trained language model using the second vocabulary, the second embedding matrix and the multimodal training data to obtain a multimodal large model.

[0179] In one implementation, the training module 2002 is specifically used to: obtain a training speech sequence; construct a third vocabulary, which is a set of audio primitives in the training speech sequence; and construct a third embedding matrix, where each row of the third embedding matrix corresponds to each audio primitive in the third vocabulary, and each column corresponds to the dimension of the embedding vector, and the second embedding matrix is ​​used to map the audio primitives to the embedding vector; obtain a first vocabulary and a first embedding matrix; and train the pre-trained language model using the first vocabulary, the third vocabulary, the first embedding matrix, the third embedding matrix, and the multimodal training data to obtain a multimodal large model.

[0180] Figure 11 A schematic structural diagram of a second multimodal large model building device for real-time speech recognition provided in an embodiment of the present application.

[0181] like Figure 11 As shown, the embodiment of the present application further provides a multimodal large model construction device for real-time speech recognition, wherein the multimodal large model is trained based on the first multimodal large model construction method for real-time speech recognition; the device includes:

[0182] The segmentation module 3001 is configured to input real-time speech data to be recognized into the multimodal large model, and use the multimodal large model to segment the real-time speech data according to a preset length to obtain a speech sequence to be recognized that includes audio primitives to be recognized; wherein the length of the real-time speech data gradually increases over time, the real-time speech data and the speech sequence to be recognized are arranged in frames, and each frame of real-time speech data corresponds to at least one frame of audio primitives to be recognized;

[0183] The prediction output module 3002 is used to use the multimodal large model to predict the current frame to obtain a predicted data frame, where the current frame is one of the frames in the speech sequence to be recognized;

[0184] If the predicted data frame is a text frame, the prediction output module 3002 is further configured to output the text frame using the multimodal large model to achieve real-time speech recognition; and insert the text frame after the current frame to form a new speech sequence to be recognized, and continue to use the multimodal large model to predict the next frame of the current frame.

[0185] If the predicted data frame is an audio frame, the prediction output module 3002 is further configured to continue using the multimodal large model to predict the next frame of the current frame.

[0186] In a specific implementation, the present invention further provides a computer storage medium, wherein the computer storage medium may store a program that, when executed, may include some or all of the steps of each embodiment of the multimodal training data construction method and the multimodal large model construction method provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0187] It is easy to understand that those skilled in the art can combine, split, reorganize, etc. the embodiments of the present application based on the several embodiments provided in the present application to obtain other embodiments, and these embodiments do not exceed the scope of protection of the present application.

[0188] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not intended to limit the scope of protection of the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the scope of protection of the embodiments of the present application.< / bos>

Claims

1. A method for constructing multimodal training data, characterized in that: Large multimodal models for real-time speech recognition; The method comprises: Acquire multiple speech sequences and text sequences corresponding to each of the speech sequences; wherein the speech sequences are composed of feature primitives obtained by extracting features from speech data, or the speech sequences are composed of audio primitives obtained by segmenting the speech data, and the text sequences are composed of text primitives obtained by segmenting text data corresponding to the speech data, each text primitive being a character, word, or sentence in the text data; For each of the speech sequences, the corresponding text primitives are sequentially inserted between the feature primitives or the audio primitives included therein to obtain multimodal training data; the multimodal training data includes training input samples and training output samples; For each of the speech sequences, the corresponding text primitives are sequentially inserted between the feature primitives or the audio primitives included therein to obtain multimodal training data, including: Adding a start marker at the start position of the speech sequence and inserting each text primitive into its corresponding insertion position to form the training input sample; the insertion position is D frames after the corresponding pronunciation end position of the text primitive in the speech sequence, D≥0; offsetting the training input sample by one frame to form the training output sample, so that the second frame of the training output sample is aligned with the first frame of the training input sample; All the multimodal training data are collected to form a training data set; when predicting, the multimodal large model trained with the training data set makes predictions based on the current frame and the data of frames before the current frame.

2. The multimodal training data construction method according to claim 1, characterized in that: The speech data and the speech sequence are arranged in frames, and each frame of the speech data corresponds to at least one frame of the feature primitives or the audio primitives; after adding a start mark at the starting position of the speech sequence and inserting each of the text primitives into its corresponding insertion position, and before forming the training input sample, the method further includes: For each text primitive corresponding to the speech sequence, determining the first frame sequence number of the pronunciation end position of the text primitive in the speech sequence; Determine a delay parameter, where the delay parameter is D frames; For each of the text elements corresponding to the speech sequence, its insertion position in the speech sequence is determined based on its corresponding first frame sequence number and the delay parameter; wherein the frame sequence number of the insertion position is the second frame sequence number, and the second frame sequence number is equal to the sum of the first frame sequence number and the delay parameter.

3. The method for constructing multimodal training data according to claim 2, wherein: The steps after inserting each text primitive into its corresponding insertion position include: Determining whether the second frame sequence number corresponding to a target text primitive is greater than the total number of frames in the speech sequence, the target text primitive being the last text primitive in the text sequence; If the number of the text primitives is not greater than the total number of frames of the speech sequence, inserting each text primitive after its corresponding insertion position; If it is greater than the total number of frames of the speech sequence, N blank frames are added to the end of the speech sequence, and each text element is inserted after its corresponding insertion position; wherein N is equal to the difference between the second frame sequence number corresponding to the target text element and the total number of frames.

4. A method for constructing a multimodal large model for real-time speech recognition, characterized in that: The method comprises: Obtaining a pre-trained language model; wherein the pre-trained language model is constructed based on a decoder-only transformer architecture; The pre-trained language model is trained using multimodal training data to obtain the multimodal large model for real-time speech recognition; wherein the multimodal training data is constructed based on the multimodal training data construction method according to any one of claims 1 to 3.

5. The method for constructing a multimodal large model for real-time speech recognition according to claim 4, characterized in that: The step of training the pre-trained language model using multimodal training data includes: Obtaining a training speech sequence, wherein the training speech sequence is a speech sequence composed of audio primitives obtained by segmenting speech data in the multimodal training data construction method according to any one of claims 1 to 3; Obtaining a first vocabulary and a first embedding matrix; wherein the first vocabulary is a set of text primitives, and the step of segmenting the text data corresponding to the speech data to obtain text primitives is performed based on the first vocabulary; each row of the first embedding matrix corresponds to each text primitive in the first vocabulary, and each column corresponds to a dimension of an embedding vector, and the first embedding matrix is ​​used to map the text primitives to embedding vectors; Adding each audio primitive in the training speech sequence to the first vocabulary of the pre-trained language model to obtain a second vocabulary, wherein the second vocabulary is a set of audio primitives and text primitives; and, expanding the rows of the first embedding matrix to obtain a second embedding matrix such that each row of the second embedding matrix corresponds to each audio primitive or text primitive in the second vocabulary; The pre-trained language model is trained using the second vocabulary, the second embedding matrix, and the multimodal training data to obtain the multimodal large model.

6. The method for constructing a multimodal large model for real-time speech recognition according to claim 5, characterized in that: The step of training the pre-trained language model using multimodal training data further includes: Acquire the training speech sequence; constructing a third vocabulary, the third vocabulary being a set of audio primitives in the training speech sequence; and constructing a third embedding matrix, wherein each row of the third embedding matrix corresponds to each audio primitive in the third vocabulary, each column corresponds to a dimension of an embedding vector, and the second embedding matrix is ​​used to map the audio primitives to the embedding vectors; Obtaining the first vocabulary and the first embedding matrix; The pre-trained language model is trained using the first vocabulary, the third vocabulary, the first embedding matrix, the third embedding matrix, and the multimodal training data to obtain the multimodal large model.

7. A method for constructing a multimodal large model for real-time speech recognition, characterized in that: The multimodal large model is trained based on the multimodal large model construction method for real-time speech recognition according to any one of claims 4 to 6; The method comprises: Inputting real-time speech data to be recognized into the multimodal large model, and using the multimodal large model to segment the real-time speech data according to a preset length to obtain a speech sequence to be recognized that includes audio primitives to be recognized; wherein the length of the real-time speech data gradually increases over time, the real-time speech data and the speech sequence to be recognized are arranged in frames, and each frame of the real-time speech data corresponds to at least one frame of the audio primitives to be recognized; Using the multimodal large model to predict a current frame to obtain a predicted data frame, the current frame being one of the frames in the speech sequence to be recognized; If the predicted data frame is a text frame, output the text frame using the multimodal large model to achieve real-time speech recognition; and insert the text frame after the current frame to form a new speech sequence to be recognized, and continue to use the multimodal large model to predict the next frame of the current frame; If the predicted data frame is an audio frame, continue to use the multimodal large model to predict the next frame of the current frame.

8. A multimodal training data construction device, characterized in that: Large multimodal models for real-time speech recognition; The device comprises: A first acquisition module is configured to acquire a plurality of speech sequences and a text sequence corresponding to each of the speech sequences; wherein the speech sequences are composed of feature primitives obtained by extracting features from speech data, or the speech sequences are composed of audio primitives obtained by segmenting the speech data, and the text sequences are composed of text primitives obtained by segmenting text data corresponding to the speech data, each text primitive being a character, word, or sentence in the text data; A data construction module is configured to sequentially insert the corresponding text primitives between the feature primitives or audio primitives included in each speech sequence to obtain multimodal training data; the multimodal training data includes training input samples and training output samples; The data construction module is specifically used for: Adding a start marker at the start position of the speech sequence and inserting each text primitive into its corresponding insertion position to form the training input sample; the insertion position is D frames after the corresponding pronunciation end position of the text primitive in the speech sequence, D≥0; offsetting the training input sample by one frame to form the training output sample, so that the second frame of the training output sample is aligned with the first frame of the training input sample; The collection module is used to collect all the multimodal training data to form a training data set; when predicting, the multimodal large model trained with the training data set makes predictions based on the current frame and the frame data before the current frame.

9. A multimodal large model construction device for real-time speech recognition, characterized in that: The device comprises: A second acquisition module is used to acquire a pre-trained language model; wherein the pre-trained language model is constructed based on a decoder-only transformer architecture; A training module, configured to train the pre-trained language model using multimodal training data to obtain the multimodal large model for real-time speech recognition; wherein the multimodal training data is constructed based on the multimodal training data construction method according to any one of claims 1 to 3; and / or, The multimodal large model is trained based on the multimodal large model construction method for real-time speech recognition according to any one of claims 4 to 6; The device comprises: a segmentation module, configured to input real-time speech data to be recognized into the multimodal large model, and segment the real-time speech data using the multimodal large model according to a preset length to obtain a speech sequence to be recognized comprising audio primitives to be recognized; wherein the length of the real-time speech data gradually increases over time, the real-time speech data and the speech sequence to be recognized are arranged in frames, and each frame of the real-time speech data corresponds to at least one frame of the audio primitives to be recognized; A prediction output module, configured to use the multimodal large model to predict a current frame to obtain a prediction data frame, wherein the current frame is one of the frames in the speech sequence to be recognized; If the predicted data frame is a text frame, the prediction output module is further configured to output the text frame using the multimodal large model to achieve real-time speech recognition; and insert the text frame after the current frame to form a new speech sequence to be recognized, and continue to use the multimodal large model to predict the next frame of the current frame. If the predicted data frame is an audio frame, the prediction output module is further configured to continue using the multimodal large model to predict the next frame of the current frame.

Images