Multi-modal training data construction method and device and multi-modal large model construction method

Abstract

The invention provides a multi-modal training data construction method and device and a multi-modal large model construction method. The multi-modal training data construction method comprises the steps of obtaining a plurality of voice sequences and a text sequence corresponding to each voice sequence; wherein the voice sequence is composed of feature primitives obtained by performing feature extraction on the voice data, or the voice sequence is composed of audio primitives obtained by segmenting the voice data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the voice data; for each voice sequence, inserting each text primitive corresponding to the voice sequence between the feature primitives or the audio primitives included in the voice sequence in sequence to obtain multi-modal training data; and gathering all the multi-modal training data to form a training data set. Therefore, the mapping relation between the short-time voice and the text data can be established, so that the understanding and processing capability of the model on cross-modal information is enhanced, and the real-time voice recognition effect is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for constructing multimodal training data and a method for constructing a multimodal large model. Background Art

[0002] Real-time speech recognition technology is a special case of speech recognition technology. It can achieve instant feedback, that is, output the corresponding text content at the same time as the speech input (allowing a small delay). This technology is suitable for scenes such as meeting minutes where the speech content needs to be displayed instantly. In other words, as the speaker speaks, his speech content can be displayed instantly.

[0003] There are currently two solutions for achieving real-time speech recognition. One of them is an end-to-end deep learning model based on Connectionist Temporal Classification (CTC). The CTC model can output recognized text during speech input. In the training phase, the CTC model relies on "data pairs" consisting of speech and text for training. In the inference phase, the CTC model can generate the entire text sequence at one time by learning the overall mapping relationship between audio and text sequences. However, this mapping method is relatively simple, does not make in-depth use of the complex structure and grammatical rules in the language, and does not take into account the continuity and randomness of the language. Therefore, the real-time speech recognition effect of the CTC model is relatively general.

[0004] Another solution is to implement it based on a large language model (LLM). In the training phase, the large language model LLM relies on ultra-large-scale text data for training and has large-scale model parameters, so that language knowledge can be fully applied. However, in the inference phase, the large language model LLM generates text units one by one for a piece of input voice data, so it can only be used to implement voice recognition functions, or in other words, it can only be used to implement real-time voice recognition functions with large delays. Summary of the invention

[0005] The embodiments of the present application provide a multimodal training data construction method, device and multimodal large model construction method to solve the problem of poor real-time speech recognition effect of traditional models.

[0006] In a first aspect, an embodiment of the present application provides a method for constructing multimodal training data, which is applied to a multimodal large model for real-time speech recognition; the method comprises: obtaining multiple speech sequences and a text sequence corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by extracting features from speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data; for each speech sequence, the corresponding text primitives are inserted in order between the feature primitives or audio primitives included therein to obtain multimodal training data; and all multimodal training data are collected to form a training data set.

[0007] In an achievable manner, speech data and speech sequences are arranged in frames, and each frame of speech data corresponds to at least one frame of feature primitives or audio primitives; multimodal training data includes training input samples and training output samples; for each speech sequence, the corresponding text primitives are inserted in sequence between the feature primitives or audio primitives included therein, and the steps of obtaining multimodal training data include: for each text primitive corresponding to the speech sequence, determining the first frame sequence number of the pronunciation end position in the speech sequence; determining a delay parameter, the delay parameter is D frames, D≥0; for each text primitive corresponding to the speech sequence, determining its insertion position in the speech sequence based on its corresponding first frame sequence number and delay parameter; wherein the frame sequence number of the insertion position is the second frame sequence number, and the second frame sequence number is equal to the sum of the first frame sequence number and the delay parameter; adding a start mark at the starting position of the speech sequence, and inserting each text primitive into its corresponding insertion position to form a training input sample; and determining a training output sample based on the training input sample.

[0008] In one achievable manner, the step of inserting each text primitive after its corresponding insertion position includes: determining whether the second frame sequence number corresponding to the target text primitive is greater than the total number of frames of the speech sequence, the target text primitive being the last text primitive in the text sequence; if it is not greater than the total number of frames of the speech sequence, inserting each text primitive after its corresponding insertion position; if it is greater than the total number of frames of the speech sequence, adding N blank frames at the end of the speech sequence, and inserting each text primitive after its corresponding insertion position; wherein N is equal to the difference between the second frame sequence number corresponding to the target text primitive and the total number of frames.

[0009] In one achievable manner, the step of determining the training output sample based on the training input sample includes: offsetting the training input sample by one frame to form the training output sample, so that the second frame of the training output sample is aligned with the first frame of the training input sample.

[0010] In a second aspect, an embodiment of the present application provides a method for constructing a multimodal large model for real-time speech recognition, the method comprising: obtaining a pre-trained language model; wherein the pre-trained language model is constructed based on a decoder-only transformer architecture; training the pre-trained language model using multimodal training data to obtain a multimodal large model for real-time speech recognition; wherein the multimodal training data is constructed by the multimodal training data construction method provided by the aforementioned first aspect and its various implementation methods.

[0011] In an achievable manner, the step of training a pre-trained language model using multimodal training data includes: obtaining a training speech sequence, the training speech sequence is a speech sequence composed of audio primitives obtained by segmenting speech data in the multimodal training data construction method provided by the first aspect and its various implementations; obtaining a first vocabulary and a first embedding matrix; wherein the first vocabulary is a collection of text primitives, and the step of segmenting the text data corresponding to the speech data to obtain text primitives is performed based on the first vocabulary; each row of the first embedding matrix corresponds to each text primitive in the first vocabulary, and each column corresponds to the dimension of the embedding vector, and the first embedding matrix is ​​used to map the text primitive to the embedding vector; each audio primitive in the training speech sequence is added to the first vocabulary of the pre-trained language model to obtain a second vocabulary, and the second vocabulary is a collection of audio primitives and text primitives; and, the rows of the first embedding matrix are expanded to obtain a second embedding matrix, so that each row of the second embedding matrix corresponds to each audio primitive or text primitive in the second vocabulary; the pre-trained language model is trained using the second vocabulary, the second embedding matrix and the multimodal training data to obtain a multimodal large model.

[0012] In one achievable manner, the step of training the pre-trained language model using multimodal training data also includes: obtaining a training speech sequence; constructing a third vocabulary, which is a set of audio primitives in the training speech sequence; and constructing a third embedding matrix, wherein each row of the third embedding matrix corresponds to each audio primitive in the third vocabulary, each column corresponds to the dimension of the embedding vector, and the second embedding matrix is ​​used to map the audio primitives to the embedding vector; obtaining a first vocabulary and a first embedding matrix; and training the pre-trained language model using the first vocabulary, the third vocabulary, the first embedding matrix, the third embedding matrix, and the multimodal training data to obtain a large multimodal model.

[0013] In a third aspect, an embodiment of the present application further provides a method for constructing a multimodal large model for real-time speech recognition, wherein the multimodal large model is trained based on the multimodal large model construction method for real-time speech recognition provided by the aforementioned second aspect and its various implementation methods; the method comprises: inputting the real-time speech data to be recognized into the multimodal large model, using the multimodal large model to segment the real-time speech data according to a preset length, and obtaining a speech sequence to be recognized including audio elements to be recognized; wherein the length of the real-time speech data gradually increases with time, and the real-time speech data and the speech sequence to be recognized are It is arranged by frame, and each frame of real-time speech data corresponds to at least one frame of audio primitives to be recognized; a multimodal large model is used to predict the current frame to obtain a predicted data frame, and the current frame is one of the frames in the speech sequence to be recognized; if the predicted data frame is a text frame, the multimodal large model is used to output the text frame to achieve real-time speech recognition; and the text frame is inserted after the current frame to form a new speech sequence to be recognized, and the multimodal large model is continued to be used to predict the next frame of the current frame; if the predicted data frame is an audio frame, the multimodal large model is continued to be used to predict the next frame of the current frame.

[0014] In a fourth aspect, an embodiment of the present application also provides a multimodal training data construction device, which is applied to a multimodal large model for real-time speech recognition; the device includes: a first acquisition module, used to obtain multiple speech sequences and a text sequence corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by feature extraction of speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data; a data construction module, used to, for each speech sequence, sequentially insert the corresponding text primitives between the feature primitives or audio primitives included therein to obtain multimodal training data; a collection module, used to collect all multimodal training data to form a training data set.

[0015] In a fifth aspect, an embodiment of the present application also provides a multimodal large model construction device for real-time speech recognition, the device comprising: a second acquisition module, used to obtain a pre-trained language model; wherein the pre-trained language model is constructed based on a decoder-only transformer architecture; a training module, used to train the pre-trained language model using multimodal training data to obtain a multimodal large model for real-time speech recognition; wherein the multimodal training data is constructed based on the multimodal training data construction method provided by the aforementioned first aspect and its various implementation methods.

[0016] In a sixth aspect, an embodiment of the present application further provides a device for constructing a multimodal large model for real-time speech recognition, wherein the multimodal large model is trained based on the multimodal large model construction method for real-time speech recognition provided in the aforementioned second aspect and its various implementation methods; the device comprises: a segmentation module, which is used to input the real-time speech data to be recognized into the multimodal large model, and use the multimodal large model to segment the real-time speech data according to a preset length to obtain a speech sequence to be recognized including audio primitives to be recognized; wherein the length of the real-time speech data gradually increases with time, and the real-time speech data and the speech sequence to be recognized are arranged in frames, and each frame The real-time speech data corresponds to at least one frame of audio primitives to be recognized; a prediction output module is used to use a multimodal large model to predict the current frame to obtain a prediction data frame, and the current frame is one of the frames in the speech sequence to be recognized; if the predicted data frame is a text frame, the prediction output module is also used to output the text frame using the multimodal large model to achieve real-time speech recognition; and, insert the text frame after the current frame to form a new speech sequence to be recognized, and continue to use the multimodal large model to predict the next frame of the current frame; if the predicted data frame is an audio frame, the prediction output module is also used to continue to use the multimodal large model to predict the next frame of the current frame.

[0017] From the above content, it can be seen that the embodiment of the present application provides a multimodal training data construction method, device and multimodal large model construction method, which includes: obtaining multiple speech sequences and text sequences corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by feature extraction of speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data; for each speech sequence, the corresponding text primitives are inserted in order between the feature primitives or audio primitives included therein to obtain multimodal training data; all multimodal training data are collected to form a training data set. In this way, a mapping relationship between short-term speech and text data can be established, thereby enhancing the model's understanding and processing capabilities of cross-modal information, thereby improving the effect of real-time speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A first flow chart of a method for constructing multimodal training data provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of segmenting voice data and text data provided in an embodiment of the present application;

[0020] Figure 3A second flow chart of the multimodal training data construction method provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of a process for inserting text primitives into a speech sequence provided in an embodiment of the present application;

[0022] Figure 5 A decomposition diagram of inserting text primitives into a speech sequence provided by an embodiment of the present application;

[0023] Figure 6 A schematic diagram of a process flow of a first multimodal large model construction method for real-time speech recognition provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a second method for constructing a multimodal large model for real-time speech recognition provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of a multimodal large model prediction or reasoning process provided in an embodiment of the present application;

[0026] Fig. 9 A schematic diagram of the structure of a multimodal training data construction device provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of the structure of a first multimodal large model building device for real-time speech recognition provided in an embodiment of the present application;

[0028] Fig.11 A schematic diagram of the structure of a second multimodal large model building device for real-time speech recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0030] Before introducing the technical solutions of the embodiments of the present application, the terms involved in the embodiments of the present application are first introduced by way of example.

[0031] 1. Multimodal Data: A data set that contains two or more different types of information.

[0032] 2. Tokenization: Also known as tokenization, it refers to dividing text into smaller units, namely tokens. In natural language processing, tokenization is a common preprocessing step. Tokenization can divide the original text into an ordered sequence of tokens according to preset rules or algorithms. The token sequence can be used as the input data basis for subsequent deep learning model training and reasoning. Common text tokenization methods include: word segmentation based on spaces, segmentation based on punctuation marks, segmentation by characters or subwords, etc. The purpose of tokenization is to convert text into a form that can be understood and processed by machines.

[0033] In the application of large models, text data (also known as corpus) can usually be divided into multiple small fragments, namely text tokens. A text token can represent the smallest semantic character unit in the text, such as a single Chinese character or word in Chinese. After removing all possible text tokens, a token list, namely the vocabulary, can be obtained. After that, a unique identifier (such as an integer ID) can be assigned to each text token in the vocabulary, which can be used as an embedding index for the embedding matrix of the large model. When processing text, the large model can first divide the text into text tokens and determine their corresponding indexes, then the index can be used to find the corresponding embedding vector in the embedding matrix. For example, if the character "cat" appears in the text, the index of "cat" (such as 0) can be found through the vocabulary, and then the embedding vector of "cat" can be found in the 0th row of the embedding matrix.

[0034] As technology develops, the application and implementation of tokenization are also evolving, providing strong support for various AI tasks. Tokens no longer refer exclusively to text fragments, but have gradually expanded to other types of data representations, such as image blocks or audio fragments. This expansion enables large models to process not only traditional text data, but also multimodal inputs, further improving the adaptability and application scope of large models.

[0035] Speech recognition technology is a technology that converts speech signals into text or commands. Speech recognition technology can identify the content expressed by the speaker by analyzing and processing human voice input. Real-time speech recognition technology is a special case of speech recognition technology, which can achieve instant feedback, that is, output the corresponding text content at the same time as the voice input (allowing a small delay). This technology is suitable for scenarios such as meeting minutes, where the content of speech needs to be displayed immediately. In other words, as the speaker speaks, the content of his speech can be displayed immediately. Corresponding to this is offline speech recognition technology, which gives a complete text at one time for the entire input speech, such as a meeting recording, a call recording, etc.

[0036] There are currently two solutions for achieving real-time speech recognition. One of them is an end-to-end deep learning model based on Connectionist Temporal Classification (CTC). The CTC model can output recognized text during speech input. In the training phase, the CTC model relies on "data pairs" consisting of speech and text for training. In the inference phase, the CTC model can generate the entire text sequence at one time by learning the overall mapping relationship between audio and text sequences. However, this mapping method is relatively simple, does not make in-depth use of the complex structure and grammatical rules in the language, and does not take into account the continuity and randomness of the language. Therefore, the real-time speech recognition effect of the CTC model is relatively general.

[0037] Another solution is to implement it based on a large language model (LLM). In the training phase, the large language model LLM relies on ultra-large-scale text data for training and has large-scale model parameters, so that language knowledge can be fully applied. However, in the inference phase, the large language model LLM generates text units one by one for a piece of input voice data and outputs them uniformly. Therefore, it can only be used to implement voice recognition functions, which is not equivalent to realizing real-time voice recognition functions, or in other words, it can only be used to realize real-time voice recognition functions with large delays.

[0038] The embodiments of the present application can provide a multimodal training data construction method, a multimodal large model construction method and corresponding devices. The method provided by the embodiments of the present application can not only utilize the powerful language modeling capabilities of the large model, but also realize real-time speech recognition function, which can solve the aforementioned problems, improve recognition accuracy and ensure recognition effect.

[0039] Before introducing the specific method provided in the embodiments of the present application, some exemplary explanations of text tokenization and speech tokenization are required.

[0040] As mentioned above, text tokenization is the process of breaking down a text string into smaller, processable units (called text tokens), usually words, subwords, or characters. For example, for the sentence "Natural language processing is an important field of artificial intelligence", the result of subword segmentation is: ["nature", "language", "processing", "is", "artificial", "intelligent", "of", "an", "important", "field", "."], where "nature" is a text token, and the result of character segmentation is: ["self", "natural", "language", "language", "processing", "is", "human", "artificial", "intelligent", "ability", "of", "one", "important", "field", "."], where "self" and "natural" are two text tokens respectively.

[0041] Speech tokenization is to decompose the audio signal into smaller, processable units, so that the continuous audio signal can be converted into a discrete, recognizable sequence of symbols. The smaller, processable units are usually phonemes, syllables or acoustic features. Acoustic features are, for example, Mel-Frequency Cepstral Coefficients (MFCC) or Filter Bank (FBank) features. After that, the phonemes, syllables or acoustic features can be mapped to feature tokens through a vocabulary, where the vocabulary includes a mapping relationship between feature tokens and embedded representations of phonemes, syllables or acoustic features. Alternatively, a more direct method is to divide the audio signal into short-term frames, each frame represents an audio signal within a period of time, for example, 10ms, 20ms, and each frame of audio signal can be called a speech token.

[0042] In order to achieve speech recognition, the token arrangement of the existing text-speech multimodal large model is [speech feature / speech token sequence][text feature / text token sequence]. It can be understood that here only the "speech feature" is equivalent to the aforementioned "feature token".

[0043] The method provided in the embodiment of the present application can modify the existing token arrangement to: [short-time speech feature / short-time speech token sequence][text feature / token][short-time speech feature / short-time speech token sequence][text feature / token], that is, interspersing text tokens in the speech feature sequence or the speech token sequence. Specifically, the text token can be inserted at the position where it is expected to appear (for example, shortly after the text token appears completely in the speech). The specific method of insertion will be described in detail below and will not be repeated here.

[0044] For example, if "*" represents a speech feature or a speech token and "+" represents a text token, then for a speech recognition task, the existing token arrangement is "****************+++++++", and the modified arrangement is "***+**+*+***+**+**+***+".

[0045] It should be emphasized that, in order to facilitate description and avoid ambiguity, in the following text tokens are uniformly referred to as text primitives, speech tokens obtained by framing are uniformly referred to as audio primitives, and feature tokens obtained based on acoustic feature mapping are uniformly referred to as feature primitives. It is understandable that the name change does not have any limiting effect on the actual processing steps of this solution.

[0046] It can be understood that the life cycle of a model includes three stages, namely, the data set construction stage, the training stage, and the reasoning stage. The embodiment of the present application will introduce a large multimodal model for real-time speech recognition in stages.

[0047] The dataset construction phase is introduced below.

[0048] Figure 1 The first flowchart of the multimodal training data construction method provided in the embodiment of the present application.

[0049] like Figure 1 As shown, the multimodal training data construction method provided in the embodiment of the present application includes the following steps S101-S104.

[0050] S101: Acquire voice data.

[0051] Among them, voice data is specifically an audio signal, which contains certain voice content, such as words, phrases or sentences, and can be used as a basic data source for subsequent processing. In practical applications, for example, the sound emitted by a speaker randomly or with reference to a corpus can be captured in real time by a recording device such as a microphone, and then converted into an audio signal in a digital format by sampling and quantization to obtain voice data. For another example, the voice data can be a public voice data set, and specifically Common Voice, LibriSpeech and other data sets can be used, which is not specifically limited in the embodiments of the present application. For another example, obtaining voice data in a real business scenario can be the sound recorded by a user using a recording device in a meeting or interview. The embodiments of the present application do not specifically limit this.

[0052] In the embodiment of the present application, the voice data is arranged in frames. For example, if the sampling rate is 44.1kHz, there are 44100 samples per second. When the length of the audio frame is 10ms, each audio frame contains 441 samples (44100*0.01). The voice data is in different formats such as WAV, MP3, etc., which is not specifically limited in the embodiment of the present application.

[0053] S102: Transcribing the voice data into text to obtain text data.

[0054] The text transcription step may be performed manually and proofread to ensure the accuracy of the text data. Alternatively, the text data may be transcribed based on an automatic speech recognition (ASR) system to generate text data, which may then be proofread manually to ensure the accuracy of the text data.

[0055] In some implementations, when voice data is captured in real time by a recording device, if the speaker usually pronounces based on existing corpus, that is, if this step is performed based on existing corpus, then the text data is known, and therefore there is no need to perform a text transcription step.

[0056] It is understandable that when a public voice data set is used, the text data corresponding to the voice data set can usually be directly obtained.

[0057] S103: segment the text data, and record the data sequence formed after segmentation as a text sequence. That is, the text sequence is composed of text primitives obtained by segmenting the text data.

[0058] The specific tokenization (segmentation) method may be space-based tokenization, punctuation-based tokenization, subword tokenization, character-level tokenization, or sentence-level tokenization, which is not specifically limited in the embodiments of the present application.

[0059] S104: Segment the speech data, and record the data sequence formed after segmentation as a speech sequence. That is, the speech sequence is composed of audio primitives obtained by segmenting the speech data.

[0060] Specifically, the step of segmenting the speech data may include segmenting the speech data into a number of short time frames, for example, each frame is 10 ms or 20 ms. Figure 2 The results of segmenting speech data and text data are exemplarily shown.

[0061] It is worth noting that the voice data and the voice sequence are arranged in frames, and each frame of voice data corresponds to at least one frame of audio primitives. If the audio frame length of the voice data is equal to the target frame length of the segmentation, then each frame of voice data corresponds to one frame of audio primitives, and if the audio frame length of the voice data is less than the target frame length of the segmentation, then each frame of voice data corresponds to multiple frames of audio primitives, and the embodiments of the present application do not specifically limit this.

[0062] Before segmenting the speech data, a preprocessing step may be performed first. The preprocessing may include denoising and normalization, etc., which is not specifically limited in the embodiment of the present application.

[0063] In some implementations, the speech sequence may also be determined based on the following step S105.

[0064] S105: Extract features from the speech data to obtain a speech sequence. That is, the speech sequence is composed of feature primitives obtained by extracting features from the speech data.

[0065] This step can include two stages: first, extract embeddings from the existing speech representation model, and then further discretize these embeddings into a limited set of feature primitives based on the vocabulary. The speech data and speech sequence are arranged in frames, and each frame of speech data corresponds to at least one frame of feature primitives.

[0066] Specifically, after preprocessing, the speech data can be subjected to feature extraction, such as Mel Spectrogram, Mel Frequency Cepstral Coefficient (MFCC), etc. After that, the audio features are input into the speech representation model, an embedding is proposed, and then the embedding vector is mapped to the closest feature primitive using a vocabulary. The vocabulary includes the mapping relationship between the feature primitive and the embedding representation of the acoustic feature.

[0067] Figure 3 A second flowchart of the multimodal training data construction method provided in an embodiment of the present application.

[0068] like Figure 3 As shown, the multimodal training data construction method provided in the embodiment of the present application is applied to a multimodal large model for real-time speech recognition, and may specifically include the following steps S200-S400.

[0069] S200: Acquire multiple speech sequences and a text sequence corresponding to each speech sequence.

[0070] The speech sequence is composed of feature primitives obtained by extracting features from speech data, which can be specifically determined based on step S105.

[0071] Alternatively, the speech sequence is composed of audio primitives obtained by segmenting the speech data, which can be specifically determined based on step S104.

[0072] The embodiment of the present application does not specifically limit the number of speech sequences. For example, dozens, hundreds or even more speech sequences can be obtained to enhance the learning and generalization capabilities of the model.

[0073] For example, in the embodiment of the present application, "*" is used to represent a feature primitive or an audio primitive. For a speech sequence whose text content is "I love you, my motherland", the corresponding speech sequence may be "***************", and the total number of frames is 15. It is understandable that the speech sequence here is only an exemplary introduction, and "*" is not an actual feature primitive or audio primitive.

[0074] The text sequence is composed of text primitives obtained by segmenting the text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data. The text sequence may be determined by step S103, and the content contained in a text primitive is different depending on the tokenization (segmentation) method.

[0075] Exemplarily, the embodiment of the present application performs the step of text tokenization in a character-by-character segmentation manner. Then, for the text data: "I love your motherland", the tokenization result is [I, love, you, motherland, country], "I" is a text primitive, "love" is a text primitive, "you" is a text primitive, "ancestor" is a text primitive, and correspondingly, "country" is also a text primitive.

[0076] S300: For each speech sequence, insert the corresponding text primitives into the feature primitives or audio primitives included therein in order to obtain multimodal training data.

[0077] The multimodal training data includes training input samples and training output samples, the training input samples may also be referred to as input or training input, and the training output samples may also be referred to as target or training output. The specific steps of inserting text primitives into speech sequences will be described in detail below and will not be repeated here.

[0078] S400: Collect all multimodal training data to form a training data set D.

[0079] The embodiment of the present application can also collect a certain amount of the above multimodal training data to form a training data set. The amount of multimodal training data in the training data set can be designed based on actual conditions, and the embodiment of the present application does not specifically limit this. The training data set D can be used to train a large multimodal model so that the large multimodal model can achieve real-time speech recognition with high recognition accuracy and good recognition effect.

[0080] In summary, based on the method provided in the embodiment of the present application, a multimodal training data that integrates text and voice information can be obtained, and in the multimodal training data, the text information and the voice information are intertwined, rather than independent of each other. Then, based on the data training model, a mapping relationship between short-term speech (such as 10ms frame data) and text data can be established to replace the mapping relationship between long speech (such as a sentence) and text data, thereby enhancing the model's understanding and processing capabilities for cross-modal information. Moreover, based on the data training model, the model can be made to predict the next output based on the previous output (whether text or voice), enhancing the model's understanding of language continuity and achieving autoregression.

[0081] From the above content, it can be seen that the embodiment of the present application provides a method for constructing multimodal training data, which includes: obtaining multiple speech sequences and text sequences corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by feature extraction of speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data; for each speech sequence, the corresponding text primitives are inserted in order between the feature primitives or audio primitives included therein to obtain multimodal training data; all multimodal training data are collected to form a training data set. In this way, a mapping relationship between short-term speech and text data can be established, thereby enhancing the model's ability to understand and process cross-modal information.

[0082] Figure 4 A schematic diagram of a flow chart of inserting text primitives into a speech sequence provided in an embodiment of the present application.

[0083] Figure 5 A decomposition diagram of inserting text primitives into a speech sequence provided in an embodiment of the present application.

[0084] like Figure 4 and Figure 5 As shown, step S300 may include the following steps S301-S305.

[0085] S301: For each text primitive corresponding to a speech sequence, determine the first frame sequence number of its pronunciation end position in the speech sequence.

[0086] Since the speech sequence is arranged in frames, each frame of audio primitives or feature primitives will have a unique frame sequence number. The starting value of the frame sequence number can be set to 1, that is, starting from the 1st frame and continuing to the Mth frame, where M is the total number of frames.

[0087] Determining the pronunciation end position may refer to, for each text primitive, such as for the text primitive "我" (I), determining the timestamp (t_end) in the speech data when its pronunciation ends, and then determining the audio primitive frame or feature primitive frame corresponding to the pronunciation end position based on the timestamp. The pronunciation end position may also be referred to as the right boundary or end boundary of the audio primitive or feature primitive corresponding to the text primitive. This step may be implemented based on an algorithm or with the help of manual proofreading, and the embodiments of the present application do not make specific limitations on this.

[0088] Exemplarily, after marking the pronunciation end position, the text data is as follows: [我3, 爱5, 你7, 祖11, 国14], where "我3" means that the pronunciation of "我" ends at the 3rd frame, that is, the first frame sequence number B corresponding to the pronunciation end position of "我" is 3.

[0089] S302: Determine the delay parameter, and the delay parameter is D frames.

[0090] Among them, D should be a non-negative integer, D≥0. The delay parameter can be set based on actual needs. For example, D is equal to 0, 3, 5, or 7, and the embodiments of the present application do not make specific limitations on this. Setting the delay parameter can help the model better understand the fluency of the language and the natural rhythm of the speech.

[0091] This is because, for Chinese, the same pronunciation can correspond to multiple Chinese characters. If the predicted Chinese characters are directly output without setting a delay, the continuity between contexts will be ignored. And during the speech recognition process, setting the delay parameter can help the model identify the appropriate context in the speech input and then correctly decode it into the corresponding Chinese characters. For example, when the speaker says "我爱你祖", the model can predict and output the character "你" to avoid prediction errors.

[0092] S303: For each text primitive corresponding to the speech sequence, based on its corresponding first frame sequence number and the delay parameter, determine its insertion position in the speech sequence; where the frame sequence number of the insertion position is the second frame sequence number.

[0093] It can be understood that the frame sequence number (the second frame sequence number) of the insertion position is the sum of the first frame sequence number and the delay parameter, that is, the second frame sequence number T = B + D, where B represents the first frame sequence number and D represents the value of the delay parameter.

[0094] Exemplarily, if the delay parameter is 3 frames, that is, D = 3, then the insertion position can be expressed as: [我3 + 3, 爱5 + 3, 你7 + 3, 祖11 + 3, 国14 + 3], that is, [我6, 爱8, 你10, 祖14, 国17], that is, the second frame sequence number T of the insertion position of "我" is 6.

[0095] Step S303 may specifically include the following steps S3031 - S3033.

[0096] S3031: Determine whether the second frame sequence number corresponding to the target text primitive is greater than the total number of frames in the speech sequence, and the target text primitive is the last text primitive in the text sequence.

[0097] It is understandable that when the delay parameter is 0 frames, that is, D=0, then the second frame sequence number corresponding to the insertion position of the target text primitive (the last text primitive in the text sequence) must not exceed the total number of frames of the speech sequence. When the delay parameter is not 0, that is, D>0, the second frame sequence number corresponding to the insertion position of the target text primitive (the last text primitive in the text sequence) may exceed the total number of frames of the speech sequence, and the speech sequence needs to be extended at this time.

[0098] S3032: If the number of frames is not greater than the total number of frames of the speech sequence, insert each text primitive after its corresponding insertion position.

[0099] It can be understood that when the second frame sequence number corresponding to the insertion position is 6, it means that the text element should be inserted after the 6th frame of the speech sequence. Correspondingly, when the second frame sequence number corresponding to the insertion position is 14, it means that the text element should be inserted after the 14th frame of the speech sequence.

[0100] For example, when the delay parameter is 0 frames, that is, D=0, the second frame sequence number corresponding to the target text primitive is 14, which does not exceed the total number of frames 14. Therefore, each text primitive can be directly inserted after its corresponding insertion position, and the insertion result is: "***I**love**your****homeland*".

[0101] S3033: If it is greater than the total number of frames of the speech sequence, N blank frames are added to the end of the speech sequence, and each text primitive is inserted after its corresponding insertion position.

[0102] Where N is equal to the difference between the second frame sequence number corresponding to the target text primitive and the total number of frames. A blank frame can also be called a silent frame, a silent frame or a speech frame without content. It refers to a frame that does not contain a valid speech signal or only contains background noise. A blank frame is represented by a region with a low amplitude or close to zero in the speech waveform to serve as a placeholder.

[0103] For example, when the delay parameter is 3 frames, that is, D>0, the second frame sequence number corresponding to the text primitive is 17, which exceeds the total number of frames 15, so N blank frames can be added at the end of the speech sequence to make the speech sequence large enough to accommodate all text primitives, where N=17-15=2. The insertion result is: "****** I** love** you **** motherland*--country", where "-" represents a blank frame.

[0104] It can be understood that the delay parameter is 3 frames, which means that it is expected that the corresponding output "I" will be output when the voice sequence is input to the 6th frame, and so on, which will not be elaborated here.

[0105] S304: Add a start mark at the start position of the speech sequence, and insert each text primitive into its corresponding insertion position to form a training input sample.

[0106] The start tag can be " ", used to indicate the beginning of the model training input sample. In some implementations, the start marker can also be " <bos>", which is not specifically limited in the embodiments of the present application.

[0107] Exemplarily, the training input sample can be "b******I **love** you ****祖*--国".

[0108] S305: Determine the training output sample based on the training input sample.

[0109] Step S305 can specifically include the following step S3051: Shift the training input sample by one frame to form the training output sample, so that the second frame of the training output sample is aligned with the first frame of the training input sample.

[0110] Among them, shifting the training input sample by one frame specifically means that relative to the training input sample, the training input sample is shifted one unit to the left.

[0111] Exemplarily, the training output sample can be "******I **love** you ****祖*--国".

[0112] It can be understood that in order to ensure that the number of frames of the training output sample and the training output sample is the same, the last frame of the training input sample can be removed, that is, the training input sample can be "b******I **love** you ****祖*--".

[0113] Then, the training input sample and the training output sample can be regarded as a multi-modal training data. A certain number of the above multi-modal training data sets can form a training data set D. This training data set D can be used to train the model so that the model can learn the mapping relationship between the feature primitives or audio primitives and the text primitives, thereby realizing the accurate understanding and processing of the speech signal.

[0114] It can be understood that for each speech sequence, steps S301 - S305 need to be executed.

[0115] Next, the model training stage will be introduced.

[0116] Figure 6 It is a schematic flowchart of the first multi-modal large model construction method for real-time speech recognition provided by the embodiments of the present application.

[0117] As Figure 6 shown, the multi-modal large model construction method provided by the embodiments of the present application can include the following steps S501 - S502.

[0118] S501: Obtain a pre-trained language model.

[0119] Among them, the pre-trained language model is built based on the decoder-only transformer architecture. In this way, the embodiment of the present application can be based on a pre-trained LLM model and retrained on the training data set D to obtain the final model. Exemplarily, the pre-trained language model is, for example, the LLAMA3 pre-trained model, which is the base LLM. It can be seen that the embodiment of the present application can reuse the language model that has been pre-trained to shorten the model building process.

[0120] S502: Using multimodal training data to train a pre-trained language model to obtain a large multimodal model for real-time speech recognition; wherein the multimodal training data is constructed based on the aforementioned multimodal training data construction method.

[0121] The embodiment of the present application can provide the following two training methods.

[0122] Specifically, in the first training method, step S502 may include the following steps S5021-S5025.

[0123] S5021: Obtain a training speech sequence, where the training speech sequence is a speech sequence composed of audio primitives obtained by segmenting speech data in the aforementioned multimodal training data construction method.

[0124] It can be understood that this step can obtain all training speech sequences corresponding to the training data set D, rather than a training speech sequence corresponding to a multimodal training data.

[0125] S5022: Obtain a first vocabulary and a first embedding matrix; wherein the first vocabulary is a set of text primitives, and the step of segmenting the text data corresponding to the speech data to obtain text primitives is performed based on the first vocabulary; each row of the first embedding matrix corresponds to each text primitive in the first vocabulary, and each column corresponds to the dimension of the embedding vector, and the first embedding matrix is ​​used to map the text primitives to the embedding vectors.

[0126] Exemplarily, the first vocabulary may include 30,000 text primitives, then the index ID of the text primitive may range from 0 to 29,999. Further, the number of rows of the first embedding matrix is ​​30,000, and the number of columns may be designed based on the actual required embedding vector dimension. If the index ID of the text primitive "I" is 3, the index ID of the text primitive "love" is 5, the index ID of the text primitive "you" is 7, the index ID of the text primitive "ancestor" is 9, and the index ID of the text primitive "country" is 11, then the text sequence [I, love, you, ancestor, country] can be converted to an index ID sequence [3,5,7,9,11]. Afterwards, the corresponding embedding vectors can be extracted from the first embedding matrix. For example, for the index ID sequence [3,5,7,9,11], the model can take out the 3rd, 5th, 7th, 9th and 11th rows from the first embedding matrix to obtain the embedding vectors of these five text primitives.

[0127] S5023: Add each audio primitive in the training speech sequence to the first vocabulary of the pre-trained language model to obtain a second vocabulary, where the second vocabulary is a collection of audio primitives and text primitives.

[0128] Exemplarily, if there are 10,000 audio primitives in the training speech sequence, then the 10,000 audio primitives can be added after the 30,000 text primitives, so that the first vocabulary is expanded to include 40,000 primitives. It is understandable that the index ID corresponding to the audio primitive can range from 30,000 to 39,999.

[0129] S5024: Expand the rows of the first embedding matrix to obtain a second embedding matrix, so that each row of the second embedding matrix corresponds to each audio primitive or text primitive in the second vocabulary.

[0130] It is understandable that this step can expand the rows of the first embedded matrix from 30,000 rows to 40,000 rows. In addition, the embodiment of the present application does not limit the columns of the first embedded matrix, and the columns can be expanded based on actual needs.

[0131] In this way, a second embedding matrix can be obtained, which not only contains the embedding vectors of the original text primitives, but also adds the embedding vectors of the audio primitives. Each row in the second embedding matrix corresponds to each audio primitive or text primitive in the second vocabulary, ensuring that the model can accurately map the audio primitive or text primitive to the embedding vector during the training process.

[0132] S5025: Train the pre-trained language model using the second vocabulary, the second embedding matrix, and the multimodal training data to obtain a large multimodal model.

[0133] In some implementations, audio primitives other than the audio primitives in the training speech sequence may be further added to the second vocabulary to expand the coverage of the audio primitives in the second vocabulary.

[0134] It can be understood that the training method provided in the embodiment of the present application can use audio primitives as expansion tokens. In this way, the multimodal large model not only inherits the text processing capabilities of the pre-trained language model, but also adds voice processing capabilities, realizing real-time conversion from voice signals to text output.

[0135] In the second training method, step S502 may include the following steps S5026-S5030.

[0136] S5026: Obtain a training speech sequence.

[0137] It can be understood that the training speech sequence is a speech sequence composed of audio primitives obtained by segmenting speech data in the aforementioned multimodal training data construction method.

[0138] S5027: Construct a third vocabulary, where the third vocabulary is a set of audio primitives in the training speech sequence.

[0139] For example, the training speech sequence involves 10,000 audio primitives, and the third vocabulary may be a set of the 10,000 audio primitives, and the index ID of the audio primitives ranges from 0 to 9999.

[0140] S5028: Construct a third embedding matrix, where each row of the third embedding matrix corresponds to each audio primitive in the third vocabulary, and each column corresponds to the dimension of the embedding vector. The second embedding matrix is ​​used to map the audio primitive to the embedding vector.

[0141] Exemplarily, the number of rows of the third embedding matrix may be 10,000, corresponding to each audio primitive, and the number of columns may be designed based on the actual required embedding vector dimension, which is not specifically limited in the embodiments of the present application.

[0142] S5029: Obtain a first vocabulary and a first embedding matrix.

[0143] It can be understood that the first vocabulary and the first embedding matrix are the same as the first vocabulary and the first embedding matrix in step S5022.

[0144] S5030: Train the pre-trained language model using the first vocabulary, the third vocabulary, the first embedding matrix, the third embedding matrix, and the multimodal training data to obtain a multimodal large model.

[0145] It is worth noting that the training steps of step S5025 and step S5030 may be the same as the pre-training, continued training and / or fine-tuning method of LLM, and the embodiment of the present application does not make any specific limitation on this.

[0146] In some implementations, audio primitives other than the audio primitives in the training speech sequence may be added to the third vocabulary to expand the coverage of the audio primitives by the third vocabulary.

[0147] It should be noted that, in practical applications, one of the training methods may be selected for training, such as using steps S5021-S5025 for training, or using steps S5026-S5030 for training.

[0148] The model inference phase is introduced below.

[0149] Figure 7 A flowchart of a second method for constructing a multimodal large model for real-time speech recognition provided in an embodiment of the present application.

[0150] Figure 8 A schematic diagram of the multimodal large model prediction or reasoning process provided in an embodiment of the present application.

[0151] like Figure 7 and Figure 8 As shown, the multimodal large model construction method for real-time speech recognition provided by the embodiment of the present application may include the following steps S600-S900. The multimodal large model is trained based on the first multimodal large model construction method for real-time speech recognition.

[0152] S600: inputting the real-time speech data to be recognized into the multimodal large model, using the multimodal large model to segment the real-time speech data according to a preset length, and obtaining a speech sequence to be recognized including audio primitives to be recognized.

[0153] Among them, the real-time voice data is dynamic, and the length of the real-time voice data gradually increases over time. For example, the real-time voice data is the voice data generated during a call, or the voice data generated during an online meeting or an offline meeting. The real-time voice data and the voice sequence to be recognized are arranged in frames, and each frame of real-time voice data corresponds to at least one frame of audio primitives to be recognized. After segmentation, the audio primitives to be recognized can be audio clips with a duration of 10ms, 20ms or 30ms, which is not specifically limited in the embodiments of the present application.

[0154] It can be understood that the step of segmenting the real-time voice data according to the preset length using the multimodal large model is the step of tokenizing the real-time voice data.

[0155] S700: Use the multimodal large model to predict the current frame to obtain a predicted data frame, where the current frame is one of the frames in the speech sequence to be recognized.

[0156] It is understandable that in the embodiment of the present application, the multimodal large model can start prediction from the first frame of the speech sequence to be recognized. In this step, the multimodal large model can make predictions based on the current frame and the frame data before the current frame, and the prediction data frame can be a text frame including text information or an audio frame including audio information.

[0157] S800: If the predicted data frame is a text frame, use the multimodal large model to output the text frame to achieve real-time speech recognition; and insert the text frame after the current frame to form a new speech sequence to be recognized, and continue to use the multimodal large model to predict the next frame of the current frame.

[0158] If the predicted data frame is a text frame, the multimodal large model can immediately convert it into readable text information and output it. This process marks the realization of real-time speech recognition, and users can instantly see the result of speech being converted into text.

[0159] like Figure 8 As shown, in the embodiment of the present application, if the predicted data frame output by the previous frame is a text frame (text primitive), then the current input is from the output of the previous frame. This step not only ensures the continuity of the recognition process, but also provides the model with the latest context information, so that it can more accurately predict the next speech content and achieve autoregression. Subsequently, the model will continue its prediction work and turn to the next frame of the current frame for analysis. It is worth noting that at this time, the "next frame of the current frame" is the text frame (prediction data frame) that has just been inserted.

[0160] S900: If the predicted data frame is an audio frame, continue to use the multimodal large model to predict the next frame of the current frame.

[0161] It is understandable that when predicting a data frame as an audio frame, the model can keep processing the real-time speech and continue to predict the next frame.

[0162] It is worth noting that if Figure 8 As shown, the input data and the inference output are aligned in the time dimension. For example, for time point A, the content on its left can be used to represent the form of the inference data and input at the current time point A. Furthermore, for each frame of data input, the multimodal large model will form an output (inference output). Specifically, after training with multimodal training data, the multimodal large model can output a text frame "I" when the input data is the first 7 frames, and the text frame "I" is located after the 6th frame of the inference output. Then, along the time direction, at the next time point (or time step), the text frame "I" appears after the 7th frame of the input data, forming a new speech sequence to be recognized.

[0163] It is understandable that the multimodal large model can output two types of reasoning outputs, one of which is internal output and the other is external output. The form of the internal output is the same as the form of the aforementioned training output samples. For example, when the speech sequence is input to the 7th frame, the corresponding output is "I". In the internal output sequence, "I" is located after the 6th frame. The external output is the result that the model finally presents to the user or application system, that is, the text result of real-time speech recognition.

[0164] It should also be noted that when the embodiment of the present application adopts the aforementioned first training method to train the model, the index ID of the predicted data frame can be determined by retrieving the first vocabulary, and the predicted data frame can be determined to be a text frame or an audio frame based on the specific data of the index ID. This is because the index IDs of the text primitives (text frames) and the audio primitives (audio frames) do not overlap. When the embodiment of the present application adopts the aforementioned second training method to train the model, other methods can be used to determine whether the predicted data frame is a text frame or an audio frame. For example, the method of extracting features from the predicted data frame determines whether it is a text frame or an audio frame.

[0165] From the above content, it can be seen that the embodiment of the present application provides a multimodal large model construction method for real-time speech recognition. Based on this method, the embodiment of the present application can add the text frame (text token / text primitive) to the input sequence of the model when the reasoning output is a text frame, so that the text information and voice information of the input sequence are intertwined, that is, the embodiment of the present application provides a new arrangement method of text frames and audio frames in the model input data, thereby achieving autoregression and improving the prediction effect. At the same time, the output mode of the model also changes. The model can generate output when a short-term speech is input. In this way, the reasoning output can be output word by word (or token by token) instead of the whole sentence output, thereby improving the effect of real-time speech recognition.

[0166] Fig. 9 A schematic diagram of the structure of a multimodal training data construction device provided in an embodiment of the present application.

[0167] like Fig. 9 As shown, the embodiment of the present application provides a multimodal training data construction device, which is applied to a multimodal large model for real-time speech recognition; the device includes:

[0168] A first acquisition module 1001 is used to acquire multiple speech sequences and a text sequence corresponding to each speech sequence; wherein the speech sequence is composed of feature primitives obtained by extracting features from speech data, or the speech sequence is composed of audio primitives obtained by segmenting speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data;

[0169] The data construction module 1002 is used to insert the corresponding text primitives into the feature primitives or audio primitives included in each speech sequence in order to obtain multimodal training data;

[0170] The collection module 1003 is used to collect all multimodal training data to form a training data set.

[0171] In some implementations, speech data and speech sequences are arranged in frames, and each frame of speech data corresponds to at least one frame of feature primitives or audio primitives; the multimodal training data includes training input samples and training output samples; for each speech sequence, the data construction module 1002 is specifically used to: for each text primitive corresponding to the speech sequence, determine the first frame sequence number of its pronunciation end position in the speech sequence; determine the delay parameter, the delay parameter is D frames, D≥0; for each text primitive corresponding to the speech sequence, based on its corresponding first frame sequence number and delay parameter, determine its insertion position in the speech sequence; wherein the frame sequence number of the insertion position is the second frame sequence number, and the second frame sequence number is equal to the sum of the first frame sequence number and the delay parameter; add a start mark at the starting position of the speech sequence, and insert each text primitive into its corresponding insertion position to form a training input sample; determine the training output sample based on the training input sample.

[0172] In some implementations, the data construction module 1002 is also used to: determine whether the second frame sequence number corresponding to the target text primitive is greater than the total number of frames of the speech sequence, the target text primitive being the last text primitive in the text sequence; if it is not greater than the total number of frames of the speech sequence, insert each text primitive after its corresponding insertion position; if it is greater than the total number of frames of the speech sequence, add N blank frames at the end of the speech sequence, and insert each text primitive after its corresponding insertion position; wherein N is equal to the difference between the second frame sequence number corresponding to the target text primitive and the total number of frames.

[0173] In some implementations, the data construction module 1002 is further configured to: offset the training input samples by one frame to form training output samples, so that the second frame of the training output samples is aligned with the first frame of the training input samples.

[0174] Fig.10 A schematic diagram of the structure of the first multimodal large model building device for real-time speech recognition provided in an embodiment of the present application.

[0175] like Fig.10 As shown, the embodiment of the present application provides a multimodal large model construction device for real-time speech recognition, the device comprising:

[0176] The second acquisition module 2001 is used to acquire a pre-trained language model; wherein the pre-trained language model is constructed based on a decoder-only transformer architecture;

[0177] The training module 2002 is used to train the pre-trained language model using multimodal training data to obtain a large multimodal model for real-time speech recognition; wherein the multimodal training data is constructed based on the aforementioned multimodal training data construction method.

[0178] In one implementation, the training module 2002 is specifically used to: obtain a training speech sequence, which is a speech sequence composed of audio primitives obtained by segmenting speech data in the aforementioned multimodal training data construction method; obtain a first vocabulary and a first embedding matrix; wherein the first vocabulary is a collection of text primitives, and the step of segmenting the text data corresponding to the speech data to obtain text primitives is performed based on the first vocabulary; each row of the first embedding matrix corresponds to each text primitive in the first vocabulary, and each column corresponds to the dimension of the embedding vector, and the first embedding matrix is ​​used to map the text primitive to the embedding vector; each audio primitive in the training speech sequence is added to the first vocabulary of the pre-trained language model to obtain a second vocabulary, and the second vocabulary is a collection of audio primitives and text primitives; and, the rows of the first embedding matrix are expanded to obtain a second embedding matrix, so that each row of the second embedding matrix corresponds to each audio primitive or text primitive in the second vocabulary; the pre-trained language model is trained using the second vocabulary, the second embedding matrix and the multimodal training data to obtain a multimodal large model.

[0179] In one implementation, the training module 2002 is specifically used to: obtain a training speech sequence; construct a third vocabulary, which is a set of audio primitives in the training speech sequence; and construct a third embedding matrix, wherein each row of the third embedding matrix corresponds to each audio primitive in the third vocabulary, and each column corresponds to the dimension of the embedding vector, and the second embedding matrix is ​​used to map the audio primitives to the embedding vector; obtain a first vocabulary and a first embedding matrix; and train the pre-trained language model using the first vocabulary, the third vocabulary, the first embedding matrix, the third embedding matrix, and the multimodal training data to obtain a multimodal large model.

[0180] Fig.11 A schematic diagram of the structure of a second multimodal large model building device for real-time speech recognition provided in an embodiment of the present application.

[0181] like Fig.11 As shown, the embodiment of the present application also provides a multimodal large model construction device for real-time speech recognition, wherein the multimodal large model is trained based on the aforementioned first multimodal large model construction method for real-time speech recognition; the device comprises:

[0182] The segmentation module 3001 is used to input the real-time speech data to be recognized into the multimodal large model, and use the multimodal large model to segment the real-time speech data according to a preset length to obtain a speech sequence to be recognized including audio primitives to be recognized; wherein the length of the real-time speech data gradually increases with time, the real-time speech data and the speech sequence to be recognized are arranged in frames, and each frame of real-time speech data corresponds to at least one frame of audio primitives to be recognized;

[0183] The prediction output module 3002 is used to predict the current frame using the multimodal large model to obtain a prediction data frame, where the current frame is one of the frames in the speech sequence to be recognized;

[0184] If the predicted data frame is a text frame, the prediction output module 3002 is further used to output the text frame using the multimodal large model to achieve real-time speech recognition; and insert the text frame after the current frame to form a new speech sequence to be recognized, and continue to use the multimodal large model to predict the next frame of the current frame;

[0185] If the predicted data frame is an audio frame, the prediction output module 3002 is also used to continue to use the multimodal large model to predict the next frame of the current frame.

[0186] In a specific implementation, the present invention further provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment of the multimodal training data construction method and the multimodal large model construction method provided by the present invention. The storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0187] It is easy to understand that those skilled in the art can combine, split, reorganize, etc. the embodiments of the present application to obtain other embodiments based on the several embodiments provided in the present application, and these embodiments do not exceed the protection scope of the present application.

[0188] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not used to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.< / bos>

Claims

1. A method for constructing multimodal training data, characterized in that: Large multimodal models for real-time speech recognition; The method comprises: Acquire multiple speech sequences and text sequences corresponding to each of the speech sequences; wherein the speech sequence is composed of feature primitives obtained by extracting features from speech data, or the speech sequence is composed of audio primitives obtained by segmenting the speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data; For each of the speech sequences, the corresponding text primitives are sequentially inserted between the feature primitives or the audio primitives included therein to obtain multimodal training data; All the multimodal training data are collected to form a training data set.

2. The method for constructing multimodal training data according to claim 1, characterized in that: The speech data and the speech sequence are arranged in frames, and each frame of the speech data corresponds to at least one frame of the feature primitive or the audio primitive; the multimodal training data includes training input samples and training output samples; for each of the speech sequences, the corresponding text primitives are inserted in sequence between the feature primitives or the audio primitives included therein, and the step of obtaining the multimodal training data includes: For each of the text primitives corresponding to the speech sequence, determining the first frame sequence number of the pronunciation end position in the speech sequence; Determine a delay parameter, where the delay parameter is D frames, D≥0; For each text primitive corresponding to the speech sequence, based on the first frame sequence number and the delay parameter corresponding to the text primitive, determining its insertion position in the speech sequence; wherein the frame sequence number of the insertion position is a second frame sequence number, and the second frame sequence number is equal to the sum of the first frame sequence number and the delay parameter; Adding a start mark at the start position of the speech sequence and inserting each text primitive into its corresponding insertion position to form the training input sample; The training output samples are determined based on the training input samples.

3. The method for constructing multimodal training data according to claim 2, characterized in that: The steps after inserting each of the text primitives into its corresponding insertion position include: Determine whether the second frame sequence number corresponding to the target text primitive is greater than the total number of frames of the speech sequence, the target text primitive being the last text primitive in the text sequence; If it is not greater than the total number of frames of the speech sequence, inserting each of the text primitives after the corresponding insertion position; If it is greater than the total number of frames of the speech sequence, N blank frames are added to the end of the speech sequence, and each of the text primitives is inserted after its corresponding insertion position; wherein N is equal to the difference between the second frame sequence number corresponding to the target text primitive and the total number of frames.

4. The method for constructing multimodal training data according to claim 2 or 3, characterized in that: The step of determining the training output sample based on the training input sample comprises: The training input sample is shifted by one frame to form the training output sample, so that the second frame of the training output sample is aligned with the first frame of the training input sample.

5. A method for constructing a multimodal large model for real-time speech recognition, characterized in that: The method comprises: Obtain a pre-trained language model; wherein the pre-trained language model is constructed based on a decoder-only transformer architecture; The pre-trained language model is trained using multimodal training data to obtain the multimodal large model for real-time speech recognition; wherein the multimodal training data is constructed based on the multimodal training data construction method according to any one of claims 1 to 4.

6. The method for constructing a multimodal large model for real-time speech recognition according to claim 5, characterized in that: The step of training the pre-trained language model using multimodal training data includes: Acquire a training speech sequence, wherein the training speech sequence is a speech sequence composed of audio primitives obtained by segmenting speech data in the multimodal training data construction method according to any one of claims 1 to 4; Obtain a first vocabulary and a first embedding matrix; wherein the first vocabulary is a set of text primitives, and the step of segmenting the text data corresponding to the speech data to obtain text primitives is performed based on the first vocabulary; each row of the first embedding matrix corresponds to each text primitive in the first vocabulary, and each column corresponds to the dimension of the embedding vector, and the first embedding matrix is ​​used to map the text primitives to the embedding vectors; Adding each audio primitive in the training speech sequence to the first vocabulary of the pre-trained language model to obtain a second vocabulary, wherein the second vocabulary is a collection of audio primitives and text primitives; and, expanding the rows of the first embedding matrix to obtain a second embedding matrix such that each row of the second embedding matrix corresponds to each audio primitive or text primitive in the second vocabulary; The pre-trained language model is trained using the second vocabulary, the second embedding matrix, and the multimodal training data to obtain the multimodal large model.

7. The method for constructing a multimodal large model for real-time speech recognition according to claim 6, characterized in that: The step of training the pre-trained language model using multimodal training data also includes: Acquire the training speech sequence; constructing a third vocabulary, the third vocabulary being a set of audio primitives in the training speech sequence; and constructing a third embedding matrix, wherein each row of the third embedding matrix corresponds to each audio primitive in the third vocabulary, each column corresponds to a dimension of an embedding vector, and the second embedding matrix is ​​used to map the audio primitive to the embedding vector; Obtaining the first vocabulary and the first embedding matrix; The pre-trained language model is trained using the first vocabulary, the third vocabulary, the first embedding matrix, the third embedding matrix and the multimodal training data to obtain the multimodal large model.

8. A method for constructing a multimodal large model for real-time speech recognition, characterized in that: The multimodal large model is trained based on the multimodal large model construction method for real-time speech recognition according to any one of claims 5 to 7; The method comprises: Inputting the real-time speech data to be recognized into the multimodal large model, and using the multimodal large model to segment the real-time speech data according to a preset length to obtain a speech sequence to be recognized including audio primitives to be recognized; wherein the length of the real-time speech data gradually increases with time, the real-time speech data and the speech sequence to be recognized are arranged in frames, and each frame of the real-time speech data corresponds to at least one frame of the audio primitives to be recognized; Using the multimodal large model to predict a current frame to obtain a predicted data frame, wherein the current frame is one of the frames in the speech sequence to be recognized; If the predicted data frame is a text frame, output the text frame using the multimodal large model to achieve real-time speech recognition; and insert the text frame after the current frame to form a new speech sequence to be recognized, and continue to use the multimodal large model to predict the next frame of the current frame; If the predicted data frame is an audio frame, continue to use the multimodal large model to predict the next frame of the current frame.

9. A multimodal training data construction device, characterized in that: Large multimodal models for real-time speech recognition; The device comprises: A first acquisition module is used to acquire a plurality of speech sequences and a text sequence corresponding to each of the speech sequences; wherein the speech sequence is composed of feature primitives obtained by extracting features from speech data, or the speech sequence is composed of audio primitives obtained by segmenting the speech data, and the text sequence is composed of text primitives obtained by segmenting text data corresponding to the speech data, and each text primitive is a character, word or sentence in the text data; A data construction module is used for inserting the corresponding text primitives in order between the feature primitives or the audio primitives included in each speech sequence to obtain multimodal training data; The collection module is used to collect all the multimodal training data to form a training data set.

10. A multimodal large model construction device for real-time speech recognition, characterized in that: The device comprises: A second acquisition module is used to acquire a pre-trained language model; wherein the pre-trained language model is constructed based on a decoder-only transformer architecture; A training module, used to train the pre-trained language model using multimodal training data to obtain the multimodal large model for real-time speech recognition; wherein the multimodal training data is constructed based on the multimodal training data construction method according to any one of claims 1 to 4; and / or, The multimodal large model is trained based on the multimodal large model construction method for real-time speech recognition according to any one of claims 5 to 7; The device comprises: A segmentation module, used for inputting the real-time speech data to be recognized into the multimodal large model, and using the multimodal large model to segment the real-time speech data according to a preset length to obtain a speech sequence to be recognized including audio primitives to be recognized; wherein the length of the real-time speech data gradually increases with time, the real-time speech data and the speech sequence to be recognized are arranged in frames, and each frame of the real-time speech data corresponds to at least one frame of the audio primitives to be recognized; A prediction output module, used to use the multimodal large model to predict the current frame to obtain a prediction data frame, wherein the current frame is one of the frames in the speech sequence to be recognized; If the predicted data frame is a text frame, the prediction output module is further used to output the text frame using the multimodal large model to achieve real-time speech recognition; and insert the text frame after the current frame to form a new speech sequence to be recognized, and continue to use the multimodal large model to predict the next frame of the current frame; If the predicted data frame is an audio frame, the prediction output module is further used to continue to use the multimodal large model to predict the next frame of the current frame.

Images