Audio encoder training method and device, voice recognition method and device, electronic equipment and storage medium

Abstract

The invention provides an audio encoder training method and device, a voice recognition method and device, electronic equipment and a storage medium, and relates to an AI or artificial intelligence technology in the technical field of intelligent cabins. The method comprises the following steps: respectively inputting training data into a first training model and a second training model; the first training model and the second training model have an incidence relation; the training data comprises audio data and text data; performing feature extraction on the training data through a first training model to obtain first feature representation; performing feature extraction on the training data through a second training model to obtain second feature representation; determining a first loss according to the corresponding first feature representation and the second feature representation; training a second training model according to the first loss; and in response to training convergence of the second training model, constructing the audio encoder based on the second training model. According to the method, the training data range is expanded, and the text data is introduced into training, so that richer language features can be learned.

Description

Technical Field

[0001] The present disclosure relates to the field of smart cockpit technology, and in particular to an audio encoder training method, a speech recognition method, a device, an electronic device, and a storage medium. Background Art

[0002] In the field of AI-based speech recognition, the application of large language models (LLMs) is gradually showing great potential. However, current speech recognition technology still faces many challenges, especially in terms of the quality and quantity of training data. High-quality training data is key to building highly accurate speech recognition models, but in reality, obtaining such data is often subject to various limitations.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides an audio encoder training method, a speech recognition method, an apparatus, an electronic device, and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for training an audio encoder is provided, comprising: inputting training data into a first training model and a second training model respectively; the first training model and the second training model are associated with each other; the training data comprises: audio data and text data; performing feature extraction on the training data through the first training model to obtain a first feature representation; performing feature extraction on the training data through the second training model to obtain a second feature representation; determining a first loss based on the corresponding first feature representation and second feature representation; training the second training model based on the first loss; and constructing an audio encoder based on the second training model in response to convergence of the second training model training.

[0006] In some exemplary embodiments of the present disclosure, the feature extraction of the training data by the second training model to obtain the second feature representation includes: extracting features from the audio data by an audio feature extractor to determine audio features; spatially mapping the audio features by an audio projector to determine an audio feature sequence; extracting features from the text data by a text feature extractor to determine text features; spatially mapping the text features by a text projector to determine a text feature sequence; inputting the audio feature sequence and the text feature sequence into a shared layer, performing context modeling by the shared layer, and determining the second feature representation.

[0007] In some exemplary embodiments of the present disclosure, the second feature representation includes: a second audio feature representation and a second text feature representation.

[0008] In some exemplary embodiments of the present disclosure, inputting the audio feature sequence and the text feature sequence into a shared layer, performing context modeling through the shared layer, and determining the second feature representation includes: alternately inputting the audio feature sequence and the text feature sequence into the shared layer, performing context modeling through the shared layer, and determining the second feature representation.

[0009] In some exemplary embodiments of the present disclosure, the method further includes: obtaining the first training model through parameter optimization migration based on the second training model.

[0010] In some exemplary embodiments of the present disclosure, the first training model is obtained through parameter optimization migration based on the second training model, including: determining the parameters of the first training model through parameter optimization migration according to parameter change information of the second training model during the training process; and obtaining the first training model based on the parameters of the first training model.

[0011] In some exemplary embodiments of the present disclosure, inputting training data into a first training model and a second training model respectively includes: inputting the training data into the first training model; performing mask processing on the training data; and inputting the masked training data into the second training model.

[0012] In some exemplary embodiments of the present disclosure, the first feature representation includes: a first audio feature representation and a first text feature representation; the second feature representation includes: a second audio feature representation and a second text feature representation; determining the first loss based on the corresponding first feature representation and second feature representation includes: determining the audio loss based on the corresponding first audio feature representation and second audio feature representation; determining the text loss based on the corresponding first text feature representation and second text feature representation; determining the first loss based on the audio loss and the text loss.

[0013] In some exemplary embodiments of the present disclosure, determining the first loss according to the audio loss and the text loss includes: determining the first loss according to the audio loss and the text loss acquired alternately.

[0014] In some exemplary embodiments of the present disclosure, constructing an audio encoder based on the second training model includes: constructing the audio encoder based on the audio feature extractor, audio projection layer and sharing layer in the second training model.

[0015] In some exemplary embodiments of the present disclosure, the audio encoder is constructed based on the audio feature extractor, audio projection layer and shared layer in the second training model, including: extracting parameters of the audio feature extractor, parameters of the audio projection layer and parameters of the shared layer of the second training model that has converged in training; and constructing the audio encoder according to the parameters of the audio feature extractor, parameters of the audio projection layer and parameters of the shared layer.

[0016] According to a second aspect of an embodiment of the present disclosure, a speech recognition method is provided, comprising: extracting features of a target speech through an audio encoder to obtain an audio feature representation; the audio encoder is trained according to any one of the above-mentioned audio encoder training methods; dimensionally adapting the audio feature representation through a linear layer to obtain an audio embedding vector; extracting features of a prompt word text through a text embedding layer to obtain a prompt word embedding vector; and performing speech recognition on the audio embedding vector based on the prompt word embedding vector through a large language model to obtain a target text.

[0017] In some exemplary embodiments of the present disclosure, feature extraction is performed on the prompt word text through a text embedding layer to obtain a prompt word embedding vector, including: performing word segmentation processing on the prompt word text to determine the prompt word segmentation; and extracting the prompt word embedding vector corresponding to the prompt word segmentation through the text embedding layer.

[0018] According to a third aspect of an embodiment of the present disclosure, an audio encoder training device is provided, comprising: a training data input module for inputting training data into a first training model and a second training model respectively; the first training model and the second training model are associated with each other; the training data include: audio data and text data; a first feature representation determination module for performing feature extraction on the training data through the first training model to obtain a first feature representation; a second feature representation determination module for performing feature extraction on the training data through the second training model to obtain a second feature representation; a first loss determination module for determining a first loss based on the corresponding first and second feature representations; a model training module for training the second training model based on the first loss; and an audio encoder construction module for constructing an audio encoder based on the second training model in response to the second training model training convergence.

[0019] According to a fourth aspect of an embodiment of the present disclosure, a speech recognition device is provided, including: an audio feature representation determination module, configured to perform feature extraction on a target speech through an audio encoder to obtain an audio feature representation; the audio encoder is trained using any one of the above-mentioned audio encoder training methods; an audio embedding vector determination module, configured to perform dimension adaptation on the audio feature representation through a linear layer to obtain an audio embedding vector; a prompt word embedding vector determination module, configured to perform feature extraction on a prompt word text through a text embedding layer to obtain a prompt word embedding vector; and a speech recognition module, configured to perform speech recognition on the audio embedding vector based on the prompt word embedding vector through a large language model to obtain a target text.

[0020] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to: implement the steps of any one of the speech recognition methods described.

[0021] According to the sixth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the steps of any one of the audio encoder training methods or speech recognition methods.

[0022] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0023] The present disclosure provides a first training model and a second training model with an associated relationship, and inputs training data into the first training model and the second training model respectively, extracts features from the training data through the first training model to obtain a first feature representation, and extracts features from the training data through the second training model to obtain a second feature representation, determines a first loss based on the corresponding first feature representation and second feature representation, trains the second training model with the first loss, and constructs an audio encoder based on the second training model in response to the convergence of the second training model. When training an artificial intelligence-based model, this embodiment, in addition to training the model through audio data, also adds a method of training through text data, thereby expanding the scope of training data. Moreover, text data is easier to obtain and has lower cost than audio data. By introducing text data into the training process, the model can learn richer language features; through the joint training of the first training model and the second training model, the audio encoder can extract useful language information from text data in the absence of sufficient audio samples, thereby improving its generalization ability. The training method that combines the features of audio data and text data is a cross-modal learning technology. It can fully utilize the complementarity between the two modalities, break through the traditional method's dependence on high-quality audio data, and give full play to the synergy of the two modalities of audio and text to improve the model's language understanding ability, robustness and adaptability.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0026] Figure 1 This is a flow chart of an audio encoder training method according to an exemplary embodiment of the present disclosure. Figure 1 .

[0027] Figure 2 This is a flow chart of an audio encoder training method according to an exemplary embodiment of the present disclosure. Figure 2 .

[0028] Figure 3 2 is a diagram of a second training model architecture of an audio encoder training method according to an exemplary embodiment of the present disclosure.

[0029] Figure 4This is a flow chart of a speech recognition method according to an exemplary embodiment of the present disclosure. Figure 1 .

[0030] Figure 5 It is an architectural diagram of a speech recognition method according to an exemplary embodiment of the present disclosure.

[0031] Figure 6 The figure is a block diagram of an audio encoder training device according to an exemplary embodiment of the present disclosure.

[0032] Figure 7 The block diagram of a speech recognition device according to an exemplary embodiment of the present disclosure is shown.

[0033] Figure 8 It is a functional block diagram of a vehicle according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] Some exemplary embodiments of the present disclosure will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications and equivalents of the methods, devices and / or systems described herein will become apparent after understanding the present disclosure. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but may be changed as becomes apparent after understanding the present disclosure, except for operations that must be performed in a specific order. In addition, descriptions of features known in the art may be omitted for clarity and brevity.

[0035] The following exemplary embodiments of the present disclosure do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0036] With the increasing prominence of large language models (LLMs) in the field of natural language processing (NLP), the practice of training and adapting large language models to other tasks has also attracted increasing attention in many other fields. By adapting the corresponding modal encoder, the large language model can understand multimodal inputs including images, text, and audio. In the field of audio processing, adapting audio encoders to large language models has been proven to be effective in many audio processing tasks (such as speech recognition ASR), which utilizes the powerful knowledge reasoning and context modeling capabilities of large language models. Models based on self-supervised pre-training are often used as audio encoders due to their powerful audio representation capabilities. The audio features extracted from these models are then input as prompts into the large language model to generate the target text.

[0037] In related technologies, the amount of high-quality training data required for model training is limited. The audio encoder is trained to align text features with speech features, primarily through paired audio-text data. However, paired audio-text data is expensive to annotate and limited in quantity. This is especially true in multilingual environments with rare languages, where the available paired audio-text data is relatively limited. This limits the training of the audio encoder, resulting in poor performance and, in turn, the accuracy of speech recognition using large language models.

[0038] Hereinafter, each step of the method in the exemplary embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings and embodiments.

[0039] Figure 1 This is a flow chart of an audio encoder training method according to an exemplary embodiment of the present disclosure. Figure 1 .like Figure 1 As shown, in some embodiments, the audio encoder training method of the present disclosure example includes:

[0040] In step S110, training data is input into the first training model and the second training model respectively; the first training model and the second training model are associated with each other; the training data includes: audio data and text data;

[0041] The embodiments of the present disclosure provide training data for model training. The training data includes audio data and text data. The audio data and text data in the training data can be paired audio-text data or unpaired audio data and text data. For example, in paired audio-text data, the content of the audio data is "I need a bottle of water", and the content of the corresponding text data is also "I need a bottle of water"; while in unpaired audio data and text data, the content of the audio data is "I need a bottle of water", and the content of the text data may be "Today's weather is sunny"; the embodiments of the present disclosure are mainly aimed at unpaired audio data and text data, that is, the content of the audio data is inconsistent with the text data.

[0042] The training data containing audio data and text data is input into a first training model and a second training model with an associated relationship. In one embodiment, the association relationship between the first training model and the second training model is the same architecture, that is, the architecture of the first training model is the same as the architecture of the second training model. For example, if the first training model includes a feature extractor, a projector and a shared layer, then the second training model also includes a feature extractor, a projector and a shared layer. Based on the above association relationship, the first training model and the second training model can adopt a teacher-student learning framework (Teacher-Student Framework). The teacher-student learning framework includes a paradigm in which the teacher model and the student model optimize the model performance through knowledge transfer and dynamic guidance, and guides the student model to learn from the teacher model, thereby achieving knowledge compression, enhanced generalization and improved training efficiency. In this embodiment, the first training model can be a teacher model and the second training model can be a student model.

[0043] In step S120, feature extraction is performed on the training data using a first training model to obtain a first feature representation;

[0044] In the disclosed embodiments, in machine learning or deep learning tasks, in order to extract high-level, abstract feature information from raw data, a constructed model is typically used to perform feature extraction on the input training data. Specifically, the training data input to the first training model is audio data and text data. The first training model performs feature extraction on the audio data to obtain a first audio feature representation, and the first training model performs feature extraction on the text data to obtain a first text feature representation. The first feature representation includes the first audio feature representation and the first text feature representation obtained above. The first training model is used to extract features from the input data.

[0045] The first feature representation obtained by extracting features from the training data through the first training model is a high-dimensional vector that contains the core semantic information of the input training data while removing redundant or irrelevant information, reflecting the contextual relationship in the training data.

[0046] In step S130, feature extraction is performed on the training data using a second training model to obtain a second feature representation;

[0047] In the disclosed embodiment, the first training model and the second training model are associated with each other. Similarly, the training data input to the second training model also consists of audio data and text data. The second training model performs feature extraction on the audio data to obtain a second audio feature representation. The second training model also performs feature extraction on the text data to obtain a second text feature representation. The second feature representation includes the second audio feature representation and the second text feature representation obtained above. The second training model is used to extract features from the input data.

[0048] The second feature representation, obtained by extracting features from the training data using the second training model, is also a high-dimensional vector. It contains the core semantic information of the input training data while removing redundant or irrelevant information. Based on the teacher-student learning framework described above, the second feature representation obtained by the second training model can learn the features corresponding to the training data.

[0049] In step S140, a first loss is determined based on the corresponding first feature representation and second feature representation;

[0050] In the disclosed embodiment, for the same piece of training data, the first feature representation obtained by inputting the training data into the first training model corresponds to the second feature representation obtained by inputting the training data into the second training model. The training data input into the second training model is processed, so the second feature representation obtained by feature extraction by the second training model is different from the first feature representation. The goal of model training is to narrow the difference between the second feature representation and the first feature representation, specifically by determining the first loss through the corresponding first feature representation and second feature representation to characterize the difference between the first feature representation and the second feature representation.

[0051] In step S150, the second training model is trained according to the first loss;

[0052] In the disclosed embodiment, the first loss represents the difference between the first feature representation and the second feature representation. When training the second training model, in order to optimize the parameters of the second training model so that it can better complete the target task, the parameters of the second training model will be updated through the back propagation algorithm based on the first loss (Loss) calculated above.

[0053] The core idea of backpropagation is to calculate the gradient of the loss function relative to each model parameter through the chain rule. Specifically: starting from the output layer, the gradient of each layer of neurons is calculated forward layer by layer. The gradient represents the sensitivity of the loss function to the model parameters, that is, how the change of the parameters affects the loss value. The calculated gradient information will be passed back to each layer of the model to guide the direction of parameter adjustment. After obtaining the gradient, the parameters of the second training model are updated using an optimization algorithm (such as stochastic gradient descent SGD, Adam, etc.) to train the second training model. As the training progresses, the model will gradually learn to extract more meaningful features from the input data and improve its performance on the target task.

[0054] In step S160 , in response to the second training model training converging, an audio encoder is constructed based on the second training model.

[0055] In the disclosed embodiment, a model training convergence condition is set; wherein the model training convergence condition includes: the first loss is reduced to a set threshold and / or the number of model training times reaches a set value. Specifically, when the first loss is reduced to the set threshold, the model training convergence condition is met, and the second training model stops training. Alternatively, when the number of training times of the second training model reaches a set value, the second training model stops training. The second training model obtained at this time is the trained second training model, and the audio encoder can be constructed based on the trained second training model.

[0056] When training the model, this embodiment not only trains the model through audio data, but also adds a method of training through text data, which expands the scope of training data. In addition, text data is easier to obtain and has lower cost than audio data. By introducing text data into the training process, the model can learn richer language features. Through the joint training of the first training model and the second training model, the audio encoder can extract useful language information from text data in the absence of sufficient audio samples, thereby improving its generalization ability. The training method that combines audio data features and text data features belongs to the category of cross-modal learning. This technology can make full use of the complementarity between the two modalities, break through the traditional method's dependence on high-quality audio data, and give full play to the synergy between the two modalities of audio and text to improve the model's language understanding ability, robustness and adaptability.

[0057] This embodiment extracts features from the training data through a first training model and a second training model that have an associated relationship. When the audio data and text data in the training data are not paired, the corresponding first feature representation and second feature representation can be extracted to determine the first loss, thereby enabling the training of the second training model based on the unpaired audio data and text data, and constructing an audio encoder based on the second training model that has converged. This embodiment can use unpaired audio data and text data to train and construct an audio encoder. Unpaired audio data and text data are easier to obtain and larger in quantity than paired audio-text data, and have the potential to improve the performance of the audio encoder through data volume. In unpaired audio data and text data, the amount of text data is far greater than that of audio data. The performance of speech recognition can be improved based on the contextual information in the text data, especially in scenarios where audio data is scarce.

[0058] The audio encoder constructed in this embodiment using the AI or artificial intelligence model training method can be applied to a variety of speech recognition carriers and scenarios. For example, it can be applied to vehicles, automobiles, home appliances, smartphones, and wearable devices for speech recognition. It can also be applied to interactive speech recognition scenarios such as smart cockpits, smart homes, online meetings, and remote training. The high-performance audio encoder can improve the accuracy of speech recognition.

[0059] Figure 2 This is a flow chart of an audio encoder training method according to an exemplary embodiment of the present disclosure. Figure 2 .like Figure 2 As shown, in this embodiment, the second training model is used to extract features from the training data to obtain a second feature representation, including:

[0060] In step S210, the audio data is subjected to feature extraction by an audio feature extractor to determine audio features;

[0061] In step S220, the audio features are spatially mapped by an audio projector to determine an audio feature sequence;

[0062] In step S230, a text feature extractor is used to extract features from the text data to determine text features;

[0063] In step S240, the text features are spatially mapped by a text projector to determine a text feature sequence;

[0064] In step S250, the audio feature sequence and the text feature sequence are input into a shared layer, context modeling is performed through the shared layer, and a second feature representation is determined.

[0065] Figure 3FIG. 1 is a second training model architecture diagram of an audio encoder training method according to an exemplary embodiment of the present disclosure. Figure 3 As shown in the embodiment, the architecture of the second training model mainly includes multiple modules such as an audio feature extractor, a text feature extractor, an audio projector, a text projector, and a shared layer. Among them, the audio feature extractor and the text feature extractor are both feature extractors, used to extract corresponding features from the input data; the audio projector and the text projector are both projectors, used to spatially map the input features to the same latent space.

[0066] The audio feature extractor is used to extract features from audio data to obtain audio features.

[0067] In audio processing tasks, extracting meaningful feature information from audio data often relies on specially designed audio feature extractors. This process is a key step in audio signal processing and machine learning tasks, transforming complex time-domain or frequency-domain signals into a set of representative feature vectors. An audio feature extractor is a specially designed tool or algorithm for extracting high-level feature representations from input audio data. Audio features can include time-domain features, frequency-domain features, and time-frequency-domain features.

[0068] The audio projector is used to spatially project audio features, mapping them to an audio latent space to produce an encoded audio feature sequence. To further enhance the model's understanding of audio data and achieve higher-level abstract representations, the audio projector module is often introduced. The audio projector spatially transforms and maps the extracted audio features, converting the original audio features into a more expressive latent space, the audio latent space, thereby generating an encoded audio feature sequence.

[0069] The text feature extractor is used to extract text features from text data.

[0070] In natural language processing tasks, text feature extractors are often used to convert text data into numerical representations that can be understood by machine learning models. The core goal of this process is to extract feature vectors from text that are capable of capturing the semantic, syntactic, and contextual information of the text.

[0071] The text projector is used to perform spatial projection on text features, map the text features to the text latent space, and obtain the encoded text feature sequence;

[0072] In order to further improve the model's ability to understand text data and achieve a higher level of abstract representation, a text projector module is usually introduced. The role of the text projector is to perform spatial transformation and mapping on the extracted text features, converting the original text features into a more expressive latent space, namely the text latent space, thereby generating an encoded text feature sequence. In an example embodiment, the goal of model training is to align the text feature representation and the speech feature representation, specifically by aligning the audio latent space with the text latent space.

[0073] The shared layer is used to perform context modeling on the audio feature sequence and the text feature sequence, obtain the relationship between the text context, and thus obtain the second feature representation.

[0074] In multimodal tasks, in order to achieve deep interaction between audio and text modalities and capture their contextual relationships, a shared layer is usually introduced. The core function of the shared layer is to jointly model the audio feature sequence and the text feature sequence, extract the correlation information between the modalities through a unified context modeling mechanism, and thus generate a more representative second feature representation.

[0075] Before entering the shared layer, audio and text data have already been processed by their respective feature extractors and projectors, and converted into structured feature sequences. Although these feature sequences have a certain degree of expressive power, they may still lack intermodal interaction information and cannot directly reflect the semantic relationship between audio and text.

[0076] Shared blocks are a general neural network module that can be composed of Transformer modules. Shared layers have the following important functions: Intra-modal modeling: Modeling audio feature sequences and text feature sequences separately to capture their respective temporal dependencies and contextual information. For example, capturing syntactic structure and semantic relationships in text, and capturing features such as rhythm and pitch in audio. Inter-modal interaction: By sharing parameters or cross-attention mechanisms, information interaction between audio and text features is promoted to discover potential correlations between the two modalities. Unified representation learning: Mapping audio and text features to a shared latent space to generate a second feature representation that can simultaneously reflect information from both modalities.

[0077] Through the above audio feature extractor, text feature extractor, audio projector, text projector and shared layer, rich features can be extracted from audio data and text data, and converted into a highly abstract and easy-to-use representation through a series of processing steps. It is not limited to a single type of data, but can effectively integrate information from multiple modalities to provide a more comprehensive perspective; by optimizing feature extraction, streamlining feature dimensions and strengthening the interaction between modalities, it can adapt to a variety of application scenarios. When the encoders and shared layers of the two modalities are optimized at the same time, the gradient conflicts between the modalities can be reduced. At the same time, this design also enables the model to constrain the audio features output by the two modal feature extractors through the shared layer, thereby achieving the purpose of aligning the output features of the two modalities.

[0078] In an embodiment, the audio feature sequence and the text feature sequence are input into a shared layer, context modeling is performed through the shared layer, and a second feature representation is determined, including: alternately inputting the audio feature sequence and the text feature sequence into a shared layer, context modeling is performed through the shared layer, and the second feature representation is determined.

[0079] In multimodal learning tasks, to achieve deep interaction between audio and text modalities and capture their contextual relationships, an alternating input approach is often designed, feeding audio and text feature sequences sequentially into a shared layer for processing. This alternating input approach not only effectively models intra-modal and inter-modal contextual information but also improves model training efficiency and stability through a gradient accumulation mechanism.

[0080] Audio feature sequences and text feature sequences represent information from two different modalities. To enable these two modal features to fully interact in the shared layer and generate a unified representation, a strategy for alternating input is designed: audio feature sequences and text feature sequences are fed into the shared layer for processing in an alternating manner (e.g., in turns or in batches). After each feature from a modality is input, the corresponding loss value is calculated and the gradient is accumulated, rather than immediately updating the model parameters. After all modal features have completed a single forward and backward pass, the model parameters are updated uniformly.

[0081] For example, taking the audio feature sequence as A and the text feature sequence as B, one alternating method is to input the audio feature sequence and the text feature sequence alternately into the shared layer, then the alternating input into the shared layer is ABAB...; another alternating method is to input the audio feature sequence and the text feature sequence alternately into the shared layer in batches, then the alternating input into the shared layer is AABB...; another alternating method is to input multiple audio feature sequences and then input a text feature sequence, then the alternating input into the shared layer is AABAA...; another alternating method is to input an audio feature sequence and then input multiple text feature sequences, then the alternating input into the shared layer is BBABB.... The several alternating methods exemplified above are only for illustration, and the embodiments of the present disclosure may also include other alternating methods input into the shared layer, all of which fall within the scope of protection of the embodiments of the present disclosure.

[0082] By alternating inputs, we ensure that audio and text features can fully interact in the shared layer, thereby capturing the semantic connections between the modalities. This avoids gradient instability caused by the limited amount of single input data and improves training effectiveness by accumulating gradient information from multiple modalities. By accumulating gradient information from multiple modalities, we reduce gradient fluctuations caused by insufficient single input data and improve training stability. When video memory is limited, alternating inputs can process data from different modalities in steps, avoiding performance bottlenecks caused by loading large amounts of data all at once.

[0083] In an embodiment, based on the above steps S110-S160, the audio encoder training method further includes:

[0084] Step S170: Optimize and migrate parameters based on the second training model to obtain the first training model.

[0085] During the model training process, in order to improve the generalization ability and stability of the model, parameter optimization migration will be adopted.

[0086] Parameter optimization transfer (PAT) involves transferring knowledge or parameters from a pre-trained source model (e.g., a teacher model) to a target model (e.g., a student model) during model training to improve the target model's performance. This method is commonly used in scenarios such as knowledge distillation, transfer learning, and model compression. Specific implementations can include direct parameter replication, exponential moving average (EMA), soft label guidance, and other techniques. Because the second training model updates its parameters during each training session, the parameters of the first training model, based on the teacher-student learning framework, are not updated in real time as the parameters of the second training model are updated. Instead, parameter optimization transfer is used to smoothly update the parameters of the first training model. For new tasks or small datasets, PAT can help the student model find a suitable parameter space more quickly, thereby accelerating convergence. Using methods such as the exponential moving average (EMA) for PAT can smooth the changes in model parameters, reduce the sharp fluctuations caused by a single update, and help maintain stability during training.

[0087] In the disclosed embodiment, the first training model is obtained by parameter optimization migration based on the second training model, including:

[0088] Determining the parameters of the first training model through parameter optimization migration based on parameter change information of the second training model during the training process;

[0089] A first training model is obtained based on the parameters of the first training model.

[0090] During the training process of the first training model, each training will bring about parameter changes and adjustments, such as the updated values of the model parameters after each iteration, the changing trend of the parameter gradient, and the convergence speed and stability of the parameters of different layers. First, it is necessary to obtain the parameter change information of the second training model during the training process. The key information of the parameter change information of the second training model during the training process is extracted. The key information may include parameter weight distribution, gradient direction and convergence characteristics; based on the above key information, the parameters of the first training model are determined by parameter optimization migration; for example, when the parameter optimization migration is the exponential moving average (EMA), the parameters of the first training model are determined by EMA; based on the obtained parameters of the first training model, the first training model can be further obtained.

[0091] Based on the changes in the student model's parameters during training, the teacher model's parameters were adjusted using the exponential moving average method, successfully achieving collaborative optimization of the teacher and student models. This approach not only improves the stability of the teacher model but also accelerates the convergence of the student model, providing strong support for building efficient and stable deep learning systems.

[0092] In the disclosed embodiment, the training data is input into the first training model and the second training model respectively, including: inputting the training data into the first training model; performing mask processing on the training data; and inputting the masked training data into the second training model.

[0093] Based on the above-mentioned teacher-student learning framework, the second training model needs to learn the feature extraction content of the first training model during feature extraction. The training data that the first training model needs to extract features from is complete training data, so the complete training data is input into the first training model, that is, the complete audio data and text data are input into the first training model for feature extraction. In order for the second training model to learn the feature extraction content of the first training model, it is necessary to mask the training data to obtain incomplete masked training data, and then input the masked training data into the second training model. Among them, masking the training data mainly involves randomly selecting target positions in the training data and replacing the target positions with special mask tags [MASK]. The goal of the model is to predict the content of these masked target positions based on the other parts of the training data. This method enables the model to learn context-related word representations, because the model needs to understand the context of the entire sentence to correctly predict the content of the masked target positions. For example, in the training data, the content of the audio data is "The weather is nice today, I went for a walk in the park with my friends." After masking the audio data, the content of the masked audio data is "Today's weather is [MASK], I went for a walk in the park with [MASK]." The masked training data is input into the second training model, which learns the feature extraction content of the first training model and recognizes the content at the [MASK] position.

[0094] The disclosed embodiment also provides another masking method, which is to randomly select a target position in the training data, delete the content of the target position to obtain masked training data, and input the masked training data into the second training model. For example, in the training data, the content of the audio data is "The weather is very good today, and my friends and I went for a walk in the park." After masking the audio data, the content of the masked audio data obtained is "The weather is very good today, and I went for a walk in the park." The masked training data is input into the second training model, and the second training model learns the feature extraction content of the first training model to fill in the deleted content of the target position. Under this masking method, the second training model still has the opportunity to infer the information of the masked word through the surrounding words, thereby filling in the target position and obtaining complete audio data.

[0095] The embodiment of the present disclosure inputs training data into a first training model, masks the training data to obtain masked training data, and inputs the masked training data into a second training model, so that the model can learn context-related word representations, improve the generalization ability of the model, learn more representative features, and enhance the robustness of the model. The model can better cope with missing or noisy input data, thereby facilitating the second training model to learn from the first training model.

[0096] In the disclosed embodiment, determining the first loss according to the corresponding first feature representation and the second feature representation includes:

[0097] determining an audio loss based on the corresponding first audio feature representation and the second audio feature representation;

[0098] Determining a text loss based on the corresponding first text feature representation and the second text feature representation;

[0099] A first loss is determined according to the audio loss and the text loss.

[0100] Based on the first audio feature representation obtained by inputting the same audio data into the first training model to extract the features of the audio data, and the second audio feature representation obtained by inputting the same audio data into the second training model to extract the features of the audio data, the above loss calculation based on the corresponding first audio feature representation and second audio feature representation obtained from the same audio data can determine the audio loss. Specifically, the first audio feature representation is obtained by feature extraction of the complete audio data, and the second audio feature representation is obtained by feature extraction of the masked audio data. In essence, the second audio feature representation is obtained by predicting the content of the masked position, so there is a difference between the corresponding second audio feature representation and the first audio feature representation. The training goal of the second training model is to reduce this difference. The audio loss is obtained by calculating the mean square error loss (MSE-LOSS) of the first audio feature representation and the second audio feature representation.

[0101] Based on the first text feature representation obtained by inputting the same text data into the first training model to extract the features of the text data, and the second text feature representation obtained by inputting the same text data into the second training model to extract the features of the text data, the above loss calculation based on the corresponding first text feature representation and second text feature representation obtained from the same text data can determine the text loss. Specifically, the first text feature representation is obtained by feature extraction of the complete text data, while the second text feature representation is obtained by feature extraction of the masked text data. In essence, the second text feature representation is obtained by predicting the content of the masked position, so there is a difference between the corresponding second text feature representation and the first text feature representation. The training goal of the second training model is to reduce this difference. The text loss is obtained by calculating the mean square error loss (MSE-LOSS) of the first text feature representation and the second text feature representation.

[0102] Based on the audio loss and text loss calculated above, the first loss can be obtained as the total loss.

[0103] In multimodal model training tasks involving both text and audio, multiple loss functions are designed during the training process to measure the learning effects of different modalities. The audio loss and text loss are used to evaluate the model's performance in the audio and text modalities, respectively, while the first loss comprehensively considers the losses of these two modalities and is used to guide the overall optimization direction of the model. During model training, the first loss is used as the optimization target. The model parameters are gradually updated by calculating the gradient of the first loss with respect to the model parameters and performing backpropagation using an optimization algorithm (such as SGD, Adam, etc.). In an embodiment, different weights can be assigned to the audio loss and text loss to balance their contributions, and the first loss is obtained by weighted summation. Alternatively, a maximum loss method can be used to calculate the maximum loss between the audio loss and the text loss as the first loss. Alternatively, a product method can be used to collaboratively optimize the audio and text losses, multiplying the audio and text losses to obtain the first loss. Alternatively, a weighted sum of squares method can be used to calculate the first loss for the audio and text losses, further amplifying larger loss values and thus focusing more attention on the poorly performing modality. The above example of obtaining the first loss from audio loss and text loss is for illustration only. Other methods may also be used to obtain the first loss, which also fall within the scope of protection of the embodiments of the present disclosure.

[0104] In the disclosed embodiment, determining the first loss according to the audio loss and the text loss includes:

[0105] A first loss is determined according to the alternately acquired audio loss and text loss.

[0106] Based on the aforementioned process of alternately inputting the audio feature sequence and the text feature sequence into the shared layer to obtain the corresponding feature representations, the audio loss and the text loss are also alternately output when determining them. Therefore, the audio loss and the text loss can be alternately obtained to determine the first loss. Specifically, the first loss can be obtained by adding the alternately obtained audio loss and text loss.

[0107] For example, the audio loss is obtained as C, and the text loss is obtained as D. There are multiple alternating ways to alternately obtain audio loss and text loss. One alternating way is to obtain audio loss and text loss in turns, then the corresponding alternating acquisition is CDCD...; another alternating way is to obtain audio loss and text loss in batches, then the corresponding alternating acquisition is CCDD...; another alternating way is to obtain multiple audio losses and then obtain a text loss, then the corresponding alternating acquisition is CCDCC...; another alternating way is to obtain an audio loss and then obtain multiple text losses, then the corresponding alternating acquisition is DDCDD.... The several alternating ways exemplified above are only for illustration, and the embodiments of the present disclosure may also include other alternating ways to obtain audio loss and text loss, all of which fall within the scope of protection of the embodiments of the present disclosure.

[0108] In order to optimize the performance of the model, this embodiment designs multiple loss functions during the training process to measure the learning effects of different modalities. Among them, alternately obtaining text loss and audio loss is a dynamic optimization strategy. By focusing on the performance of text modality and audio modality respectively at different stages of training, the first loss is gradually constructed to guide the overall optimization direction of the model. In multimodal tasks, audio modality and text modality may have different complexities or convergence speeds. By alternately obtaining the losses of the two modalities, the performance of audio modality and text modality can be optimized separately at different stages, thereby achieving a more balanced training process.

[0109] In a disclosed embodiment, constructing an audio encoder based on the second training model includes:

[0110] Build an audio encoder based on the audio feature extractor, audio projection layer, and shared layer in the second trained model.

[0111] The architecture of the second training model includes: audio feature extractor, text feature extractor, audio projector, text projector and shared layer; for the audio encoder, its function is to convert audio data into audio feature representation, so the main modules that can be used to construct the audio encoder in the second training model are audio feature extractor, audio projection layer and shared layer.

[0112] In multimodal learning or audio processing tasks, the audio encoder is a key component for converting raw audio signals into high-level semantic representations. To build an efficient and powerful audio encoder, we can leverage the audio feature extractor, audio projection layer, and shared layers from the second trained model. These modules each perform distinct functions and work together to achieve a complete mapping process from raw audio to encoded features.

[0113] The audio feature extractor, audio projection layer, and shared layer in the second training model have been fully trained on related tasks and possess strong feature extraction and semantic modeling capabilities. Reusing these modules significantly reduces the training time and computational cost of the new audio encoder while improving its performance. This achieves knowledge transfer and modular design, significantly improving the performance and flexibility of the audio encoder.

[0114] In the disclosed embodiment, an audio encoder is constructed based on the audio feature extractor, the audio projection layer, and the sharing layer in the second training model, including:

[0115] Extracting parameters of the audio feature extractor, the audio projection layer, and the shared layer of the second training model whose training converges;

[0116] Construct an audio encoder based on the parameters of the audio feature extractor, the parameters of the audio projection layer, and the parameters of the shared layer.

[0117] To quickly build an efficient and high-performance audio encoder, key modules and their parameters from the already trained and converged second model can be leveraged. These modules include the audio feature extractor, audio projection layer, and shared layer, which are responsible for extracting low-level features from the raw audio signal, generating high-level semantic representations, and capturing contextual information, respectively. Training of the second model is completed when it reaches convergence, achieving optimal processing performance. The parameters of the audio feature extractor, audio projection layer, and shared layer of the converged second model are then extracted. The audio feature extractor extracts audio features from the audio data and inputs them into the audio projection layer, which performs spatial mapping to generate an audio feature sequence. This is then input into the shared layer, which performs context modeling to generate the corresponding feature representation. The parameters of the audio feature extractor, audio projection layer, and shared layer are saved, ensuring compatibility with the subsequently constructed audio encoder, such as dimensionality matching and data type consistency. All necessary parameters are extracted to avoid omissions that could cause the model to malfunction. The audio encoder is constructed based on the parameters of the audio feature extractor, audio projection layer, and shared layer.

[0118] Using the audio feature extractor parameters, load the extracted convolution kernel weights and other parameters to initialize the audio feature extractor. Using the audio projection layer parameters, load the weight matrix and activation function parameters of the fully connected layer to initialize the audio projection layer. Using the shared layer parameters, load the self-attention weights or hidden state parameters of the recurrent neural network to initialize the shared layer. During initialization, ensure that the input and output dimensions of each module match and verify that the parameters are loaded correctly. Finally, construct the audio encoder.

[0119] The audio feature extractor, audio projection layer, and shared layer in the second training model have been fully trained on related tasks and have strong feature extraction and semantic modeling capabilities. By reusing the parameters of these modules, the training time and computational cost of the new audio encoder can be significantly reduced while improving its performance. The process of extracting parameters and constructing the encoder is more efficient than retraining a complete model, which can significantly reduce the consumption of computing resources. With a modular design, each module can be optimized or replaced independently, which facilitates flexible adjustment according to task requirements. The constructed audio encoder can be fine-tuned according to the requirements of the new task and is suitable for a variety of application scenarios (such as speech recognition, sentiment analysis, cross-modal retrieval, etc.). Using model parameters that have been trained and converged can ensure that the initial state of the audio encoder has high stability and reliability.

[0120] In the prior art, for the training of audio encoders, the training data all uses paired audio-text data to achieve alignment between text-speech modalities, which has strong limitations. The disclosed embodiment pre-trains the audio encoder in LLM-based ASR (speech recognition task based on large language model) based on unpaired data, so as to achieve the effect of modal alignment. Unpaired audio-text data is easier to obtain than paired data, and the quantity is huge, and there is potential to improve model performance through data volume. At the same time, the amount of text data is far greater than that of audio data, and the contextual information in the text has the potential to improve the performance of ASR, especially in scenarios where voice data is scarce, such as multilingual environments, where a large amount of unpaired data can be used. Unpaired data is very easy to obtain, thereby training a high-performance audio encoder.

[0121] The training process of the disclosed embodiments is guided by mask prediction within a teacher-student learning framework. The student model has the same architecture as the teacher model, and during training, the student model should closely match the output of the teacher model.

[0122] For unpaired speech-text data, each audio and text is alternately fed into the audio feature extractor and text feature extractor, respectively. For audio input, the output of the masked part of the teacher model is finally predicted, and the resulting loss is recorded as the audio loss (loss-audio); for text input, the output of the masked part of the teacher model is finally predicted, and the resulting loss is recorded as the text loss (loss-text). The final backpropagation of the first loss is to obtain loss-audio and loss-text respectively, add the two losses to obtain the first loss as the total loss, and perform backpropagation together. When optimizing the encoders and shared layers of the two modalities at the same time, this training method can reduce gradient conflicts between the modalities. At the same time, this design also allows the model to constrain the embedded coding of the outputs of the two modal feature extractors through the shared layer, thereby achieving the purpose of aligning the output features of the two modalities.

[0123] Figure 4 This is a flow chart of a speech recognition method according to an exemplary embodiment of the present disclosure. Figure 1 .like Figure 4 As shown, in one embodiment, a speech recognition method of the present disclosure example includes:

[0124] In step S410, feature extraction is performed on the target speech by an audio encoder to obtain an audio feature representation; the audio encoder is trained according to the above-mentioned audio encoder training method.

[0125] During speech recognition based on a large language model, a target speech and a prompt text are input. The target speech is then fed into an audio encoder trained using the aforementioned audio encoder training method. The audio encoder then extracts features from the target speech to generate an audio feature representation. Specifically, the core modules of the audio encoder include an audio feature extractor, an audio projection layer, and a shared layer. These modules work together to gradually encode the audio signal from low-level features to high-level semantic representations.

[0126] The target speech signal is usually represented in the form of digital audio, such as a WAV file or PCM data. The original audio signal is a one-dimensional time series, which represents the change of the amplitude of the sound wave over time. The audio feature extractor is responsible for extracting low-level audio features (such as time domain features, frequency domain features, etc.) from the original target speech. The audio projection layer is responsible for mapping low-level audio features to a higher-level audio latent space to generate a compact and semantically meaningful audio feature sequence. The shared layer is responsible for context modeling the audio feature sequence, capturing temporal dependencies and semantic associations, and outputting audio feature representations. After processing by the audio encoder, the target speech is converted into a high-level audio feature representation. It achieves efficient feature extraction and provides flexible and powerful support for downstream tasks.

[0127] In step S420, the audio feature representation is dimensionally adapted through a linear layer to obtain an audio embedding vector.

[0128] The audio feature representation generated by the target speech after being processed by the audio encoder typically has a specific dimension, while the input dimension of the large language model often matches its pre-training design (for example, a fixed-length vector dimension). In order to enable the speech embedding vector to seamlessly integrate into the input space of the large language model, a linear layer (Adapter) is required to adapt the audio feature representation to the dimension, thereby generating an audio embedding vector that meets the input requirements of the large language model.

[0129] The audio feature representation generated by the audio encoder usually has a specific dimension (for example, 512 dimensions or 1024 dimensions), while the input dimension of the large language model may be different (for example, 768 dimensions or 1280 dimensions). If the audio feature representation is directly input into the large language model, it may cause dimension mismatch problems, resulting in calculation errors or performance degradation. The linear layer is a simple and efficient tool for mapping between different dimensions. By introducing the linear layer, the audio feature representation can be mapped from the original dimension to the target dimension to obtain the audio embedding vector, so that it can adapt to the input requirements of the large language model.

[0130] In step S430, feature extraction is performed on the prompt word text through the text embedding layer to obtain a prompt word embedding vector.

[0131] Prompt text is an important form of input used to guide the model to generate specific outputs or complete specific tasks. Prompt text is a natural language input, usually in the form of short sentences or paragraphs. In the task, prompt text is used to provide contextual information, specify task objectives, or guide the model's behavior. Deep learning models cannot directly process raw text data. In order for prompt text to be effectively utilized by deep learning models (such as large language models or cross-modal models), it needs to be converted into a high-level semantic representation, namely a prompt embedding vector. This step is usually implemented by a text embedding layer, which extracts features from the prompt text and generates a fixed-dimensional prompt embedding vector. The main function of the text embedding layer is to extract features from the prompt text and generate an embedding vector with semantic meaning.

[0132] In step S440, speech recognition is performed on the audio embedding vector based on the prompt word embedding vector through the large language model to obtain the target text.

[0133] Leveraging the powerful semantic understanding and generation capabilities of the large language model, combined with cue word embeddings and audio embeddings, the conversion of speech signals into text is accomplished. The task of speech recognition is to convert speech signals into human-readable text. Pre-trained on massive amounts of text data, the large language model possesses powerful semantic understanding and generation capabilities. By converting speech signals into audio embeddings and then using the cue word embeddings to guide the model in generating target text, speech recognition performance can be significantly improved. The audio embeddings serve as a high-level representation of the speech signal, and the cue word embeddings serve as task-guided input, both of which are fed into the large language model. By fusing these two, the large language model generates semantically consistent target text.

[0134] The large language model has been pre-trained and has strong semantic understanding capabilities, which can accurately capture the semantic information in the audio embedding vector. The prompt word embedding vector further enhances the model's understanding of the task objectives, thereby improving the relevance and accuracy of the generated text. The combination of audio embedding vectors and prompt word embedding vectors fully utilizes the information of both modalities to achieve collaborative optimization of multimodal tasks. This synergy is particularly suitable for complex scenarios (such as noisy environments or multi-language support). The design of the prompt word embedding vector can be flexibly adjusted according to task requirements to adapt to different application scenarios. Combined with the powerful capabilities of the large language model, the accuracy and robustness of speech recognition are significantly improved, especially in complex tasks.

[0135] In the disclosed embodiment, feature extraction is performed on the prompt word text through the text embedding layer to obtain the prompt word embedding vector, including:

[0136] Perform word segmentation on the prompt word text to determine the prompt word segmentation;

[0137] The prompt word embedding vector corresponding to the prompt word segmentation is extracted through the text embedding layer.

[0138] Effective processing of prompt text is a key step in ensuring the model can accurately understand and generate high-quality output. To achieve this, it is usually necessary to segment the prompt text and then use a text embedding layer to convert the segmented results into a high-level semantic representation, namely the prompt word embedding vector.

[0139] Natural language is composed of continuous characters, and deep learning models cannot directly process raw text data. Therefore, the prompt word text needs to be segmented into the smallest semantic units (such as words, subwords or characters) for subsequent processing. The quality of word segmentation directly affects the model's ability to understand the prompt words and the accuracy of the generated text. The purpose of word segmentation is to decompose the prompt word text into smaller semantic units to facilitate subsequent feature extraction and semantic modeling. Select a suitable word segmentation tool or tokenizer according to the task requirements, such as NLTK, spaCy, BERT's WordPiece, GPT's Byte PairEncoding (BPE), RoBERTa's SentencePiece, etc.

[0140] The selected segmentation tool is used to segment the prompt word text, producing a series of tokens. To enhance the model's understanding of sentence structure, special tokens are often added to the segmentation results. The final result is a list of tokens, known as the prompt word segmentations. The main function of the text embedding layer is to extract features from the prompt word segmentations and generate semantically meaningful embedding vectors. For each prompt word segmentation, the corresponding prompt word embedding vector is searched from the pre-trained word embedding table.

[0141] The text embedding layer is a standard component in modern deep learning frameworks. Its implementation is highly efficient and easy to integrate. This seamless integration makes the text embedding layer a valuable tool for natural language processing tasks. Leveraging pre-trained word embedding and language models, the text embedding layer is robust and maintains stable performance across a wide range of scenarios. Even in noisy environments or when faced with non-standard language expressions, the model can still generate relatively accurate hint word embedding vectors.

[0142] The text embedding layer converts the prompt word text into an embedding vector of fixed dimension, obtaining the prompt word embedding vector. This method not only achieves efficient feature extraction but also provides strong support for the implementation of multimodal tasks.

[0143] The speech recognition method provided by the embodiments of the present disclosure can be applied to various types of speech recognition carriers and scenarios. For example, it can be applied to carriers such as vehicles, automobiles, home appliances, smartphones, and wearable devices for speech recognition. It can also be applied to interactive scenarios such as smart cockpits, smart homes, online meetings, and remote training for speech recognition. The high-performance audio encoder and linear layer obtained through the above construction provide a more accurate audio embedding vector for the large language model. The prompt word embedding vector obtained by feature extraction of the prompt word text using the text embedding layer is also input into the large language model for accurate speech recognition to achieve accurate target text output. Taking the smart cockpit of a vehicle as an example, the speech recognition method of the embodiment of the present disclosure is applied to the smart cockpit scenario; specifically, a large language model is deployed in the smart cockpit, and when a user in the smart cockpit speaks and generates user speech, the audio acquisition device in the smart cockpit collects the user speech, and performs feature extraction on the user speech through an audio encoder to obtain an audio feature representation, and performs dimension adaptation on the audio feature representation through a linear layer to obtain an audio embedding vector; further, a text input device in the smart cockpit receives a prompt word text input by the user, performs word segmentation on the prompt word text to obtain prompt word segmentations, and extracts a prompt word embedding vector corresponding to the above prompt word segmentations through a text embedding layer; the audio embedding vector and the prompt word embedding vector are input into the large language model, and the large language model performs speech recognition on the audio embedding vector based on the prompt word embedding vector to obtain a target text, and the target text is output through an output device; the output device can be a display screen in the smart cockpit, and the target text is displayed on the display screen. The output device may also be a speaker in the smart cockpit, which uses text-to-speech (TTS) technology to convert the target text into a playback voice, and plays the playback voice through the speaker in the smart cockpit.

[0144] Based on the natural language capabilities of a large language model, the disclosed embodiments can accurately analyze user intent, support contextual memory, and enable multiple rounds of conversation. Users no longer need to repeat background information, and can generate coherent responses based on previous conversations. By supporting both voice and text input, the system meets the needs of various scenarios and seamlessly integrates voice and text input to maintain conversational coherence. Voice input reduces the need for drivers to manually operate devices, avoiding distractions and allowing them to focus more on road conditions, reducing distractions caused by operating devices and ensuring driving safety.

[0145] Figure 5 FIG. 1 is an architecture diagram of a speech recognition method according to an exemplary embodiment of the present disclosure. Figure 5As shown in the figure, the overall framework of the large language model-based ASR (LLM-based ASR) mainly includes: audio encoder, linear layer, text embedding layer, large language model, etc. The audio encoder mainly includes: audio feature extractor, audio projection layer and sharing layer, etc. When performing speech recognition, the target speech and target prompt word text are input; specifically, the target speech is input into the audio encoder for feature extraction to obtain audio feature representation; the audio feature representation is dimensionally adapted through the linear layer to obtain an audio embedding vector; the input prompt word text is segmented to obtain prompt word segmentation, and the prompt word embedding vector corresponding to the prompt word segmentation is extracted through the text embedding layer; the prompt word embedding vector and the audio embedding vector are input into the large language model together, and the large language model performs speech recognition on the audio embedding vector based on the prompt word embedding vector to obtain the target text.

[0146] In the overall framework of the large language model (LLM-based ASR) described above, the main training process involves the following: the large language model predicts the vector representation of the next token based on the prompt word embedding vector, the audio embedding vector, and the historical output tokens. After a softmax transformation, a probability distribution is formed over the total vocabulary dimension. Subsequently, the model is trained using a cross-entropy loss function to maximize the probability of the correct next token. During inference, the model selects the token with the highest probability from the probability distribution output by the softmax layer as the model output.

[0147] The speech recognition method of the disclosed embodiment can be applied to various speech recognition scenarios, such as smart devices such as mobile phones, speakers, televisions, and tablets. It can also be used for in-vehicle speech recognition, using large model technology to assist smart cockpits.

[0148] The audio encoder trained by the audio encoder training method provided by the embodiment of the present disclosure is applied to the speech recognition task of a large language model. When the overall automatic speech recognition is actually evaluated on the development set and the test set, compared with the speech recognition model trained with paired text-audio data, the embodiment of the present disclosure uses unpaired text data and audio data, and the error rate of speech recognition is lower. This is because the embodiment of the present disclosure can find a large amount of non-matching text-speech data, while the paired data is very limited; further, compared with the speech recognition model trained with only audio data, the embodiment of the present disclosure uses unpaired text data and audio data, and the error rate of speech recognition is lower. The above two comparisons prove the effectiveness of the embodiment of the present disclosure, that is, using the method of the embodiment of the present disclosure, the final speech recognition effect can be improved with the help of text data that does not match the audio annotation.

[0149] It can be seen that the method of the disclosed embodiment is even better than the method using paired data training. This is actually due to the fact that the disclosed embodiment can find a large amount of non-matching text-speech data, while paired data is very limited. At the same time, the method of the disclosed embodiment is better than using audio data alone without adding text data for audio encoder pre-training, which also proves the effectiveness of the disclosed embodiment. That is, using the method of the disclosed embodiment, the final speech recognition effect can be improved by using text data that does not match the audio annotations.

[0150] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed herein are in compliance with the relevant provisions of national laws and regulations. The various types of data such as personal identity data, operation data, behavioral data, etc. related to individuals, customers, and groups obtained in the embodiments of the present disclosure have all been authorized.

[0151] Figure 6 FIG1 is a block diagram of an audio encoder training device according to an exemplary embodiment of the present disclosure. The device of this embodiment can be applied to electronic devices.

[0152] like Figure 6 As shown, the audio encoder training device may include:

[0153] A training data input module 610 is configured to input training data into a first training model and a second training model respectively; the first training model and the second training model are associated with each other; the training data includes audio data and text data;

[0154] A first feature representation determination module 620 is configured to extract features from the training data using the first training model to obtain a first feature representation;

[0155] A second feature representation determination module 630 is configured to extract features from the training data using the second training model to obtain a second feature representation;

[0156] A first loss determination module 610 is configured to determine a first loss based on the corresponding first feature representation and second feature representation;

[0157] A model training module 650, configured to train the second training model according to the first loss;

[0158] The audio encoder construction module 660 is configured to construct an audio encoder based on the second training model in response to the second training model training converging.

[0159] Figure 7 FIG2 is a block diagram of a speech recognition device according to an exemplary embodiment of the present disclosure. The device of this embodiment can be applied to electronic devices.

[0160] like Figure 7 As shown, the speech recognition device may include:

[0161] An audio feature representation determination module 710 is configured to extract features of the target speech using an audio encoder to obtain an audio feature representation; the audio encoder is trained using any of the above-mentioned audio encoder training methods;

[0162] An audio embedding vector determination module 720 is configured to perform dimension adaptation on the audio feature representation through a linear layer to obtain an audio embedding vector;

[0163] A prompt word embedding vector determination module 730 is configured to extract features of the prompt word text through a text embedding layer to obtain a prompt word embedding vector;

[0164] The speech recognition module 740 is configured to perform speech recognition on the audio embedding vector based on the prompt word embedding vector through a large language model to obtain a target text.

[0165] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0166] Figure 8 This is a functional block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Examples of electronic devices include mobile phones, wearable devices, and vehicles. Taking a vehicle as an example, vehicle 800 can be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or another type of vehicle. Vehicle 800 can be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0167] Reference Figure 8 Vehicle 800 may include various subsystems, such as an infotainment system 810, a perception system 820, a decision-making and control system 830, a drive system 840, and a computing platform 850. Vehicle 800 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of vehicle 800 may be interconnected via wired or wireless means.

[0168] In some embodiments, the infotainment system 810 may include a communication system, an entertainment system, a navigation system, and the like.

[0169] The perception system 820 may include several sensors for sensing information about the environment surrounding the vehicle 800. For example, the perception system 820 may include a global positioning system (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera.

[0170] The decision control system 830 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0171] The drive system 840 may include components that provide power to the vehicle 800. In one embodiment, the drive system 840 may include an engine, a power source, a transmission system, and wheels. The engine may be an internal combustion engine, an electric motor, an air compression engine, or a combination thereof. The engine is capable of converting energy provided by the power source into mechanical energy.

[0172] Some or all functions of the vehicle 800 are controlled by a computing platform 850. The computing platform 850 may include at least one processor 851 and a memory 852. The processor 851 may execute instructions 853 stored in the memory 852.

[0173] The processor 851 can be any conventional processor, such as a commercially available CPU. The processor can also include a graphics processor (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0174] The memory 852 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0175] In addition to instructions 853 , memory 852 may also store data, such as road maps, route information, and vehicle location, direction, speed, etc. The data stored in memory 852 may be used by computing platform 850 .

[0176] In the embodiment of the present disclosure, the processor 851 may execute the instruction 853 to complete all or part of the steps of the above-mentioned data transmission method.

[0177] In some embodiments of the present disclosure, a non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform all or part of the steps of the above-mentioned road audio encoder training method or speech recognition method.

[0178] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0179] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training an audio encoder, characterized in that: include: Inputting the training data into the first training model and the second training model respectively; The first training model and the second training model have an association relationship; The training data includes: audio data and text data; Performing feature extraction on the training data using the first training model to obtain a first feature representation; Performing feature extraction on the training data using the second training model to obtain a second feature representation; Determine a first loss based on the corresponding first feature representation and second feature representation; Training the second training model according to the first loss; In response to the second training model training converging, an audio encoder is constructed based on the second training model.

2. The audio encoder training method according to claim 1, wherein The extracting features from the training data using the second training model to obtain a second feature representation includes: Extracting features from the audio data using an audio feature extractor to determine audio features; spatially mapping the audio features through an audio projector to determine an audio feature sequence; Performing feature extraction on the text data by a text feature extractor to determine text features; Performing spatial mapping on the text features by a text projector to determine a text feature sequence; The audio feature sequence and the text feature sequence are input into a shared layer, context modeling is performed through the shared layer, and the second feature representation is determined.

3. The audio encoder training method according to claim 2, wherein: The second feature representation includes: a second audio feature representation and a second text feature representation.

4. The audio encoder training method according to claim 2, wherein The step of inputting the audio feature sequence and the text feature sequence into a shared layer, performing context modeling through the shared layer, and determining the second feature representation includes: The audio feature sequence and the text feature sequence are alternately input into the shared layer, context modeling is performed through the shared layer, and the second feature representation is determined.

5. The audio encoder training method according to claim 1, wherein The method further comprises: The first training model is obtained by parameter optimization migration based on the second training model.

6. The audio encoder training method according to claim 5, characterized in that The first training model is obtained by performing parameter optimization migration based on the second training model, including: Determining the parameters of the first training model through parameter optimization migration according to parameter change information of the second training model during the training process; Based on the parameters of the first training model, the first training model is obtained.

7. The audio encoder training method according to claim 1, wherein: Inputting the training data into the first training model and the second training model respectively includes: inputting the training data into the first training model; Performing mask processing on the training data; The masked training data is input into the second training model.

8. The audio encoder training method according to claim 1 or 7, characterized in that: The first feature representation includes: a first audio feature representation and a first text feature representation; the second feature representation includes: a second audio feature representation and a second text feature representation; The determining of the first loss according to the corresponding first feature representation and second feature representation includes: determining audio loss based on the corresponding first audio feature representation and second audio feature representation; Determining text loss based on the corresponding first text feature representation and second text feature representation; The first loss is determined according to the audio loss and the text loss.

9. The audio encoder training method according to claim 8, wherein: Determining the first loss according to the audio loss and the text loss includes: The first loss is determined according to the alternately acquired audio loss and text loss.

10. The audio encoder training method according to claim 2, wherein: Building an audio encoder based on the second training model, including: Construct the audio encoder based on the audio feature extractor, audio projection layer and shared layer in the second training model.

11. The audio encoder training method according to claim 10, wherein: Constructing the audio encoder based on the audio feature extractor, the audio projection layer, and the shared layer in the second training model, including: Extracting parameters of the audio feature extractor, the audio projection layer, and the shared layer of the second training model that has reached convergence; The audio encoder is constructed according to the parameters of the audio feature extractor, the parameters of the audio projection layer, and the parameters of the shared layer.

12. A speech recognition method, characterized in that: include: Extract features of the target speech through an audio encoder to obtain audio feature representation; The audio encoder is trained according to any one of the methods of claims 1-11; Performing dimension adaptation on the audio feature representation through a linear layer to obtain an audio embedding vector; The prompt word text is feature extracted through the text embedding layer to obtain the prompt word embedding vector; The audio embedding vector is subjected to speech recognition based on the prompt word embedding vector by a large language model to obtain a target text.

13. The speech recognition method according to claim 12, wherein: The text embedding layer is used to extract features from the prompt word text to obtain the prompt word embedding vector, including: Performing word segmentation processing on the prompt word text to determine the prompt word segmentation; The prompt word embedding vector corresponding to the prompt word segmentation is extracted through the text embedding layer.

14. An audio encoder training device, characterized in that include: A training data input module, used to input training data into the first training model and the second training model respectively; The first training model and the second training model have an association relationship; The training data includes: audio data and text data; A first feature representation determination module is configured to extract features from the training data using the first training model to obtain a first feature representation; A second feature representation determination module is used to extract features from the training data using the second training model to obtain a second feature representation; A first loss determination module, configured to determine a first loss based on the corresponding first feature representation and second feature representation; A model training module, configured to train the second training model according to the first loss; An audio encoder construction module is used to construct an audio encoder based on the second training model in response to the second training model training convergence.

15. A speech recognition device, characterized in that: include: An audio feature representation determination module is used to extract features of the target speech through an audio encoder to obtain an audio feature representation; The audio encoder is trained according to any one of the methods of claims 1-11; an audio embedding vector determination module, configured to perform dimension adaptation on the audio feature representation through a linear layer to obtain an audio embedding vector; A prompt word embedding vector determination module is used to extract features of the prompt word text through the text embedding layer to obtain the prompt word embedding vector; A speech recognition module is used to perform speech recognition on the audio embedding vector based on the prompt word embedding vector through a large language model to obtain a target text.

16. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Implement the steps of the speech recognition method described in any one of claims 12-13.

17. A non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the steps of the audio encoder training method according to any one of claims 1 to 11 or the speech recognition method according to any one of claims 12 to 13.

Images