A training method and device of an audio recognition model, an equipment, a storage medium and a program product

Abstract

The application provides a training method and device of an audio recognition model, equipment, a storage medium and a program product. The method comprises: performing encoding processing on an audio data sample through an encoding network in the audio recognition model to obtain first encoding features; performing bias mapping on the first encoding features based on preset words through a first bias network to obtain second encoding features; performing decoding processing on the second encoding features through a decoding network to obtain a first text probability distribution; performing bias mapping on the first text probability distribution based on preset words through a second bias network to obtain a second text probability distribution; determining a loss value based on a text label of the audio data sample, the first encoding features, the first text probability distribution and the second text probability distribution; and updating parameters of the audio recognition model based on the loss value to obtain a trained audio recognition model. Through the application, the accuracy of audio recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a training method, apparatus, device, storage medium, and program product for an audio recognition model. Background Technology

[0002] Significant progress has been made in related technologies, particularly in end-to-end audio recognition systems based on neural networks. These systems directly convert audio signals into text using deep learning models, reducing the complexity of multiple steps in traditional audio recognition systems. However, despite the overall performance improvement, audio recognition accuracy remains low when dealing with rare words not included in the training dataset or contextual terms such as proper nouns. Summary of the Invention

[0003] This application provides a training method, apparatus, device, storage medium, and program product for an audio recognition model, which can improve the accuracy of audio recognition.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a method for training an audio recognition model, the method comprising:

[0006] The audio data samples are encoded using the encoding network in the audio recognition model to obtain the first encoded feature;

[0007] The first bias network in the audio recognition model performs bias mapping on the first encoded features based on preset words to obtain the second encoded features;

[0008] The second encoded feature is decoded through the decoding network in the audio recognition model to obtain the first text probability distribution.

[0009] The second text probability distribution is obtained by bias mapping the first text probability distribution based on the preset words through the second bias network in the audio recognition model.

[0010] Based on the text annotations of the audio data samples, the first encoding feature, the first text probability distribution, and the second text probability distribution, the loss value is determined;

[0011] The parameters of the audio recognition model are updated based on the loss value to obtain the trained audio recognition model.

[0012] This application provides a training device for an audio recognition model, comprising:

[0013] The encoding module is used to encode audio data samples through the encoding network in the audio recognition model to obtain the first encoded feature;

[0014] The encoding module is further configured to perform bias mapping on the first encoding feature based on preset words through the first bias network in the audio recognition model to obtain the second encoding feature;

[0015] The decoding module is used to decode the second encoded feature through the decoding network in the audio recognition model to obtain the first text probability distribution;

[0016] The decoding module is further configured to perform a bias mapping on the first text probability distribution based on the preset words through the second bias network in the audio recognition model to obtain the second text probability distribution;

[0017] The training module is used to determine the loss value based on the text annotation of the audio data sample, the first encoding feature, the first text probability distribution, and the second text probability distribution;

[0018] The training module is also used to update the parameters of the audio recognition model based on the loss value, so as to obtain the trained audio recognition model.

[0019] This application provides an electronic device, the electronic device comprising:

[0020] Memory is used to store executable instructions or computer programs.

[0021] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method of the audio recognition model provided in the embodiments of this application.

[0022] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the training method of the audio recognition model provided in this application when executed by a processor.

[0023] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the training method of the audio recognition model provided in this application.

[0024] The embodiments of this application have the following beneficial effects:

[0025] By introducing a first bias network, the first encoded features obtained from the encoding process are biased and mapped based on preset words, thus guiding the audio recognition model to focus on specific words (i.e., preset words). By introducing a second bias network, the first text probability distribution obtained after decoding is biased and mapped, further optimizing the recognition of preset words. Furthermore, by combining the first encoded features, the first text distribution probability, the second text distribution probability, and the loss value design of multiple supervision signals such as text annotation, the audio recognition model has stronger semantic focusing ability and robustness, thereby improving the accuracy of audio recognition. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the audio recognition system architecture provided in the embodiments of this application;

[0027] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0028] Figure 3A This is a schematic diagram of the first process of training the audio recognition model provided in the embodiments of this application;

[0029] Figure 3B This is a schematic diagram of the second process of the training method for the audio recognition model provided in the embodiments of this application;

[0030] Figure 3C This is a schematic diagram of the third process of the training method for the audio recognition model provided in the embodiments of this application;

[0031] Figure 3D This is a schematic diagram of the fourth process of the training method for the audio recognition model provided in the embodiments of this application;

[0032] Figure 3E This is a schematic diagram of the fifth step in the training method of the audio recognition model provided in the embodiments of this application;

[0033] Figure 3F This is a schematic diagram of the sixth step in the training method of the audio recognition model provided in the embodiments of this application;

[0034] Figure 3G This is a schematic diagram of the seventh step in the training method of the audio recognition model provided in the embodiments of this application;

[0035] Figure 3H This is a schematic diagram of the eighth step in the training method of the audio recognition model provided in the embodiments of this application;

[0036] Figure 4 This is a schematic diagram illustrating the training principle of the audio recognition model provided in the embodiments of this application;

[0037] Figure 5This is a schematic diagram illustrating the calculation principle of the first sub-loss value provided in the embodiments of this application;

[0038] Figure 6 This is a schematic diagram of the bias network structure provided in the embodiments of this application;

[0039] Figure 7 This is a schematic diagram of the word prediction network provided in the embodiments of this application.

[0040] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0044] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0045] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0046] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0047] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0048] 1) Audio recognition (or speech recognition) is a technology that converts human speech into text. Its goal is to enable computers to "understand" human language and accurately convert speech information into text output.

[0049] 2) Long Short-Term Memory (LSTM) is a variant of Recurrent Neural Network (RNN). It solves the problems of gradient vanishing and gradient explosion in traditional RNN by introducing a gating mechanism, which enables the model to better capture long-term dependencies in the sequence.

[0050] 3) Transformer: A temporal model based on self-attention. In the encoder part, it can effectively encode temporal information, and its processing capability for temporal information is far superior to LSTM, while also being faster. It is widely used in natural language processing, computer vision, machine translation, speech recognition, and other fields.

[0051] 4) Connectionist Temporal Classification (CTC) is a commonly used loss function in audio recognition tasks. The basic principle of the CTC loss function is to map continuous audio signals into continuous sequences of text characters without needing to know the word boundaries in the audio beforehand. In this way, the model's task is simplified to directly predicting the text without the need for additional word segmentation steps.

[0052] 5) Preset words refer to words that are pre-set for specific fields or application scenarios, such as proper nouns, high-frequency words, hot words, and business terms (such as product names) in the financial field.

[0053] 6) Contextual Phrase Prediction Network (CPP Network) is a neural network that predicts related phrases based on contextual information.

[0054] 7) Convolution-augmented Transformer (Conformer) refers to a model architecture that combines Transformer and Convolutional Neural Network (CNN). Transformer models are good at capturing content-based global interactions, while CNNs effectively utilize local features, making the Conformer model better at modeling both long-term global interaction information and local features.

[0055] 8) Context-enhanced high-efficiency automatic speech recognition framework (SEQ-former) is an end-to-end speech recognition framework that combines the advantages of CTC with attention-based encoder-decoder (AED) and recurrent neural network transducer (RNN-T).

[0056] Significant progress has been made in related technologies for end-to-end audio recognition systems based on neural networks. These systems directly convert audio signals into text using deep learning models, reducing the complexity of multiple steps in traditional audio recognition systems. However, despite the overall performance improvement, audio recognition accuracy remains low when dealing with rare words or contextual terms (such as proper nouns) not included in the training dataset. This application provides a training method, apparatus, device, computer-readable storage medium, and computer program product for an audio recognition model, which can improve the accuracy of audio recognition.

[0057] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. The electronic devices provided in the embodiments of this application can be implemented as various types of terminal devices such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals, or they can be implemented as servers.

[0058] See Figure 1 , Figure 1 This is a schematic diagram of the audio recognition system architecture provided in the embodiments of this application. Figure 1 The system involves server 100, terminal device 200, and network 300. Terminal device 200 is connected to server 100 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0059] In some embodiments, the present application embodiments can be implemented collaboratively by a server and a terminal device. For example, the terminal device 200 sends an audio data sample and preset words to the server 100. The server 100 receives the audio data sample and preset words, obtains a trained audio recognition model using the audio recognition model training method provided in the present application embodiments, and sends the trained audio recognition model to the terminal device 200.

[0060] In other embodiments, the embodiments of this application can be implemented independently by a terminal device. The terminal device 200 acquires local audio data samples and preset words, and obtains a trained audio recognition model through the training method of the audio recognition model provided in the embodiments of this application.

[0061] In some embodiments, the terminal device or server can implement the audio recognition model training method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application program, module, or plug-in. Terminal devices include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, and in-vehicle terminals.

[0062] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The illustrated electronic device 400 includes at least one processor 410, a memory 430, and at least one network interface 420. The various components of the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0063] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0064] The memory 430 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 430 may optionally include one or more storage devices physically located away from the processor 410.

[0065] The memory 430 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 430 described in this application embodiment is intended to include any suitable type of memory.

[0066] In some embodiments, memory 430 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0067] Operating system 431 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0068] The network communication module 432 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0069] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A training device 433 for an audio recognition model stored in memory 430 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: encoding module 4331, decoding module 4332, and training module 4333. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0070] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the audio recognition model provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0071] The following will describe the training method of the audio recognition model provided in this application embodiment, with the server as the execution subject, using exemplary applications and implementations of the server provided in the embodiments of this application. See also Figure 3A , Figure 3A This is a schematic diagram of the first process of training the audio recognition model provided in the embodiments of this application, which will be combined with Figure 3A The steps shown are explained.

[0072] In step 101, the audio data samples are encoded using the encoding network in the audio recognition model to obtain the first encoded feature.

[0073] Here, an audio recognition model is an artificial intelligence model used to process and analyze audio signals, capable of converting audio data into text data. Audio recognition models can be used in various scenarios requiring audio recognition, such as transcribing recorded telephone customer service conversations, generating real-time captions for meetings, voice assistants, and smart speakers.

[0074] In some embodiments, step 101 is implemented by performing the following processes through the coding network in the audio recognition model: performing frame processing on the audio data samples to obtain audio frames; performing frequency domain transformation processing on the audio frames to obtain the frequency domain representation of the audio frames; performing acoustic feature extraction processing on the frequency domain representation to obtain spectral features; and performing feature encoding processing on the spectral features to obtain first coded features.

[0075] For example, audio data samples can be preprocessed before encoding, such as removing noise, adjusting sampling rate and gain, to ensure audio quality. The preprocessed audio data samples are then divided into frames of fixed length. Hamming or Hanning windows can be applied to each audio frame to reduce spectral leakage. Frequency domain transformation is achieved by performing a Fast Fourier Transform (FFT) on each audio frame to obtain a frequency domain representation. Based on the frequency domain representation, Mel-Frequency Cepstral Coefficients (MFCCs) or other acoustic features are calculated to extract acoustic features to represent the spectral features of the audio data samples. The spectral features are then encoded using an encoding network to obtain the first encoded feature. For example, the encoding network can be built based on a Conformer Encoder, which includes components such as convolutional layers, self-attention layers, and feedforward network layers to process and transform spectral features to obtain the first encoded feature. The embodiments of this application do not limit the specific number of layers or the specific network structure of the encoding network.

[0076] In step 102, the first coding feature is biased and mapped based on preset words through the first bias network in the audio recognition model to obtain the second coding feature.

[0077] In some embodiments, the number of coding networks is multiple, which may include a head coding network, a middle coding network, and a tail coding network. The head coding network refers to the coding network located in the first layer interval of the audio recognition model; the middle coding network refers to the coding network located in the second layer interval of the audio recognition model; and the tail coding network refers to the coding network located in the third layer interval of the audio recognition model. The maximum layer number in the first layer interval is less than the minimum layer number in the second layer interval, and the maximum layer number in the first layer interval and the minimum layer number in the second layer interval are consecutive positive integers. The maximum layer number in the second layer interval is less than the minimum layer number in the third layer interval, and the maximum layer number in the second layer interval and the minimum layer number in the third layer interval are consecutive positive integers.

[0078] Here, the layer number refers to the sequential numbering of the encoding networks in the audio recognition model from input to output.

[0079] For example, see Figure 4 , Figure 4This is a schematic diagram illustrating the training principle of the audio recognition model provided in this application embodiment. Taking a total of 12 layers of multiple coding networks as an example, the multiple coding networks may include a head coding network (for example, the head coding network may be any coding network from Encoder Layer 1 to Encoder Layer 6 (corresponding to the first layer range), a middle coding network (for example, the middle coding network may be any coding network from Encoder Layer 7 to Encoder Layer 9 (corresponding to the second layer range), and a tail coding network (for example, the tail coding network includes any coding network from Encoder Layer 10 to Encoder Layer 12 (corresponding to the third layer range)).

[0080] Here, the preset words refer to words pre-set for specific fields or application scenarios, such as proper nouns in the financial field, high-frequency words, hot words, and business terms in financial scenarios (such as product names).

[0081] Following the previous example, see [link / reference]. Figure 3B , Figure 3A Step 102 shown can be implemented through steps 1021 to 1022, which will be explained in detail below.

[0082] In step 1021, for the first encoding network among multiple encoding networks, if the first encoding network is the head encoding network among multiple encoding networks, attention processing is performed on the preset words and the first encoding features to obtain the first attention features, and the first encoding features and the first attention features are fused to obtain the second encoding features.

[0083] In some embodiments, attention processing is performed on preset words and first encoding features to obtain first attention features, which can be achieved by: encoding preset words to obtain word features; and performing attention encoding processing on word features and first encoding features to obtain first attention features.

[0084] For example, see Figure 6 , Figure 6 This is a schematic diagram of the bias network structure provided in the embodiments of this application, based on the first encoding feature and the word features of preset words (through...). Figure 5 The text encoder shown encodes preset words and performs attention processing to obtain the first attention feature. The first encoding feature and the first attention feature are then fused (concatenated) and input into a linear layer. The second encoding feature is then output through the linear mapping of the linear layer.

[0085] For example, encoding preset words to obtain word features can be achieved in the following way: perform token embeddings on the preset words to obtain token embedding features, and then encode the token embedding features to obtain word features.

[0086] For example, word embedding models (such as Word2Vec, GloVe, etc.) can be used to encode the preset words, representing each preset word as a fixed-length vector, and then forming word embedding features (sequences) in order; word features can be obtained by encoding the word embedding features using an LSTM network.

[0087] For example, the first attention feature can be obtained by performing attention encoding on word features and first encoding features. This can be achieved by: determining the attention score matrix based on word features and first encoding features; and then weighting the word features using the attention score matrix to obtain the first attention feature.

[0088] For example, the preset words include multiple words (e.g., represented as p0, p1, ..., pN), and the first encoded feature (corresponding to...) Figure 6 The one shown Let the first encoded feature corresponding to the audio frame at time step t be used as the query vector (Query, Q), and let the word features of multiple words (corresponding to) be used as the query vector. Figure 6 The one shown The i-th word (representing the word feature) is used as both a key vector (Key, K) and a value vector (Value, V). The attention score of each word in multiple words is obtained by calculating the dot product of Q and K. Next, the attention scores are normalized, for example by applying a normalization function (such as the softmax function), so that the attention score of each word in multiple words becomes a probability distribution, which is used as the attention score matrix.

[0089] Continuing with the previous example, the word features (corresponding to V above) are weighted using an attention score matrix to obtain the first attention feature (corresponding to V above). Figure 6 The one shown Let represent the attention feature corresponding to the audio frame at time step t), where the first attention feature includes the attention feature corresponding to each audio frame in the audio data sample.

[0090] In some embodiments, fusing the first encoding feature and the first attention feature to obtain the second encoding feature can be achieved by concatenating the first attention feature and the first encoding feature, and then performing a linear transformation on the concatenated feature representation through a linear layer to obtain the second encoding feature (corresponding to...). Figure 6 The one shown (This represents the second encoded feature corresponding to the audio frame at time step t).

[0091] In other embodiments, the first attention feature and the first encoding feature can be weighted and summed using preset weight coefficients to obtain the second encoding feature. This application does not limit the specific implementation of fusing the first attention feature and the first encoding feature.

[0092] In step 1022, if the first encoding network is any encoding network after the head encoding network, then the first word is selected from the preset words, attention processing is performed on the first word and the first encoding feature to obtain the second attention feature, and the first encoding feature and the second attention feature are fused to obtain the second encoding feature.

[0093] like Figure 4 As shown, any coding network after the head coding network (such as the 1st to 6th Encoder Layers of the encoder) refers to any one of the middle coding networks (such as the 7th to 9th Encoder Layers) and the tail coding network (such as the 10th to 12th Encoder Layers).

[0094] In some embodiments, the preset words include multiple words; the first attention feature includes the attention feature corresponding to each audio frame in the audio data sample; see also Figure 3C , Figure 3B The step 1022 shown, in which the first word is selected from the preset words, can be achieved through the following steps 10221 to 10223, which are explained in detail below.

[0095] In step 10221, for each audio frame, an attention score matrix corresponding to the attention feature is determined, wherein the attention score matrix includes the attention score of each word in multiple words.

[0096] Following the example in step 1021, for each audio frame, the first encoded feature output by the head encoding network is used as the query vector (Q), and the word features of multiple words are used as both the key vector (K) and the value vector (V). The attention score of each word in the multiple words is obtained by calculating the dot product of Q and K.

[0097] In step 10222, a preset number of words are determined from multiple words in descending order of attention scores, and multiple second words are determined for each audio frame.

[0098] For example, for each audio frame, the K (preset number) words with the highest attention scores among the preset words are obtained as multiple second words corresponding to the audio frame.

[0099] In step 10223, the second words corresponding to multiple audio frames are formed into a set, the frequency of each second word in the set is counted, and the second word whose frequency of occurrence is greater than a preset threshold is determined as the first word.

[0100] For example, suppose the second words (3 words) corresponding to multiple (3 frames) audio frames are represented as {(frame 1: word c, word d, word b), (frame 2: word c, word a, word b), (frame 3: word e, word d, word b)}, then the set can be represented as {word c, word d, word b, word c, word a, word b, word e, word d, word b}. The second word that appears more than a preset threshold (e.g., 2 times) is determined as the first word, then the first word is {word c, word d, word b}.

[0101] Through steps 10221 to 10223, a two-stage screening strategy using attention mechanism and frequency statistics is implemented to select the first word from a massive number of preset words. First, the local correlation between audio segments and words is quantified by frame-level attention scores to select the Top-K candidates for each frame (avoiding misjudgment in a single frame). Next, through cross-frame word frequency statistics, words with strong semantic association (the first word) are retained, which greatly reduces redundant calculations (only the first word is processed instead of all preset words).

[0102] See also Figure 3A In step 103, the second encoded feature is decoded through the decoding network in the audio recognition model to obtain the first text probability distribution.

[0103] In some embodiments, the second encoded feature is decoded by a decoding network to obtain the first text probability distribution. For example, the decoding network can be constructed using a Transformer Decoder structure. Here, decoding can be performed by stacking multiple Transformer Decoder structures. The embodiments of this application do not limit the specific number of layers in the decoding network.

[0104] For example, the second encoded features can be mapped to the dimension of the vocabulary size through a fully connected (Dense Layer), and the linearly transformed output can be converted into the first text probability distribution through the Softmax function.

[0105] Here, the first text probability distribution represents the probability value of all possible text units (such as characters, sub-words, words, etc.) given the second encoding feature of the input.

[0106] For example, the first text probability distribution can be represented as a vector of length K, where K is the vocabulary size and each element corresponds to the probability of a text unit. If the vocabulary is [a, b, c, d, e], then the first text probability distribution at time step t may be: [0.7, 0.1, 0.05, 0.1, 0.05], indicating that the character "a" is most likely to be output at time step t (probability 70%).

[0107] In step 104, the second text probability distribution is obtained by bias mapping the first text probability distribution based on preset words through the second bias network in the audio recognition model.

[0108] In some embodiments, based on a first text probability distribution and preset word features (via... Figure 5 The text encoder shown encodes preset words and performs attention processing to obtain attention features. The first text probability distribution and attention features are then fused (concatenated) and input into a linear layer. The second text probability distribution is then output through the linear mapping of the linear layer.

[0109] For a detailed explanation of the implementation of bias mapping, please refer to step 102 above. It will not be repeated here.

[0110] In step 105, the loss value is determined based on the text annotation of the audio data sample, the first encoding feature, the first text probability distribution, and the second text probability distribution.

[0111] For example, see Figure 4 The loss values ​​include the loss values ​​on the encoding side and the loss values ​​on the decoding side. The loss values ​​on the encoding side include the first loss value, the second loss value, and the third loss value, while the loss values ​​on the decoding side include the fourth loss value and the fifth loss value.

[0112] In some embodiments, the number of encoding networks is multiple; see [link to documentation]. Figure 3D , Figure 3A Step 105 shown can be implemented through steps 1051 to 1055, which will be explained in detail below.

[0113] In step 1051, for the first coding network among multiple coding networks, if the first coding network is the head coding network among multiple coding networks, the first loss value is determined based on the first coding features and text annotations output by the head coding network.

[0114] For example, see Figure 4 The first loss value corresponding to the head encoding network includes the first sub-loss value (L ctc_bias_6 ) and second sub-loss value (L ctc_mid_6 ).

[0115] In some embodiments, see Figure 3E , Figure 3D Step 1051 shown can be implemented through steps 10511 to 10514, which will be explained in detail below.

[0116] In step 10511, a first sub-loss value is determined based on the first encoded features and word annotations output by the head encoding network, wherein the word annotations include words in the text annotations that match preset words.

[0117] In some embodiments, attention processing is performed on the preset words and the first encoding features output by the head encoding network to obtain attention features (see the description of step 1021 above). Based on the attention features and word annotations, the first sub-loss value is determined.

[0118] For example, see Figure 5 , Figure 5 This is a schematic diagram illustrating the calculation principle of the first sub-loss value provided in the embodiments of this application. The preset words are encoded by a text encoder (e.g., an LSTM network) to obtain word features. Attention features are obtained by performing attention processing (implemented through a first bias network) on the word features and the first encoded features. The first sub-loss value is determined by a word prediction network based on the attention features and word annotations.

[0119] For example, see Figure 7 , Figure 7 This is a schematic diagram of the word prediction network provided in the embodiment of this application. The attention features are linearly transformed by the linear layer in the word prediction network to obtain the linearly transformed features. The first sub-loss value is determined based on the linearly transformed features and word annotations by connecting the temporal classification linear layer (CTC Linear).

[0120] Here, word tagging includes words in the text tagging that match preset words. For example, if the text tagging is "I want to query **flower repayment information", and the preset words also include "**flower", then the word tagging is "**flower".

[0121] Continuing with the previous example, based on the features after linear transformation and word annotations, the first sub-loss value can be represented by formula (1):

[0122] L ctc_bias_6 =-logP CTC (y P |x A1 (1)

[0123] Among them, y P Indicates word tagging, x A1 P represents the feature after linear transformation. CTCIndicates that given x A1 Under the condition of predicting word tagging y P The probability, this probability is obtained through the CTC decoder (such as... Figure 7 The loss is calculated using the Connected Temporal Classification Linear Layer (CT CLinear) in the model. The -log represents taking the negative logarithm of the probability, so that when the probability is close to 0, the loss value becomes very large, which helps to impose greater weight on mispredictions during backpropagation.

[0124] It should be noted that L shown in formula (1) ctc_bias_6 The calculation method is only an example of one implementation method. Other loss calculation methods, such as CTC Spike Reduce, can also be used to reduce the number of spikes.

[0125] In step 10512, the second sub-loss value is determined based on the first encoded features and text annotations output by the head encoding network.

[0126] In some embodiments, a second sub-loss value is calculated based on the first encoded features and text annotations output by the head encoding network using a preset loss function. For example, the second sub-loss value L... ctc_mid_6 It can be expressed by formula (2):

[0127]

[0128] in, Indicates text annotation, P represents the first encoded feature output by the head encoding network. CTC Indicates that in a given Under the condition of predicting the sequence The probability is calculated by the CTC decoder.

[0129] In step 10513, the first sub-loss value and the second sub-loss value are fused to obtain the first loss value.

[0130] In some embodiments, the first sub-loss value and the second sub-loss value are weighted and summed according to preset weights to obtain the first loss value.

[0131] See also Figure 3D In step 1052, if the first coding network is the middle coding network among multiple coding networks, then the second loss value is determined based on the first coding feature, the first text probability distribution and the text annotation output by the middle coding network.

[0132] For example, see Figure 4 The second loss value corresponding to the middle coding network includes the third sub-loss value (L). ctc_bias_9 ), fourth sub-loss value (Lctc_pre_9 ) and the fifth sub-loss value (L sha_pre_dec_9 ).

[0133] In some embodiments, see Figure 3F , Figure 3D Step 1052 shown can be implemented through steps 10521 to 10525, which will be explained in detail below.

[0134] In step 10521, a third sub-loss value is determined based on the first coding features and word annotations output by the central coding network, wherein the word annotations are words that match the text annotations with preset words.

[0135] In some embodiments, attention processing is performed on preset words and the first encoded features output by the middle coding network to obtain attention features. Based on the attention features and word annotations, a third sub-loss value (L) is determined. ctc_bias_9 ).

[0136] For specific implementation details, please refer to the description in step 10511 above, which will not be repeated here.

[0137] In step 10522, the first coding feature output by the middle coding network and the first text probability distribution are fused to obtain the third coding feature, and the fourth sub-loss value is determined based on the third coding feature and text annotation.

[0138] In some embodiments, the first encoded feature output by the middle coding network and the first text probability distribution are concatenated (Concatenate), and the concatenated feature is linearly mapped (which can be achieved through a linear layer) to obtain the third encoded feature. Then, based on the third encoded feature and text annotations, a second sub-loss value is calculated using a preset loss function. For example, a fourth sub-loss value L is... ctc_pre_9 It can be expressed by formula (3):

[0139]

[0140] in, Indicates text annotation, x E9 P represents the third coding feature. CTC Indicates that given x E9 Under the condition of predicting the sequence The probability is calculated by the CTC decoder.

[0141] In step 10523, the first encoded feature output by the middle encoding network is decoded through the decoding network in the audio recognition model to obtain the third text probability distribution, and the fifth sub-loss value is determined based on the third text probability distribution and text annotation.

[0142] In some embodiments, the first encoded features output by the middle encoding network are decoded by the decoding network to obtain the third text probability distribution. For example, the decoding network can be constructed using a Transformer Decoder structure. Here, decoding can be performed by stacking multiple Transformer Decoder structures. The embodiments of this application do not limit the specific number of layers of the decoding network.

[0143] For example, the first encoded features output by the middle encoding network can be mapped to the dimension of the vocabulary size through a fully connected layer (Dense Layer), and the linearly transformed output can be converted into a third text probability distribution through the Softmax function.

[0144] In some embodiments, a fifth sub-loss value is determined based on a third text probability distribution and text annotation using a pre-set loss function. For example, the fifth sub-loss value L... sha_pre_dec_9 It can be expressed by formula (4):

[0145]

[0146] Where label_smoothing_loss(·) represents the label-smoothing cross-entropy loss. This represents the first encoded feature output by the middle coding network. This represents the probability distribution of the third text. This indicates a text annotation.

[0147] For example, given the third probability distribution for each time step (the third text probability distribution includes the probability distribution for each time step), for instance, the probability of each character in the vocabulary (e.g., 4 characters a, b, c, d) output at time step 1 is {0.2, 0.3, 0.4, 0.1}. The text label corresponding to the current time step 1 is "b", so the probability distribution of the text label is {0, 1, 0, 0}. Calculate the label smoothing cross-entropy loss between {0.2, 0.3, 0.4, 0.1} and {0, 1, 0, 0}. And so on, calculate the loss corresponding to all time steps, and use the average loss as the fifth sub-loss value.

[0148] In step 10524, the third sub-loss value, the fourth sub-loss value, and the fifth sub-loss value are fused to obtain the second loss value.

[0149] In some embodiments, the third sub-loss value, the fourth sub-loss value, and the fifth sub-loss value are weighted and summed according to preset weights to obtain the second loss value.

[0150] Through steps 10521 to 10524, the first text probability distribution is integrated into the loss calculation of the intermediate coding network, and the prediction results of the decoding stage are injected back into the encoding process, which promotes bidirectional information interaction between the encoder and decoder and enhances the model's ability to model context dependencies.

[0151] See also Figure 3D In step 1053, if the first coding network is the tail coding network among multiple coding networks, then the third loss value is determined based on the first coding feature output by the tail coding network, the text annotation, and the second coding feature obtained by bias mapping of the first coding feature output by the tail coding network.

[0152] In some embodiments, a sixth sub-loss value is determined based on the first coding features output by the tail coding network and word annotations, wherein the word annotations include words in the text annotations that match preset words (see the description of step 10511 above); a seventh sub-loss value is determined based on the second coding features obtained by mapping the first coding features output by the tail coding network and the text annotations (see the description of step 10512 above, which corresponds to calculating the predicted sequence under the given condition of the second coding features obtained by mapping the first coding features output by the tail coding network). The probability of the loss is calculated by fusing the sixth and seventh sub-loss values ​​to obtain the third loss value (see the explanation of step 10513 above).

[0153] In step 1054, the loss value on the encoding side is determined based on the first loss value, the second loss value, and the third loss value, and the loss value on the decoding side is determined based on the first text probability distribution, the second text probability distribution, and the text annotation.

[0154] In some embodiments, the loss value on the encoding side can be expressed as formula (5):

[0155] L enc_total =λL bias_enc +L enc (5)

[0156] Where λ represents the pre-set weighting coefficient.

[0157] Continuing from the previous example, L bias_enc It can be expressed by formula (6):

[0158] L bias_enc =αL ctc_bias_6 +αL ctc_bias_9 +(1-α)L ctc_bias_12 (6)

[0159] Where α represents the pre-set weighting coefficient, L ctc_bias_6L represents the first sub-loss value in the first loss value. ctc_bias_9 L represents the third sub-loss value in the second loss value. ctc_bias_12 This represents the sixth sub-loss value in the third loss value.

[0160] Continuing from the previous example, L enc It can be expressed by formula (7):

[0161] L enc =L ctc_mid_6 +L ctc_pre_9 +L sha_pre_dec_9 +L ctc_enc_12 (7)

[0162] Among them, L ctc_mid_6 L represents the second sub-loss value in the first loss value. ctc_pre_9 L represents the fourth sub-loss value in the second loss value. sha_pre_dec_9 L represents the fifth sub-loss value in the second loss value. ctc_enc_12 This represents the seventh sub-loss value in the third loss value.

[0163] It should be noted that formulas (5) to (7) are merely examples of one implementation of determining the loss value on the encoding side, and the embodiments of this application do not limit the specific loss value calculation method.

[0164] In some embodiments, the loss value on the decoding side is determined based on the first text probability distribution, the second text probability distribution, and the text annotation. This can be achieved by: determining a fourth loss value based on the first text probability distribution and the text annotation; determining a fifth loss value based on the second text probability distribution and the text annotation; and fusing the fourth loss value and the fifth loss value to obtain the loss value on the decoding side.

[0165] For example, see Figure 4 The loss value on the decoding side includes the fourth loss value and the fifth loss value, and the loss value on the decoding side can be expressed by formula (8):

[0166] L dec_total =αL bias_dec +L dec (8)

[0167] Among them, L bias_dec L represents the fourth loss value. dec This represents the fifth loss value, and α represents the pre-set weighting coefficient.

[0168] Continuing from the previous example, L dec It can be expressed by formula (9):

[0169]

[0170] in, This represents the probability distribution of the first text. This represents the text annotation, and label_smoothing_loss(·) represents the label smoothing cross-entropy loss.

[0171] Continuing from the previous example, L bias_dec It can be expressed by formula (10):

[0172]

[0173] in, This represents the probability distribution of the second text. This represents the text annotation, and label_smoothing_loss(·) represents the label smoothing cross-entropy loss.

[0174] It should be noted that formulas (8) to (10) are merely examples of one way to determine the loss value on the decoding side, and the embodiments of this application do not limit the specific method of calculating the loss value.

[0175] In step 1055, the loss value from the encoding side and the loss value from the decoding side are fused to obtain the loss value.

[0176] In some embodiments, the loss value on the encoding side and the loss value on the decoding side are weighted and summed according to preset weighting coefficients to obtain the loss value.

[0177] Steps 1051 to 1054 achieve multi-level joint loss calculation. Specifically, by designing differentiated loss calculation strategies for the head, middle, and tail encoding networks (such as emphasizing basic feature alignment in the head / tail and introducing the text probability distribution on the decoding side as a supervision signal in the middle), targeted optimization of features at different levels of the encoding network is achieved, ensuring that shallow networks capture acoustic details and deep networks focus on semantic context, ultimately achieving the beneficial effect of improving the accuracy of audio recognition.

[0178] See also Figure 3A In step 106, the parameters of the audio recognition model are updated based on the loss value to obtain the trained audio recognition model.

[0179] In some embodiments, gradient information is obtained through the loss value, and the parameters of each network in the audio recognition model are updated according to the gradient information to obtain the trained audio recognition model.

[0180] For example, the gradient information of the loss value with respect to each parameter of the audio recognition model can be obtained through the backpropagation algorithm. The parameters of the audio recognition model can be updated using the obtained gradient information according to the gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.). The above process is repeated until a certain number of iterations are reached or the audio recognition model converges, thereby obtaining the trained audio recognition model.

[0181] Through steps 101 to 106, a first bias network is introduced into the audio recognition model. Based on preset words, the first encoded features obtained by encoding processing are biased and mapped, thereby guiding the audio recognition model to focus on specific words (i.e., preset words). By introducing a second bias network, the first text probability distribution obtained after decoding is biased and mapped, further optimizing the recognition of preset words. Furthermore, by combining the first encoded features, the first text distribution probability, the second text distribution probability, and the loss value design of multiple supervision signals of text annotation, the audio recognition model has stronger semantic focusing ability and robustness, thereby improving the accuracy of audio recognition.

[0182] In some embodiments, after obtaining the trained audio recognition model, audio recognition can be performed on the audio data to be recognized based on the trained audio recognition model to obtain the first text corresponding to the audio data to be recognized.

[0183] In some embodiments, see Figure 3G The audio recognition model trained on the audio data to be recognized is used to perform audio recognition and obtain the first text corresponding to the audio data to be recognized. This can be achieved through the following steps 201 to 206, which are explained in detail below.

[0184] In step 201, the audio data to be recognized is encoded using the encoding network in the trained audio recognition model to obtain the third encoded feature.

[0185] In some embodiments, the audio data to be identified is segmented into frames to obtain audio frames; the audio frames are transformed in the frequency domain to obtain a frequency domain representation of the audio frames; the frequency domain representation is subjected to acoustic feature extraction to obtain spectral features; and the spectral features are subjected to feature encoding to obtain a third encoded feature.

[0186] For specific implementation details, please refer to the description of step 101 above, which will not be repeated here.

[0187] In step 202, the third coding feature is biased and mapped based on preset words through the first bias network in the trained audio recognition model to obtain the fourth coding feature.

[0188] In some embodiments, there are multiple encoding networks. For the first encoding network among the multiple encoding networks, if the first encoding network is the head encoding network among the multiple encoding networks, attention processing is performed on the preset words and the third encoding feature to obtain the third attention feature, and the third encoding feature and the third attention feature are fused to obtain the fourth encoding feature (see the description in step 1021 above); if the first encoding network is any encoding network after the head encoding network, the first word is selected from the preset words, attention processing is performed on the first word and the third encoding feature to obtain the fourth attention feature, and the third encoding feature and the fourth attention feature are fused to obtain the fourth encoding feature (see the description in step 1022 above).

[0189] In step 203, the fourth encoded feature is decoded through the decoding network in the trained audio recognition model to obtain the fourth text probability distribution.

[0190] In some embodiments, the fourth encoded feature is decoded by a decoding network to obtain the fourth text probability distribution. For example, the decoding network can be constructed using a Transformer Decoder structure. Here, decoding can be performed by stacking multiple Transformer Decoder structures. The embodiments of this application do not limit the specific number of layers in the decoding network.

[0191] For example, the fourth encoding feature can be mapped to the dimension of the vocabulary size through a fully connected (Dense Layer), and the linearly transformed output can be converted into a fourth text probability distribution through the Softmax function.

[0192] Here, the fourth text probability distribution represents the probability value of all possible text units (such as characters, sub-words, words, etc.) given the fourth encoding feature of the input.

[0193] In step 204, the second bias network in the trained audio recognition model performs bias mapping on the fourth text probability distribution based on preset words to obtain the fifth text probability distribution.

[0194] In some embodiments, based on a fourth text probability distribution and preset word features (via... Figure 5 The text encoder shown encodes preset words and performs attention processing to obtain attention features. The fourth text probability distribution and attention features are then fused (concatenated) and input into a linear layer. The fifth text probability distribution is then output through the linear mapping of the linear layer.

[0195] For a detailed explanation of the implementation of bias mapping, please refer to step 102 above. It will not be repeated here.

[0196] In step 205, multiple second texts are generated based on the probability distribution of the fifth text.

[0197] In some embodiments, a fixed-size (bundle width) set of candidate text sequences is maintained by beam search. At each time step, all candidate text sequences are expanded according to the probability distribution of the fifth text, and their cumulative probability (or the sum of log probabilities) is calculated. The top k candidate text sequences with the highest cumulative probability are retained. Through layer-by-layer expansion and dynamic pruning, multiple high-probability candidate text sequences are finally output as the second text.

[0198] In step 206, a first text is selected from multiple second texts based on the first evaluation score of each second text.

[0199] In some embodiments, the second text with the highest first evaluation score among a plurality of second texts is used as the first text.

[0200] Through steps 201 to 206, the probability distribution of the fourth text is fused with attention based on preset words through the second bias network. The probability distribution is dynamically adjusted using the word features generated by the text encoder, which significantly enhances the model's sensitivity to the preset words. Multiple candidate sequences are maintained through beam search to avoid the local optimum problem caused by single greedy decoding, thereby improving the semantic rationality and domain adaptability of the output text. Candidate texts are screened based on the first evaluation score, which effectively balances recognition accuracy and semantic fluency and reduces the risk of misidentification or logical errors.

[0201] In some embodiments, see Figure 3H Before selecting the first text from multiple second texts based on the first evaluation score of each second text, the following steps 301 to 304 can also be performed, which are explained in detail below.

[0202] In step 301, for each second text, the alignment between the second text and the audio data to be identified is scored based on the second text and the fourth coding features to determine a second evaluation score.

[0203] In some embodiments, the second evaluation score S ctc It can be expressed by formula (11):

[0204]

[0205] Where xt represents the fourth encoded feature corresponding to the t-th audio frame in the audio data to be identified, and yt represents the label corresponding to the t-th frame of the second text in the CTC alignment path.

[0206] Here, S ctcThe score is used to evaluate whether each word in the second text (e.g., "hello") matches a second encoded feature for the corresponding time period in the audio data to be recognized. For example, if frames 5-10 of the audio data to be recognized contain the pronunciation of "you," and "you" appears in the 3rd position in the second text, S... ctc It will calculate how well these frames match "you".

[0207] For example, the second text (e.g., "Hello") is expanded into possible alignment paths (e.g., "you-you-empty-good-good") according to the CTC rule. For each time step, the probability P(yt|xt) is calculated, and the probabilities of all frames are logarithmic (to avoid numerical underflow) and summed to obtain the final S. ctc Scoring. For example, if the second encoded features corresponding to 5 audio frames are represented as x1, x2, x3, x4, x5, then S ctc The rating can be expressed as: logP(you|x1)+logP(you|x2)+logP(empty|x3)+logP(good|x4)+logP(good|x5).

[0208] In step 302, for each second text, the semantic coherence of the second text is scored based on the second text and the fourth encoding features to determine the third evaluation score.

[0209] In some embodiments, the third evaluation score S dec It can be expressed by formula (12):

[0210]

[0211] Where wi represents the i-th word in the second text, w 1:i-1 This represents the first i-1 words that have been generated, and Henc represents the fourth encoding feature.

[0212] Here, S dec The scoring is used to assess whether the second text is fluent, logical, and conforms to language conventions. For example, the second text's phrase "collecting applications" received an S grade because the combination of "collection" and "application" felt awkward. dec The rating is low, and "mobile app" is more in line with everyday language. dec Higher rating.

[0213] For example, suppose the second text is "Please click on the mobile application". When generating this second text word by word, the probability of generating "please" can be represented as P(please|empty, Henc), the probability of generating "dot" can be represented as P(dot|please, Henc), the probability of generating "click" can be represented as P(click|please dot, Henc), and so on, to obtain the probability corresponding to each word. After taking the logarithm of the probabilities of all words, sum them up to get S. dec score.

[0214] In step 303, for each second text, the relevance between the second text and the preset words is scored based on the second text and preset words to determine the fourth evaluation score.

[0215] In some embodiments, the fourth evaluation score S bias_dec It can be expressed by formula (13):

[0216] S bias_dec =∑ k∈Bias_ PhraseslogPbias(k|Hdec) (13)

[0217] Where k represents a word in the preset vocabulary, Bias_Phrases represents the preset vocabulary, Hdec represents the probability distribution of the fifth text, and Pbias(k|Hdec) represents the probability of k appearing in the second text. This probability value can be calculated by CPPNetwork.

[0218] Here, S bias_dec The scoring is used to check whether the second text contains preset words (such as "**flower") and to "add points" for the presence of these words. For example, if the preset words include "**flower", the appearance of "**flower" in the second text will significantly improve the Score of this second text. bias_dec score.

[0219] In step 304, the second evaluation score, the third evaluation score, and the fourth evaluation score are combined to determine the first evaluation score.

[0220] In some embodiments, the first evaluation score S final It can be expressed by formula (14):

[0221]

[0222] Among them, S ctc S represents the second evaluation score. dec S represents the third evaluation score. bias_dec This indicates the fourth evaluation score.

[0223] It should be noted that S can also be calculated using the method shown in the Unified Two-pass Bidirectional End-to-end Model for Speech Recognition (U2++). ctc Rating and S dec The scoring is not limited in this application embodiment.

[0224] Through steps 301 to 304, a multi-dimensional joint optimization text filtering mechanism was implemented, achieving a beneficial effect of balancing speech alignment accuracy, semantic rationality, and adaptability to business needs. Specifically, through S... ctc The scoring assesses the temporal alignment between the second text and the audio frame, avoiding local misalignment issues caused by the global dependency of the attention mechanism; through S... dec Scoring incorporates prior knowledge from the language model to filter text with grammatical errors or semantic contradictions; through S... bias_dec The scoring system incorporates the weights of pre-defined words (such as "flower") into the scoring process, improving the adaptability of audio recognition to business needs.

[0225] Significant progress has been made in neural network-based end-to-end speech recognition systems. However, recognition accuracy drops significantly for rare words or contextual terms (e.g., proper nouns) not included in the training dataset. Previous research has attempted to improve the performance of Automatic Speech Recognition (ASR) on uncommon phrases through language model (LM) fusion. While this approach improves the recognition of relevant words, improvements to the bias list remain limited. Although language model fusion can optimize contextual output, it does not significantly improve overall performance, especially when dealing with phrases outside the training set.

[0226] Related techniques have proposed deep bias methods based on neural networks, which optimize the prediction of specific phrases by integrating lexical knowledge from the training set into the speech recognition network. However, these methods typically only append bias vectors to the last layer of the network. Many studies have attempted to inject deep biases into intermediate layers of the encoder. Among these, some schemes only utilize the outputs of these intermediate encoder layers as context queries, and the final generated context embeddings are still integrated with the final output of the encoder in the last layer. Although some methods have explored strategies for injecting contextual information into earlier layers, they have not attempted to introduce any additional shared text supervision loss beyond the bias loss.

[0227] The training method for the audio recognition model provided in this application has the following advantages:

[0228] 1) Shared intermediate encoder bias network (corresponding to the first bias network mentioned above)

[0229] This application provides a shared intermediate encoder bias network. By introducing a neural bias cross-attention mechanism in layers 6, 9, and 12 of the SEQ-former encoder (corresponding to the encoding network above) and combining it with a multi-level loss supervision method, it achieves multi-level bias enhancement and parameter sharing supervision for audio embedding. This effectively utilizes SEQ-former prior parameters, strengthens the role of the bias module, and optimizes multi-level embedding information sharing.

[0230] 2) Biased decoder network with shared attention loss (corresponding to the second biased network above)

[0231] This application provides a biased decoder network with shared attention loss. By integrating the biased decoder network into the decoder (corresponding to the decoding network mentioned above) and introducing a shared attention loss method, it achieves efficient modeling of semantic context and enhanced attention to phrase information, thereby improving phrase recognition accuracy and optimizing decoder prediction performance. Furthermore, by combining a comprehensive strategy of CTC decoding score, decoder score, and biased decoder score, the accuracy of the final text prediction is further improved.

[0232] 3) Shared bias attention phrase filtering (corresponding to the above-mentioned filtering from preset words to obtain the first word)

[0233] This application provides a shared bias attention phrase filtering method, which achieves efficient filtering and enhanced attention concentration on large-scale phrase lists, thereby reducing performance degradation, improving recognition accuracy, and accelerating decoding. Furthermore, by replacing the multi-head attention mechanism (MHA) with a multi-head latent attention mechanism (MLA), support for ultra-long inputs and processing efficiency are further enhanced.

[0234] The training method of the audio recognition model provided in this application significantly improves the performance and generalization ability of the audio recognition system, enabling it to maintain high accuracy and efficiency in different fields and complex scenarios.

[0235] The following describes the specific implementation of the training method for the audio recognition model provided in the embodiments of this application.

[0236] 1) Shared intermediate encoder bias network

[0237] like Figure 4As shown, for the 12-layer SEQ-former encoder network (corresponding to the encoding network above), the outputs are extracted from layers 6, 9, and 12, and after layer normalization, the audio embedding is obtained. (corresponding to the first encoded feature output by the head encoding network above) (corresponding to the first encoded feature output by the coding network in the middle of the text above) and

[0238] (Corresponding to the first encoded feature output by the tail encoding network above). These embeddings are respectively related to P e (Combined with the word features corresponding to the pre-set words above), through the neural bias cross-attention mechanism (i.e.) Figure 6 After attention processing, element-wise addition is performed to form a biased audio embedding. (i.e., the second coding feature obtained by bias mapping the first coding feature output by the head coding network) (i.e., the first encoded feature output from the middle coding network is biased and mapped)

[0239] The second encoded feature obtained by the shooting) and (That is, the second encoded feature obtained by bias mapping the first encoded feature output by the tail-coding network). Wherein... The neural bias cross-attention mechanism used can be a multi-head potential attention mechanism or a multi-head attention mechanism, etc. The embodiments of this application do not limit the specific attention mechanism used.

[0240] During training, encoder bias enhancement loss is used to enhance the model's supervision of the bias module, preventing the audio recognition model from ignoring the influence of the bias module parameters. Furthermore, all loss terms in the SEQ-former are retained, which serve as shared supervision for the bias network after the bias is added. The total encoder loss (corresponding to the encoding-side loss value mentioned above) can be expressed by formula (15):

[0241] L enc_total =λL bias_enc +L SEQ-former_enc_total (15)

[0242] Among them, L SEQ-former_enc_total This represents the total encoder loss in SEQ-former, including intermediate layer loss L. ctc_mid_6 CTC loss L ctc_enc_12 The predictive decoder network supervision loss L ctc_pre_9 And shared predictive decoder network loss L sha_pre_dec_9 L bias_enc Including the loss of biased audio embedding in layers 6, 9, and 12, L bias_encThe calculation method can be found in the explanation of formula (6) above. Here, all terms ending with a number represent the output of the corresponding coding network layer.

[0243] Following the example above, L ctc_mid_6 The calculation method can be found in the explanation of formula (2) above. L ctc_pre_9 The calculation method can be found in the explanation of formula (3) above. L sha_pre_dec_9 The calculation method can be found in the explanation of formula (4) above, L ctc_enc_12 The calculation method for (corresponding to the seventh sub-loss value above) can be found in the explanation of step 1053 above.

[0244] Due to loss sharing and decoder parameter sharing, the bias network effectively utilizes the pre-shared parameters in the SEQ-former and maintains supervision through shared loss, thereby enhancing multi-level embedding information in the form of shared supervision. 2) Biased decoder network with shared attention loss

[0245] Accurate phrase recognition (i.e., recognizing predefined words in the preceding text) requires combining both acoustic and semantic context. Guided by attention loss, the decoder can efficiently model the semantic context. Therefore, by sharing attention loss, the bias decoder network (corresponding to the second bias network mentioned above) is integrated into the decoder (corresponding to the decoding network mentioned above), as follows: Figure 4 The second bias network shown in the figure has a learning objective of transcription tags (corresponding to the text annotations above), and the tag smoothing cross-entropy loss is used as the loss function to achieve accurate phrase recognition based on the semantic context of the decoder. The loss function of the bias decoder can be expressed by formula (10) above, where, This indicates the audio transcription content (text annotation) of the current batch. This represents the weighted sum of the attention mechanism of the bias decoder (i.e., the attention features obtained after inputting the output of the decoder (such as the Transformer Decoder) into the bias decoder (the second bias network), where N is the number of transcription tags in that batch. All phrases in each batch are consistent with the phrases in the encoder. This loss design aims to enhance phrase information in semantic modeling, making the decoder's predictions more focused on bias information. The learning objective is the final text, with the goal of improving prediction accuracy by introducing phrase bias.

[0246] The total loss of the decoder (corresponding to the loss value on the decoding side above) can be found in the explanation of formula (8) above.

[0247] It should be noted that during the inference phase of the audio recognition model, the CPP network part in the decoder is not discarded (the CPP network connected to the encoder is only used for the training phase). Instead, the posterior probability score output by the CPP network (corresponding to Pbias(k|Hdec) in formula (13)) is used to integrate the phrase decoding into the attention rescoring mechanism and introduce adjustable weights to obtain the final score (corresponding to the first evaluation score above). The formula for calculating the final score can be found in the explanation of formula (14) above.

[0248] 3) Shared bias attention phrase filtering

[0249] When dealing with large lists of phrases, bias enhancement strategies in related technologies often suffer from performance degradation due to distraction. Thanks to the attention map (corresponding to the attention score matrix in step 10221 above) in the intermediate bias network (first bias network), it clearly defines the relationship between each speech frame and each phrase, selecting the top -k phrases for each frame (corresponding to determining a preset number of words from multiple words in descending order of attention score in step 10222 above, and identifying multiple second words corresponding to each audio frame). Then, the frequency of phrase occurrences is counted across all frames, and phrases occurring more than n times are considered key phrases (corresponding to constructing a set of second words corresponding to multiple audio frames in step 10223 above, counting the frequency of each second word in the set, and identifying second words with a frequency greater than a preset threshold as first words). These phrases (first words) are further extracted for bias embedding (corresponding to the bias mapping above) and passed to subsequent bias networks, thereby focusing attention and improving recognition accuracy. Layer 6 (header encoding network) was chosen for bias filtering because earlier layers typically provide finer low-level information, while layer 6 can effectively control the impact of long lists on speed.

[0250] The following describes the data and experimental results of the training method for the audio recognition model provided in the embodiments of this application.

[0251] 1) Data

[0252] The LibriSpeech dataset and a private 11,000-hour Chinese telephone audio training set were used, with a 10.32-hour test set. For the LibriSpeech dataset, training was performed on the full dataset, and validation was conducted using dev-clean / other, while test-clean / other was used for evaluation. During the training of the bias network, byte-pair encoding (BPE) was used, randomly selecting 1-3 words from each speech transcription as positive samples and choosing 57 unseen phrases as negative samples to simulate interference. In the testing phase, a list of contextual phrases from a private dataset, ranging in length from 100 to 2000, was used. For the private dataset, character modeling was used, randomly selecting 2-8 characters from each transcription as positive samples. A list of 100,000 phrases was also constructed to validate the effectiveness of the shared bias attention phrase filtering. Other settings remained consistent with the LibriSpeech dataset configuration.

[0253] The bias network has an attention dimension of 384, containing 8 attention heads and 2048 feedforward neural network (FFN) units, including a 384-dimensional single-layer BLSTM encoder (corresponding to...). Figure 5 The text encoder shown in the figure has two linear layers and an 8-head attention layer used to introduce bias.

[0254] See Figure 7 The first linear layer in a CPP Network ( Figure 7 The linear layer shown uses the tanh activation function and maintains a 384-dimensional output, while the second linear layer ( Figure 7 The CTC Linear shown projects the input to a vocabulary size and shares parameters with the CTC layer of the SEQ-former encoder. Due to different training objectives, the bias decoding network (second bias network) and the bias encoding network (first bias network) share the structure but not the parameters; that is, the first bias network and the second bias network have the same network structure but different network parameters.

[0255] Word Error Rate (WER), biased word error rate (B-WER, for words in the bias list), and unbiased word error rate (U-WER, for words not in the bias list) were used as evaluation metrics. The SEQ-former-M model on LibriSpeech and private telephone data was used as a baseline, with its parameters frozen during bias network training. The Adam optimizer was used with an initial learning rate of 0.001, and training was conducted for 50 epochs on the LibriSpeech dataset and 10 epochs on the private data.

[0256] 2) Experimental Results and Analysis

[0257] The experimental results on the librispeech dataset are as follows:

[0258] Table 1

[0259]

[0260] Here, SI stands for Speaker Identification, SD stands for Speaker Diarization, which is suitable for complex scenarios with multiple speakers, and N represents the size of the bias list (i.e., the number of preset words).

[0261] As shown in Table 1, when N=100, compared with CPPN, the training method of the audio recognition model provided in this application reduces WER by 56.76% and 52.90% on test-clean and test-other, respectively. Similar trends in WER, U-WER, and B-WER were observed for N=500 and N=1000. Under text perturbation conditions, the training method of the audio recognition model provided in this application outperforms the optimal LibriSpeech scheme on test-other at N=100. Although it fails to achieve an overall WER advantage at other N values, it performs well in B-WER, indicating that bias information is enhanced.

[0262] To demonstrate the generalization ability of shared information, experiments were conducted on the Conformer. Specifically, a Conformer with a parameter size similar to the SEQ-former was selected and divided into two groups: one group used a shared decoder at layer 9, and the other group did not. Both groups applied depth-biased shared information at layer 9. The results show that introducing shared decoder supervision into the Conformer enhances the effect of depth bias, thus validating the effectiveness of the shared information.

[0263] Table 2

[0264]

[0265] Table 2 shows the performance of the shared bias attentional phrase filtering method provided in this application on the LibriSpeech dataset, where the size of the bias list is N=2000, and SSI+TP represents the combination of Speaker-Specific Information (SSI) and Text Post-processing (TP). Clearly, using SBA-PF significantly improves the performance of WER, U-WER, and B-WER. On the test-clean dataset, the relative improvements are: WER improved by 6.16%, U-WER by 6.43%, and B-WER by 5.41%. Similar improvements were observed on the test-other dataset.

[0266] Table 3

[0267]

[0268]

[0269] The ablation experiment results shown in Table 3 demonstrate that the shared intermediate encoder bias network brings the most significant performance improvement, enhancing the neural bias effect of the encoder by utilizing the shared information of the SEQ-former intermediate layers. Introducing the Biasing Decoder Network provides an additional relative performance improvement of approximately 8.70%, explicitly strengthening the depth bias of the decoder by sharing decoder attention loss information and employing probabilistic rescoring. Furthermore, combining text perturbation and shallow fusion further improves the performance, indicating that the training method of the audio recognition model provided in this application can effectively complement existing methods.

[0270] Table 4

[0271] method CER (U-CER / B-CER) RTF Shared bias attention phrase filtering used in 6-layer 8.37(7.93 / 12.27) 0.127 Shared bias attention phrase filtering used in 12-layer 8.39(7.95 / 12.31) 0.156 Shared bias attention phrase filtering 8.37(7.93 / 12.27) 0.127 Unshared shared bias attention phrase filtering 8.57(8.02 / 13.46) 0.126 MLA-based shared bias attention phrase filtering 8.37(7.93 / 12.27) 0.127 MHA-based shared bias attention phrase filtering 8.39(7.95 / 12.31) 0.131

[0272] Table 4 illustrates the robust performance of the shared bias attention phrase filtering provided in this application embodiment, particularly its speed, in the case of a list of 100,000 phrases. Here, CER (Character Error Rate) represents the character error rate, B-CER represents the bias word character error rate (for words in the bias list), U-CER represents the non-bias word character error rate (for words not in the bias list), and RTF (Real-Time Factor) represents the real-time factor used to measure the real-time performance of the audio recognition system. The shared bias attention phrase filtering used in layer 6 outperforms the application in layer 12 in both speed and accuracy, demonstrating the performance improvement effect of intermediate layer information. For non-shared shared bias attention phrase filtering (i.e., phrases obtained from shared bias attention phrase filtering corresponding to different coding layers are not shared), configuring an independent phrase attention layer for phrase filtering shows that the shared bias attention phrase filtering is simpler to implement and performs better. Furthermore, MLA exhibits higher efficiency and accuracy in phrase list filtering, providing powerful support for handling large-scale list inputs.

[0273] 3) Model Application

[0274] Objective: To apply the trained audio recognition model to real-world scenarios for speech recognition.

[0275] Model deployment: Deploy the trained audio recognition model to the target device, such as a server or embedded device.

[0276] Real-time processing: The input speech signal is processed in real time, features are extracted, and then fed into the audio recognition model for recognition. Output results: The recognition results are output as text for subsequent applications.

[0277] The following description continues to illustrate the exemplary structure of the training device 433 for the audio recognition model provided in this application embodiment as a software module. In some embodiments, such as... Figure 2 As shown, the software modules in the training device 433 of the audio recognition model stored in the memory 430 may include:

[0278] The encoding module 4331 is used to encode audio data samples through the encoding network in the audio recognition model to obtain the first encoded feature.

[0279] In some embodiments, the encoding module 4331 is further configured to perform bias mapping on the first encoding feature based on preset words through the first bias network in the audio recognition model to obtain the second encoding feature.

[0280] The decoding module 4332 is used to decode the second encoded feature through the decoding network in the audio recognition model to obtain the first text probability distribution.

[0281] In some embodiments, the decoding module 4332 is further configured to perform bias mapping on the first text probability distribution based on the preset words through the second bias network in the audio recognition model to obtain the second text probability distribution.

[0282] Training module 4333 is used to determine the loss value based on the text annotation of the audio data sample, the first encoding feature, the first text probability distribution and the second text probability distribution.

[0283] In some embodiments, the training module 4333 is further configured to update the parameters of the audio recognition model based on the loss value to obtain the trained audio recognition model.

[0284] In some embodiments, the number of encoding networks is multiple. The training module 4333 is further configured to, for a first encoding network among the multiple encoding networks, if the first encoding network is a head encoding network among the multiple encoding networks, determine a first loss value based on the first encoding feature output by the head encoding network and the text annotation; if the first encoding network is a middle encoding network among the multiple encoding networks, determine a second loss value based on the first encoding feature output by the middle encoding network, the first text probability distribution, and the text annotation; if the first encoding network is a tail encoding network among the multiple encoding networks, determine a third loss value based on the first encoding feature output by the tail encoding network and the text annotation; determine an encoding-side loss value based on the first loss value, the second loss value, and the third loss value, and determine a decoding-side loss value based on the first text probability distribution, the second text probability distribution, and the text annotation; and fuse the encoding-side loss value and the decoding-side loss value to obtain the final loss value.

[0285] In some embodiments, the training module 4333 is further configured to determine a first sub-loss value based on the first encoding features and word annotations output by the head encoding network, wherein the word annotations include words in the text annotations that match the preset words; determine a second sub-loss value based on the first encoding features and the text annotations output by the head encoding network; and fuse the first sub-loss value and the second sub-loss value to obtain the first loss value.

[0286] In some embodiments, the training module 4333 is further configured to: determine a third sub-loss value based on the first encoding features and word annotations output by the mid-encoding network, wherein the word annotations are words that match the text annotations with the preset words; fuse the first encoding features and the first text probability distribution output by the mid-encoding network to obtain a third encoding feature, and determine a fourth sub-loss value based on the third encoding features and the text annotations; decode the first encoding features output by the mid-encoding network through the decoding network in the audio recognition model to obtain a third text probability distribution, and determine a fifth sub-loss value based on the third text probability distribution and the text annotations; and fuse the third sub-loss value, the fourth sub-loss value, and the fifth sub-loss value to obtain the second loss value.

[0287] In some embodiments, the number of encoding networks is multiple; the encoding module 4331 is further configured to, for a first encoding network among the multiple encoding networks, if the first encoding network is a head encoding network among the multiple encoding networks, perform attention processing on the preset words and the first encoding feature to obtain a first attention feature, and fuse the first encoding feature and the first attention feature to obtain a second encoding feature; if the first encoding network is any encoding network after the head encoding network, select a first word from the preset words, perform attention processing on the first word and the first encoding feature to obtain a second attention feature, and fuse the first encoding feature and the second attention feature to obtain a second encoding feature.

[0288] In some embodiments, the preset words include multiple words; the first attention feature includes the attention feature corresponding to each audio frame in the audio data sample; the number of encoding networks is multiple; the encoding module 4331 is further configured to determine the attention score matrix corresponding to the attention feature for each audio frame, wherein the attention score matrix includes the attention score of each word in the multiple words; determine a preset number of words from the multiple words in descending order of the attention scores, and determine multiple second words corresponding to each audio frame; form a set of the second words corresponding to the multiple audio frames, count the frequency of each second word in the set, and determine the second words whose frequency of occurrence is greater than a preset threshold as the first word.

[0289] In some embodiments, the training module 4333 is further configured to perform audio recognition on the audio data to be recognized based on the trained audio recognition model, and obtain the first text corresponding to the audio data to be recognized.

[0290] In some embodiments, the training module 4333 is further configured to: encode the audio data to be recognized using the encoding network in the trained audio recognition model to obtain a third encoding feature; perform a bias mapping on the third encoding feature based on the preset words using the first bias network in the trained audio recognition model to obtain a fourth encoding feature; decode the fourth encoding feature using the decoding network in the trained audio recognition model to obtain a fourth text probability distribution; perform a bias mapping on the fourth text probability distribution based on the preset words using the second bias network in the trained audio recognition model to obtain a fifth text probability distribution; generate multiple second texts based on the fifth text probability distribution; and select the first text from the multiple second texts based on the first evaluation score of each second text.

[0291] In some embodiments, the training module 4333 is further configured to: for each second text, score the alignment between the second text and the audio data to be identified based on the second text and the fourth encoding feature to determine a second evaluation score; for each second text, score the semantic coherence of the second text based on the second text and the fourth encoding feature to determine a third evaluation score; for each second text, score the relevance between the second text and the preset words based on the second text and the preset words to determine a fourth evaluation score; and fuse the second evaluation score, the third evaluation score, and the fourth evaluation score to determine the first evaluation score.

[0292] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the audio recognition model training method described above in this application.

[0293] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the training method of the audio recognition model provided in this application. For example, ... Figure 3A The training method for the audio recognition model is shown.

[0294] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0295] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0296] For example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0297] For example, computer-executable instructions may be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0298] In summary, through the embodiments of this application, a first bias network is introduced into the audio recognition model. Based on preset words, the first encoded features obtained by encoding processing are biased and mapped, thereby guiding the audio recognition model to focus on specific words (i.e., preset words). By introducing a second bias network, the first text probability distribution obtained after decoding is biased and mapped, further optimizing the recognition of preset words. Furthermore, by combining the first encoded features, the first text distribution probability, the second text distribution probability, and the loss value design of multiple supervision signals such as text annotation, the audio recognition model has stronger semantic focusing ability and robustness, thereby improving the accuracy of audio recognition.

[0299] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A training method for an audio recognition model, characterized in that, The method includes: The audio data samples are encoded using the encoding network in the audio recognition model to obtain the first encoded feature; The first bias network in the audio recognition model performs bias mapping on the first encoded features based on preset words to obtain the second encoded features; The second encoded feature is decoded through the decoding network in the audio recognition model to obtain the first text probability distribution. The second text probability distribution is obtained by bias mapping the first text probability distribution based on the preset words through the second bias network in the audio recognition model. Based on the text annotations of the audio data samples, the first encoding feature, the first text probability distribution, and the second text probability distribution, the loss value is determined; The parameters of the audio recognition model are updated based on the loss value to obtain the trained audio recognition model.

2. The method according to claim 1, characterized in that, The number of the encoding networks is multiple; The step of determining the loss value based on the text annotation of the audio data sample, the first encoded feature, the first text probability distribution, and the second text probability distribution includes: For a first coding network among multiple coding networks, if the first coding network is a head coding network among the multiple coding networks, then a first loss value is determined based on the first coding feature output by the head coding network and the text annotation; If the first coding network is the middle coding network among the plurality of coding networks, then the second loss value is determined based on the first coding feature output by the middle coding network, the first text probability distribution, and the text annotation; If the first coding network is the tail coding network among the plurality of coding networks, then a third loss value is determined based on the first coding feature output by the tail coding network and the text annotation; Based on the first loss value, the second loss value, and the third loss value, the loss value on the encoding side is determined, and based on the first text probability distribution, the second text probability distribution, and the text annotation, the loss value on the decoding side is determined. The loss value is obtained by fusing the loss value from the encoding side and the loss value from the decoding side.

3. The method according to claim 2, characterized in that, The determination of the first loss value based on the first encoded features output by the head encoding network and the text annotation includes: Based on the first encoding features and word annotations output by the head encoding network, a first sub-loss value is determined, wherein the word annotations include words in the text annotations that match the preset words; Based on the first encoded features output by the head encoding network and the text annotation, a second sub-loss value is determined. The first sub-loss value is obtained by fusing the first sub-loss value and the second sub-loss value.

4. The method according to claim 2, characterized in that, The determination of the second loss value based on the first encoded features output by the central coding network, the first text probability distribution, and the text annotation includes: Based on the first coding features and word annotations output by the central coding network, a third sub-loss value is determined, wherein the word annotations are words that match the text annotations with the preset words; By fusing the first coding feature output by the middle coding network and the first text probability distribution, a third coding feature is obtained, and a fourth sub-loss value is determined based on the third coding feature and the text annotation. The first encoded feature output by the middle encoding network is decoded through the decoding network in the audio recognition model to obtain the third text probability distribution, and the fifth sub-loss value is determined based on the third text probability distribution and the text annotation. The second loss value is obtained by fusing the third sub-loss value, the fourth sub-loss value, and the fifth sub-loss value.

5. The method according to claim 1, characterized in that, The number of the encoding networks is multiple; The step of using the first bias network in the audio recognition model to perform bias mapping on the first encoded features based on preset words to obtain the second encoded features includes: For the first encoding network among multiple encoding networks, if the first encoding network is the head encoding network among the multiple encoding networks, then attention processing is performed on the preset words and the first encoding features to obtain the first attention features, and the first encoding features and the first attention features are fused to obtain the second encoding features; If the first encoding network is any encoding network after the head encoding network, then a first word is selected from the preset words, attention processing is performed on the first word and the first encoding feature to obtain a second attention feature, and the first encoding feature and the second attention feature are fused to obtain the second encoding feature.

6. The method according to claim 5, characterized in that, The preset words include multiple words; the first attention feature includes the attention feature corresponding to each audio frame in the audio data sample; The step of selecting the first word from the preset words includes: For each audio frame, an attention score matrix is ​​determined corresponding to the attention features, wherein the attention score matrix includes the attention score of each word among the plurality of words; Based on the attention scores from high to low, a preset number of words are determined from the plurality of words, and a plurality of second words are determined for each audio frame; The second words corresponding to multiple audio frames are formed into a set. The frequency of each second word in the set is counted. The second words whose frequency of occurrence is greater than a preset threshold are determined as the first words.

7. The method according to any one of claims 1-6, characterized in that, After obtaining the trained audio recognition model, the method further includes: Based on the trained audio recognition model, audio recognition is performed on the audio data to be recognized to obtain the first text corresponding to the audio data to be recognized.

8. The method according to claim 7, characterized in that, The step of performing audio recognition on the audio data to be recognized based on the trained audio recognition model to obtain the first text corresponding to the audio data to be recognized includes: The audio data to be identified is encoded using the encoding network in the trained audio recognition model to obtain a third encoded feature. Using the first bias network in the trained audio recognition model, the third coding feature is biased and mapped based on the preset words to obtain the fourth coding feature; The fourth encoded feature is decoded using the decoding network in the trained audio recognition model to obtain the fourth text probability distribution; The fifth text probability distribution is obtained by bias mapping the fourth text probability distribution based on the preset words through the second bias network in the trained audio recognition model. Multiple second texts are generated based on the probability distribution of the fifth text; The first text is selected from the plurality of second texts based on the first evaluation score of each second text.

9. The method according to claim 8, characterized in that, Before selecting the first text from the plurality of second texts based on the first evaluation score of each second text, the method further includes: For each second text, based on the second text and the fourth encoded feature, the alignment degree between the second text and the audio data to be identified is scored to determine a second evaluation score; For each second text, based on the second text and the fourth encoding feature, the semantic coherence of the second text is scored to determine a third evaluation score; For each second text, based on the second text and the preset words, the relevance between the second text and the preset words is scored to determine a fourth evaluation score; The first evaluation score is determined by combining the second evaluation score, the third evaluation score, and the fourth evaluation score.

10. A training device for an audio recognition model, characterized in that, The device includes: The encoding module is used to encode audio data samples through the encoding network in the audio recognition model to obtain the first encoded feature; The encoding module is further configured to perform bias mapping on the first encoding feature based on preset words through the first bias network in the audio recognition model to obtain the second encoding feature; The decoding module is used to decode the second encoded feature through the decoding network in the audio recognition model to obtain the first text probability distribution; The decoding module is further configured to perform a bias mapping on the first text probability distribution based on the preset words through the second bias network in the audio recognition model to obtain the second text probability distribution; The training module is used to determine the loss value based on the text annotation of the audio data sample, the first encoding feature, the first text probability distribution, and the second text probability distribution; The training module is also used to update the parameters of the audio recognition model based on the loss value, so as to obtain the trained audio recognition model.

11. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method of the audio recognition model according to any one of claims 1-9.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the training method of the audio recognition model according to any one of claims 1-9 is implemented.

13. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the training method of the audio recognition model according to any one of claims 1-9 is implemented.

Images