Audio recognition method, content display method, apparatus, device, medium, and product

Abstract

This application relates to an audio recognition method, content display method, apparatus, device, medium, and product. The method includes: acquiring target audio; extracting valid audio segments from the target audio; determining language recognition audio based on the valid audio segments, and performing language recognition on the language recognition audio to obtain a language recognition result; determining a target transcription audio to be transcribed based on the language recognition audio, and determining a target language tag for the target transcription audio based on the language recognition result; performing text conversion on the target transcription audio based on the target language tag to generate initial text, the initial text being free of punctuation; and restoring punctuation from the initial text to obtain the target text. This method can improve audio recognition efficiency.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio recognition method, content display method, apparatus, device, medium and product. Background Technology

[0002] With the continuous development of speech recognition technology, the application scenarios of speech recognition are becoming more and more widespread, including but not limited to voice search, live streaming or short videos.

[0003] In traditional solutions, speech can usually be converted into corresponding text by training a model; for example, speech can be input into a model to obtain text.

[0004] However, the process of converting speech into correct text is often time-consuming, resulting in low audio recognition efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide an audio recognition method, content display method, device, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency of audio recognition in response to the above-mentioned technical problems.

[0006] On one hand, this application provides an audio recognition method, comprising: acquiring target audio; extracting valid audio segments from the target audio; determining language recognition audio based on the valid audio segments, and performing language recognition on the language recognition audio to obtain a language recognition result of the language recognition audio; determining a target transcription audio to be transcribed based on the language recognition audio, and determining a target language tag for the target transcription audio based on the language recognition result; performing text conversion on the target transcription audio based on the target language tag to generate initial text, wherein the initial text has no punctuation; and restoring punctuation from the initial text to obtain the target text.

[0007] On the other hand, this application also provides an audio recognition device, comprising: a segment extraction module for acquiring target audio and extracting valid audio segments from the target audio; a language recognition result acquisition module for determining language recognition audio based on the valid audio segments, and performing language recognition on the language recognition audio to obtain the language recognition result of the language recognition audio; a tag determination module for determining target transcription audio to be transcribed based on the language recognition audio, and determining target language tags for the target transcription audio based on the language recognition result; a first text generation module for performing text conversion on the target transcription audio based on the target language tags to generate initial text, wherein the initial text has no punctuation; and a second text generation module for restoring punctuation from the initial text to obtain target text.

[0008] On the other hand, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described audio recognition method.

[0009] On the other hand, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described audio recognition method.

[0010] On the other hand, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described audio recognition method.

[0011] The aforementioned audio recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire target audio, extract valid audio segments from the target audio, determine the language recognition audio based on the valid audio segments, perform language recognition on the language recognition audio to obtain the language recognition result, determine the target audio to be transcribed based on the language recognition audio, determine the target language tag of the target transcribed audio based on the language recognition result, perform text conversion on the target transcribed audio based on the target language tag to generate initial text (which has no punctuation), and perform punctuation recovery on the initial text to obtain the target text. This achieves an integrated audio recognition solution from audio, valid audio segment recognition, language recognition, text conversion to punctuation recovery, which helps reduce the time consumed in the audio recognition process and improve the efficiency of audio recognition.

[0012] On the other hand, this application provides a content display method, including: displaying a content details page of target content; displaying, on the content details page, an input voice associated with the target content and a transcribed text corresponding to the input voice; wherein, the transcribed text is determined based on target text, the target text is obtained by processing target audio according to an audio recognition method, and the target audio is determined based on the input voice.

[0013] On the other hand, this application provides a content display device, including: a first display module for displaying a content details page of a target content; and a second display module for displaying, on the content details page, an input voice associated with the target content and a transcribed text corresponding to the input voice; wherein the transcribed text is determined based on a target text, the target text is obtained by processing a target audio according to an audio recognition method, and the target audio is determined based on the input voice.

[0014] On the other hand, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described content display method.

[0015] On the other hand, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described content display method.

[0016] On the other hand, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described content display method.

[0017] The above-mentioned content displays methods, devices, computer equipment, computer-readable storage media, and computer program products, displaying a content details page of the target content. On the content details page, the input speech associated with the target content and the transcribed text corresponding to the input speech are displayed. The transcribed text is determined based on the target text, which is obtained by processing the target audio according to an audio recognition method. The target audio is determined based on the input speech. Since the audio recognition method improves the efficiency of audio recognition, it can improve the efficiency of transcribed text, that is, improve the efficiency of speech transcription. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a diagram illustrating the application environment of an audio recognition method in one embodiment;

[0020] Figure 2 This is a flowchart illustrating an audio recognition method in one embodiment;

[0021] Figure 3A This is a schematic diagram of the structure of the first model and the second model in one embodiment;

[0022] Figure 3B This is a schematic diagram of a content details page in one embodiment;

[0023] Figure 4 This is a schematic diagram of the content details page in another embodiment;

[0024] Figure 5 This is a schematic diagram of the structure of an audio detection model in one embodiment;

[0025] Figure 6 This is a schematic diagram illustrating the principle of identifying the audio category of an audio segment in one embodiment;

[0026] Figure 7 This is a schematic diagram illustrating the principle of audio processing in both streaming and non-streaming scenarios in one embodiment.

[0027] Figure 8 This is a schematic diagram of the structure of the first detection model in one embodiment;

[0028] Figure 9 This is a schematic diagram of the structure of the second detection model in one embodiment;

[0029] Figure 10 This is a schematic diagram of an audio processing flow in one embodiment;

[0030] Figure 11 This is a schematic diagram illustrating the principle of obtaining a target punctuation prediction model in one embodiment;

[0031] Figure 12 This is a flowchart illustrating the overall process from generative model training to punctuation prediction in one embodiment.

[0032] Figure 13 This is a structural block diagram of an audio recognition device in one embodiment;

[0033] Figure 14 This is an internal structural diagram of a computer device in one embodiment;

[0034] Figure 15 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0036] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0037] The terms used in the embodiments of this application include:

[0038] ASR (Automatic Speech Recognition) refers to automatic speech recognition, a technology used to convert human speech into text.

[0039] VAD (Voice Activity Detection) is a technique used in target audio processing to detect whether audio contains valid speech (human voice) and to distinguish between silence / noise and valid speech. In other words, it is used to detect whether there are valid speech segments in a speech stream, thereby distinguishing the target audio from silent or noisy parts.

[0040] LID (Language Identification) refers to language identification, which is used to automatically identify the language or dialect category to which an audio segment belongs.

[0041] Punc (Punctuation Restoration) is a pointer point restoration tool used to add correct punctuation marks to a sequence of plain text.

[0042] CER (Character Error Rate) is a metric used to measure ASR performance; a lower value is better.

[0043] SOTA (State of the Art) refers to the best performance at the current level of technology.

[0044] LLM (Large Language Model) refers to a large language model.

[0045] AED (Attention-based Encoder-Decoder): refers to an encoder-decoder model architecture based on attention mechanisms, used to build the backbone network of LID models.

[0046] BGM (Background Music) refers to background music.

[0047] aED (audio event detection) is a technology that identifies specific events (such as instrumental music, background music, and noise) in audio signals. It achieves automatic event detection by extracting and analyzing audio features.

[0048] AUC-ROC (Area Under the ROC Curve - Receiver Operating Characteristic): The area under the ROC curve is an important metric used in machine learning and pattern recognition to evaluate the performance of classification models. It measures the overall ability of a model to distinguish between positive and negative samples at different thresholds by calculating the area under the ROC curve (the curve showing the relationship between the true positive rate and the false positive rate).

[0049] F1-Score: The harmonic mean of precision and recall, used to comprehensively evaluate the performance of a classification model. It is especially suitable for handling imbalanced datasets, balancing the model's ability to identify positive samples with its ability to avoid false positives.

[0050] RTF (Real Time Factor): refers to the real-time rate, which is used to measure the processing speed of a speech recognition system.

[0051] BERT (Bidirectional Encoder Representations from Transformers) refers to bidirectional encoder representations based on Transformers.

[0052] CNN (Convolutional Neural Network) refers to a convolutional neural network.

[0053] The VGG (Visual Geometry Group) model refers to the visual geometry group model.

[0054] Conformer: Combines CNN with Transformer to capture both local features and global interaction information in audio.

[0055] LoRA (Low-Rank Adaptation) is a technique used to freeze the weights of a pre-trained model and then introduce a low-rank matrix to simulate the changes in parameters during model updates, thereby reducing the number of parameters that need to be trained.

[0056] Fbank (Filter Bank features): Filter bank features are frequency domain acoustic features that mimic the characteristics of human hearing and can be used as audio feature forms for model input.

[0057] CMVN (Cepstral Mean and Variance Normalization): A feature normalization method used to eliminate channel noise and speaker differences.

[0058] DFSMN (Deep Feedforward Sequence Memory Network) is a neural network architecture specifically designed for temporal data. Through feedforward connections and sequence memory mechanisms, it can effectively capture the temporal dependencies of audio signals, making it particularly suitable for audio content with complex temporal characteristics, such as speech, singing, and music. The advantage of this architecture is that it can process complete audio segments (non-streaming VAD) as well as support real-time segmented processing (streaming VAD), balancing detection accuracy and real-time performance.

[0059] Token: A token, referring to the discrete category identifier (such as <|zh|> or <|yue|>) corresponding to the model's output layer.

[0060] CE (Cross Entropy): A loss function used to measure the difference between the predicted probability distribution and the true label.

[0061] SNR (Signal-to-Noise Ratio): The signal-to-noise ratio is the ratio of signal power to noise power, used in data augmentation to control noise intensity.

[0062] RIR (Room Impulse Response): A data augmentation technique used to simulate reverberation effects in different spaces.

[0063] Acoustic features: For an audio frame, acoustic features can be the spectral features of the audio frame, or the result of CMVN or feature normalization of the spectral features of the audio frame; for audio (or audio segment), acoustic features can be the spectrogram (or Fbank feature) of the audio (or audio segment), or the result of CMVN or feature normalization of the spectrogram (or Fbank feature).

[0064] Transformer model: a deep learning model based on self-attention mechanism.

[0065] The conformer model is a hybrid deep learning model that combines the transformer model and the CNN model.

[0066] This application provides a state-of-the-art (SOTA) industrial-grade integrated multilingual speech recognition system, such as... Figure 1As shown, the system includes a terminal 102 and a server 104. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted displays, etc. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal 102 communicates with the server 104 via a network. A data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104 or located in the cloud or on other network servers.

[0067] Specifically, terminal 102 can upload audio, configure parameters, and display / save results. Server 104 includes a front-end audio processing module, a voice activity detection module (also known as an audio detection module), a language recognition module, a speech-to-text module, a punctuation recovery module, and a system scheduling and result output module. The front-end audio processing module → voice activity detection module → language recognition module → speech-to-text module → punctuation recovery module → system scheduling and result output module enables a state-of-the-art (SOTA) industrial-grade integrated multilingual speech recognition system. This system supports independent / combined module calls to adapt to diverse needs. The modules are linked serially, with the output of one module serving as the input of the next, and are uniformly managed by the scheduling module. Among these:

[0068] The input to the front-end audio processing module includes: audio (e.g., raw speech audio) in any format (e.g., mp3, flac, or wav) uploaded by the client, and user-specified basic processing rules. The front-end audio processing module's process includes: 1. Format verification: detecting audio format, sampling rate, and number of channels, and removing invalid files; 2. Standardization conversion: uniformly converting to 16kHz 16-bit mono PCM wav format using the ffmpeg tool. The conversion command is: ffmpeg -i<input_audio_path> -ar 16000 -ac 1 -acodec pcm_s16le -f wav<output_wav_path> 3. Invalid segment removal: Automatically removes blank silent segments that exceed 2 seconds at the beginning and end (customizable). The output of the front-end audio processing module includes: standard format voice audio files, along with basic audio information (duration, sampling rate, number of channels).

[0069] The input to the voice activity detection module includes standard audio output from the front-end audio processing module and user-configured VAD detection parameters. The voice activity detection module supports three modes: non-streaming VAD, streaming VAD, and audio event detection (aED). The processing steps of the voice activity detection module include: 1. Loading preset parameters (customizable) and segmenting audio according to a fixed frame length; 2. Calculating frame-level speech / singing / music probability (AED mode) or speech probability only; 3. Determining human voice boundaries by combining speech thresholds and smoothing the process to avoid misjudgments; 4. Segmenting valid human voice audio segments and removing short segments shorter than 0.3 seconds. The output of the voice activity detection module includes: valid human voice audio segments, frame-level probability values ​​(AED mode), human voice segmentation timestamps, and segment duration information.

[0070] The language identification module's input includes the valid audio segments output by the speech activity detection module and a confidence threshold (e.g., 0.8) for language detection. The language identification module's processing includes: 1. Audio feature extraction: Extracting acoustic features such as Mel spectrum and fundamental frequency from valid human voice audio segments to generate fixed-dimensional sentence-level feature vectors, without additional redundant preprocessing steps; 2. Language / dialect feature matching: Matching the feature vectors with the model's built-in feature library (covering 100+ common languages ​​and 20+ Chinese dialects / accents), calculating the similarity with each language / dialect; 3. Result judgment and confidence calculation: Selecting the language / dialect with the highest similarity as the detection result, calculating the confidence value of the result using a normalization algorithm, without subjective adjustment logic; 4. Low confidence labeling: If the confidence value is lower than the confidence threshold, the result is specially labeled, prompting the user for manual review, without setting additional filtering rules. The output of the language recognition module includes: the language / dialect label corresponding to the audio (the label format follows the built-in specification of the model), the confidence value (calculated based on the distribution of measured data) and / or the low confidence mark.

[0071] The input to the speech-to-text module includes valid audio segments annotated by the language recognition module, language / dialect tags, and ASR inference parameters. The processing steps of the speech-to-text module include: 1. Loading the corresponding architecture model; 2. Decoding and inference: AED architecture beam search, LLM architecture sampling and decoding; 3. AED architecture generates word-level timestamps and confidence scores, LLM architecture only outputs confidence scores; 4. Semantic correction, removing meaningless errors. It provides both LLM-based and AED-based ASR, implementing a dual-architecture ASR, optimized for Chinese dialects and mixed Chinese-English speech. The output of the speech-to-text module includes plain text without punctuation, word-level timestamps (AED architecture), whole-sentence confidence scores, and basic transcription information (inference duration and / or RTF value).

[0072] The punctuation recovery module (a punctuation recovery model applicable to both Chinese and English) takes as input the plain text without punctuation output from the speech-to-text module and language tags. The processing steps of the punctuation recovery module include: 1. Loading the pre-trained Chinese / English universal punctuation recovery model and matching corresponding adaptation parameters based on the language tags; 2. Inputting the plain text without punctuation and performing text preprocessing; 3. The model predicts the punctuation type at each position in the text (using a unified punctuation system for both Chinese and English, supporting common punctuation marks such as commas, periods, question marks, and exclamation marks); 4. Post-processing the prediction results and integrating them into complete text with punctuation. The output of the punctuation recovery module includes: the recognized text with punctuation and a detailed list of punctuation additions.

[0073] This system can be applied to various audio processing scenarios, including but not limited to: voice search (scenarios where searches are conducted via voice), voice commenting, voice note-taking, and video caption generation. In voice search, it can convert spoken input into text. In voice commenting, it can transcribe user voice comments into text. In voice note-taking, it can transcribe user-sent voice notes into corresponding text notes. In video caption generation, it can automatically generate captions for videos or live streams.

[0074] This system can achieve integrated high-accuracy speech recognition, supporting multilingual scenarios and 20+ Chinese dialects. For any user-inputted speech audio, it can complete effective speech segment detection, language / dialect recognition, speech-to-text conversion, punctuation restoration, and output of punctuated text in one stop. The performance of each stage can approach or reach the industry's state-of-the-art (SOTA) level, adapt to low-computing-power deployment environments, and support the recognition of a cappella lyrics and mixed Chinese and English speech.

[0075] This system integrates a language recognition module and a speech-to-text module, which can solve the problems of independent language detection and speech-to-text modules and cumbersome data interaction, reduce data transmission and adaptation work between modules, and improve the overall speech processing efficiency.

[0076] The speech-to-text module in this system can include a first speech-to-text model and a second speech-to-text model. The second speech-to-text model is implemented based on a pre-trained LLM architecture, while the first speech-to-text model is implemented based on an AED architecture. In other words, the speech-to-text module employs a dual architecture. Because the second speech-to-text model, based on the pre-trained LLM architecture, can meet high-precision transcription requirements, while the first speech-to-text model, based on the AED architecture, can adapt to low-computing-power and real-time transcription scenarios, the speech-to-text module can solve the problem that a single model cannot simultaneously achieve both recognition accuracy and inference efficiency, thus meeting diverse business needs.

[0077] This system integrates a voice activity detection module, a voice transcription module, a language recognition module, and a punctuation recovery module. It can address the problem of cumbersome connections caused by the independent deployment of multiple modules, and can achieve end-to-end full-process voice recognition, thereby reducing the technical threshold for system deployment and debugging.

[0078] This system is compatible with multilingual and Chinese dialect recognition, which can address the problem of low accuracy of traditional models in recognizing Chinese dialects or minority languages, and can expand the application scope of the system in cross-border and local characteristic scenarios.

[0079] This system is adapted to speech mixed with Chinese and foreign languages. It can address the problem of low recognition accuracy in mixed Chinese and English speech or special speech scenarios, improve the system's adaptability to complex speech input, and meet the recognition needs of cross-border interaction and other scenarios.

[0080] The speech activity detection module in this system can distinguish between speech, singing, and music (pure music), and it takes into account multi-mode detection and low false negative rate. It can solve the problems of unbalanced detection effect of traditional VAD and lack of support for distinguishing complex audio events (such as singing or music). It can more accurately segment effective audio segments, reduce unnecessary computing power consumption, and improve the accuracy of subsequent recognition.

[0081] This system provides a standardized configuration framework and supports multiple calling methods, which can address the problems of complex deployment and limited calling methods in traditional ASR systems. It adapts to the different usage needs of algorithm and non-algorithm users and can improve deployment efficiency.

[0082] In one exemplary embodiment, such as Figure 2 As shown, an audio recognition method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0083] Step 202: Extract valid audio segments from the target audio.

[0084] The target audio can include speech. The speech can be spoken by a real person, a virtual person (e.g., a robot), or it can be played by a device.

[0085] Step 204: Determine the language identification audio based on the valid audio segments, and perform language identification on the language identification audio to obtain the language identification result of the language identification audio.

[0086] This can be achieved by using valid audio segments as language identification audio, or by segmenting valid audio segments to obtain language identification audio, or by merging multiple valid audio segments. The language identification result can include language tags for the primary language and / or the secondary language. Language tags can also be called language identifiers.

[0087] Step 206: Determine the target audio to be transcribed based on the language recognition audio, and determine the target language tag of the target audio to be transcribed based on the language recognition result.

[0088] Among these methods, language-identified audio can be used as target transcription audio, or target transcription audio can be extracted from language-identified audio.

[0089] Step 208: Based on the target language tag, perform text conversion on the target transcribed audio to generate initial text, which has no punctuation.

[0090] The initial text is generated using either a first audio transcription model or a second audio transcription model. The first audio transcription model consists of an encoder and decoder linked by an attention mechanism, while the second audio transcription model consists of a pre-trained large language model. The pre-trained large language model can be implemented using a decoder-only model and can be an open-source pre-trained large language model.

[0091] For example, the first audio transcription model is obtained by training a first model. In the first model, the encoder and decoder are associated through an attention mechanism. The encoder can employ a transformer or conformer architecture. The decoder can also employ a transformer or conformer architecture. For example, if the encoder uses a conformer architecture, it combines the ability of a convolutional neural network to extract local features with the ability of a transformer to capture global dependencies. The decoder can be an autoregressive decoder that receives the encoder's output through a cross-attention mechanism, combining it with preceding context information to predict the next character or word. For example... Figure 3A In the first model, encoder A and decoder A are included.

[0092] The process of training the first model includes: inputting the acoustic features of the sample audio and the text vector features corresponding to the sample audio into the first model; using the encoder in the first model to encode the acoustic features to obtain encoded features; using the decoder to decode the encoded features and text vector features to obtain the decoding result; the decoding result contains the probability distribution corresponding to each audio frame in the sample audio; generating the cross-entropy loss of the first model based on the probability distribution and the cross-entropy loss function; and updating the parameters of the first model using the cross-entropy loss.

[0093] For example, the process of obtaining the first audio transcription model includes: inputting the acoustic features of the sample audio and the text vector features corresponding to the sample audio into the first model to be trained, obtaining the output of the first model, the output of which includes the first probability distribution corresponding to each audio frame in the sample audio; generating cross-entropy loss and alignment loss according to the first probability distribution corresponding to each audio frame; updating the parameters of the first model based on the cross-entropy loss and alignment loss, and obtaining the first audio transcription model.

[0094] The first model predicts the next character using an autoregressive approach, optimizing its generation capability by minimizing the difference between the predicted distribution and the actual text. It can generate auxiliary text based on the text corresponding to the sample audio. This auxiliary text can include the text corresponding to the sample audio, the language tag of the sample audio, and a start tag. For example, the auxiliary text could be " <sos>"[en] Hello world", where <sos>[en] is the start label, [en] is the language label, and "Hello world" is the text corresponding to the sample audio. The text vector feature can be a vectorized representation of the auxiliary text.

[0095] The first probability distribution corresponding to the audio frame includes the probability of each word and whitespace character in the vocabulary. The cross-entropy loss is generated based on the cross-entropy loss function and the first probability distribution, while the alignment loss is generated based on CTC (Connectionist Temporal Classification) and the first probability distribution.

[0096] For example, alignment loss for:

[0097]

[0098] in, T represents the acoustic characteristics, and T represents the number of audio frames contained in the sample audio. This represents the feature vector corresponding to the t-th audio frame in the sample audio. K represents the target label sequence corresponding to the sample audio. Each label in the target label sequence represents a word in the vocabulary. The sequence of words represented by each label in the target label sequence is the real text of the sample audio. K is the length of the target label sequence.

[0099] Represents all that can be mapped to A set of paths. Represents the audio frame for frame t. The first model outputs symbols The probability. At least one path can be determined based on the probability distribution corresponding to each speech frame in the decoding result. . Represents a line of length The path, Let t be the t-th symbol on the path. It belongs to a word in the vocabulary or a blank character. It can be mapped to The path. For each path, the probabilities of each path are multiplied consecutively to obtain the cumulative probability of that path. The alignment loss is determined based on the sum of the cumulative probabilities of each path.

[0100] For example, the cross-entropy loss and alignment loss can be weighted to obtain a weighted loss, and the parameters of the first model can be updated using the weighted loss.

[0101] In this embodiment, training the first model by combining cross-entropy loss and alignment loss can improve training accuracy. Alignment loss addresses the issue of length mismatch or misalignment between audio input and text output, enabling the encoder to autonomously learn temporal alignment relationships. The introduction of alignment loss accelerates the encoder's convergence speed and improves the robustness of the features extracted by the encoder.

[0102] For example, the second audio transcription model is obtained by training a second model. The encoder in the second model has the same structure as the encoder in the first model. The parameters of the encoder in the second model are initialized using the parameters of the encoder in the first audio transcription model, thus combining the encoder's ability to extract audio features in the first audio transcription model with the text generation and understanding capabilities of LLM. The second model includes a prediction layer, which comprises a pre-trained large language model and an adaptation layer, such as an adapter. Since the output dimension of the encoder in the second model may not be consistent with the word embedding dimension of LLM, the adaptation layer can project (i.e., map) the output of the encoder in the second model to a semantic space that LLM can understand, achieving modality alignment. The adapter can be implemented using a linear layer or a multilayer perceptron.

[0103] During the training of the second model, the pre-trained large language model is frozen to preserve the general knowledge and reasoning ability acquired by the LLM during large-scale text pre-training, avoiding catastrophic forgetting. To enable the frozen LLM to better handle speech recognition tasks, LoRA (Low-Rank Adaptation) modules are inserted into the LLM layers of the speech-to-text model. Inserting LoRA modules into the LLM is a parameter-efficient fine-tuning (PEFT) technique that allows the model to adapt to new modalities by updating only a small number of parameters. Using LoRA for lightweight fine-tuning enables the LLM to process audio at a lower training cost while retaining its instruction-following ability. During the training of the second model, the parameters of the LoRA module, adapter, and encoder in the second model are updated. The second model can be trained iteratively multiple times. During training, the negative log-likelihood, i.e., the cross-entropy loss, of the generated initial text can be calculated based on the "next token prediction" paradigm.

[0104] like Figure 3A The diagram shows the structure of the second model, which can be trained using sample audio. The sample audio used to train the second model can be the same as or different from the sample audio used to train the first model. The transcription prompt, the acoustic features of the sample audio, and the text vector features of the sample audio can be input into the second model. Encoder B encodes the acoustic features, and an adaptation layer transforms the dimensions of the features output by encoder B to obtain deep acoustic features. Then, the transcription prompt, the deep acoustic features, and the text vector features of the sample audio are input into the prediction layer of the second model to obtain the prediction result. The loss can be calculated based on the prediction result, and then the parameters of the second model are updated using the loss. The transcription prompt is an instruction to convert audio to text, such as "Speech To Text".

[0105] For example, the target language tag and the acoustic features of the target transcribed audio can be input into a first audio transcription model to obtain the initial text output by the first audio transcription model. Alternatively, the target language tag and the acoustic features of the target transcribed audio can be input into a second audio transcription model to obtain the initial text output by the second audio transcription model. Or, the transcription prompt, the target language tag, and the acoustic features of the target transcribed audio can be input into a second audio transcription model to obtain the initial text output by the second audio transcription model.

[0106] In an exemplary embodiment, text conversion of target transcribed audio based on target language tags to generate initial text includes: obtaining acoustic features of the target transcribed audio; extracting features from the acoustic features to obtain target extracted features; and generating initial text based on transcription prompts, target extracted features, and target language tags.

[0107] For example, the acoustic features of the target audio can be input into a second audio transcription model. Through the processing of the encoder and adaptation layer in the second audio transcription model, the target extraction features are obtained. Then, the transcription prompt, the target extraction features, and the target language label are input into the prediction layer to obtain the initial output text.

[0108] Step 210: Perform punctuation restoration on the initial text to obtain the target text.

[0109] This process involves recognizing punctuation marks in the initial text, obtaining punctuation recognition results, and then adding punctuation marks to the initial text using these results to obtain the target text.

[0110] The aforementioned audio recognition method involves acquiring target audio, extracting valid audio segments from the target audio, determining the language of the target audio based on the valid audio segments, performing language recognition on the target audio, obtaining the language recognition result, determining the target audio to be transcribed based on the target audio, determining the target language tag of the target audio based on the language recognition result, performing text conversion on the target audio based on the target language tag to generate initial text (which lacks punctuation), and restoring punctuation from the initial text to obtain the target text. This achieves an integrated audio recognition solution encompassing audio, valid audio segment recognition, language recognition, text conversion, and punctuation restoration, helping to reduce time consumption and improve the efficiency of audio recognition.

[0111] In an exemplary embodiment, text conversion of target transcribed audio based on target language tags to generate initial text includes: selecting a target audio transcription model from a first audio transcription model and a second audio transcription model; inputting the acoustic features of the target transcribed audio and the target language tags into the target audio transcription model to obtain the output initial text; wherein the first audio transcription model includes an encoder and decoder associated with an attention mechanism, and the second audio transcription model includes a pre-trained large language model.

[0112] For example, transcription configuration parameters can be obtained, which can be user-configured. Based on the transcription configuration parameters, a target audio transcription model can be selected from a first audio transcription model and a second audio transcription model; that is, the target audio transcription model can be manually selected.

[0113] In this embodiment, the first audio transcription model includes an encoder and a decoder associated with an attention mechanism. Therefore, the first audio transcription model is based on the AED architecture. Since the second audio transcription model includes a pre-trained large language model, the second audio transcription model is based on the LLM architecture, thus realizing a dual-architecture ASR.

[0114] In an exemplary embodiment, a content display method is provided, which is illustrated by taking the application of the method to a terminal as an example, and includes the following steps:

[0115] Step 202A: Display the content details page of the target content.

[0116] The target content can be content published on a content publishing platform, and it can contain at least one of the following: text, images, video, or audio. The content details page is used to display the information contained in the target content.

[0117] Step 202B: On the content details page, display the input speech associated with the target content and the corresponding transcribed text.

[0118] The transcribed text is determined based on the target text, which is obtained by processing the target audio using an audio recognition method. The target audio is determined based on the input speech. The transcribed text either contains the target text or is the target text.

[0119] For example, a terminal can send input speech to a server, which can perform format standardization processing on the input speech to obtain the target audio. The terminal can then extract valid audio segments from the target audio, determine the language recognition audio based on the valid audio segments, and perform language recognition on the language recognition audio to obtain the language recognition result. Based on the language recognition audio, the terminal can determine the target audio to be transcribed, and based on the language recognition result, determine the target language tag for the target transcribed audio. Based on the target language tag, the terminal can perform text conversion on the target transcribed audio to generate initial text. The initial text has no punctuation, and punctuation is restored from the initial text to obtain the target text. The transcribed text contains the target text or is the target text itself.

[0120] For example, the input voice can be entered when publishing the target content, such as voice uploaded to a content publishing page. The content details page can display the input voice using a playback control and show the transcribed text corresponding to the input voice. For example... Figure 3B The content details page 300 displays a playback control 302 corresponding to the input audio uploaded when publishing the target content, and presents the transcribed text 304 of the input speech corresponding to the playback control.

[0121] For example, the input voice can also be entered after the target content is published; for instance, it can be voice used to interact with the target content, such as a voice comment, i.e., a comment made in voice form. For example... Figure 4 The content details page 306 displays the audio commentary 308 of the target content, and the transcribed text 310 of the audio commentary is displayed accordingly.

[0122] In the above content display method, the content details page of the target content is displayed. On the content details page, the input voice associated with the target content and the transcribed text corresponding to the input voice are displayed. The transcribed text is determined based on the target text, which is obtained by processing the target audio according to the audio recognition method. The target audio is determined based on the input voice. Since the audio recognition method improves the audio recognition efficiency, it can improve the efficiency of transcribed text, that is, improve the speech transcription efficiency.

[0123] Among related technologies, speech activity detection has many shortcomings in terms of multi-scenario adaptability, performance and functional scalability, including: (1) Limited multilingual support: Traditional VAD models are mainly optimized for specific languages ​​(such as Chinese and English), which makes it difficult to cover speech / singing / music scenarios of 100+ languages, resulting in a decrease in VAD accuracy in cross-language applications (such as multilingual intelligent assistants and cross-border voice interaction systems); (2) Insufficient audio type adaptability: Traditional VAD focuses on pure speech scenarios and is not suitable for speech activity detection in complex audio such as singing (such as melody, separation of vocals and accompaniment) and music (such as music fragments of different styles). Poor robustness, prone to misjudgment (such as misjudging human voices in music as silent, or failing to distinguish effective human voices from background accompaniment in singing); (3) Performance indicators need improvement: Existing models generally have an F1 score (combined precision and recall) of less than 95%, which is difficult to meet the requirements of high-precision VAD (real-time transcription of meetings, voice interaction and other scenarios require nearly 100% accuracy in recognizing voice activities); (4) Single function: Traditional VADs can only output the time interval of voice activities and lack the ability to detect specific events in audio (such as singing, music or accompaniment), which cannot meet the extended requirements of intelligent audio monitoring, multimodal event perception and so on. Based on this, the embodiments of this application provide an audio detection method, which can be used to implement a voice activity detection module.

[0124] In one exemplary embodiment, an audio detection method is provided for extracting valid audio segments from target audio, the method comprising:

[0125] Step 302: Perform category recognition on the audio frames in the target audio to obtain the target probability sequence of the audio frames. The target probability sequence contains the probability corresponding to multiple target audio categories. The multiple target audio categories include speech categories and at least two non-speech categories.

[0126] The target audio can be any audio file, such as audio containing speech (human speech or voice), which can be emitted by a device or a real person. Audio frames are obtained by segmenting the target audio; the length of each audio frame can be set as needed, for example, 10ms (milliseconds). The target audio can include one or more audio frames, with "multiple" meaning at least two. The target audio can be long or short. The length can be determined based on a duration threshold, which can be set as needed, for example, 30 minutes. The target audio can be formatted standard audio. For example, it can be standard audio output from a front-end audio processing module.

[0127] Among related technologies, VAD (Voice Awareness Detection) focuses on pure speech scenarios. It has poor robustness in detecting speech activities in complex audio such as singing (e.g., separating melody, vocals and accompaniment) and music (e.g., music fragments of different styles). It is prone to misjudgment (e.g. misjudging vocals in music as silent, or being unable to distinguish effective vocals from background accompaniment in singing).

[0128] Based on this, for example, at least two non-speech categories include a singing category and a music category. Music refers to instrumental music, without lyrics or vocals. Singing includes lyrics. The difference between speech and singing is that speech has the intonation of human speech, while singing has the rhythm of a song. Therefore, speech, singing, and music can be distinguished, and for audio that contains both singing and speech (human speech), the singing portion and the speaking portion of the audio can be identified, achieving the distinction between human speech and accompaniment (or songs played in the environment).

[0129] For example, acoustic features of the target audio can be obtained, and based on these features, the audio frames in the target audio can be classified to obtain a target probability sequence for each audio frame. For instance, the acoustic features of the target audio can be input into a target audio detection model to obtain its output, which includes the target probability sequence for each audio frame in the target audio. The target audio detection model can obtain the target probability sequence for each audio frame in parallel.

[0130] In an exemplary embodiment, the target probability sequence is obtained by inputting the acoustic features of the target audio into a target audio detection model. The process of obtaining the target audio detection model includes: acquiring sample acoustic features, which are the acoustic features of the sample audio; inputting the sample acoustic features into the audio detection model to be trained, and obtaining the output result of the audio detection model, the output result containing the predicted probability sequence of each sample audio frame in the sample audio, the predicted probability sequence of the sample audio frame containing the predicted probability corresponding to multiple target audio categories respectively; and training the audio detection model based on the predicted probability sequence in the output result to obtain the target audio detection model.

[0131] Here, a sample audio frame refers to an audio frame within the sample audio. The prediction probability corresponding to the target audio category represents the probability that the sample audio frame belongs to that target audio category. The audio detection model can obtain the prediction probability sequence for each sample audio frame in parallel. For example, a prediction probability sequence of [0.9, 0.1, 0.0], where 0.9 corresponds to the speech category, 0.1 corresponds to the singing category, and 0.0 corresponds to the music category, indicates a 90% probability of speech (human speech). In this scheme, the probability can be, but is not limited to, a probability.

[0132] For example, an audio detection model can be iteratively trained using a sample audio set. A target audio detection model can then be selected from the trained models based on evaluation metrics such as the F1 score. The sample audio set can include sample audio for each target audio category. Each sample audio corresponds to a category label sequence, where each label corresponds one-to-one with the target audio category. If a label value is 0 or 1, a value of 1 indicates that the sample audio belongs to the target audio category corresponding to that label, while a value of 0 indicates that the sample audio does not belong to the target audio category corresponding to that label. Therefore, the loss corresponding to the sample audio can be calculated using the cross-entropy loss function. For example, the loss corresponding to the sample audio might be:

[0133]

[0134] Where C is the total number of categories. For example, if the number of target audio categories is 3, then C=3. This represents the prediction probability corresponding to the i-th target audio category. This represents the label (0 or 1) corresponding to the i-th target audio category in the category label sequence corresponding to the sample audio. The sample audio belongs to the k-th target audio category. , This represents the prediction probability corresponding to the k-th target audio category.

[0135] For example, the losses corresponding to multiple sample audio samples can be comprehensively calculated to obtain the model loss of the audio detection model. The model parameters of the audio detection model can then be updated using the model loss to achieve the purpose of training the audio detection model. The comprehensive calculation includes, but is not limited to, at least one of mean or summation calculations.

[0136] For example, the audio detection model can be a model based on a DFSMN architecture. For instance, the audio detection model may include a first acoustic model, which comprises at least one layer of DFSMN. Figure 5 The diagram illustrates the structure of an audio detection model, which includes a first projection layer, a first acoustic model, a second projection layer, and a first output layer. The first projection layer transforms the dimension of the spectral features to the input dimension of the first acoustic model. The first projection layer may include a dimension transformation layer, or it may include a feature extraction layer and a dimension transformation layer. The feature extraction layer can be implemented using at least one of CNN, VGG, or VGG-like models. The dimension transformation layer is used for feature flattening / reshaping. The DFSMN layer in the first acoustic model includes a memory module that can look forward L steps (loo-back L) and / or look backward R steps (loo-back R). For example, the input to the memory module at step t is... Then the output of the memory module for: The second projection layer can be used to implement dimensionality transformation and can be a pooling layer. The first output layer can contain a classification layer and a normalization layer. The normalization layer can be implemented using sigmoid or softmax. The probability matrix output by the first output layer contains probabilities, which are the likelihood degrees. The probability matrix is ​​T×C, where T represents the number of audio frames, C represents the number of target audio categories, and each row in the probability matrix represents the likelihood degree sequence (e.g., the prediction likelihood degree sequence) corresponding to an audio frame.

[0137] Typically, VAD models (such as Silero-VAD and FunASR-VAD) are primarily optimized for specific languages ​​(e.g., Chinese and English), making it difficult to cover speech / singing / music scenarios in 100+ languages. This leads to decreased VAD accuracy in cross-language applications (e.g., multilingual intelligent assistants and cross-border voice interaction systems). VAD models generally achieve an F1 score (precision × recall / (precision + recall)) below 95%, failing to meet the requirements for high-precision VAD (real-time meeting transcription, voice interaction, etc., require near 100% speech activity recognition accuracy). However, the sample audio set in this application covers speech / singing / music in 100+ languages, achieving an F1 score ≥ 97.5%, thus meeting the requirements for high-precision VAD.

[0138] In this embodiment, the target probability sequence is determined by the trained target audio detection model, which can improve the reliability and acquisition efficiency of the target probability sequence.

[0139] Step 304: Determine the matching audio category of the audio frame based on the probability in the target probability sequence of the audio frame. The matching audio category is any target audio category or the silence category.

[0140] The matched audio category can be the target audio category corresponding to the highest probability in the target probability sequence.

[0141] In an exemplary embodiment, determining the matching audio category of an audio frame based on the probability in the target probability sequence of the audio frame includes at least one of the following: if the probability corresponding to the speech category reaches a speech probability threshold, then the matching audio category of the audio frame is determined to be a speech category; if the probability corresponding to the speech category does not reach a speech probability threshold, but the probability corresponding to the singing category reaches a singing probability threshold, then the matching audio category of the audio frame is determined to be a singing category; if the probability corresponding to the speech category does not reach a speech probability threshold, the probability corresponding to the singing category does not reach a singing probability threshold, but the probability corresponding to the music category reaches a music probability threshold, then the matching audio category of the audio frame is determined to be a music category; if the probability corresponding to the speech category does not reach a speech probability threshold, the probability corresponding to the singing category does not reach a singing probability threshold, and the probability corresponding to the music category does not reach a music probability threshold, then the matching audio category of the audio frame is determined to be a silence category.

[0142] Here, "reaching" means greater than or equal to. The speech probability threshold, singing probability threshold, and music probability threshold can be determined based on the target audio detection model after training. Taking the determination of the speech probability threshold as an example, multiple audio segments corresponding to different speech categories are identified. The acoustic features of the audio segments are input into the target audio detection model to obtain the probability sequence corresponding to each audio frame in the audio segment. The probability sequence corresponding to each audio frame contains the probability that the audio frame belongs to each target audio category. The mean of the probability corresponding to the speech category in the probability sequence corresponding to each audio frame in the audio segment is taken as the probability that the audio segment belongs to the speech category. The probability of these multiple audio segments belonging to the speech category is formed into a probability set. The minimum, average, mode, or maximum value in the probability set is taken as the speech probability threshold. Alternatively, the speech probability threshold can be determined based on the target proportion. The target proportion can be set according to actual needs, for example, 90% or 95%. The probability of the target proportion in the probability set reaches the speech probability threshold. For example, 90% of the probability in the probability set reaches the speech probability threshold.

[0143] In this embodiment, the distinction between voice, song, music and silence categories is achieved through step-by-step determination.

[0144] Step 306: Determine valid audio segments from the target audio based on the matching audio categories of the audio frames in the target audio.

[0145] The matched audio category can be one of multiple target audio categories or a silence category. Valid audio segments can be non-silent segments. Therefore, boundary detection can be performed on the target audio based on the matched audio category of the audio frame to identify silent segments. The target audio can then be segmented from the boundary positions of the silent segments, and the non-silent segments other than the silent segments in each segmentation can be considered as valid audio segments. Alternatively, valid audio segments can also be determined based on the target scene.

[0146] For example, if the duration of the segmented non-silent segment is greater than or equal to a duration threshold, the non-silent segment is considered a valid audio segment, thereby eliminating short segments that are too short. The duration threshold can be set according to actual needs, for example, it can be 0.1s (seconds).

[0147] In one exemplary embodiment, determining a valid audio segment from the target audio based on the matching audio category of the audio frames in the target audio includes: determining a valid audio segment in the target scene from the target audio based on the target scene and the matching audio category of the audio frames in the target audio.

[0148] The target scene can be determined based on the source of the target audio, or it can be flexibly configured.

[0149] For example, the effective audio category can be determined based on the target scene. Then, based on the matching audio category of the target scene and audio frames in the target audio, the segment corresponding to the effective audio category can be determined from the target audio, thus obtaining the effective audio segment. For instance, in a voice input scenario, the effective audio category is the speech category. In a lyrics recognition scenario, the effective audio category could be the singing category.

[0150] In this embodiment, determining the effective audio segments through the target scene can improve the flexibility of effective audio segment recognition.

[0151] In the aforementioned audio detection method, target audio is acquired, and category recognition is performed on audio frames within the target audio to obtain a target probability sequence for each audio frame. This sequence contains the probability values ​​corresponding to multiple target audio categories. Based on the probability values ​​in the target probability sequence, the matching audio category of each audio frame is determined. Based on the matching audio category of each audio frame in the target audio, valid audio segments are identified from the target audio. Since multiple target audio categories include speech categories and at least two non-speech categories, and the matching audio category is any target audio category or the silence category, the method can distinguish between speech categories, at least two non-speech categories, and the silence category, improving the accuracy of category differentiation and thus enhancing the accuracy of audio detection.

[0152] In an exemplary embodiment, class identification is performed on audio frames in the target audio to obtain a target probability sequence of audio frames, including: acquiring acoustic features of the target audio; class identification is performed on each audio frame in the target audio in parallel based on the acoustic features to obtain a target probability sequence of audio frames; and determining valid audio segments from the target audio based on the matching audio categories of the audio frames in the target audio includes: performing boundary detection on the target audio based on the matching audio categories of the audio frames in the target audio to determine the boundary positions of valid audio segments; and segmenting the target audio at the boundary positions of the valid audio segments to obtain valid audio segments.

[0153] For example, based on the matching audio category of each audio frame, boundary detection is performed on the target audio. The detected boundaries are flanked by segments of different types. Segmentation at the boundaries yields audio segments of different categories, such as speech segments, singing segments, audio segments, or silence segments. For example, as... Figure 6 As shown, the system first determines the matching audio category of each audio frame. Then, based on the matching audio category of each audio frame, boundary detection is performed on the target audio to segment it into audio segments of different audio categories. From these segmented audio segments, audio segments belonging to valid audio categories can be selected to obtain valid audio segments.

[0154] Among them, such as Figure 7 The diagram illustrates the principle of audio processing in both streaming and non-streaming scenarios. A detection mode can be selected based on the target audio detection indication. When the indication indicates a non-streaming detection mode, the complete audio can be read, and frame-level probability calculations can be performed. Frame-level probability calculation refers to calculating the target probability sequence on a per-audio-frame basis. Then, boundary detection and segmentation are performed to obtain valid segments, i.e., valid audio segments.

[0155] In this embodiment, by performing category recognition on each audio frame in the target audio in parallel, the recognition efficiency can be improved in non-streaming scenarios.

[0156] In an exemplary embodiment, the target audio is an audio segment acquired in real time, and the audio category of a valid audio segment is a valid audio category. Category identification is performed on the audio frames in the target audio to obtain a target probability sequence of the audio frames. This includes: sequentially performing category identification on each audio frame in the target audio to obtain a target probability sequence for each audio frame; determining a valid audio segment from the target audio based on the matching audio categories of the audio frames in the target audio, including: determining the audio category to which the target audio belongs based on the matching audio categories of the audio frames in the target audio; the audio category to which the target audio belongs is any target audio category or a silence category; if the audio category to which the target audio acquired this time belongs is an invalid audio category, then multiple target audio segments acquired previously whose consecutive audio categories belong to valid audio categories are merged to obtain a valid audio segment.

[0157] The valid audio categories are voice or song. For example... Figure 7 In the case of a streaming detection mode indicated by the detection indication information, audio can be read in segments to obtain audio segments. Frame-level probability calculation is performed on the audio segments (i.e., the probability is calculated for each audio frame in turn), and boundary detection and segment integration are performed in real time. Segment integration refers to merging consecutive segments of valid audio categories to obtain valid segments, i.e., valid audio segments.

[0158] In this embodiment, each audio frame in the target audio is sequentially classified to obtain the target probability sequence of each audio frame, which can improve the real-time performance of recognition in streaming scenarios.

[0159] This application supports two modes: non-streaming (offline batch processing) and streaming (real-time processing). The non-streaming mode supports batch processing of long audio files (such as speech-to-text preprocessing of several hours) to ensure processing efficiency. The streaming mode supports real-time processing of short-frame audio streams (such as real-time voice streams) with a latency of ≤50ms, meeting the needs of real-time interactive scenarios. It can meet the needs of different application scenarios (such as speech-to-text preprocessing, real-time voice interaction, and intelligent audio monitoring), and has low latency and high robustness. It can be integrated into multilingual voice interaction platforms, intelligent conferencing systems, cross-border voice content review tools, and other products to improve the accuracy of voice processing in complex audio environments.

[0160] In an exemplary embodiment, frame-level probability calculation in a non-streaming scenario can also be a row-wise calculation of the probability that each audio frame belongs to a speech category. Specifically, the acoustic features of the target audio can be input into a non-streaming detection model to obtain the output of the non-streaming detection model. The output includes the probability that each audio frame in the target audio belongs to a speech category. Based on the probability that the audio frames in the target audio belong to a speech category, silent segments are determined from the target audio, and non-silent segments are segmented from the target audio as valid audio segments.

[0161] The non-stream detection model can be obtained by training the first detection model. The structure of the first detection model can be similar to that of the audio detection model. Figure 8 The diagram shows the structure of the first detection model, which may include a third projection layer, a second acoustic model, a fourth projection layer, and a second output layer. The second output layer outputs the probability of each frame, that is, the likelihood of each audio frame belonging to a speech category. The explanation of the second acoustic model can be found in the explanation of the first acoustic model, and will not be repeated here.

[0162] In an exemplary embodiment, frame-level probability calculation in a streaming scenario can also involve sequentially calculating the probability of each audio frame belonging to a speech category. Specifically, audio frames can be sequentially determined from the target audio, and the spectral features of the audio frames can be input into a streaming detection model to obtain the output of the streaming detection model. This output includes the probability of a single audio frame belonging to a speech category. The probability of each audio frame belonging to a speech category is obtained. Based on the probability of audio frames in the target audio belonging to a speech category, the probability of the target audio belonging to a speech category is determined. Based on the probability of the target audio belonging to a speech category, it is determined whether the target audio belongs to the speech category or the silence category. For the target audio belonging to the silence category, valid speech segments are determined from the target audio.

[0163] The streaming detection model can be obtained by training a second detection model. The structure of the second detection model can be similar to that of the audio detection model. For example... Figure 9 The diagram shows the structure of the second detection model, which may include a fifth projection layer, a third acoustic model, a sixth projection layer, and a third output layer. The third output layer outputs the probability of a single audio frame, i.e., the likelihood that a single audio frame belongs to a speech category. Activation layers can be implemented using activation functions (such as ReLU or Swish). The explanation of the third acoustic model can refer to the first acoustic model. It should be noted that the memory module in the third acoustic model may include a look-back step (L steps) instead of a look-back step (R steps).

[0164] The audio detection method provided in this application can segment, demute, or remove noise from long audio input by users in voice search scenarios; segment, demute, or remove noise from long audio input by users in voice comment scenarios; segment, demute, or remove noise from long audio input by users in voice note scenarios; segment, demute, or remove noise from long audio input by users in video CC subtitle scenarios; segment, demute, or remove noise from audio sources in medium- to long videos; and segment, demute, or remove noise from long audio input by users in voice private messaging scenarios.

[0165] The models used in this application support deployment on edge devices (such as mobile devices or embedded devices). The models used meet the following requirements: by compressing the models (such as quantization or pruning) or designing a lightweight architecture, the consumption of computing resources is reduced while ensuring performance (such as model size ≤ 10MB, inference speed real-time rate RTF <= 0.01).

[0166] The aforementioned audio detection method, based on a DFSMN-based VAD model design, leverages DFSMN's ability to capture audio temporal dependencies, addressing the shortcomings of traditional VAD models in modeling complex temporal features (such as dynamic changes in speech / music) and improving detection accuracy. It supports over 100 languages ​​and speech / singing / music scenarios, overcoming the poor generalization and weak scenario adaptability of traditional single-language VADs, thus expanding application coverage. It achieves integrated support for non-streaming and streaming VADs: through the DFSMN architecture, it simultaneously supports complete audio processing (non-streaming) and real-time streaming processing (streaming), overcoming the limitation of traditional VAD models with single scenario adaptation (such as offline only or real-time only), and improving system flexibility. It supports extended audio event detection (aED) functionality: on top of VAD, it supports the detection of non-speech events such as music / ambient noise, overcoming the limitation of traditional VADs focusing only on speech / silence classification, and enhancing analysis capabilities in complex audio scenarios. The VAD model (e.g., target audio detection model) in this application outperforms traditional models in metrics such as F1 and AUC-ROC, and can address the problem of insufficient accuracy in multilingual VAD tasks, thereby improving detection reliability.

[0167] In an exemplary embodiment, the method further includes: determining language identification audio based on a valid audio segment; performing language identification on the language identification audio to obtain a target language identifier, the target language identifier representing one of multiple languages; performing sub-language identification on the language identification audio based on the target language identifier to obtain a target sub-language identifier, the sub-language represented by the target sub-language identifier belonging to the language represented by the target language identifier; and determining the language identification result of the language identification audio based on the target language identifier and the target sub-language identifier.

[0168] For example, valid audio segments can be used as speech recognition audio, or multiple valid audio segments can be integrated to obtain speech recognition audio, or valid audio segments can be segmented to obtain speech recognition audio.

[0169] In this embodiment, the accuracy of language recognition can be improved because the reliability of the valid audio segments is enhanced.

[0170] In related technologies, there is a lack of usable Chinese dialect recognition capabilities. For example, although the vocabulary includes dialect tokens, it is almost impossible to output the corresponding tokens when inputting Wu or Minnan dialects; the online performance is "all degraded to zh or misjudged as other languages". The reasons include: (1) Dialect tokens lack effective supervision: The vocabulary "has categories", but the dialect samples in the training data / labeling / sampling are scarce or unstable, which causes the model to not learn the dialect decision boundary. (2) The training target is biased towards language rather than dialect: Many models mainly optimize "language recognition", and the internal subdivision of Chinese is treated as a weak task or not included in the main loss at all, so the dialect ability naturally degrades.

[0171] In an exemplary embodiment, language identification is performed on the language identification audio to obtain the language identification result of the language identification audio, including: matching the acoustic features of the language identification audio with the language acoustic features corresponding to multiple languages, wherein the multiple languages ​​include a first language and a second language, and the first language includes multiple sub-languages; if the successfully matched language acoustic feature corresponds to the first language, then the acoustic features of the language identification audio are matched with the sub-language acoustic features corresponding to the multiple sub-languages; and the language identification result of the language identification audio is determined based on the language corresponding to the successfully matched language acoustic feature and / or the sub-language corresponding to the successfully matched sub-language acoustic feature.

[0172] The acoustic features corresponding to each of the multiple languages ​​and the acoustic features corresponding to each of the multiple sub-languages ​​were pre-extracted. The acoustic features corresponding to each language reflect the acoustic characteristics of the audio in that language. The acoustic features corresponding to each sub-language reflect the acoustic characteristics of the audio in that sub-language.

[0173] For example, if the successfully matched acoustic features of a language correspond to a second language, the language identification result is determined based on the language identifier (i.e., language label) of the second language.

[0174] In this embodiment, by matching with pre-generated language acoustic features or sub-language acoustic features, language recognition results can be generated accurately and quickly.

[0175] In one exemplary embodiment, a language identification method is provided for identifying the language of audio and obtaining the language identification result of the audio. The method includes:

[0176] Step 402: Perform language identification on the language identification audio to obtain the target language identifier, which represents one of multiple languages.

[0177] The language identification audio can be long or short. The language identifier is used to uniquely identify the language. The language identifier can be, but is not limited to, a token. The target language identifier is the language identifier identified from the language identification audio. The length of the audio (long or short) can be determined based on a duration threshold, which can be, for example, 30 minutes. The target audio can be formatted standard audio. For example, it can be standard audio output from a front-end audio processing module.

[0178] The language refers to the primary language. Primary languages ​​can be, but are not limited to: zh (Chinese), en (English), ja (Japanese), ko (Korean), fr (French), es (Spanish), ru (Russian), de (German), and pt (Portuguese). This primary language can be divided into second and third languages. The first language corresponds to several sub-languages ​​(secondary languages).

[0179] In an exemplary embodiment, the first language represents the language used in the target region, which includes multiple sub-regions, and the sub-languages ​​cover the dialects used in at least two sub-regions of the target region.

[0180] The target region can be a country. Sub-regions can be cities or provinces within the country. Sub-languages ​​can be a combination of dialects from multiple provinces or cities.

[0181] Taking "zh" as the first language as an example, the sub-languages ​​can be, but are not limited to: Mandarin: Standard Mandarin; Yue: Cantonese; Wu: Wu dialect (such as Shanghainese, Suzhou dialect, etc.); Min: Min dialect (including Southern Min, Chaoshan dialect, etc.); Xiang: Xiang dialect (such as Changsha dialect, etc.); North: Northern dialects, such as the dialects of Shandong, Gansu, Ningxia, Hebei, Shanxi, Liaoning, Shaanxi, etc.; Xinan: Southwestern dialects, such as the dialects of Sichuan, Yunnan, Guizhou, Hubei, Chongqing, etc. (Southwestern Mandarin); Jianghuai: Jianghuai dialects, such as the dialects of Anhui, Jiangsu, etc. (Jianghuai Mandarin); Dongbei: Northeastern dialects, referring to the dialects of Northeast China.

[0182] The characteristics of Northern dialects include: geographical continuity, significant influence from population migration and modern common language → wide coverage and strong interoperability; differences from Standard Mandarin are mostly at the accent level (phonetic details and a few vocabulary differences), and generally, communication is possible without systematic learning. These dialects can be considered as a broad category.

[0183] The characteristics of Wu, Min, Yue, Pinghua, Gan, and Xiang dialects include: geographical separation by mountains and rivers, deep historical layers → small regional areas, significant differences, and weak intercommunication; they often have systematic differences from Standard Mandarin (common features include preserved entering tones, more tones, complex tone sandhi, and different systems of commonly used words / auxiliary words), often requiring learning to understand. Therefore, they need to be classified separately.

[0184] Southwestern dialects are characterized by being relatively similar to each other and easily mutually intelligible, but distinct from Northern dialects. These dialects can be classified as a broad category.

[0185] The characteristics of the Jianghuai dialect include: it differs significantly from the southwestern and northern dialects, but it is closer to Standard Mandarin than Wu, Min, Yue, Pinghua, Gan, Xiang, and Pinghua dialects, so these dialects can be considered as a major category.

[0186] In this embodiment, dialects are classified according to "province level", which is closer to the category boundaries that can be acoustically distinguished.

[0187] In an exemplary embodiment, language recognition and sub-language recognition are implemented based on a target language recognition model. The process of obtaining the target language recognition model includes: obtaining a first language training sample, which contains a first acoustic feature corresponding to the first language audio and a language identifier pair corresponding to the first language audio, wherein the language of the first language audio is the first language, and the language identifier pair contains a language identifier of the first language and a labeled sub-language identifier; obtaining a second language training sample, which contains a second acoustic feature corresponding to the second language audio and a language identifier of the second language, wherein the language of the second language audio is the second language; and training the language recognition model to be trained using the first language training sample and the second language training sample to obtain the target language recognition model.

[0188] This can involve one primary language and multiple different secondary languages. The training samples for the primary language contain both primary language identifiers and secondary language identifiers (e.g., dialect identifiers). The training samples for the secondary languages ​​do not contain secondary language identifiers. The sub-language identifier represents the actual sub-language to which the primary language audio belongs; any sub-language identifier can be used. Sub-language classification can be based on the degree of dialectal difference. For example, using the zh symbol, "Chinese dialects closer to Mandarin" can be classified as one sub-language, while "Chinese dialects with significant differences" can be classified as different sub-languages.

[0189] The first language training samples can be obtained from a first language training sample set. The first language training sample set contains multiple first language training samples, and also contains first language training samples corresponding to each sub-language identifier. Specifically, if a first language training sample contains a sub-language identifier, then the sub-language identifier is considered to correspond to a first language training sample.

[0190] In language tagging, the language tag for the first language can be placed before the sub-language tags (this reduces boundary noise and allows for learning the language first, followed by internal subdivisions, such as in Chinese). For example, the first language training sample set could be: {(acoustic feature 1: (zh, mandarin)); (acoustic feature 2: (zh, north)); (acoustic feature 3: (zh, xinan))}. Here, (acoustic feature 1: (zh, mandarin) is a first language training sample.

[0191] Second language training samples can be selected from a second language training sample set. This set contains multiple second language training samples, and each second language training sample contains a second language training sample corresponding to its language identifier. If a second language training sample contains a second language identifier, then that language identifier is considered a second language training sample. For example, the second language training sample set could be: {(acoustic feature 4, en); (acoustic feature 5, ja); (acoustic feature 6, ko)}. Here, (acoustic feature 4, en) is a second language training sample.

[0192] For example, the second language can be selected from the primary languages ​​other than the first language, such as high-frequency primary languages.

[0193] Among them, the loss function used in the language recognition model training process can be the cross-entropy loss function, which is used to optimize the discrimination of "final category token (language / dialect)". The backbone of the language recognition model can adopt an Encoder-Decoder structure. And the encoder of the language recognition model can be initialized with parameters from the decoder in the first audio transcription model or the second audio transcription model. This can enhance the audio understanding ability and stability of the encoder in the language recognition model. That is to say, when training the language recognition model, instead of starting from random initialization, the Encoder parameters that have been trained on the large-scale speech recognition (ASR) task are reused / transferred as the initialization parameters of the language recognition model. The benefits of this include: (1) Enhanced acoustic understanding: By utilizing the powerful extraction ability of ASR model for phonemes and prosody, the feature extraction accuracy of the language recognition model for short audio and ambiguous speech can be improved; (2) Accelerated convergence and stability: In the case of relatively scarce dialect data, this transfer learning makes the model training converge faster and has stronger generalization ability, which can avoid overfitting as much as possible.

[0194] In an exemplary embodiment, a language recognition model to be trained is obtained by using training samples in a first language and training samples in a second language. The process includes: inputting training samples in the first language into the language recognition model to be trained to obtain a first sample confidence sequence, the first sample confidence sequence containing first sample confidence scores corresponding to multiple language identifiers and multiple sub-language identifiers; inputting training samples in the second language into the language recognition model to obtain a second sample confidence sequence, the second sample confidence sequence containing second sample confidence scores corresponding to multiple language identifiers and multiple sub-language identifiers; generating a first loss based on the first sample confidence scores corresponding to the multiple language identifiers and multiple sub-language identifiers, and generating a second loss based on the second sample confidence scores corresponding to the multiple language identifiers; and training the language recognition model based on the first loss and the second loss to obtain the target language recognition model.

[0195] The first sample confidence sequence includes the first sample confidence scores corresponding to multiple language identifiers and the first sample confidence scores corresponding to multiple sub-language identifiers. The first sample confidence score corresponding to a language identifier reflects the probability that the first language audio belongs to the language represented by that language identifier. The first sample confidence score corresponding to a sub-language identifier reflects the probability that the first language audio belongs to the sub-language represented by that sub-language identifier.

[0196] The second sample confidence sequence includes the second sample confidence scores corresponding to multiple language identifiers and the second sample confidence scores corresponding to multiple sub-language identifiers. The second sample confidence score corresponding to a language identifier reflects the probability that the second language audio belongs to the language represented by that language identifier. The second sample confidence score corresponding to a sub-language identifier reflects the probability that the second language audio belongs to the sub-language represented by that sub-language identifier.

[0197] For example, when the confidence level of the first sample is logits, the first loss can be generated by combining the cross-entropy loss function and the normalized confidence levels of each first sample. When the confidence level of the first sample is a probability, the first loss can be generated by combining the cross-entropy loss function and the confidence levels of each first sample.

[0198] For example, when the second sample confidence level is logits, a second loss can be generated by combining the cross-entropy loss function and the normalized second sample confidence levels corresponding to the multiple language identifiers. When the second sample confidence level is probability, a second loss can be generated by combining the cross-entropy loss function and the second sample confidence levels corresponding to the multiple language identifiers. That is, the second sample confidence levels corresponding to the multiple sub-language identifiers are not used when generating the second loss.

[0199] In this embodiment, for the first language (e.g., Chinese), the confidence scores of the first samples corresponding to the multiple sub-language identifiers are used to calculate the loss, while for the second language, the confidence scores of the second samples corresponding to the multiple sub-language identifiers are not used to calculate the loss, thereby avoiding mutual interference between the sub-languages ​​and the second language.

[0200] In an exemplary embodiment, the first acoustic feature corresponding to the first language audio is an acoustic feature extracted from the first language audio, or an acoustic feature extracted from the perturbed first language audio after perturbing the first language audio; the second acoustic feature corresponding to the second language audio is an acoustic feature extracted from the second language audio, or an acoustic feature extracted from the perturbed second language audio after perturbing the second language audio.

[0201] The perturbation can be a random augmentation of the audio. For example, perturbation can be adding noise. Noise includes, but is not limited to, additive noise (such as street noise, crowds, car interior noise, or wind noise) and SNR random noise. Perturbation can be speed perturbation and / or volume perturbation. Perturbation can also be mixing in background music / accompaniment (to match the commentary scene), etc. Perturbation can also be achieved through reverb / room response (RIR).

[0202] For example, the language recognition model can be trained iteratively multiple times. During the iteration, the first acoustic feature and / or the second acoustic feature can be updated. For example, after one round of training, a new first acoustic feature can be generated by updating the perturbation.

[0203] In this embodiment, by obtaining the first acoustic feature or the second acoustic feature through perturbation, the language recognition model can maintain a stable recall rate in real-world scenarios (such as voice comments) and improve its robustness to short audio clips and accents.

[0204] For example, acoustic features of the language identification audio can be extracted, and language identification can be performed based on these acoustic features to obtain the target language identifier. Alternatively, the acoustic features of the language identification audio can be input into a target language identification model for language identification, and the target language identifier can be determined based on the output of the target language identification model.

[0205] In this embodiment, by training a language recognition model, multiple languages ​​plus Chinese dialects can be identified (and subsequent expansion to include new categories are supported). The accuracy for each language / dialect on the business test set is ≥90%, and the recall rate is ≥95%. Applied to the speech recognition pipeline, it can reduce pipeline latency and support online real-time inference and high-concurrency deployment. Robustness to short audio clips: Voice comments are often short, noisy, and have heavy accents; the model needs to output stably in short durations and complex environments.

[0206] In an exemplary embodiment, obtaining a target language identifier by performing language identification on language-identified audio includes: segmenting the language-identified audio to obtain multiple target audio segments; performing language identification on each target audio segment to obtain a first segment confidence sequence corresponding to the target audio segment, the first segment confidence sequence containing the first segment confidence corresponding to multiple language identifiers respectively; combining the first segment confidence sequences corresponding to at least some of the target audio segments to determine the language confidence sequence corresponding to the language-identified audio; and determining the target language identifier based on the language confidence in the language confidence sequence.

[0207] Here, "multiple languages" refers to multiple first-level languages. Language identifiers can be, but are not limited to, language tokens. Confidence reflects probability. The confidence level of the first segment corresponding to the language identifier reflects the probability that the target audio segment belongs to the language represented by that language identifier.

[0208] The language confidence sequence contains the language confidence scores corresponding to multiple language identifiers. Each language identifier's corresponding language confidence score reflects the probability that the audio belongs to the language represented by that identifier. The confidence score can be a probability or logits. When segmenting the language recognition audio, it can be done according to a fixed-size time window. The size of the time window can be set according to actual needs, for example, 10 seconds, meaning it can be segmented every 10 seconds.

[0209] For example, the confidence sequences of the first segments corresponding to all target audio segments can be comprehensively calculated (e.g., by mean calculation or weighted calculation) to obtain the language confidence sequence.

[0210] For example, low-energy or low-confidence target audio segments can be filtered out from the multiple target audio segments, and the confidence sequences of the first segments corresponding to the remaining target audio segments can be comprehensively calculated (e.g., mean calculation or weighted calculation) to obtain the language confidence sequence.

[0211] For example, the language identifier corresponding to the highest language confidence in the language confidence sequence can be used as the target language identifier.

[0212] For example, acoustic features of a target audio segment can be extracted, and these features can be input into a target language recognition model for language recognition to obtain a first recognition result for the target audio segment. The first recognition result includes a first segment confidence sequence and a language identifier corresponding to the highest confidence level in the first segment confidence sequence. Specifically, the first recognition result may include the first segment confidence level corresponding to each language identifier and each sub-language identifier. The first segment confidence sequence includes the first segment confidence levels corresponding to each language identifier in the first recognition result, but does not include the first segment confidence levels corresponding to sub-language identifiers in the first recognition result.

[0213] In this embodiment, the target language identifier is determined by combining the language identification of multiple target audio segments, which can improve the accuracy and reliability of the target language identifier.

[0214] In one exemplary embodiment, at least some target audio segments are selected from multiple target audio segments according to their confidence level. In this way, by filtering target audio segments with low confidence, the interference of background music (BGM) or sudden noise on the overall judgment can be automatically eliminated, thereby improving noise resistance.

[0215] Step 404: Based on the target language identifier, perform sub-language identification on the language identification audio to obtain the target sub-language identifier. The sub-language represented by the target sub-language identifier belongs to the language represented by the target language identifier.

[0216] The sub-language identifier is used to uniquely identify a sub-language. The target sub-language identifier is the sub-language identifier identified from the audio for language identification.

[0217] For example, the target language identifier and the acoustic features of the language recognition audio can be input into the target language recognition model to perform sub-language recognition and obtain the output target sub-language identifier.

[0218] In an exemplary embodiment, sub-language identification is performed on language identification audio based on a target language identifier to obtain a target sub-language identifier, including: when the identified language identifier represents a first language, sub-language identification is performed on language identification audio based on the target language identifier to obtain a target sub-language identifier; the method further includes: when the identified language identifier represents a second language, determining the language identification result of the language identification audio based on the target language identifier.

[0219] The first language corresponds to multiple sub-languages, and the target sub-language identifier represents any one of these sub-languages. The second language is different from the first language.

[0220] In this embodiment, it can support the identification of two levels of languages ​​as well as the identification of a single level of languages, thus improving the flexibility of language identification. A conditionally dependent output structure is implemented: the first level first performs coarse-grained language classification of the audio (such as Chinese, English, Japanese, etc.), and the second level of dialect subdivision result (such as Cantonese, Wu dialect, Southwestern dialect, or Northern dialect, etc.) is activated or output only when the first level classification result is the first language, such as "Chinese (zh)".

[0221] It retains the general language recognition capability while also achieving in-depth understanding of Chinese dialects, facilitating flexible model loading by downstream ASR based on hierarchical results (e.g., if the first level is not zh, it directly follows the foreign language link; if the first level is zh but there is no dialect, it follows the Mandarin link), improving business flexibility. It can minimize the confusion caused by traditional flat classification (juxtaposing dialects with foreign languages) (e.g., no longer misclassifying foreign languages ​​with similar pronunciations as a certain type of Chinese dialect). Addressing the issue of "dialect tokens existing but recall rate of 0" in related technologies, by enforcing hierarchical constraints, the model focuses on distinguishing dialect features only when it is confirmed to be Chinese, thus improving the recall rate of dialect tokens.

[0222] In an exemplary embodiment, when the target language identifier represents a first language, sub-language identification is performed on the language identification audio based on the target language identifier to obtain a target sub-language identifier, including: determining at least two reference audio segments from at least a portion of the target audio segments, wherein the first segment confidence sequence corresponding to the reference audio segments indicates that the reference audio segments belong to the first language; combining the second segment confidence sequences corresponding to the at least two reference audio segments respectively to determine a sub-language confidence sequence corresponding to the language identification audio, wherein the second segment confidence sequence contains the second segment confidence corresponding to multiple sub-language identifiers respectively, and the multiple sub-language identifiers represent multiple sub-languages ​​belonging to the first language; and determining the target sub-language identifier based on the sub-language confidence in the sub-language confidence sequence.

[0223] In this process, if the first segment confidence sequence of the target audio segment indicates that the target audio segment belongs to the first language (i.e., the language represented by the language identifier corresponding to the highest confidence in the first segment confidence sequence is the first language), then sub-language identification is performed on the target audio segment based on the language identifier of the first language to obtain the second segment confidence sequence corresponding to the target audio segment. For example, the language identifier of the first language and the acoustic features of the target audio segment are input into the target language identification model to obtain the second identification result of the target audio segment. The second identification result may include the second segment confidence corresponding to each language identifier and each sub-language identifier, but the second segment confidence sequence includes the second segment confidence corresponding to each sub-language identifier in the second identification result, but does not include the second segment confidence corresponding to the language identifier in the second identification result. The second segment confidence corresponding to the sub-language identifier reflects the probability that the reference audio segment belongs to the language represented by the sub-language identifier.

[0224] The sub-language confidence sequence contains the sub-language confidence scores corresponding to multiple sub-language identifiers. The sub-language confidence score corresponding to a sub-language identifier reflects the probability that the audio belongs to the language represented by the sub-language identifier.

[0225] For example, the sub-language identifier corresponding to the highest sub-language confidence in the sub-language confidence sequence can be used as the target sub-language identifier.

[0226] For example, for each sub-audio segment, the acoustic features of the sub-audio segment can be determined, the acoustic features of the sub-audio segment can be input into the target language recognition model, and the target sub-language identifier can be determined based on the output results.

[0227] In this embodiment, the target sub-language identifier is determined by comprehensively considering the sub-language identification results of at least two reference audio segments, which can improve the accuracy and reliability of the target sub-language identifier. A multi-window aggregation and weighted decision mechanism based on short audio window slicing is implemented. Specifically, considering the varying duration of voice comments, the audio is divided into multiple windows of fixed duration for inference. In the post-processing stage, "window-level confidence filtering" and "weighted voting" are combined to generate the final result, which can improve the robustness of short audio and solve the problem of misjudgment that might occur if a single inference occurs precisely in a silent or noisy segment.

[0228] Step 406: Determine the language identification result of the audio based on the target language identifier and the target sub-language identifier.

[0229] The language identification result may include a target language identifier and a target sub-language identifier, and may also include at least one of a first confidence level, a second confidence level, or a third confidence level. The first confidence level refers to the confidence level corresponding to the language represented by the target language identifier (e.g., the largest element in the first comprehensive confidence level sequence). The second confidence level refers to the confidence level corresponding to the sub-language represented by the target sub-language identifier (e.g., the largest element in the sub-language confidence level sequence). The third confidence level is the product of the first and second confidence levels.

[0230] In some exemplary embodiments, language identification of the audio is further performed by obtaining a language confidence sequence, which includes multiple language confidence levels corresponding to different language identifiers. The target language identifier is the language identifier corresponding to the highest language confidence level in the language confidence sequence. Based on the target language identifier and the target sub-language identifier, the language identification result of the audio is determined, including: comparing the highest and second-highest language confidence levels in the language confidence sequence; if the confidence difference between the highest and second-highest language confidence levels is less than a difference threshold, then the language identification result of the audio is generated based on the language identifiers corresponding to the highest and second-highest language confidence levels; if the confidence difference reaches the difference threshold, then the language identification result of the audio is generated based on the language identifier corresponding to the highest language confidence level and the target sub-language identifier.

[0231] The highest and second-highest language confidence scores refer to the two (Top 2) language confidence scores obtained from the language confidence score sequence in descending order of language confidence score. The difference threshold can be set according to actual needs.

[0232] If "the language identification result of the audio is generated based on the language identifiers corresponding to the highest and second-highest language confidence scores", then the language identification result may include: the language identifier corresponding to the highest language confidence score and the language identifier corresponding to the second-highest language confidence score, and may also include the highest and / or second-highest language confidence scores.

[0233] If "based on the language identifier corresponding to the maximum language confidence and the target sub-language identifier, generate the language identification result of the language identification audio", then the language identification result may include: the language identifier corresponding to the maximum language confidence and the target sub-language identifier, and may also include the maximum language confidence and / or the sub-language confidence corresponding to the target sub-language identifier.

[0234] For example, if the confidence difference is less than the difference threshold, the language identification result of the language identification audio is determined to be a mixed language tag (e.g., MIX tag). The mixed language tag is used to characterize that the language identification audio covers at least two languages.

[0235] In this embodiment, the judgment of mixed languages ​​is achieved by comparing the confidence scores of the highest and second-highest languages. In cases of obvious language mixing, the language identification result is determined based on the language identifiers corresponding to the highest and second-highest language confidence scores, which can improve the reliability of the language identification result. It can identify complex situations such as "mixed Chinese and English" (Conflict Detection), supports mixed language detection, and provides richer information for downstream applications.

[0236] In some exemplary embodiments, comparing the highest and second-highest language confidence scores in the language confidence score sequence includes: when the highest language confidence score reaches a confidence threshold, comparing the highest and second-highest language confidence scores in the language confidence score sequence; the method further includes: when the highest language confidence score is less than the confidence threshold, determining the language recognition result of the language recognition audio as an unknown language identifier, or determining the language recognition result of the language recognition audio based on the language identifier corresponding to the highest language confidence score.

[0237] The confidence threshold can be set according to actual needs. The language unknown is indicated as "unknown".

[0238] If "the language identification result of the audio is determined based on the language identifier corresponding to the maximum language confidence score", then the language identification result includes the language identifier corresponding to the maximum language confidence score and the maximum language confidence score, but does not include the target sub-language identifier.

[0239] In this embodiment, language recognition results are generated according to the relationship between the maximum language confidence score and the confidence score threshold, which can maximize the reliability of language recognition results under different circumstances.

[0240] In the above language identification method, the following steps are taken: First, language identification audio is acquired. Then, language identification is performed on the audio to obtain a target language identifier, which represents one of multiple languages. Next, sub-language identification is performed on the audio based on the target language identifier to obtain a target sub-language identifier. The sub-language represented by the target sub-language identifier belongs to the language represented by the target language identifier. Finally, the language identification result of the audio is determined based on the target language identifier and the target sub-language identifier. This two-level / hierarchical language identification approach minimizes the confusion between sub-languages ​​and the main language, and also avoids the situation where only the main language is identified, rendering the sub-language unusable, thus improving the accuracy and reliability of language identification.

[0241] The language recognition method provided in this application can be used in audio processing workflows (such as speech recognition workflows), such as... Figure 10 As shown, a schematic diagram of an audio processing flow is provided, including:

[0242] The front-end processing module standardizes the input audio to generate standard audio, featuring resampling, normalization, audio segmentation, and feature extraction. For example, it can acquire audio streams or files, convert them to mono with a fixed sampling rate (e.g., 16kHz), and perform amplitude normalization and filter out silence or abnormal segments. It can determine the language based on the duration of the standard audio. If determined to be long, it can perform VAD (Voice over Audio) detection to extract valid audio segments (e.g., segmenting suspected human voices). If determined to be short, it can directly input the standard audio (which is valid audio, such as voice commentary) into the language recognition module. This reduces the amount of audio from different sources while ensuring acoustic consistency and minimizing interference from invalid silence on the Language Identification (LID) module.

[0243] The language identification module takes as input the audio or audio segment output from the front-end processing module. It determines the language identification audio based on the received audio or audio segment, segments the audio (e.g., windowing), extracts acoustic features from the segmented target audio segments, and then uses the target language identification model to identify the language of the target audio segment, determining its primary language (primary output: language (token level)). If the primary language is zh (Chinese), it combines zh (Chinese) and the acoustic features of the target audio segment to determine its secondary language (secondary output: dialect), and then proceeds to the post-processing aggregation module. If the primary language is not zh (Chinese), there is no need to identify the secondary language, and the audio proceeds directly to the post-processing aggregation module. Optional business parameters for the language identification module include: desired output granularity (language only / language + Chinese dialect) and confidence threshold.

[0244] The post-processing aggregation module filters out low-energy / low-confidence target audio segments from each target audio segment. It then performs weighted fusion on the first segment confidence sequences corresponding to the remaining target audio segments to obtain a language confidence sequence. If the highest language confidence in the language confidence sequence reaches a confidence threshold, mixed language detection is performed; otherwise, an unknown language identifier is output or a backoff strategy is executed. If significant conflicts are detected in the mixed language detection, a mixed language label or language identifiers corresponding to the highest and second-highest language confidence are output, i.e., a mixed language label and Top 2 candidates are output. If the mixed language detection results show high consistency, the language identifier corresponding to the highest language confidence and the target sub-language identifier (i.e., Top 1 language and dialect) are output. The language recognition module can output the language recognition results in a structured format.

[0245] Language recognition results can be applied to downstream operations. Downstream ASR routing modules can load corresponding recognition models and select appropriate post-processing workflows based on language / dialect. Content tagging, review, or recommendation processes can distribute and statistically analyze data based on language / dialect. It can also be used to identify erroneous or low-confidence samples downstream.

[0246] By identifying erroneous / low-confidence samples downstream, a self-circulating iterative system (data closure and continuous upgrades) can be achieved. This self-circulating iterative system is a periodic closed loop established to ensure the long-term usability and continuous improvement of the model, encompassing online sampling → manual label submission → incremental training / retraining → gray-scale evaluation → release. Specifically:

[0247] 1. Online data retrieval (period T): Audio is sampled online according to rules at regular intervals (covering new regions, new entry points, new noise conditions, low confidence samples, and downstream ASR high error samples).

[0248] 2. Submission and Quality Inspection: Manually label the language / Chinese dialect (including mixed language labels); conduct consistency verification and arbitration for disputed samples.

[0249] 3. Training set update: Add newly labeled data to the training set; resample historical data to maintain class balance; and add a new round of noise augmentation (noise / reverberation / speed change / BGM) strategies.

[0250] 4. Training and Evaluation: Evaluate the recall rate of each language / dialect on the voice comment service test set to ensure that it meets the launch threshold (such as 95%); track specific indicators for easily confused pairs.

[0251] 5. Gray release: small-scale verification → full release; continuous online monitoring of confidence distribution, confusion matrix and downstream ASR revenue.

[0252] 6. Reflow and Iteration: Reflow failed cases from the online environment back to step 1 to form a continuous loop.

[0253] The self-looping iterative system can transform low-confidence online samples into high-value training data, address cold start and long-tail issues (as the business progresses, the model can automatically adapt to new accents or new noise environments), and continuously ensure high performance metrics (ensuring that the recall rate does not decay over time after the model goes live, supporting the long-term full operation of the business).

[0254] The language recognition method provided in this application can be applied to the "voice comment" service, enabling the "voice comment" service to be expanded to multilingual, multi-regional and Chinese dialect scenarios. In the "voice comment" service, the language / dialect of the user's audio can be identified first, and then the ASR link to be followed can be determined (such as selecting the corresponding language recognition model, sensitive word strategy or content tagging system, etc.).

[0255] The language recognition method provided in this application, when applied to the "voice comment" service, also enables the "voice comment" service to provide language / dialect tags. This allows the voice comment service to generate reliable "language / dialect" tags for user display, increasing the interactivity of the voice comment section. The language recognition method provided in this application achieves state-of-the-art (SOTA) performance in multilingual and Chinese dialect classification.

[0256] In this application, punctuation restoration can be achieved based on a target punctuation prediction model. In an exemplary embodiment, a punctuation prediction method is provided, which can be used to obtain a target punctuation prediction model and perform punctuation restoration. Taking the application of this method to a server as an example, the method includes the following steps:

[0257] Step 502: Obtain the set of user texts corresponding to the target punctuation mark, where the user texts in the set contain the target punctuation mark.

[0258] The target punctuation mark can be any punctuation mark, including but not limited to commas (","), question marks ("."), exclamation marks ("!"), colons (":"), pause marks ("、"), and semicolons (";"). User text is text written or published by users.

[0259] For example, the user text set includes user text of type 1 and user text of type 2, where user text of type 1 has lower sentence structure completeness than user text of type 2, and / or user text of type 1 has lower logical rigor than user text of type 2.

[0260] The first type of user text can be collected from the first type of scenario. The second type of user text can be collected from the second type of scenario. The first type of scenario can be everyday conversational scenarios, including but not limited to daily communication, content evaluation, and casual chat, and can include song subtitles, live stream comments, or text transcribed from speech. The second type of scenario can be standard written language scenarios, including but not limited to business consultation, work coordination, formal explanations, and project communication. User text in the first type of scenario mainly consists of everyday conversational statements with incomplete syntactic structures but clear semantics, while user text in the second type of scenario mainly consists of standardized expressions with complete sentence structures and rigorous logic. Both types of scenarios exhibit stable punctuation usage preferences.

[0261] In this embodiment, different types of user text can cover everyday spoken language scenarios and standard written language scenarios, making punctuation prediction more closely match the corresponding scenarios.

[0262] Step 504: Extract the usage preferences of target punctuation marks in the user text contained in the user text set to obtain the usage preference features corresponding to the target punctuation marks.

[0263] The usage preference features include at least one of scenario features, semantic features, and sentence structure features. A mapping relationship can be established between usage preference features and the type of target punctuation, for example, (scenario type - semantic feature - sentence structure - punctuation type). This clarifies the differences and commonalities in the use of different punctuation marks in spoken and written language scenarios. For example, the exclamation mark is suitable for both spoken emotional evaluation and written formal notification scenarios, while the question mark frequently appears in spoken rhetorical questions and written formal consultation scenarios.

[0264] Step 506: Based on usage preference features, filter target user texts that match the usage preference features from the user text set.

[0265] In an exemplary embodiment, the usage preference features include at least one of scenario features, semantic features, and sentence structure features. Based on the usage preference features, target user text matching the usage preference features is filtered from a set of user texts. This includes: generating text features for each user text in the set, whereby the text features reflect the user text's characteristics in terms of scenario, semantics, and sentence structure; determining the feature similarity between the text features and the usage preference features; and identifying the user text as target user text if the feature similarity is greater than a similarity threshold. The similarity threshold can be set according to actual needs. In this embodiment, filtering target user text based on scenario, semantics, or sentence structure can improve the accuracy of text filtering, making the filtered target user text more consistent with user habits.

[0266] Step 508: Generate text based on target user text to obtain target generated text corresponding to target punctuation. Target user text and target generated text are used to train a target punctuation prediction model, which is used to predict punctuation for text.

[0267] In an exemplary embodiment, text generation based on target user text to obtain target generated text corresponding to target punctuation includes: generating a target prompt based on the target punctuation, punctuation constraints, and target user text, wherein the target prompt is used to indicate that text containing the target punctuation and conforming to the punctuation constraints is generated based on the target user text; and generating text according to the target prompt to obtain target generated text corresponding to the target punctuation.

[0268] The text features include, but are not limited to, at least one of scene features, semantic features, and sentence structure features. The generative model can be a pre-trained large language model.

[0269] For example, the target prompt can include the type of the target punctuation mark, a real-world reference example (target user text), and generation requirements, including punctuation constraints. Taking an exclamation mark as the target punctuation mark, the target prompt (Prompt) could be:

[0270] {

[0271] Referring to the language style, sentence structure, punctuation usage logic, and scenario adaptation habits of the [Real User Sentence Examples with Punctuation] below, generate n entirely new Chinese sentences, with exclamation marks as the core punctuation target. The generated content must cover both everyday colloquial expressions and standard written expressions.

[0272] # Detailed generation requirements

[0273] 1. Punctuation constraints: Each sentence must end with the target punctuation mark exclamation mark. Within the sentence, commas, pause marks, and other standard punctuation marks may be used appropriately according to semantic logic and expression needs. The position and type of punctuation marks must fully comply with Chinese punctuation standards and users' actual usage habits. The core target punctuation mark of a single sentence must be unique and clear.

[0274] 2. Scene and sentence structure coverage requirements: The generated sentences must cover two types of core expressions at the same time. The first is the standard written sentence structure with a complete subject-verb-object structure (such as formal notification, business explanation, project communication, official notice, etc.), and the second is the colloquial sentence structure that conforms to the logic of daily communication (including a small number of semantically clear and reasonable omission sentences). The proportion of the two types of sentence structures should match the distribution of real user input. Short fragmented sentences of 3 characters or less are prohibited.

[0275] 3. Length and fluency requirements: The length of a single sentence should be controlled between 5 and 30 characters, matching the mainstream online short text input habits of users. The sentence should be semantically complete, logically coherent, free of grammatical errors, and conform to the expression logic of the corresponding scenario.

[0276] # Real user examples of sentences with punctuation

[0277] 1. The sunrise I saw at the beach today was absolutely breathtaking!

[0278] 2. The project proposal submitted this time has been fully optimized and adjusted!

[0279] 3. The food at the restaurant you recommended really exceeded my expectations!

[0280] }

[0281] For example, the target prompt can be input into a pre-trained large language model to obtain multiple candidate texts output by the large language model. At least one of the following can be performed on the candidate texts: punctuation usage standardization verification, scene matching degree verification, semantic fluency verification, and deduplication and content compliance verification. The candidate texts that pass the verification are used as the target generated text.

[0282] In this embodiment, since the target prompt is used to instruct the generation of text containing target punctuation and conforming to text features based on the text features presented by the target user, text similar to the target user's text can be generated using the generation model and the target prompt, thus enabling rapid corpus expansion.

[0283] For example, the target user text and the target generated text can be stored in a punctuation prediction training set. The text in the punctuation prediction training set is used to train the target punctuation prediction model. Specifically, for the target user text (or target generated text) in the punctuation prediction training set, punctuation can be removed from the target user text (or target generated text) to obtain the punctuation-free text and punctuation annotation information corresponding to the target user text (or target generated text). Based on the punctuation-free text and punctuation annotation information, punctuation training samples are constructed. The punctuation prediction model to be trained is then trained using the punctuation training samples to obtain the target punctuation prediction model. The punctuation prediction model can be based on a BERT (e.g., BERT-base) architecture. In the punctuation prediction model, the BERT layer connects to the output layer (containing a fully connected layer + activation function). The activation function can be, but is not limited to, Softmax. Figure 11 As shown, a schematic diagram of the principle of obtaining the target punctuation prediction model is provided.

[0284] For example, the punctuation prediction training set can be preprocessed, and the model can be trained using the preprocessed punctuation prediction training set. For instance, the punctuation prediction training set may contain text obtained from open-source databases. This text may not contain punctuation, making it impossible to obtain text pairs. Therefore, before training, this type of text without punctuation can be removed from the punctuation prediction training set. Additionally, semantically incomplete fragmented text, text without corresponding punctuation annotations, and invalid data of unsuitable length can be removed from the punctuation prediction training set.

[0285] For example, there can be multiple different target punctuation marks. These target punctuation marks can be selected based on their usage frequency. For instance, punctuation marks with a usage frequency greater than a frequency threshold can be selected as target punctuation marks. For example, the target punctuation marks could be commas, periods, question marks, and exclamation marks. Thus, the punctuation prediction model can perform 5-class classification, with the 5 categories being no punctuation, commas, periods, question marks, and exclamation marks. The input to the punctuation prediction model can include text without punctuation and punctuation annotation information. The output includes the probability distribution for each word in the text without punctuation, containing the probabilities for no punctuation, commas, periods, question marks, and exclamation marks, respectively. The parameters of the punctuation prediction model can then be updated using cross-entropy loss.

[0286] For example, the punctuation prediction model during training can be evaluated using at least one of precision (P), recall (R), and F1 score to obtain the optimal punctuation prediction model, which is then used as the target punctuation prediction model.

[0287] If the target user text (or target generated text) contains punctuation marks other than the target punctuation marks, then the remaining punctuation marks can be mapped to the target punctuation marks based on the mapping relationship between the remaining punctuation marks and the target punctuation marks (for example, mapping the ending punctuation mark to a period). If the remaining punctuation marks have no mapping relationship with any of the target punctuation marks, then the remaining punctuation marks are deleted.

[0288] For example, the punctuation prediction training set may also include punctuation-free text output by the audio transcription module and punctuation annotation information corresponding to the punctuation-free text. Such punctuation-free text and corresponding punctuation annotation information can also be used to train the punctuation prediction model.

[0289] For example, such as Figure 12 The diagram illustrates an overall flowchart from generative model training to punctuation prediction. After data preprocessing, a punctuation prediction model is trained using punctuation prediction samples to obtain a target punctuation prediction model. This target model is then used to perform punctuation inference (punctuation prediction) on the initial text. Based on the punctuation prediction results, punctuation is added to the initial text to obtain the target text. The punctuation prediction results may include the probability distribution for each word in the initial text. The punctuation type corresponding to the highest probability in the probability distribution is determined as the punctuation type for the word, such as a period. A period is then added after that word in the initial text.

[0290] The target punctuation prediction model can be deployed in the punctuation recovery module. After receiving the initial text without punctuation, the punctuation recovery module can automatically remove garbled and invalid characters, perform compliant truncation processing according to the character threshold, and use the same BERT word segmenter as in the training phase to perform word segmentation encoding on the initial text to generate an input vector that the model can recognize.

[0291] For example, the initial text is the text obtained by converting the target transcribed audio into text. The language tag corresponding to the target transcribed audio can be used as the language tag corresponding to the initial text. The initial text and the language tag are input into the target punctuation prediction model to obtain the punctuation prediction result. Punctuation is added to the initial text according to the punctuation prediction result to obtain the target text.

[0292] In an exemplary embodiment, text generation based on target user text to obtain target generated text corresponding to target punctuation includes: identifying replaceable items from the target user text; replacing at least one of the replaceable items contained in the target user text to obtain target generated text corresponding to the target punctuation.

[0293] The types of replaceable items include, but are not limited to, nouns, adjectives, and verbs. The target user's text may contain one or more replaceable items; "multiple" means at least two. The punctuation and sentence structure of the generated text must be consistent with the target user's text.

[0294] For example, if the target user text is "The desserts at this shop are so delicious, I must come again next time! (colloquial)," and the alternatives are nouns "this shop, desserts, next time," adjective "delicious," and verb "come again," then the target generated text could be "The cakes at this restaurant are so amazing, I must come here often in the future!" or "The coffee at this cafe is so delicious, I must come here this weekend!"

[0295] For example, if the target user text is "It's so late, why haven't you left yet? (The syntax is incomplete but the meaning is clear)," and the alternatives are the noun "this time" and the verb "leave," then the target generated text could be "It's so late, why haven't you left yet?" or "It's the middle of the night, why haven't you replied to my message yet?".

[0296] For example, if the target user text is "The acceptance materials for this project have been fully organized and archived as required! (Standard written formal scenario sentence)," and the replaceable items are the nouns "project, acceptance materials" and the verbs "organize, archive," then the target generated text could be "The application materials for this activity have been fully written and submitted as required!" or "The debriefing documents for this meeting have been fully sorted and summarized as required!"

[0297] In this embodiment, by replacing at least one of the replaceable items contained in the target user's text, text similar to the target user's text can be generated, which can quickly expand the corpus.

[0298] In an exemplary embodiment, the method further includes: inputting the initial text into a target punctuation prediction model to perform punctuation prediction and obtaining punctuation prediction results; adding punctuation to the initial text according to the punctuation prediction results to obtain updated text, wherein the form of the punctuation in the updated text matches the first language; and, if the initial text contains target text content in a second language, updating the punctuation corresponding to the target text content in the updated text to a form matching the second language to obtain the target text.

[0299] For example, the target punctuation prediction model predicts punctuation in the first language form by default, such as full-width Chinese characters. If the initial text contains non-Chinese content, such as English content, the punctuation corresponding to the English content can be updated to half-width characters.

[0300] For example, the updated text can be detected for consecutive punctuation marks, absence of punctuation at the end of sentences, and unbroken sentences with excessively long sentences. For consecutive punctuation marks, duplicates can be removed; for sentences without punctuation at the end, punctuation can be added to the end of the sentence; and for unbroken sentences with excessively long sentences, punctuation can be added to break them up, making the punctuation in the obtained target text more accurate. In this embodiment, the output punctuation can be made to conform to the usage habits of both Chinese and English languages.

[0301] In the above punctuation prediction method, a set of user texts corresponding to the target punctuation is obtained. This set of user texts contains the target punctuation. The usage preferences of the target punctuation in the user texts within the set are extracted to obtain the usage preference features corresponding to the target punctuation. Based on these features, target user texts matching the usage preference features are selected from the user text set. Text generation is then performed based on these target user texts to obtain the target generated text corresponding to the target punctuation. The target user texts and the target generated texts are used to train a target punctuation prediction model, which is then used to predict punctuation in text. Since the target user texts are selected based on usage preference features, they better align with users' punctuation usage habits. Furthermore, since the target generated texts are generated with reference to the target user texts, they also align with users' punctuation usage habits. Therefore, training the model using the target user texts and the target generated texts allows the model to learn users' punctuation usage habits, thereby improving the accuracy of punctuation prediction.

[0302] This application supports joint prediction of four types of punctuation: commas, periods, question marks, and exclamation marks. It can fully reproduce the intonation features of speech-to-text, improving the completeness and naturalness of text expression. Specific data and training optimization strategies are applied to target punctuation points to address the pain point of low overall recall rate for internal punctuation, minimizing the loss of key punctuation points related to pauses and emotional tone. The model architecture is built based on pre-trained BERT-base, fully utilizing semantic feature extraction capabilities and ensuring that punctuation predictions fit the semantics of the text while reducing training and inference computational costs, thus improving model efficiency. Training is conducted using ASR speech-to-text scenario-specific data and a customized preprocessing workflow, solving the problem of mismatch between general datasets and actual business scenario data distribution, improving the model's adaptability and prediction accuracy in real speech-to-text scenarios. An early stopping mechanism based on validation set loss can be introduced during the training phase to minimize model overfitting, improve model generalization ability, and ensure stable prediction performance even on unknown speech-to-text. Post-processing corrects punctuation (full-width or half-width), removes abnormal punctuation, and completes sentence-end punctuation. This corrects prediction flaws and improves the readability of the output text without increasing computational overhead, and does not disrupt the model's original semantic prediction logic. On multi-domain Chinese-English bilingual test sets, it achieves superior accuracy compared to similar models, addressing the weaknesses of traditional models in adapting to multiple scenarios and their poor English punctuation prediction, thus enabling high-precision punctuation recovery in both Chinese and English.

[0303] The punctuation prediction method provided in this application can be applied to scenarios such as speech comment transcription, live / video / song subtitle generation, and voice note creation, providing standardized punctuation recovery capabilities for various audio and video textification needs, and improving the readability and content display effect of transcribed text.

[0304] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0305] Based on the same inventive concept, this application also provides an audio recognition device for implementing the audio recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more audio recognition device embodiments provided below can be found in the limitations of the audio recognition method described above, and will not be repeated here.

[0306] In one exemplary embodiment, such as Figure 13 As shown, an audio recognition device is provided, including: a segment extraction module 1302, a language recognition result acquisition module 1304, a tag determination module 1306, a first text generation module 1308, and a second text generation module, wherein:

[0307] The segment extraction module 1302 is used to acquire the target audio and extract valid audio segments from the target audio.

[0308] The language recognition result acquisition module 1304 is used to determine the language recognition audio based on the valid audio segments, and to perform language recognition on the language recognition audio to obtain the language recognition result of the language recognition audio.

[0309] The tag determination module 1306 is used to determine the target audio to be transcribed based on the language recognition audio, and to determine the target language tag of the target audio to be transcribed based on the language recognition result.

[0310] The first text generation module 1308 is used to convert the target transcribed audio into text based on the target language tags and generate initial text, which has no punctuation.

[0311] The second text generation module 1310 is used to restore the punctuation of the initial text to obtain the target text.

[0312] In some embodiments, the first text generation module 1308 is further configured to acquire the acoustic features of the target transcribed audio, extract features from the acoustic features to obtain target extracted features, and generate initial text based on the transcription prompts, target extracted features, and target language tags.

[0313] In some embodiments, the first text generation module 1308 is further configured to select a target audio transcription model from the first audio transcription model and the second audio transcription model; input the acoustic features of the target transcribed audio and the target language label into the target audio transcription model to obtain the output initial text; wherein the first audio transcription model includes an encoder and a decoder associated with an attention mechanism, and the second audio transcription model includes a pre-trained large language model.

[0314] In some embodiments, the module for obtaining the first audio transcription model is used to input the acoustic features of the sample audio and the text vector features corresponding to the sample audio into the first model to be trained, and obtain the output result of the first model, the output result including the first probability distribution corresponding to each audio frame in the sample audio; generate cross-entropy loss and alignment loss according to the first probability distribution corresponding to each audio frame; update the parameters of the first model based on the cross-entropy loss and alignment loss, and obtain the first audio transcription model.

[0315] In some embodiments, the segment extraction module 1302 is configured to perform category recognition on audio frames in the target audio to obtain a target probability sequence of the audio frames. The target probability sequence contains the probability corresponding to multiple target audio categories, which include speech categories and at least two non-speech categories. Based on the probability in the target probability sequence of the audio frames, the matching audio category of the audio frames is determined. The matching audio category is any target audio category or a silence category. Based on the matching audio category of the audio frames in the target audio, a valid audio segment is determined from the target audio.

[0316] In some embodiments, the language recognition result acquisition module 1304 is further configured to match the acoustic features of the language recognition audio with the language acoustic features corresponding to multiple languages ​​respectively, wherein the multiple languages ​​include a first language and a second language, and the first language includes multiple sub-languages; if the successfully matched language acoustic features correspond to the first language, then the acoustic features of the language recognition audio are matched with the sub-language acoustic features corresponding to the multiple sub-languages ​​respectively; based on the language corresponding to the successfully matched language acoustic features and / or the sub-language corresponding to the successfully matched sub-language acoustic features, the language recognition result of the language recognition audio is determined.

[0317] In some embodiments, the language recognition result acquisition module 1304 is further configured to perform language recognition on the language recognition audio to obtain a target language identifier, the target language identifier representing one of multiple languages; perform sub-language recognition on the language recognition audio based on the target language identifier to obtain a target sub-language identifier, the sub-language represented by the target sub-language identifier belonging to the language represented by the target language identifier; and determine the language recognition result of the language recognition audio based on the target language identifier and the target sub-language identifier.

[0318] In some embodiments, punctuation recovery is achieved through a target punctuation prediction model. The apparatus is further configured to: acquire a set of user texts corresponding to the target punctuation, wherein the user texts in the set contain the target punctuation; extract the usage preference of the target punctuation in the user texts contained in the set to obtain the usage preference features corresponding to the target punctuation; filter target user texts that match the usage preference features from the set of user texts based on the usage preference features; generate text based on the target user texts to obtain target generated text corresponding to the target punctuation, wherein the target user texts and target generated texts are used to train a target punctuation prediction model, and the target punctuation prediction model is used to predict punctuation for text.

[0319] Each module in the aforementioned audio recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0320] In one exemplary embodiment, a content display device is provided, comprising: a first display module and a second display module, wherein:

[0321] The first display module is used to display the content details page of the target content.

[0322] The second display module is used to display the input speech associated with the target content and the transcribed text corresponding to the input speech on the content details page; wherein, the transcribed text is determined based on the target text, which is obtained by processing the target audio according to the audio recognition method, and the target audio is determined based on the input speech.

[0323] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores at least some of the data involved in the audio recognition method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an audio recognition method.

[0324] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 15 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an audio recognition method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0325] Those skilled in the art will understand that Figure 14 and 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0326] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0327] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0328] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0329] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0330] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0331] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0332] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.< / sos> < / sos>

Claims

1. An audio recognition method, characterized in that, The method includes: Acquire the target audio and extract valid audio segments from the target audio; Based on the valid audio segment, the language identification audio is determined, and the language identification is performed on the language identification audio to obtain the language identification result of the language identification audio; The target audio to be transcribed is determined based on the language-identified audio, and the target language tag of the target audio to be transcribed is determined based on the language identification result. Based on the target language tags, the target transcribed audio is converted into text to generate initial text, which has no punctuation. The target text is obtained by restoring the punctuation in the initial text.

2. The method according to claim 1, characterized in that, The step of converting the target transcribed audio into text based on the target language tags to generate initial text includes: Acquire the acoustic features of the target transcribed audio, and perform feature extraction on the acoustic features to obtain the target extracted features; The initial text is generated based on the transcription prompt, the target extracted features, and the target language tag.

3. The method according to claim 1, characterized in that, The step of converting the target transcribed audio into text based on the target language tags to generate initial text includes: Select the target audio transcription model from the first audio transcription model and the second audio transcription model; The acoustic features of the target audio to be transcribed and the target language tag are input into the target audio transcription model to obtain the initial output text; The first audio transcription model includes an encoder and a decoder associated with an attention mechanism, and the second audio transcription model includes a pre-trained large language model.

4. The method according to claim 3, characterized in that, The process of obtaining the first audio transcription model includes: The acoustic features of the sample audio and the text vector features corresponding to the sample audio are input into the first model to be trained, and the output result of the first model is obtained. The output result contains the first probability distribution corresponding to each audio frame in the sample audio. Based on the first probability distribution corresponding to each audio frame, generate cross-entropy loss and alignment loss; Based on the cross-entropy loss and the alignment loss, the parameters of the first model are updated to obtain the first audio transcription model.

5. The method according to any one of claims 1 to 4, characterized in that, Extracting valid audio segments from the target audio includes: The audio frames in the target audio are classified to obtain a target probability sequence of the audio frames. The target probability sequence contains the probability corresponding to multiple target audio categories, and the multiple target audio categories include speech categories and at least two non-speech categories. Based on the probability in the target probability sequence of the audio frame, the matching audio category of the audio frame is determined, wherein the matching audio category is any of the target audio categories or the silence category; Valid audio segments are determined from the target audio based on the matching audio category of the audio frames in the target audio.

6. The method according to any one of claims 1 to 4, characterized in that, The step of performing language identification on the language-identified audio to obtain the language identification result of the language-identified audio includes: The acoustic features of the language identification audio are matched with the language acoustic features corresponding to multiple languages, including a first language and a second language, and the first language includes multiple sub-languages. If the matched language acoustic features correspond to the first language, then the acoustic features of the language recognition audio are matched with the sub-language acoustic features corresponding to the multiple sub-languages ​​respectively. Based on the language corresponding to the successfully matched language acoustic features and / or the sub-language corresponding to the successfully matched sub-language acoustic features, the language recognition result of the language recognition audio is determined.

7. The method according to any one of claims 1 to 4, characterized in that, The step of performing language identification on the language-identified audio to obtain the language identification result of the language-identified audio includes: The target language identifier is obtained by performing language identification on the language-identified audio, and the target language identifier represents one of multiple languages; Based on the target language identifier, the language recognition audio is subjected to sub-language recognition to obtain a target sub-language identifier, wherein the sub-language represented by the target sub-language identifier belongs to the language represented by the target language identifier; Based on the target language identifier and the target sub-language identifier, the language identification result of the language-identified audio is determined.

8. The method according to any one of claims 1 to 4, characterized in that, The punctuation restoration is achieved through a target punctuation prediction model, and the method further includes: Obtain the set of user texts corresponding to the target punctuation mark, wherein the user texts in the set of user texts contain the target punctuation mark; The usage preferences of the target punctuation mark in the user text contained in the user text set are extracted to obtain the usage preference features corresponding to the target punctuation mark; Based on the usage preference features, filter target user texts that match the usage preference features from the user text set; Based on the target user text, text is generated to obtain the target generated text corresponding to the target punctuation. The target user text and the target generated text are used to train a target punctuation prediction model, which is used to predict punctuation for the text.

9. A content display method, characterized in that, The method includes: Displays the content details page of the target content; On the content details page, the input voice associated with the target content and the transcribed text corresponding to the input voice are displayed; Wherein, the transcribed text is determined based on the target text, which is obtained by processing the target audio according to any one of claims 1 to 8, and the target audio is determined based on the input speech.

10. An audio recognition device, characterized in that, The device includes: The segment extraction module is used to acquire target audio and extract valid audio segments from the target audio. The language identification result acquisition module is used to determine the language identification audio based on the valid audio segment, and to perform language identification on the language identification audio to obtain the language identification result of the language identification audio; The tag determination module is used to determine the target audio to be transcribed based on the language-identified audio, and to determine the target language tag of the target audio based on the language identification result; The first text generation module is used to perform text conversion on the target transcribed audio based on the target language tag to generate initial text, which has no punctuation. The second text generation module is used to restore the punctuation of the initial text to obtain the target text.

11. A content display device, characterized in that, The device includes: The first display module is used to display the content details page of the target content; The second display module is used to display the input voice associated with the target content and the transcribed text corresponding to the input voice on the content details page; Wherein, the transcribed text is determined based on the target text, which is obtained by processing the target audio according to any one of claims 1 to 8, and the target audio is determined based on the input speech.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images