Speech recognition method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a speech recognition method and device, an electronic device and a storage medium. The method comprises: obtaining a transcribed text, the transcribed text being obtained by transcribing a to-be-processed speech based on a language prediction label; calling a large language model to perform semantic judgment on the transcribed text to obtain a first judgment result; and outputting a transcribed result of the to-be-processed speech based on a confirmation result of the language prediction label according to the first judgment result. The embodiments of the present disclosure can improve the reliability and stability of speech recognition.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a speech recognition method and apparatus, electronic device and storage medium. Background Technology

[0002] In the field of speech content processing, speech content understanding aims to extract multi-dimensional features such as semantics, pragmatics, and linguistic attributes from speech signals through processing and analysis, thereby supporting downstream applications such as semantic retrieval, content recommendation, speech summarization, and speaker recognition. Language recognition, as a crucial component of speech content understanding, can automatically identify and determine the language category of each speech segment in an audio stream. Its accuracy and reliability directly impact the effectiveness of subsequent processing stages. Summary of the Invention

[0003] This disclosure presents a speech recognition method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a speech recognition method is provided, comprising: acquiring transcribed text, the transcribed text being obtained by transcribing speech to be processed based on language prediction labels; calling a large language model to perform semantic judgment on the transcribed text to obtain a first judgment result; and outputting a transcribed result of the speech to be processed based on the confirmation result of the language prediction labels according to the first judgment result, wherein the transcribed result of the speech to be processed includes at least the transcribed text.

[0005] According to one aspect of this disclosure, a speech recognition device is provided, comprising: an acquisition module for acquiring transcribed text, wherein the transcribed text is obtained by transcribing speech to be processed based on language prediction labels; an invocation module for invoking a large language model to perform semantic judgment on the transcribed text and obtain a first judgment result; and an output module for outputting the transcribed result of the speech to be processed based on the confirmation result of the language prediction labels according to the first judgment result, wherein the transcribed result of the speech to be processed includes at least the transcribed text.

[0006] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above-described method.

[0007] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0008] In this embodiment, a transcribed text is obtained, which is obtained by transcribing the speech to be processed based on language prediction tags. A large language model is invoked to perform semantic judgment on the transcribed text, obtaining a first judgment result. Based on the confirmation result of the language prediction tags based on the first judgment result, the transcribed result of the speech to be processed is output, and the transcribed result of the speech to be processed includes at least the transcribed text. In this way, a large language model can be invoked to analyze whether the transcribed text constitutes a reasonable and coherent language expression, thereby assisting in judging whether the language prediction tags are true and reliable. The transcribed text is only adopted when the large language model determines that the transcribed text has semantic validity. This semantic-level verification mechanism can effectively filter out misjudgments in speech recognition, greatly reduce the pollution of semantic analysis and content processing processes by pseudo-language tags, significantly reduce the risk of error propagation to subsequent processes (such as semantic understanding, recommendation, and classification processes based on language tags), and improve the reliability and stability of speech recognition.

[0009] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0011] Figure 1 A flowchart illustrating a speech recognition method according to an embodiment of the present disclosure is shown.

[0012] Figure 2 A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown.

[0013] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0014] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0015] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0016] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0017] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0018] In related technologies, most speech recognition solutions are based on pre-trained deep neural network models. Speech segments are input into a multi-classifier, which directly outputs the corresponding language labels. While this approach based on pre-trained multi-classification models possesses a certain level of language discrimination capability, it still faces technical challenges in real-world scenarios. On the one hand, these models are prone to misclassifying invalid or ambiguous speech segments, leading to label drift or error accumulation. On the other hand, due to the lack of semantic-level verification mechanisms, they cannot determine whether the predicted language labels correspond to readable language expressions, resulting in insufficient stability and low reliability of the speech content understanding system in practical applications.

[0019] In view of this, embodiments of the present disclosure provide a speech recognition method that can obtain transcribed text, which is obtained by transcribing the speech to be processed based on language prediction labels; call a large language model to perform semantic judgment on the transcribed text to obtain a first judgment result; based on the confirmation result of the language prediction labels based on the first judgment result, output the transcription result of the speech to be processed, wherein the transcription result of the speech to be processed includes at least the transcribed text.

[0020] In this way, a large language model can be invoked to analyze whether the transcribed text constitutes a reasonable and coherent language expression, thereby assisting in determining whether the language prediction labels are true and reliable. Only when the large language model determines that the transcribed text has semantic validity will it be adopted. This semantic-level verification mechanism can effectively filter out false judgments in speech recognition, greatly reducing the pollution of semantic analysis and content processing processes by pseudo-language labels. It significantly reduces the risk of error propagation to subsequent processes (such as semantic understanding, recommendation, and classification based on language labels), thus improving the reliability and stability of speech recognition.

[0021] Figure 1 A flowchart of a speech recognition method according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the speech recognition method may include:

[0022] In step S11, the transcribed text is obtained, which is obtained by transcribing the speech to be processed based on language prediction labels;

[0023] In step S12, the large language model is invoked to perform semantic judgment on the transcribed text, and the first judgment result is obtained;

[0024] In step S13, based on the confirmation result of the language prediction label based on the first judgment result, the transcription result of the speech to be processed is output, and the transcription result of the speech to be processed includes at least the transcribed text.

[0025] In one possible implementation, the speech recognition method of this disclosure embodiment can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The speech recognition method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server.

[0026] In one possible implementation, the speech recognition method of this disclosure can be applied to the annotation needs of long or short audio in real-world scenarios such as podcasts, audiobooks, and voice customer service. The source of the speech to be processed can include actual business scenarios such as podcasts, audiobooks, voice customer service systems, conference recordings, and telephone calls; the embodiments of this disclosure are not limited in scope.

[0027] In one possible implementation, obtaining the transcribed text includes: calling a pre-trained Language Identification (LID) model to perform preliminary language classification prediction on the speech to be processed, and obtaining language prediction labels; calling the speech recognition model (Automatic Speech Recognition, ASR) corresponding to the language prediction labels to perform speech transcription operation on the speech to be processed, and obtaining the transcribed text.

[0028] The language recognition model can automatically identify and determine the language category of each speech segment in an audio stream. The model can directly output the language category of the speech being processed and its corresponding confidence score, providing candidate language labels as language prediction labels. It should be understood that this language recognition model can employ publicly available multilingual recognition models, such as Whisper, XLS-R LID, and Wav2Vec2-LID. The specific selection of the language recognition model, the definition of the language set, and the source of training data are not limited to a particular scheme and can be flexibly configured according to specific application scenarios. The embodiments disclosed herein do not impose such limitations.

[0029] However, most speech recognition models are based on acoustic feature learning and lack the ability to understand language at the language level. Their recognition accuracy for "low semantic information segments" is unstable, and they are prone to misjudging segments with a high proportion of silence, ambiguous tone, extremely fast speech rate, or incomplete semantics. For example, for some segments that only contain filler words, interjections (such as "uh-huh", "um", etc.) or incomplete syllables, speech recognition models will still output clear language labels, resulting in false and invalid annotation results.

[0030] For example, when the speech to be processed is a short speech segment of 5 to 10 seconds, its information density is low, the context is limited, and the speaking habits are diverse, which can easily cause the language recognition model to overfit the training set distribution and lack generalization ability. Especially for "low-resource languages" (such as languages ​​that lack large-scale parallel corpora and other resource support) or "similar languages" (such as Cantonese and Mandarin, Dutch and German), the language recognition model is easily interfered with by non-linguistic factors such as noise, intonation, and rhythm, and the error rate of speech language recognition increases significantly.

[0031] Furthermore, after outputting language prediction labels, language recognition models lack a mechanism to verify whether the identified language matches the speech content, resulting in a lack of semantic verifiability. This means that even if a language recognition model generates a high-confidence language prediction label, it cannot determine whether the language prediction label truly reflects the semantic content of the speech segment. For example, a segment misidentified as English may actually contain dialect, non-linguistic background noise, or invalid speech.

[0032] Therefore, a speech recognition model for the corresponding language can be found through language prediction tags. The found speech recognition model is then used to perform speech-to-text transcription on the speech to be processed, generating transcribed text suitable for semantic analysis. This transcribed text can then be used for subsequent semantic verification. The speech recognition model can be a locally deployed model (e.g., Whisper-ASR model, wav2vec2-CTC model, etc.) or a model provided by a remote application programming interface (API) service (e.g., a speech recognition model provided by a social platform or open-source community platform). It should be understood that any model with stable transcription capabilities and support for the target language can be used as a speech recognition model; this disclosure does not impose specific limitations on the speech recognition model.

[0033] By introducing a label-driven transcription mechanism (e.g., dynamically selecting the corresponding speech recognition model based on candidate language prediction labels), the two independent models, the language recognition model and the speech recognition model, are linked together. The transcribed text output by the speech recognition model is used to back-verify the language prediction labels output by the language recognition model, thus establishing a verifiable link between language recognition and semantic content, thereby reducing the generalization of language label misjudgments on non-semantic fragments.

[0034] After obtaining the transcribed text in step S11, a large language model can be invoked in step S12 to perform semantic judgment on the transcribed text, obtaining a first judgment result. This first judgment result is used to characterize whether the transcribed text is a "complete and natural sentence expression" in the language. The large language model can include models such as GPT-4, Claude, and Gemini, as long as it possesses cross-language semantic understanding capabilities. The embodiments of this disclosure do not limit the type of large language model. Because the large language model has cross-language semantic understanding capabilities, it can identify accented "fragmented sentences" and "mixed speech" in speech transcription. Even if the language prediction labels generated by the language recognition model have large deviations, they can be corrected by the semantic layer's large language model, compensating for the weakness of acoustic feature models alone, which suffer from a sharp drop in accuracy in low-resource languages ​​and strong accents.

[0035] During the semantic judgment of transcribed text using a large language model, a target prompt can be set to send a semantic judgment request to the large language model. The specific format, language settings, and calling interface of the prompt can be set according to the actual application scenario, and the embodiments disclosed herein do not limit this.

[0036] In step S12, the first judgment result of the transcribed text is obtained. In step S13, the confirmation result of the language prediction label can be based on the first judgment result, and the transcribed result of the speech to be processed can be output. The transcribed result of the speech to be processed includes at least the transcribed text.

[0037] In one possible implementation, based on the first judgment result, the confirmation result of the language prediction label is obtained, including: if the first judgment result indicates that the semantics of the transcribed text is valid, the language prediction label is confirmed as the target label of the speech to be processed; if the first judgment result indicates that the semantics of the transcribed text is invalid, the language prediction label is discarded and the target label of the speech to be processed is marked as invalid.

[0038] Based on the first judgment result of the transcribed text, a decision is made on whether to retain the language prediction label as the target label for the speech to be processed. If the first judgment result indicates that the transcribed text is semantically valid, it means that the transcribed text is a reasonable expression, and the language prediction label for the speech to be processed is valid, and the language prediction label is confirmed as the target label for the speech to be processed; if the first judgment result indicates that the transcribed text is semantically invalid, it means that the transcribed text is an invalid or ambiguous expression, and the language prediction label is discarded, and the target label for the speech to be processed is marked as invalid.

[0039] This strategy is equivalent to introducing a semantic understanding-based label filtering module outside the acoustic language recognition path, adding a semantic layer filtering, which can effectively filter out the misclassification output of speech fragments and non-language segments by the language recognition model. This dual-layer discrimination mechanism of acoustic classification results and semantic verification mechanism improves the label robustness and practical usability of the speech recognition method of this embodiment.

[0040] In one possible implementation, if the first judgment result indicates that the semantics of the transcribed text are valid, the transcribed text can be output as the transcription result. This simplified output method is suitable for application scenarios where traceability requirements are not high and efficiency and simplicity are more important, reducing the exposure of sensitive metadata (such as language prediction labels, first judgment results, and confirmation results).

[0041] In one possible implementation, the language prediction label, the transcribed text, the first judgment result, and the confirmation result (e.g., the target label) can be output as the transcription result of the speech to be processed.

[0042] In this way, four levels of meta-information can be retained for each speech to be processed: the language prediction label generated by the speech recognition model, the transcribed text generated by the speech recognition model, the first judgment result generated by the large language model, and the final target label. This end-to-end traceability improves the efficiency of review and quality tracking, and helps to quickly identify anomalies and conduct manual review.

[0043] It should be understood that the language recognition model, speech recognition model, and large language model in this embodiment are all independently encapsulated, and the models can be flexibly replaced, combined, or upgraded. Each model can support multiple methods, such as open-source models, models provided by commercial remote application programming interface (API) services, and private deployment models, and they are decoupled from each other. Due to the decoupling of the model structure, each model has high replaceability and platform adaptability. This modular structural design enables this solution to have broad adaptability in different hardware conditions, compliance requirements, and language resource scenarios, providing technical support for its large-scale commercial deployment.

[0044] Through steps S11 to S13, after the speech recognition model performs preliminary language classification prediction on the speech to be processed, the speech to be processed is transcribed using the speech recognition model of the language to which the speech to be processed belongs. A large language model is then called to analyze whether the transcribed text constitutes a reasonable and coherent language expression, thereby assisting in verifying the authenticity and reliability of the language prediction labels. This joint language determination link, which constructs "language recognition model outputs language prediction labels + speech recognition model transcribes the speech to be processed + large language model verifies the transcribed text," overcomes the limitations of using only a speech recognition model for short speech segment recognition. It effectively solves technical problems such as false labels, semantic incomprehensibility, and insufficient robustness, significantly improving the practical application value and accuracy stability of language annotation in large-scale audio data scenarios. In this approach, the language prediction label is adopted as the target label only if the large language model determines that the transcribed text has semantic validity. This semantic-level verification mechanism effectively filters out misjudgments from the language recognition model (such as acoustic misjudgments of short sentences or noisy segments), greatly reduces the pollution of subsequent semantic analysis and content processing by pseudo-labels, significantly reduces the risk of error propagation to subsequent processes (such as semantic understanding, recommendation, and classification based on language labels), and improves the reliability and stability of annotation in actual audio processing.

[0045] Furthermore, compared to related technologies that only output final labels, they lack the ability to locate errors and trace data sources. The speech recognition method of this disclosure can retain four levels of meta-information for each speech to be processed, namely: the language prediction label generated by the language recognition model, the transcribed text generated by the speech recognition model, the first judgment result generated by the large language model, and the finally generated target label (i.e., the confirmation result), which helps to quickly identify anomalies and perform manual review.

[0046] Furthermore, in related technologies, improving speech recognition performance requires retraining the model, for example, by collecting new labeled samples of a specific language and retraining the speech recognition model. In contrast, the speech recognition method in this disclosure adopts a verification rather than relearning approach. It eliminates the need for new labeled data or model retraining. Through collaborative verification between the speech recognition model and the large language model, pseudo-labels are filtered out without changing the weights of the speech recognition model, thereby improving the accuracy and stability of the final language labels. This method is suitable for performance enhancement tasks in low-resource scenarios and environments where the model is uncontrollable (such as models provided by external application programming interfaces).

[0047] The speech recognition method of this disclosure will now be described in detail.

[0048] In one possible implementation, the method further includes acquiring the speech to be processed, wherein the acquisition method includes: performing preprocessing operations on the acquired raw audio to obtain a preprocessed first audio; and performing speech activity detection on the first audio to obtain the speech to be processed corresponding to each voice interval in the first audio.

[0049] For example, a target audio dataset can be obtained, such as a collection of raw audio samples from podcasts, audiobooks, or voice customer service systems. The sampling method can be adjusted based on specific data themes, language distribution, user activity, and other indicators; however, the embodiments disclosed herein do not impose limitations on this.

[0050] The length of each original audio file can be the same or different. For example, the original audio file can be audio data within a certain range (e.g., 5 minutes to 2 hours). The format of the original audio file can include uncompressed formats (e.g., WAV, AIFF), lossless compression formats (e.g., FLAC, APE), and lossy compression formats (e.g., MP3, AAC, OGG, etc.). The format of the original audio file in the embodiments of this disclosure is not limited.

[0051] The system can perform uniform preprocessing operations on the original audio to obtain a preprocessed first audio. These preprocessing operations may include at least one of the following: sampling rate normalization, volume normalization, noise reduction, channel unification, and format conversion.

[0052] Optionally, sampling rate normalization can uniformly adjust the sampling rate of the original audio signal to a preset value (e.g., 16kHz or 24kHz) to ensure that the pre-processed first audio has a consistent temporal resolution, which is convenient for subsequent model processing.

[0053] Optionally, volume normalization is achieved by adjusting the volume level of the original audio to make the volume of different original audios more consistent (e.g., conforming to the -23 LUFS broadcast loudness standard).

[0054] Optionally, the noise reduction process can suppress background noise in the original audio. The noise reduction methods may include denoising methods based on spectral subtraction, and denoising methods based on deep learning models (such as DeepFilterNet, RNNoise, etc.), and the embodiments disclosed herein are not limited thereto.

[0055] Optionally, channel unification is used to convert multi-channel stereo audio to mono audio.

[0056] Optionally, audio format conversion is used to unify the original audio into a first audio in a lossless format, such as WAV format using pulse code modulation, so that the first audio can be stored in a unified format.

[0057] Preprocessing the raw audio can effectively improve the consistency and stability of subsequent processing.

[0058] In one possible implementation, to improve the consistency and stability of the audio, the acquired raw audio is preprocessed to obtain a preprocessed first audio, including: normalizing the sampling rate of the raw audio to obtain a second audio with a target sampling rate (e.g., 16kHz or 24kHz); normalizing the volume of the second audio to obtain a third audio with a target volume (e.g., -23LUFS); denoising the third audio to obtain a denoised fourth audio; unifying the channels of the fourth audio (e.g., unifying stereo to mono) to obtain a channel-unified fifth audio; and converting the format of the fifth audio to obtain a first audio with a target format (e.g., WAV format).

[0059] To extract segments containing valid human speech from the first audio, a voice activity detection (VAD) method can be used to detect and segment the first audio, removing silent segments, pure background music segments, or environmental sound segments, to obtain the speech to be processed corresponding to each voice interval in the first audio.

[0060] Among them, the speech activity detection method can extract features from each block of the first audio, and then use a classifier to classify each block to determine whether it is a speech signal.

[0061] Optionally, the voice activity detection method can be an energy threshold-based voice activity detection method, such as the voice activity detection module used in Web Real Time Communications (WebRTC VAD).

[0062] Optionally, speech activity detection methods can be based on neural network models, such as pre-trained enterprise-grade speech activity detectors (Silero VAD) and pyannote-audio VAD modules, which can better adapt to complex backgrounds.

[0063] It should be understood that the implementation of speech activity detection can be flexibly configured according to the speech quality and background complexity of the application scenario, and the embodiments disclosed herein do not limit this.

[0064] For example, after performing speech activity detection on the first audio, a set of time interval markers can be output: {(start_1,end_1), (start_2, end_2), ..., (start_n, end_n)}, where each time interval corresponds to a detected human voice segment; among them, the human voice interval (start_1, end_1) can correspond to the first speech to be processed, the human voice interval (start_2, end_2) can correspond to the second speech to be processed, and so on, with the human voice interval (start_n, end_n) corresponding to the nth speech to be processed.

[0065] By performing speech activity detection on the first audio, silent segments and pure background segments can be removed, while the human voice area is preserved, generating a series of time-marked segments for subsequent slicing.

[0066] In one possible implementation, a pre-trained language recognition model is invoked to perform preliminary language classification prediction on the speech to be processed, and language prediction labels are obtained. This includes: performing a first segmentation process on the speech to be processed to obtain at least one speech segment; and invoking the pre-trained language recognition model to perform language classification prediction on each speech segment to obtain a language prediction label for each speech segment.

[0067] This approach facilitates the subsequent implementation of segment-level verification mechanisms, thereby breaking the propagation chain of mislabeled segments to adjacent segments, isolating errors locally, and ensuring that the analysis of the entire language structure is not affected. This also mitigates the distortion of the entire segment's analysis caused by the spread of misjudgments in long recordings.

[0068] For example, the speech to be processed may include the human voice range identified from the first audio using a speech activity detection method. The speech to be processed may be segmented into speech segments of appropriate length for subsequent language recognition processing.

[0069] Optionally, the first segmentation processing method may include fixed window segmentation, adaptive semantic segmentation, etc. The fixed window segmentation method divides each speech segment into a fixed-length window of a preset duration (e.g., 5 to 10 seconds), allowing for appropriate overlap (e.g., 50%) to improve coverage. The adaptive semantic segmentation method can be combined with automatic speech recognition or speech rate estimation to dynamically segment based on pause points or semantic boundaries. It should be understood that a suitable first segmentation processing method can be selected according to the actual application scenario, and the embodiments of this disclosure do not limit this selection.

[0070] Each audio segment is recorded in a structured manner, for example: {"audio_id":"session_123", "segment_index":5, "start_time":157.3, "end_time":163.8, "waveform":[...]}. Here, "audio_id": "session_123" indicates the audio segment number to be processed is "session_123"; "segment_index": 5 indicates the audio segment's position in the processed audio sequence is 5; "start_time": 157.3 indicates the start time of the audio segment is 157.3 seconds into the processed audio; "end_time": 163.8 indicates the end time of the audio segment is 163.8 seconds into the processed audio; and "waveform": [...] indicates the audio file format of the audio segment.

[0071] It should be understood that the purpose of performing the first segmentation of the speech to be processed within a human voice time period is to generate analyzable speech units of a preset duration (e.g., 5-10 seconds). The first segmentation method can be a fixed-length sliding window (e.g., a 6-second window with 3-second overlap), or it can employ adaptive semantic segmentation by combining speech rate estimation and pause boundaries. The resulting sliced ​​speech segments can include audio segment content, start and end times, and the encoded information of the speech to be processed, to facilitate subsequent parallel processing.

[0072] Each speech segment can be input into a pre-trained language recognition model, and preliminary language classification prediction can be performed on each speech segment to obtain a language prediction label for each speech segment. This language prediction label is used to indicate the language to which the speech segment belongs.

[0073] The language recognition models include, but are not limited to: end-to-end speech recognition models based on the Transformer architecture (encoder-decoder) (e.g., the Whisper model that supports automatic language detection), language recognition models provided by social platforms (or open-source community platforms), self-supervised audio pre-trained models (e.g., the Wav2Vec2-LID model, the XLS-R-LID model), and training custom language recognition classifiers (e.g., convolutional neural networks, self-attention networks, long short-term memory networks). The embodiments of this disclosure do not limit the type of language recognition model.

[0074] The language recognition model can make a preliminary judgment on the input speech segment, identify the speech segment as a language label in a predefined language set (such as English, Chinese, Russian, French, etc.), and output the corresponding confidence score.

[0075] The structured representation of the language prediction label output by the language recognition model is: { "segment_id": "session_123# 5", "predicted_language": "fr", "confidence": 0.87}. Here, "segment_id": "session_123#5" represents the fifth speech segment in the speech to be processed, numbered "session_123"; "predicted_language": "fr" indicates the language prediction label is French; and "confidence": 0.87 indicates that the confidence level for the language prediction label being French is 0.87.

[0076] It should be understood that the language prediction label is not the final, decisive result, but rather serves as a candidate input for the subsequent semantic verification stage.

[0077] It is known that in typical long audio scenarios such as audiobooks, voice conferencing, and customer service calls, the language recognition models in related technologies directly recognize long audio segments. If a segment is misjudged as language, the language structure analysis of the entire audio segment will fail. The embodiments of this disclosure employ a segment-by-segment recognition method for subsequent segment-by-segment verification. This segment-by-segment structural method can build a verifiable tagging system at the language level, effectively preventing the propagation of mislabels into the context, significantly reducing the propagation and cumulative error rate of mislabels in long audio scenarios, thereby reducing the risk of error coupling in the language recognition processing.

[0078] Once the language prediction label of each speech segment in the speech to be processed is obtained, the speech recognition model corresponding to the language prediction label can be called to perform speech-to-text operation on each speech segment in the speech to be processed, and obtain the transcribed text.

[0079] In one possible implementation, the speech recognition model corresponding to the language prediction label can be found based on the language prediction label and a pre-configured mapping table; the speech to be processed is input into the speech recognition model corresponding to the language prediction label to generate the transcribed text of the speech to be processed.

[0080] By constructing a pre-configured mapping table, the speech recognition method of this disclosure can achieve accurate positioning and efficient matching of speech recognition models, significantly improving the calling efficiency of speech recognition models.

[0081] For example, a pre-configured mapping table can be looked up based on the language prediction label, such as: { "en": "Whisper-ASR-English", "zh":"Whisper-ASR-Chinese", "fr":"Wav2Vec2-French-ASR",...}. Here, "en":"Whisper-ASR-English" represents the speech recognition model for English, "zh":"Whisper-ASR-Chinese" represents the speech recognition model for Chinese, and "fr":"Wav2Vec2-French-ASR" represents the speech recognition model for French.

[0082] The corresponding speech segment is input into the speech recognition model of that language to obtain the transcribed text. The structured representation of the processing result output by the speech recognition model is: {"segment_id": "session_123#5", "predicted_language": "fr", "transcript_text": "Je vais à la bibliothèque cetaprès-midi", "asr_model": "Wav2Vec2-French-ASR"}. In this context, "segment_id": "session_123#5" represents the fifth speech segment in the speech to be processed, numbered "session_123"; "predicted_language": "fr" indicates that the language prediction label is French; "transcript_text": "Je vais àla bibliothèque cet après-midi." indicates that the French transcribed text of this speech segment is "Je vais à labibliothèque cet après-midi"; and "asr_model": "Wav2Vec2-French-ASR" indicates that the model category used is the French-specific speech recognition model.

[0083] After obtaining the transcribed text of the speech segment in step S11, in step S12, the large language model can be called to make a semantic rationality judgment on the transcribed text and obtain the first judgment result.

[0084] For each speech segment's transcribed text, a large language model can be invoked to perform semantic judgment on the transcribed text, determining whether the transcribed text is a reasonable and complete sentence expression in the target language.

[0085] This process can be achieved using specially designed prompts; for example, the prompts could be designed as follows:

[0086] Please determine whether the following text is a complete natural language expression that conforms to the grammar and semantics of the language. If yes, please return Y; if not (e.g., garbled text, broken sentences, non-linguistic noise, etc.), please return N.

[0087] Language: French

[0088] Text: It's the end of the world.

[0089] The large language model can output a structured first judgment result based on the prompt words, for example: {"segment_id":"session _123#5", "valid_sentence":true}. Here, "segment_id": "session_123#5" represents the fifth speech segment in the speech to be processed, numbered "session_123"; "valid_sentence": true indicates that the first judgment result is that the transcribed text is semantically valid.

[0090] In step S12, the first judgment result of the transcribed text is obtained. In step S13, based on the first judgment result, the confirmation result of the language prediction tag is obtained, and the language tag is filtered according to the semantic judgment result.

[0091] In one possible implementation, if the first judgment result indicates that the transcribed text semantics corresponding to each speech segment in the speech to be processed are valid, the language prediction label is used as the target label of the speech segment; if the first judgment result indicates that the transcribed text semantics are invalid, the language prediction label is discarded and the speech segment is marked as "undeterminable language" or "no language label" to improve fault tolerance and robustness.

[0092] For example, if the first judgment result is "valid_sentence": true, it means that the semantics of the transcribed text corresponding to the speech segment are valid. Then the language prediction label of the speech segment is considered to be valid, and the language prediction label can be used as the final language label of the speech segment, that is, the target label.

[0093] If the first judgment result is "valid_sentence": false, it means that the semantics of the transcribed text corresponding to the speech segment is invalid. Therefore, the language prediction label of the speech segment is directly discarded, and it is considered "unable to determine language" or "semantically invalid segment," which can be marked as undefined or null, where undefined represents undefined and null represents an empty value. The structured representation of each speech segment is: {"segment_id":"session_123#5","final_language_label":"fr", "transcript_text": "...", "llm_verified": true}. Here, "segment_id": "session_123#5" represents the fifth speech segment in the speech segment numbered "session_123" to be processed; "final_language_label": "fr" indicates that the target label of the speech segment is French; "transcript_text": "..." represents the transcribed text of the speech segment; and "llm_verified": true indicates that the first judgment result is that the text is valid.

[0094] In this way, misclassification of non-linguistic audio, incomplete semantic segments, and noise segments in language recognition models can be effectively filtered out, thereby improving the stability and semantic credibility of the labeling system.

[0095] Furthermore, compared to related technologies, speech recognition systems often force the output of a language label for each speech segment. For example, even if a speech segment lacks linguistic structure or has disorganized content, a label is still output, leading to numerous misclassifications and mislabelings of uncertain segments. This hard-labeling mechanism in related technologies severely impacts the accuracy and interpretability of downstream applications, especially in real-world business scenarios with uneven data quality and highly unbalanced language distribution, easily resulting in the accumulation of a large number of erroneous labels.

[0096] In the embodiments disclosed herein, after the language recognition model generates preliminary language prediction labels, a judgment mechanism from a large language model is introduced to support the output of language-undecidable states, such as "unable to determine language" or "refuse labeling." This mechanism of refusing recognition (e.g., when the large language model determines that the transcribed text is invalid, it can explicitly return the language-undecidable state instead of forcibly assigning a language label) significantly improves the fault tolerance of the speech recognition method and the safety boundary for practical application. This mechanism is highly adaptable to scenarios such as data cleaning, label reliability assessment, and low-resource language processing, and can significantly reduce the cascading errors caused by the propagation of incorrect labels.

[0097] After filtering the target labels using the first judgment result, the segment-level language recognition annotation results of the dataset can be summarized to obtain the target labels of the speech to be processed.

[0098] In one possible implementation, the target labels of different speech segments in the same speech to be processed can be summarized to obtain the target labels of the speech to be processed.

[0099] For example, target labels for each speech segment can be aggregated based on the segment_id number of each speech segment, forming a segment-level language distribution annotation for the audio dataset. The target labels for the speech to be processed may include: a timeline of language labels corresponding to the audio file of each speech to be processed, a visualization of language distribution (e.g., language switching points, language ratio analysis), and various exportable data storage file formats (e.g., JSON, CSV, ELAN, etc., for use by downstream systems).

[0100] For example, the schematic data structure for target labels is: "audio_id": "session_123", "segments":[{"start": 0.0, "end": 7.8, "language": "en"},{"start": 7.8, "end": 15.2, "language": null},{"start": 15.2, "end": 22.6, "language": "zh"}]. Here, "audio_id": "session_123" represents the speech segment to be processed, numbered "session_123". In this speech segment, the target label for the segment [0, 7.8) is English, the target label for the segment [7.8, 15.2) is invalid, and the target label for the segment [15.2, 22.6) is Chinese.

[0101] In this way, the filtered fragment-level language recognition results can be integrated to form a fragment-level language label timeline for the entire dataset. The output can be in a structured, readable format, supporting subsequent tasks such as semantic understanding, audio retrieval, label clustering, and language switch point detection. In specific engineering implementations, the structured, readable format can be flexibly adjusted according to the platform's technology selection.

[0102] In one possible implementation, the target labels of different speech segments in the same speech to be processed are summarized to obtain the target labels of the speech to be processed. This includes: when the target labels of different speech segments in the same speech to be processed are inconsistent, obtaining the target labels of each speech sub-segment in the first speech segment with inconsistent target labels; and summarizing the target labels of different speech segments and / or different speech sub-segments in the same speech to be processed to obtain the target labels of the speech to be processed.

[0103] For example, if the target labels of different speech segments in the speech to be processed are inconsistent, such as speech segment 1, speech segment 2, speech segment 3, and speech segment 4, where the target label for speech segment 1, speech segment 3, and speech segment 4 is Chinese, and the target label for speech segment 2 is Japanese, speech segment 2 can be taken as the first speech segment. This first speech segment can then be further subdivided to determine the target label for each sub-segment within it. In this way, if the speech segment contains other languages ​​or dialects, such as two Japanese sentences mixed in with Chinese, further subdivision can break the Japanese into smaller segments, thereby further improving the accuracy of speech recognition.

[0104] In one possible implementation, when the target labels of different speech segments in the same speech to be processed are inconsistent, the target label of each speech sub-segment in the first speech segment with inconsistent target labels is obtained, including: when the target labels of different speech segments in the same speech to be processed are inconsistent, performing a second segmentation process on the first speech segment with inconsistent target labels to obtain at least one speech sub-segment; calling a pre-trained language recognition model to perform language classification prediction on each speech sub-segment in the first speech segment to obtain a language prediction sub-label for each speech sub-segment; calling the speech recognition model corresponding to the language prediction sub-label to perform speech transcription operation on each speech sub-segment in the first speech segment to obtain the transcribed text of each speech sub-segment; calling a large language model to perform semantic judgment on the transcribed text of each speech sub-segment to obtain a second judgment result; and confirming the language prediction sub-label as the target label of the speech sub-segment if the second judgment result indicates that the semantics of the transcribed text of the speech sub-segment is valid.

[0105] It should be understood that the method of obtaining the target label of each speech sub-segment in the first speech segment using language recognition models, speech recognition models, and large language models is similar to the method of obtaining the target label of speech segments mentioned above. For details, please refer to the above text, and it will not be repeated here.

[0106] In this way, speech segments with inconsistent target labels can be further broken down into sub-segments, narrowing down the scope of the problem and facilitating targeted processing. Furthermore, by using finer-grained segmentation and combining it with technologies such as language classification and semantic judgment, the accuracy and consistency of the labels for each sub-segment can be further ensured.

[0107] In the above process, the intermediate results of each step can be recorded in a structured manner, so that the path of each target label generation is traceable. For example, it can be in the order of original audio, language prediction label output by the language recognition model, transcribed text output by the speech recognition model, first judgment result output by the large language model, and target label. Each step has clear intermediate data and execution standards, and the technical link has high interpretability. This provides strong support for the quality traceability, annotation review, and anomaly investigation of the speech recognition system that adopts the speech recognition method of this disclosure after it goes online.

[0108] When manual review or low-confidence screening is required, only the segments that LLM identifies as "boundary statements" or "semantic ambiguity" can be considered, significantly reducing the cost of manual intervention.

[0109] In summary, compared with related technologies based on language recognition models, which rely solely on acoustic features for language classification, misclassification of segments lacking semantic information (such as interjections, noise, and non-verbal utterances) is highly prone to occur. The speech recognition scheme of this disclosure, by introducing a verification mechanism of speech-to-text transcription and semantic judgment using a large language model after the language recognition model classifies the speech to be processed, effectively eliminates pseudo-labels that are "speech-present but semantically invalid," significantly improving the semantic interpretability and authenticity of the target labels.

[0110] In real-world testing on podcast and customer service recording datasets containing multiple dialects, cross-language accent interference, and background noise, this solution significantly improves accuracy in overall language structure consistency assessment (e.g., by 18 to 22 percentage points) and significantly reduces label misclassification rate (e.g., by approximately 23% to 35%) compared to recognition methods that only use large language models.

[0111] Furthermore, this solution is more effective for short speech segments (e.g., including short speech segments of less than 7 seconds), breaking through the performance bottleneck of traditional speech recognition methods in short segments, noisy environments, and mixed language environments. It has stronger semantic credibility, better practical fault tolerance, higher deployment adaptability and maintainability, and has significant technological progress and application value in the field of multilingual audio content understanding and processing.

[0112] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0113] In addition, this disclosure also provides a speech recognition device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any of the speech recognition methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section and will not be repeated here.

[0114] Figure 2 A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown, such as Figure 2 As shown, the device includes:

[0115] The acquisition module 21 is used to acquire the transcribed text, which is obtained by transcribing the speech to be processed based on language prediction labels.

[0116] Module 22 is invoked to call the large language model to perform semantic judgment on the transcribed text and obtain the first judgment result.

[0117] The output module 23 is used to confirm the language prediction label based on the first judgment result and output the transcription result of the speech to be processed. The transcription result of the speech to be processed includes at least the transcribed text.

[0118] In one possible implementation, the output module 23 is configured to: confirm the language prediction label as the target label of the speech to be processed if the first judgment result indicates that the transcribed text semantics are valid; and discard the language prediction label and mark the target label of the speech to be processed as invalid if the first judgment result indicates that the transcribed text semantics are invalid.

[0119] In one possible implementation, the acquisition module 21 is used to: call a pre-trained language recognition model to perform preliminary language classification prediction on the speech to be processed, and obtain language prediction labels; call the speech recognition model corresponding to the language prediction labels to perform speech transcription operation on the speech to be processed, and obtain transcribed text; and output the transcription result of the speech to be processed based on the confirmation result of the language prediction labels according to the first judgment result, including: outputting the language prediction labels, transcribed text, first judgment result, and confirmation result as the transcription result of the speech to be processed.

[0120] In one possible implementation, the speech recognition model corresponding to the language prediction label is invoked to perform a speech-to-text operation on the speech to be processed, and the transcribed text is obtained. This includes: finding the speech recognition model corresponding to the language prediction label based on the language prediction label and a pre-configured mapping table; inputting the speech to be processed into the speech recognition model corresponding to the language prediction label to generate the transcribed text of the speech to be processed.

[0121] In one possible implementation, a pre-trained language recognition model is invoked to perform preliminary language classification prediction on the speech to be processed, and language prediction labels are obtained. This includes: performing a first segmentation process on the speech to be processed to obtain at least one speech segment; and invoking the pre-trained language recognition model to perform language classification prediction on each speech segment to obtain a language prediction label for each speech segment.

[0122] In one possible implementation, if the first judgment result indicates that the transcribed text semantics are valid, the language prediction label is confirmed as the target label of the speech to be processed, including: if the first judgment result indicates that the transcribed text semantics corresponding to each speech segment in the speech to be processed are valid, the language prediction label is used as the target label of the speech segment; the target labels of different speech segments in the same speech to be processed are summarized to obtain the target label of the speech to be processed.

[0123] In one possible implementation, the target labels of different speech segments in the same speech to be processed are summarized to obtain the target labels of the speech to be processed. This includes: when the target labels of different speech segments in the same speech to be processed are inconsistent, obtaining the target labels of each speech sub-segment in the first speech segment with inconsistent target labels; and summarizing the target labels of different speech segments and / or different speech sub-segments in the same speech to be processed to obtain the target labels of the speech to be processed.

[0124] In one possible implementation, when the target labels of different speech segments in the same speech to be processed are inconsistent, the target label of each speech sub-segment in the first speech segment with inconsistent target labels is obtained, including: when the target labels of different speech segments in the same speech to be processed are inconsistent, performing a second segmentation process on the first speech segment with inconsistent target labels to obtain at least one speech sub-segment; calling a pre-trained language recognition model to perform language classification prediction on each speech sub-segment in the first speech segment to obtain a language prediction sub-label for each speech sub-segment; calling the speech recognition model corresponding to the language prediction sub-label to perform speech transcription operation on each speech sub-segment in the first speech segment to obtain the transcribed text of each speech sub-segment; calling a large language model to perform semantic judgment on the transcribed text of each speech sub-segment to obtain a second judgment result; and confirming the language prediction sub-label as the target label of the speech sub-segment if the second judgment result indicates that the semantics of the transcribed text of the speech sub-segment is valid.

[0125] In one possible implementation, the acquisition module 21 is further configured to acquire the speech to be processed. The acquisition method of the speech to be processed includes: performing preprocessing operations on the acquired original audio to obtain a preprocessed first audio; and performing speech activity detection on the first audio to obtain the speech to be processed corresponding to each voice interval in the first audio.

[0126] In one possible implementation, the acquired raw audio is preprocessed to obtain a preprocessed first audio, including: normalizing the sampling rate of the raw audio to obtain a second audio with a target sampling rate; normalizing the volume of the second audio to obtain a third audio with a target volume; denoising the third audio to obtain a denoised fourth audio; unifying the channels of the fourth audio to obtain a unified fifth audio; and converting the format of the fifth audio to obtain a first audio with a target format.

[0127] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0128] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the described method. The computer-readable storage medium can be volatile or non-volatile.

[0129] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0130] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device performs the above-described method.

[0131] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0132] Figure 3 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 3 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0133] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958. Electronic device 1900 can operate on an operating system, such as a server operating system, stored in memory 1932. TM ), based on a graphical user interface operating system (OS X) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), an open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0134] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0135] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0136] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0137] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0138] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0139] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0140] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0141] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0143] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0144] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0145] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0146] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0147] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A speech recognition method, characterized in that, The method includes: Obtain the transcribed text, which is obtained by transcribing the speech to be processed based on language prediction labels; The large language model is invoked to perform semantic judgment on the transcribed text to obtain a first judgment result. The large language model has cross-language semantic understanding capabilities. Based on the confirmation result of the language prediction label based on the first judgment result, the transcription result of the speech to be processed is output. The transcription result of the speech to be processed includes the language prediction label, the transcribed text, the first judgment result, and the confirmation result. The language prediction label includes the target label of the speech to be processed, which is obtained by summarizing the target labels of different speech segments in the speech to be processed. The speech segment is obtained by splitting the first speech segment with inconsistent target labels in the speech to be processed when the target labels of different speech segments in the same speech to be processed are inconsistent. The target label of the speech segment includes the language prediction sub-label corresponding to the speech segment when the transcribed text semantics of the speech segment are valid.

2. The method according to claim 1, characterized in that, Based on the first judgment result, the confirmation result of the language prediction label is obtained, including: If the first judgment result indicates that the transcribed text is semantically valid, the language prediction label is confirmed as the target label of the speech to be processed; If the first judgment result indicates that the transcribed text is semantically invalid, the language prediction label is discarded and the target label of the speech to be processed is marked as invalid.

3. The method according to claim 1, characterized in that, The process of obtaining the transcribed text includes: A pre-trained language recognition model is invoked to perform preliminary language classification prediction on the speech to be processed, and the language prediction label is obtained. The speech recognition model corresponding to the language prediction label is invoked to perform speech-to-text transcription on the speech to be processed, and the transcribed text is obtained. Based on the confirmation result of the language prediction label according to the first judgment result, the transcription result of the speech to be processed is output, including: Output the transcription result of the speech to be processed.

4. The method according to claim 3, characterized in that, The speech recognition model corresponding to the language prediction label is invoked to perform a speech-to-text operation on the speech to be processed, resulting in the transcribed text, including: Based on the language prediction label and the pre-configured mapping table, find the speech recognition model corresponding to the language prediction label; The speech to be processed is input into the speech recognition model corresponding to the language prediction label to generate the transcribed text of the speech to be processed.

5. The method according to claim 3, characterized in that, A pre-trained language recognition model is invoked to perform preliminary language classification prediction on the speech to be processed, and the language prediction label is obtained, including: The speech to be processed is first segmented to obtain at least one speech segment; The pre-trained language recognition model is invoked to perform language classification prediction for each speech segment, and the language prediction label for each speech segment is obtained.

6. The method according to claim 5, characterized in that, If the first judgment result indicates that the transcribed text is semantically valid, the language prediction label is confirmed as the target label of the speech to be processed, including: If the first judgment result indicates that the transcribed text semantics corresponding to each speech segment in the speech to be processed are valid, the language prediction label is used as the target label of the speech segment; The target labels of different speech segments in the same speech to be processed are summarized to obtain the target labels of the speech to be processed.

7. The method according to claim 1, characterized in that, The method for obtaining the target label of the speech sub-segment includes: When the target labels of different speech segments in the same speech to be processed are inconsistent, the first speech segment with inconsistent target labels is subjected to second segmentation processing to obtain at least one speech sub-segment. The pre-trained language recognition model is invoked to perform language classification prediction on each speech sub-segment in the first speech segment, and the language prediction sub-label of each speech sub-segment is obtained. The speech recognition model corresponding to the language prediction sub-label is invoked to perform speech transcription operation on each speech sub-segment in the first speech segment, so as to obtain the transcribed text of each speech sub-segment; The large language model is invoked to perform semantic judgment on the transcribed text of each speech sub-segment, and a second judgment result is obtained; If the second judgment result indicates that the transcribed text semantics of the speech sub-segment are valid, the language prediction sub-label is confirmed as the target label of the speech sub-segment.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes acquiring the speech to be processed, wherein the acquisition method for the speech to be processed includes: The acquired raw audio is preprocessed to obtain the preprocessed first audio. Speech activity detection is performed on the first audio to obtain the speech to be processed corresponding to each voice interval in the first audio.

9. The method according to claim 8, characterized in that, The acquired raw audio is preprocessed to obtain the preprocessed first audio, including: The original audio is subjected to sampling rate normalization to obtain a second audio with the target sampling rate; The second audio is subjected to volume normalization processing to obtain a third audio with the target volume; The third audio is subjected to noise reduction processing to obtain the noise-reduced fourth audio; The fourth audio signal is subjected to channel unification processing to obtain the fifth audio signal after channel unification. The fifth audio file is converted to obtain the first audio file with the target format.

10. A voice recognition device, characterized in that, include: The acquisition module is used to acquire the transcribed text, which is obtained by transcribing the speech to be processed based on language prediction labels; The calling module is used to call the large language model to perform semantic judgment on the transcribed text and obtain a first judgment result. The large language model has cross-language semantic understanding capabilities. The output module is used to output the transcription result of the speech to be processed based on the confirmation result of the language prediction label of the first judgment result, wherein the transcription result of the speech to be processed includes the language prediction label, the transcribed text, the first judgment result, and the confirmation result; The language prediction label includes the target label of the speech to be processed, which is obtained by summarizing the target labels of different speech segments in the speech to be processed. The speech segment is obtained by splitting the first speech segment with inconsistent target labels in the speech to be processed when the target labels of different speech segments in the same speech to be processed are inconsistent. The target label of the speech segment includes the language prediction sub-label corresponding to the speech segment when the transcribed text semantics of the speech segment are valid.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 9.

Images