Speech recognition method and device, electronic equipment and storage medium

Abstract

The invention relates to a voice recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a transliteration text which is obtained through the transliteration of a to-be-processed voice based on a language prediction label; calling a large language model to perform semantic judgment on the transcription text to obtain a first judgment result; and outputting a transcription result of the to-be-processed voice based on a confirmation result of the first judgment result on the language prediction tag. According to the embodiment of the invention, the reliability and stability of speech recognition can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a speech recognition method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the field of speech content processing, speech content understanding aims to extract multi-dimensional features such as semantics, pragmatics and language properties through processing and analysis of speech signals, and then support downstream applications such as semantic retrieval, content recommendation, speech summary and speaker recognition. As an important part of speech content understanding, language recognition can automatically identify and determine the language category of each speech in the audio stream, and its accuracy and reliability will directly affect the effect of the subsequent processing link. SUMMARY

[0003] The present disclosure provides a speech recognition method and device, electronic equipment and storage medium.

[0004] According to an aspect of the present disclosure, a speech recognition method is provided, comprising: obtaining a transcription text, the transcription text being obtained by transcribing a to-be-processed speech based on a language prediction label; calling a large language model to perform semantic judgment on the transcription text to obtain a first judgment result; and outputting a transcription result of the to-be-processed speech based on a confirmation result of the language prediction label on the first judgment result, the transcription result of the to-be-processed speech at least including the transcription text.

[0005] According to an aspect of the present disclosure, a speech recognition device is provided, comprising: an obtaining module configured to obtain a transcription text, the transcription text being obtained by transcribing a to-be-processed speech based on a language prediction label; a calling module configured to call a large language model to perform semantic judgment on the transcription text to obtain a first judgment result; and an output module configured to output a transcription result of the to-be-processed speech based on a confirmation result of the language prediction label on the first judgment result, the transcription result of the to-be-processed speech at least including the transcription text.

[0006] According to an aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0007] According to an aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the above method.

[0008] In the embodiments of the present disclosure, the transcribed text is obtained, the transcribed text is obtained by transcribing the to-be-processed speech based on the language prediction label; a large language model is called to perform semantic judgment on the transcribed text, and a first judgment result is obtained; based on a confirmation result of the language prediction label according to the first judgment result, a transcribed result of the to-be-processed speech is output, and the transcribed result of the to-be-processed speech at least includes the transcribed text. In this way, the large language model can be called to analyze whether the transcribed text constitutes a reasonable and coherent language expression, thereby assisting in judging whether the language prediction label is real and reliable. In the case where the large language model determines that the transcribed text has semantic validity, the transcribed text will be adopted. This verification mechanism at the semantic level can effectively filter the misjudgment results of speech recognition, greatly reduce the pollution of pseudo language labels to the semantic analysis and content processing process, significantly reduce the error transmission risk of subsequent processes (such as language label-based semantic understanding, recommendation, classification, etc.), and improve the reliability and stability of speech recognition.

[0009] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the technical solutions of the present disclosure.

[0011] Figure 1 A flowchart of a speech recognition method according to an embodiment of the present disclosure is shown.

[0012] Figure 2 A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown.

[0013] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0014] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference signs in the drawings represent functionally identical or similar elements. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0015] The word "exemplary" used herein means "serving as an example, an implementation, or being illustrative". Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0016] The term "and / or", as used herein, merely describes association between associated objects, and can indicate that three conditions can exist together, for example, A and / or B can mean that only A exists, both A and B exist, or only B exists. In addition, the term "at least one of", as used herein, means any one of or a combination of a plurality of the objects in the set, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0017] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. Those skilled in the art should understand that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, elements and circuits well known to those skilled in the art are not described in detail in order to highlight the main idea of the present disclosure.

[0018] In the related art, most of the speech language recognition schemes are based on pre-trained deep neural network models, and the speech segments are input into a multi-classifier to directly output the corresponding language label. Although this pre-trained multi-classification model has certain language discrimination ability, there are still technical difficulties in real scenarios: on the one hand, such a model is prone to misjudgment of invalid or ambiguous speech segments, resulting in label drift or error accumulation; on the other hand, due to the lack of semantic verification mechanism, it cannot be determined whether the predicted language label corresponds to the real readable language expression, resulting in insufficient stability and low reliability of the speech content understanding system in actual application.

[0019] Therefore, the embodiments of the present disclosure provide a speech recognition method, which can obtain a transcribed text, the transcribed text being obtained by transcribing a to-be-processed speech based on a language prediction label; calling a large language model to perform semantic judgment on the transcribed text to obtain a first judgment result; and outputting a transcribed result of the to-be-processed speech based on a confirmation result of the language prediction label according to the first judgment result, the transcribed result of the to-be-processed speech at least including the transcribed text.

[0020] In this way, the large language model can be called to analyze whether the transcribed text constitutes a reasonable and coherent language expression, thereby assisting in determining whether the language prediction label is real and reliable. Only when the large language model determines that the transcribed text has semantic validity, the transcribed text will be adopted. This semantic verification mechanism can effectively filter the misjudgment results of speech recognition, greatly reduce the pollution of pseudo language labels to the semantic analysis and content processing process, significantly reduce the error transmission risk of subsequent processes (such as language label-based semantic understanding, recommendation, classification, etc.), and improve the reliability and stability of speech recognition.

[0021] Figure 1 A flowchart of a speech recognition method according to an embodiment of the present disclosure is shown as follows:Figure 1 As shown, the speech recognition method can include:

[0022] In step S11, a transcription text is obtained, which is obtained by transcribing the to-be-processed speech based on a language prediction label;

[0023] In step S12, a large language model is called to perform semantic judgment on the transcription text, and a first judgment result is obtained.

[0024] In step S13, based on the first judgment result and the confirmation result of the language prediction label, a transcription result of the to-be-processed speech is output, and the transcription result of the to-be-processed speech at least includes the transcription text.

[0025] In a possible implementation, the speech recognition method of the embodiments of the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The speech recognition method can be realized by a processor calling computer-readable instructions stored in a memory. Alternatively, the method can be executed by a server.

[0026] In a possible implementation, the speech recognition method of the embodiments of the present disclosure can be applied to the labeling requirements of long-time audio or short-time audio in real scenarios such as podcasts, audiobooks, and voice customer service. The source of the to-be-processed speech can include podcasts, audiobooks, voice customer service systems, conference recordings, telephone calls, and other actual business scenarios, and the embodiments of the present disclosure do not limit the comparison.

[0027] In a possible implementation, obtaining the transcription text includes: calling a pre-trained language identification model (LID) to perform preliminary language classification prediction on the to-be-processed speech to obtain a language prediction label; and calling a speech recognition model (ASR) corresponding to the language prediction label to perform speech transcription operation on the to-be-processed speech to obtain the transcription text.

[0028] The language recognition model can automatically identify and determine the language category to which each segment of speech belongs in the audio stream. The language recognition model can directly output the language category to which the to-be-processed speech belongs and the corresponding confidence score, to provide a candidate language label as the language prediction label. It should be understood that the language recognition model can adopt a publicly available multilingual recognition model, for example, models including Whisper, XLS-R LID, Wav2Vec2-LID, etc. The specific selection of the language recognition model, the definition of the language set, and the source of the training data are not limited to a specific scheme, and can be flexibly configured according to the specific application scenario, and embodiments of the present disclosure do not limit this.

[0029] However, the language recognition model is mostly based on acoustic feature learning, lacks language-level understanding ability, and has unstable recognition accuracy for “low semantic information segments”. It is easy to misjudge segments with high silent proportion, ambiguous tone, extremely fast speech speed, or incomplete semantics. For example, for some segments containing only catchphrases, tone words (such as “uh-huh”, “hum” and the like), or incomplete syllables, the language recognition model will still output an explicit language label, resulting in false and invalid labeling results.

[0030] For example, when the to-be-processed speech is a short speech segment of 5 to 10 seconds, its information density is low, the context is limited, and the speaking habits are diverse, which easily causes the language recognition model to overfit the training set distribution and lack generalization ability. Especially for “low-resource languages” (for example, languages that lack large-scale parallel corpora and other resource support) or “similar languages” (such as Cantonese and Mandarin, Dutch and German), the language recognition model is easily disturbed by non-language factors such as noise, tone, rhythm, and the like, and the error rate of speech language recognition rises significantly.

[0031] Moreover, the language recognition model has no mechanism to verify whether the language it determines matches the speech content after outputting the language prediction label, and lacks semantic verifiability, which leads to the fact that even if the language recognition model produces a language prediction label with high confidence, it cannot determine whether the language prediction label really reflects the semantic content of the speech segment. For example, a segment misjudged as English may actually be a dialect, non-language background sound, or invalid speech.

[0032] Therefore, the speech recognition model corresponding to the language can be found through the language prediction label, and the speech recognition model found is used to perform speech transcription on the to-be-processed speech to generate a transcribed text for semantic analysis, so that subsequent semantic verification is performed based on the transcribed text. The speech recognition model can be a locally deployed model (for example, a Whisper-ASR model, a wav2vec2-CTC model, etc.), or a model provided by a remote application programming interface (API) service (for example, a speech recognition model provided by a social platform or an open source community platform). It should be understood that as long as the model has stable transcription capability and supports the target language, it can be used as a speech recognition model, and the embodiments of the present disclosure do not make specific limitations on the speech recognition model.

[0033] By introducing a label-driven transcription mechanism (for example, dynamically selecting a corresponding speech recognition model based on a candidate language prediction label), the language recognition model and the speech recognition model, which are two independent models, are connected in series, the transcribed text output by the speech recognition model is used to verify the language prediction label output by the language recognition model in reverse, and a verifiable link between language recognition and semantic content is established, thereby reducing the generalization and propagation of language label misjudgment on the non-semantic segment.

[0034] After obtaining the transcribed text in step S11, a large language model can be called in step S12 to perform semantic judgment on the transcribed text to obtain a first judgment result, which is used to represent whether the transcribed text is a "complete and natural sentence expression" in the language. The large language model can include GPT-4, Claude, Gemini, etc., as long as it has cross-language semantic understanding capability, and the embodiments of the present disclosure do not limit the category of the large language model. Since the large language model has cross-language semantic understanding capability, it can recognize the broken sentences, mixed languages, etc. in the speech transcription, and even if the language prediction label generated by the language recognition model is biased, it can be corrected by the large language model at the semantic layer, and the weakness of the acoustic feature model in low-resource languages and strong accents is compensated.

[0035] In the process of using the large language model to perform semantic judgment on the transcribed text, a target prompt can be set to send a semantic discrimination request to the large language model. The specific format, language setting, and calling interface of the prompt can be set according to the actual application scenario, and the embodiments of the present disclosure do not limit this.

[0036] After obtaining the first judgment result of the transcribed text in step S12, the first judgment result of the transcribed text is used to output a transcribed result of the to-be-processed speech in step S13, and the transcribed result of the to-be-processed speech at least includes the transcribed text.

[0037] In a possible implementation, based on the first determination result, the confirmation result of the language prediction label is obtained, including: in a case where the first determination result indicates that the semantic of the transcribed text is valid, the language prediction label is confirmed as the target label of the to-be-processed speech; in a case where the first determination result indicates that the semantic of the transcribed text is invalid, the language prediction label is discarded, and the target label of the to-be-processed speech is marked as invalid.

[0038] According to the first determination result of the transcribed text, it is determined whether to retain the language prediction label as the target label of the to-be-processed speech. If the first determination result indicates that the semantic of the transcribed text is valid, it means that the transcribed text is a reasonable expression, and the language prediction label of the to-be-processed speech is valid, and the language prediction label is confirmed as the target label of the to-be-processed speech; if the first determination result indicates that the semantic of the transcribed text is invalid, it means that the transcribed text is an invalid or ambiguous expression, and the language prediction label is discarded, and the target label of the to-be-processed speech is marked as invalid.

[0039] This strategy is equivalent to introducing a label screening module based on semantic understanding outside the acoustic language recognition path, increasing semantic layer filtering, which can effectively filter the misclassification output of the language recognition model for speech residual segments and non-language segments. The double-discrimination mechanism of acoustic classification results and semantic verification mechanism improves the label robustness and actual availability of the speech recognition method in the embodiments of the present disclosure.

[0040] In a possible implementation, in a case where the first determination result indicates that the semantic of the transcribed text is valid, the transcribed text can be output as a transcribed result. This simplified output method is suitable for application scenarios that do not have high requirements for traceability, and pay more attention to efficiency and simplicity, reducing the exposure of sensitive meta-information (such as the language prediction label, the first determination result, and the confirmation result).

[0041] In a possible implementation, the language prediction label, the transcribed text, the first determination result, and the confirmation result (such as the target label) can be output as the transcribed result of the to-be-processed speech.

[0042] In this way, four levels of meta-information can be retained for each to-be-processed speech, i.e., the language prediction label generated by the language recognition model, the transcribed text generated by the speech recognition model, the first determination result generated by the large language model, and the finally generated target label. This full-link traceable method improves the efficiency of auditing and quality tracking, and helps to quickly troubleshoot and manually review.

[0043] It should be understood that the language recognition model, the speech recognition model and the large language model in the embodiments of the present disclosure are independently encapsulated, and the models can be flexibly replaced, combined or upgraded. Each model can support open source models, models provided by commercial remote application programming interface (API) services, private deployment models and other modes, and is decoupled from each other. Due to the decoupling of the model structure, each model can have high replaceability and platform adaptability. This modular structure design can make the present scheme have wide adaptability in different hardware conditions, compliance requirements and language resource scenarios, and provide technical support for large-scale commercial deployment.

[0044] Through steps S11-S13, after performing preliminary language classification prediction on the to-be-processed speech using the language recognition model, the to-be-processed speech is transcribed using the speech recognition model of the language to which the to-be-processed speech belongs, and the large language model is called to analyze whether the transcribed text constitutes a reasonable and coherent language expression, thereby assisting in verifying whether the language prediction label is real and reliable. This joint language judgment link of "language recognition model output language prediction label + speech recognition model transcribe to-be-processed speech + large language model verify transcribed text" breaks through the limitations of the language recognition model scheme for short speech segment recognition, effectively solves technical problems such as false labels, semantically incomprehensible, and insufficient robustness, and significantly improves the practical application value and precision stability of language labeling in large-scale audio data scenarios. In the present scheme, the language prediction label will only be adopted as the target label when the large language model determines that the transcribed text has semantic effectiveness. This semantic level verification mechanism effectively filters the misjudgment results of the language recognition model (such as acoustic misjudgment of short sentences and noise segments), greatly reduces the pollution of false labels on subsequent semantic analysis and content processing processes, significantly reduces the error transmission risk of subsequent processes (such as language label-based semantic understanding, recommendation, classification, etc.), and improves the reliability and stability of labeling in actual audio processing.

[0045] Moreover, compared with related technologies that only output the final label, the present disclosure lacks error positioning and data tracing capabilities. The speech recognition method of the present disclosure can retain four levels of meta-information for each to-be-processed speech, i.e., the language prediction label generated by the language recognition model, the transcribed text generated by the speech recognition model, the first judgment result generated by the large language model, and the finally generated target label (i.e., the confirmation result), which is helpful for quickly troubleshooting and manual review.

[0046] In addition, in the related art, in order to improve the speech language recognition performance, the model needs to be retrained, for example, new labeled samples of a specific language are collected to retrain the language recognition model. In contrast, in the speech recognition method of the embodiments of the present disclosure, the verification path instead of the relearning path can be adopted, without adding new labeled data or retraining the model, through the collaborative verification of the speech recognition model and the large language model, the pseudo label can be filtered without changing the weight of the language recognition model, that is, the accuracy and stability of the final language label can be improved, which is suitable for performance enhancement tasks in low resource scenarios and uncontrollable model environments (such as models provided by external application program interfaces).

[0047] The speech recognition method of the embodiments of the present disclosure will be described below.

[0048] In a possible implementation, the method further includes obtaining the to-be-processed speech, and the obtaining manner of the to-be-processed speech includes: performing a preprocessing operation on the obtained original audio to obtain the first audio after preprocessing; and performing speech activity detection on the first audio to obtain the to-be-processed speech corresponding to each voice interval in the first audio.

[0049] For example, a target audio data set can be obtained, for example, a summary of original audio sampled from a podcast, an audiobook or a voice customer service system. The sampling manner can be regulated in combination with indexes such as the theme of the specific data, the language distribution and the user activity, and the embodiments of the present disclosure do not limit this.

[0050] The length of each original audio can be the same or different. For example, the original audio can be audio data within a certain range (for example, 5 minutes to 2 hours), and the format of the original audio can include a non-compressed format (for example, including WAV, AIFF), a lossless compression format (for example, including FLAC, APE), and a lossy compression format (for example, including MP3, AAC, OGG, etc. The format of the original audio in the embodiments of the present disclosure is not limited.

[0051] The original audio can be subjected to a unified preprocessing operation to obtain the first audio after preprocessing. The preprocessing operation can include at least one of sampling rate standardization, volume standardization, noise reduction processing, channel unification, and format conversion.

[0052] Optionally, the sampling rate standardization can uniformly adjust the sampling rate of the original audio signal to a preset value (for example, 16 kHz or 24 kHz), so that the first audio after preprocessing has consistent timing resolution, facilitating subsequent model processing.

[0053] Optionally, the volume standardization is to adjust the volume level of the original audio, so that the volumes of different original audios tend to be consistent (for example, conform to the -23 LUFS broadcast level loudness standard).

[0054] Optionally, the noise reduction processing can suppress the background noise of the original audio. The noise reduction processing method can include a spectral subtraction-based noise reduction method, a deep learning model-based noise reduction method (such as DeepFilterNet, RNNoise, etc.), and embodiments of the present disclosure do not limit this.

[0055] Optionally, the channel unification is used to convert the stereo sound of the multi-channel audio into single-channel audio.

[0056] Optionally, the audio format conversion is used to unify the original audio into the first audio in a lossless format, such as a WAV format using pulse code modulation, so that the first audio can be stored in a unified format.

[0057] By performing the preprocessing operation on the original audio, the consistency and stability of the subsequent processing flow can be effectively improved.

[0058] In a possible implementation, to improve the consistency and stability of the audio, the preprocessing operation is performed on the obtained original audio to obtain the first audio after preprocessing, including: performing a sampling rate standardization processing on the original audio to obtain a second audio with a target sampling rate (such as 16 kHz or 24 kHz); performing a volume standardization processing on the second audio to obtain a third audio with a target volume (such as -23 LUFS); performing a noise reduction processing on the third audio to obtain a fourth audio after noise reduction; performing a channel unification processing (such as unifying stereo sound into single-channel sound) on the fourth audio to obtain a fifth audio after channel unification; and performing a format conversion on the fifth audio to obtain the first audio with a target format (such as a WAV format).

[0059] To extract the segment containing the effective human voice in the first audio, a voice activity detection (VAD) method can be used to detect and segment the first audio, remove the silent segment, pure background music segment, or environmental sound segment, and obtain the to-be-processed voice corresponding to each voice interval in the first audio.

[0060] The voice activity detection method can extract features from each block of the first audio, and then use a classifier to classify each block to determine whether it is a voice signal.

[0061] Optionally, the voice activity detection method can use an energy threshold-based voice activity detection method, such as a voice activity detection module (Web Real Time Communications Voice Activity Detection, WebRTC VAD) used in a web real-time communication.

[0062] Optionally, the voice activity detection method can adopt a voice activity detection method based on a neural network model, such as a pre-trained enterprise-level voice activity detector (Silero VAD), a pyannote-audio VAD module, which can better adapt to complex backgrounds.

[0063] It should be understood that the implementation of voice activity detection can be flexibly configured according to the voice quality and background complexity of the application scenario, and the embodiments of the present disclosure do not limit this.

[0064] For example, after voice activity detection is performed on the first audio, a set of time interval labels {(start_1, end_1), (start_2, end_2),..., (start_n, end_n)} can be output, each time interval corresponding to a detected human voice paragraph; wherein the human voice interval (start_1, end_1) can correspond to the first to-be-processed voice, the human voice interval (start_2, end_2) can correspond to the second to-be-processed voice, and so on, and the human voice interval (start_n, end_n) can correspond to the nth to-be-processed voice.

[0065] By performing voice activity detection on the first audio, the silent and pure background segments in the first audio can be removed, and the human voice region is retained to generate a series of time-labeled segments for subsequent slicing.

[0066] In one possible implementation, a pre-trained language recognition model is called to perform preliminary language classification prediction on the to-be-processed voice to obtain a language prediction label, including: performing first segmentation processing on the to-be-processed voice to obtain at least one voice segment; and calling a pre-trained language recognition model to perform language classification prediction on each voice segment to obtain a language prediction label of each voice segment.

[0067] In this way, it is beneficial to implement a segment-level verification mechanism to break the propagation chain of mislabeled labels to adjacent segments, so that errors are locally isolated, do not affect the analysis of the language structure of the whole segment, and alleviate the distortion of the whole segment analysis caused by misjudgment diffusion in long recording.

[0068] For example, the to-be-processed voice can include a human voice interval identified from the first audio by using the voice activity detection method, and the to-be-processed voice can be first segmented to divide the to-be-processed voice into voice segments with appropriate lengths for subsequent language recognition processing.

[0069] Optionally, the first segmentation processing method can include fixed window segmentation, adaptive semantic segmentation, etc. The fixed window segmentation method can segment each speech paragraph into a fixed length window with a preset time length (e.g., 5-10 seconds), and can allow appropriate overlap (e.g., 50%) to improve coverage; the adaptive semantic segmentation method can be combined with automatic speech recognition or speech rate estimation, and can perform dynamic segmentation according to pause points or semantic boundaries. It should be understood that a suitable first segmentation processing method can be selected according to the actual application scenario, and the embodiments of the present disclosure do not limit this.

[0070] Each speech segment is recorded in a structured manner, for example: {"audio_id":"session_123", "segment_index":5, "start_time":157.3, "end_time":163.8, "waveform":[...]}. Among them, "audio_id":"session_123" indicates that the number of the speech to be processed is "session_123"; "segment_index":5 indicates that the order position of the speech segment in the speech to be processed is 5, "start_time":157.3 indicates that the start time of the speech segment is 157.3 seconds of the speech to be processed, "end_time":163.8 indicates that the end time of the speech segment is 163.8 seconds of the speech to be processed, and "waveform": [...] indicates the audio file format of the speech segment.

[0071] It should be understood that the purpose of performing speech first segmentation processing on the speech to be processed in the vocal time period is to generate an analyzable speech unit with a preset time length (e.g., 5-10 seconds). The first segmentation method can be a fixed length sliding window (e.g., a 6 second window with a 3 second overlap), or adaptive semantic segmentation combined with speech rate estimation and pause boundary. The speech segment formed by the slice structure can include audio segment content, start and end time, and encoding information of the corresponding speech to be processed, to facilitate subsequent parallel processing.

[0072] Each speech segment can be input into a pre-trained language recognition model to perform preliminary language classification prediction on each speech segment respectively, to obtain a language prediction label of each speech segment, which is used to indicate the language to which the speech segment belongs.

[0073] The language recognition model includes but is not limited to an end-to-end speech recognition model based on a Transformer architecture (encoder-decoder) (for example, a Whisper model supporting automatic language detection), a language recognition model provided by a social platform (or an open source community platform), a self-supervised audio pre-training model (for example, a Wav2Vec2-LID model, an XLS-R-LID model), and a custom language recognition classifier (for example, including a convolutional neural network, a self-attention network, and a long short-term memory network). Embodiments of the present disclosure do not limit the type of language recognition model.

[0074] The language recognition model can preliminarily determine the input speech segment as a language label (such as English, Chinese, Russian, French, etc.) in a predefined language set and output a corresponding confidence score.

[0075] The structured representation of the language prediction label output by the language recognition model is: { "segment_id": "session_123# 5", "predicted_language": "fr", "confidence": 0.87}. Wherein "segment_id": "session_123# 5" indicates the fifth speech segment in the speech to be processed numbered "session_123"; "predicted_language": "fr" indicates that the language prediction label is French, and "confidence": 0.87 indicates that the confidence of the language prediction label is 0.87.

[0076] It should be understood that the language prediction label is not a final decision result, but a candidate input for the subsequent semantic verification stage.

[0077] It can be seen that in typical long audio scenarios such as audiobooks, voice conferences, and customer service calls, the language recognition model directly recognizes long audio in the related art. If a segment is misjudged in this process, the language structure analysis of the entire audio will fail. Embodiments of the present disclosure can use a segment-by-segment recognition method for subsequent segment-by-segment verification. This segment-by-segment structural method can construct a segment-by-segment verifiable label system at the language level, effectively block the propagation of mislabeled labels to the context, and significantly reduce the mislabeling propagation and cumulative error rate in long audio scenarios, thereby reducing the error coupling risk in the language recognition process.

[0078] After obtaining the language prediction label of each speech segment in the speech to be processed, a speech recognition model corresponding to the language prediction label can be called to perform speech transcription on each speech segment in the speech to be processed to obtain a transcribed text.

[0079] In a possible implementation, the speech recognition model corresponding to the language prediction label can be found according to the language prediction label and the preconfigured mapping table; and the speech to be processed is input into the speech recognition model corresponding to the language prediction label to generate the transcription text of the speech to be processed.

[0080] ‌By constructing the preconfigured mapping table, the speech recognition method of the embodiment of the present disclosure can realize accurate positioning and efficient matching of the speech recognition model, and significantly improve the calling efficiency of the speech recognition model.

[0081] For example, the preconfigured mapping table can be found according to the language prediction label, for example: { "en": "Whisper-ASR-English", "zh":"Whisper-ASR-Chinese", "fr":"Wav2Vec2-French-ASR",...}. Wherein, "en":"Whisper-ASR- English" represents the speech recognition model corresponding to English, "zh": "Whisper-ASR-Chinese" represents the speech recognition model corresponding to Chinese, and "fr": "Wav2Vec2-French-ASR" represents the speech recognition model corresponding to French.

[0082] The corresponding speech segment is input into the speech recognition model of the language as input to obtain the transcription text. The structured representation of the processing result output by the speech recognition model is: {"segment_id": "session_123#5", "predicted_language": "fr", "transcript_text": "Je vais à la bibliothèque cetaprès-midi", "asr_model": "Wav2Vec2-French-ASR"}. Wherein, "segment_id": "session_123#5" represents the fifth speech segment in the order in the speech to be processed numbered "session_123"; "predicted_language": "fr" represents that the language prediction label is French, "transcript_text": "Je vais à la bibliothèque cetaprès-midi." represents the French transcription text of the speech segment is "Je vais à la bibliothèque cetaprès-midi", and "asr_model": "Wav2Vec2-French-ASR" represents the model class adopted is the speech recognition model corresponding to French.

[0083] The transcribed text of the voice segment obtained in step S11 can be subjected to semantic rationality judgment by a large language model in step S12 to obtain a first judgment result.

[0084] For the transcribed text of each voice segment, a large language model can be called to perform semantic judgment on the transcribed text to determine whether the transcribed text is a reasonable and complete sentence expression in the target language.

[0085] This process can be implemented by a specially designed prompt word, for example, the prompt word can be designed as:

[0086] Please determine whether the following text is a complete, grammatically and semantically correct natural language expression. If yes, return Y; if not (such as garbled code, sentence break, non-language noise, etc.), return N.

[0087] Language: French

[0088] Text: Je vais à la bibliothèque cet après-midi.

[0089] The large language model can output a structured first judgment result according to the prompt word, for example: {"segment_id": "session_123#5", "valid_sentence": true}. Wherein, "segment_id": "session_123#5" represents the fifth voice segment in the order in the voice to be processed numbered "session_123"; "valid_sentence": true indicates that the first judgment result is that the transcribed text is semantically valid.

[0090] The first judgment result of the transcribed text obtained in step S12 can be used to obtain a confirmation result of the language prediction label in step S13, so as to filter the language label according to the semantic judgment result.

[0091] In one possible implementation, when the first judgment result indicates that the transcribed text corresponding to each voice segment in the voice to be processed is semantically valid, the language prediction label is taken as the target label of the voice segment; when the first judgment result indicates that the transcribed text is semantically invalid, the language prediction label is discarded, and the voice segment is marked as "language cannot be judged" or "no language label" to improve fault tolerance and robustness.

[0092] For example, if the first judgment result is "valid_sentence": true, indicating that the transcribed text corresponding to the voice segment is semantically valid, it is considered that the language prediction label of the voice segment is valid, and the language prediction label can be taken as the final language label of the voice segment, i.e. the target label;

[0093] If the first determination result is "valid_sentence": false, indicating that the semantic of the transcribed text corresponding to the voice segment is invalid, the language prediction label of the voice segment is directly discarded, and is regarded as "language cannot be determined" or "semantic invalid segment", which can be marked as undefined or null. Wherein, undefined means undefined, and null means null. The structured representation of each voice segment is: {"segment_id": "session_123#5", "final_language_label": "fr", "transcript_text": "...", "llm_verified": true}. Wherein, "segment_id": "session_123#5" indicates that the voice segment with the fifth order in the session_123 to be processed voice; "final_language_label": "fr" indicates that the target label of the voice segment is French; "transcript_text": "...", indicates the transcribed text of the voice segment; "llm_verified": true indicates that the first determination result is valid.

[0094] In this way, the misclassification of the language recognition model to non-language audio, semantic segment defects, noise segments and the like can be effectively filtered, and the stability and semantic reliability of the label system are improved.

[0095] In addition, compared with the related art, the language recognition system in the related art forces to output a language label for each voice segment, for example, even if a voice segment itself lacks language structure or content is chaotic, a label is still output, which can cause a large number of uncertain segments to be misjudged and mislabeled. The mechanism of hard decision label in the related art can seriously affect the accuracy and interpretability of downstream applications, especially in real business scenarios where data quality is uneven and language distribution is extremely unbalanced, which can easily cause a large number of error labels to accumulate.

[0096] In the embodiment of the present disclosure, after the language recognition model generates a preliminary language prediction label, the judgment mechanism of the large language model is introduced, and the language undeterminable state output is supported, for example, the state of "language cannot be determined" or "refuse to label" can be output. This mechanism of refusing to recognize (for example, when the large language model determines that the transcribed text is invalid, the language cannot be determined state is returned explicitly, instead of forcibly assigning a language label) greatly improves the fault tolerance and safety boundary of the actual landing of the speech recognition method. This mechanism has strong adaptability to data cleaning, label reliability evaluation, low-resource language processing and other scenarios, and can significantly reduce the chain errors caused by the propagation of error labels.

[0097] After the target label is screened by using the first judgment result, the segment-level language recognition label result of the data set can be aggregated to obtain the target label of the to-be-processed speech.

[0098] In a possible implementation, the target labels of different speech segments in the same to-be-processed speech can be aggregated to obtain the target label of the to-be-processed speech.

[0099] For example, the target labels of the speech segments can be aggregated based on the number segment_id of each speech segment to form a language distribution label of the audio data set at the segment level. The target label of the to-be-processed speech can include a language label time axis corresponding to an audio file of each to-be-processed speech, visual language distribution (for example, language switching points, language proportion analysis), and various data storage file formats (for example, JSON, CSV, ELAN, and the like) that can be exported for use by a downstream system.

[0100] For example, the schematic data structure of the target label is as follows: "audio_id": "session_123", "segments": [{"start": 0.0, "end": 7.8, "language": "en"}, {"start": 7.8, "end": 15.2, "language": null}, {"start": 15.2, "end": 22.6, "language": "zh"}]. Wherein "audio_id": "session_123" indicates that the to-be-processed speech numbered "session_123", in which the target label of the speech segment [0, 7.8] is English, the target label of the speech segment [7.8, 15.2] is an invalid label, and the target label of the speech segment [15.2, 22.6] is Chinese.

[0101] In this way, the filtered segment-level language recognition result can be integrated to form a segment-level language label time axis of the entire data set. The output can be in a structured readable format, supporting subsequent semantic understanding, audio retrieval, label clustering, language switching point detection, and the like. In a specific engineering implementation, the structured readable format can be flexibly adjusted according to platform technology selection.

[0102] In a possible implementation, the target labels of different speech segments in the same to-be-processed speech are aggregated to obtain the target label of the to-be-processed speech, including: in the case where the target labels of different speech segments in the same to-be-processed speech are inconsistent, obtaining the target label of each speech subsegment in a first speech segment in which the target labels are inconsistent; and aggregating the target labels of different speech segments and / or different speech subsegments in the same to-be-processed speech to obtain the target label of the to-be-processed speech.

[0103] For example, if the target labels of different speech segments in the to-be-processed speech are inconsistent, for example, the to-be-processed speech can include speech segment 1, speech segment 2, speech segment 3, and speech segment 4, where the target labels of the speech segment 1, the speech segment 3, and the speech segment 4 are all Chinese, and the target label of the speech segment 2 is Japanese, the speech segment 2 can be taken as the first speech segment, and the first speech segment can be further divided to determine the target label of each speech subsegment in the first speech segment. In this way, if there are some other languages or dialects in the speech segment, for example, 2 Japanese sentences are mixed in Chinese, the Japanese can be further divided into smaller segments through further division, so as to further improve the accuracy of speech recognition.

[0104] In a possible implementation, in the case where the target labels of different speech segments in the same to-be-processed speech are inconsistent, the target label of each speech subsegment in a first speech segment in which the target labels are inconsistent is obtained, including: in the case where the target labels of different speech segments in the same to-be-processed speech are inconsistent, performing second segmentation processing on the first speech segment in which the target labels are inconsistent to obtain at least one speech subsegment; invoking a pre-trained language recognition model to perform language classification prediction on each speech subsegment in the first speech segment to obtain a language prediction sublabel of each speech subsegment; invoking a speech recognition model corresponding to the language prediction sublabel to perform speech transcription on each speech subsegment in the first speech segment to obtain a transcription text of each speech subsegment; invoking a large language model to perform semantic judgment on the transcription text of each speech subsegment to obtain a second judgment result; and in the case where the second judgment result represents that the transcription text of the speech subsegment is semantically valid, confirming the language prediction sublabel as the target label of the speech subsegment.

[0105] It should be understood that the method of obtaining the target label of each speech subsegment in the first speech segment by using the language recognition model, the speech recognition model, and the large language model is similar to the method of obtaining the target label of the speech segment described above, and details can be referred to the above description, which will not be repeated here.

[0106] In this way, the target label inconsistent speech segment can be further split into sub-segments, narrowing down the problem range and facilitating subsequent targeted processing. Moreover, through finer-grained segmentation, combined with language classification, semantic judgment and other technologies, the label accuracy and consistency of each sub-segment can be further ensured.

[0107] In the above process, the intermediate results of each step can be recorded in a structured manner, so that each target label generation path can be traced, for example, according to the original audio, the language prediction label output by the language recognition model, the transcribed text output by the speech recognition model, the first judgment result output by the large language model, and the order of the target label. Each step has clear intermediate data and execution standards, and the technical link has high interpretability, providing strong support capabilities for quality traceability, annotation auditing, and exception troubleshooting for a speech recognition system that uses the speech recognition method of the embodiments of the present disclosure after going online.

[0108] When manual review or low confidence screening is required, only the segments determined by the LLM as "boundary sentences" or "semantically ambiguous" can be focused on, greatly reducing the cost of manual intervention.

[0109] In summary, compared with the related art solution based on a language recognition model, which only relies on acoustic features for language classification, it is easy to misclassify segments with insufficient semantic information (such as tone words, noise, and non-linguistic utterances). The speech recognition solution of the embodiments of the present disclosure introduces a verification mechanism of speech transcription text and large language model semantic judgment after the language recognition model classifies the speech to be processed, effectively eliminating the "speech exists but semantic invalid" false label, and greatly improving the semantic interpretability and authenticity of the target label.

[0110] The solution was tested on a podcast and customer service recording dataset containing multiple dialects, cross-lingual accent interference, and background noise. Compared with the recognition method using only a large language model, the accuracy of the solution in the whole sentence language structure consistency evaluation was greatly improved (e.g., by 18 to 22 percentage points), and the label misjudgment rate was greatly reduced (e.g., by about 23% to 35%).

[0111] Moreover, the solution has more significant effects on short speech segments (e.g., including short speech segments less than 7 seconds), breaking through the performance bottleneck of traditional speech language recognition methods in short segments, noise interference, and language mixed environments, and has stronger semantic credibility, better actual fault tolerance, higher deployment adaptability and maintainability, and has significant technical progress and application value in the field of multi-lingual audio content understanding and processing.

[0112] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form combined embodiments without deviating from the principle logic. Limited by the length of the present disclosure, the present disclosure will not be described again. Those skilled in the art can understand that in the above-mentioned method of the specific embodiment, the specific execution order of each step should be determined according to its function and possible internal logic.

[0113] In addition, the present disclosure also provides a speech recognition device, an electronic device, a computer readable storage medium, and a program, all of which can be used to implement any of the speech recognition methods provided by the present disclosure. The corresponding technical solutions and descriptions are described in the method section and are not repeated.

[0114] Figure 2 A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown as follows, Figure 2 As shown, the device includes:

[0115] The acquisition module 21 is configured to acquire a transcribed text, the transcribed text being obtained by transcribing a to-be-processed speech based on a language prediction label;

[0116] The calling module 22 is configured to call a large language model to perform semantic judgment on the transcribed text to obtain a first judgment result.

[0117] The output module 23 is configured to output a transcribed result of the to-be-processed speech based on a confirmation result of the language prediction label according to the first judgment result, the transcribed result of the to-be-processed speech at least including the transcribed text.

[0118] In a possible implementation, the output module 23 is configured to: in a case where the first judgment result represents that the transcribed text is semantically valid, confirm the language prediction label as a target label of the to-be-processed speech; and in a case where the first judgment result represents that the transcribed text is semantically invalid, discard the language prediction label and mark a target label of the to-be-processed speech as invalid.

[0119] In a possible implementation, the acquisition module 21 is configured to: call a pre-trained language recognition model to perform preliminary language classification prediction on the to-be-processed speech to obtain a language prediction label; call a speech recognition model corresponding to the language prediction label to perform a speech transcription operation on the to-be-processed speech to obtain the transcribed text; and output the transcribed result of the to-be-processed speech based on the confirmation result of the language prediction label, including: outputting the language prediction label, the transcribed text, the first judgment result, and the confirmation result as the transcribed result of the to-be-processed speech.

[0120] In a possible implementation, the speech recognition model corresponding to the language prediction label is called to perform speech transcription on the to-be-processed speech, and the transcription text is obtained, including: searching for the speech recognition model corresponding to the language prediction label according to the language prediction label and a preconfigured mapping table; inputting the to-be-processed speech into the speech recognition model corresponding to the language prediction label to generate the transcription text of the to-be-processed speech.

[0121] In a possible implementation, the pre-trained language recognition model is called to perform preliminary language classification prediction on the to-be-processed speech, and the language prediction label is obtained, including: performing first segmentation processing on the to-be-processed speech to obtain at least one speech segment; calling the pre-trained language recognition model to perform language classification prediction on each speech segment to obtain the language prediction label of each speech segment.

[0122] In a possible implementation, in a case where the first judgment result represents that the semantics of the transcription text is valid, the language prediction label is confirmed as the target label of the to-be-processed speech, including: in a case where the first judgment result represents that the semantics of the transcription text corresponding to each speech segment in the to-be-processed speech is valid, the language prediction label is taken as the target label of the speech segment; and the target labels of different speech segments in the same to-be-processed speech are summarized to obtain the target label of the to-be-processed speech.

[0123] In a possible implementation, the target labels of different speech segments in the same to-be-processed speech are summarized to obtain the target label of the to-be-processed speech, including: in a case where the target labels of different speech segments in the same to-be-processed speech are inconsistent, the target label of each speech sub-segment in a first speech segment in which the target labels are inconsistent is obtained; and the target labels of different speech segments and / or different speech sub-segments in the same to-be-processed speech are summarized to obtain the target label of the to-be-processed speech.

[0124] In a possible implementation, in the case that the target labels of different speech segments in the same to-be-processed speech are inconsistent, the target label of each speech sub-segment in the first speech segment with inconsistent target labels is obtained, including: in the case that the target labels of different speech segments in the same to-be-processed speech are inconsistent, performing second segmentation processing on the first speech segment with inconsistent target labels to obtain at least one speech sub-segment; calling a pre-trained language recognition model to perform language classification prediction on each speech sub-segment in the first speech segment to obtain a language prediction sub-label of each speech sub-segment; calling a speech recognition model corresponding to the language prediction sub-label to perform speech transcription on each speech sub-segment in the first speech segment to obtain a transcription text of each speech sub-segment; calling a large language model to perform semantic judgment on the transcription text of each speech sub-segment to obtain a second judgment result; in the case that the second judgment result represents that the transcription text of the speech sub-segment is semantically valid, the language prediction sub-label is confirmed as the target label of the speech sub-segment.

[0125] In a possible implementation, the obtaining module 21 is further configured to obtain to-be-processed speech, and the to-be-processed speech is obtained in the following manner: performing preprocessing on the obtained original audio to obtain first audio after preprocessing; performing voice activity detection on the first audio to obtain to-be-processed speech corresponding to each voice interval in the first audio.

[0126] In a possible implementation, the preprocessing on the obtained original audio to obtain first audio after preprocessing includes: performing sampling rate standardization processing on the original audio to obtain second audio with a target sampling rate; performing volume standardization processing on the second audio to obtain third audio with a target volume; performing noise reduction processing on the third audio to obtain fourth audio after noise reduction; performing channel unification processing on the fourth audio to obtain fifth audio after channel unification; and performing format conversion on the fifth audio to obtain the first audio with a target format.

[0127] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be described here.

[0128] The embodiments of the present disclosure also provide a computer-readable storage medium having computer program instructions stored therein, and the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0129] The embodiments of the present disclosure also provide an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0130] The embodiment of the disclosure further provides a computer program product, comprising computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is executed in the processor of the electronic device, the processor in the electronic device executes the above method.

[0131] The electronic device can be provided as a terminal, a server or other forms of devices.

[0132] Figure 3 A block diagram of an electronic device 1900 according to an embodiment of the disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 3 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0133] The electronic device 1900 can also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a server operating system (Server TM ), a graphical user interface operating system (OS X TM ), a multi-user multi-process computer operating system (Unix TM ), a free and open source code Unix-like operating system (Linux TM ), an open source code Unix-like operating system (FreeBSD TM ) or the like.

[0134] In an exemplary embodiment, a non-volatile computer readable storage medium, such as the memory 1932 comprising computer program instructions, is also provided, and the above computer program instructions can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

[0135] The disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions embodied therewith, which cause a processor to implement various aspects of the disclosure.

[0136] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0137] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0138] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0139] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0140] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0141] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0142] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0143] The computer program product can be embodied in a tangible medium of

[0144] The above description of the various embodiments is intended to be illustrative in all aspects, rather than being restrictive. Those skilled in the art can refer to the description of the various embodiments to make modifications and / or improvements.

[0145] Those skilled in the art can understand that, in the above-described method of the specific embodiments, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible inherent logic.

[0146] If the technical solutions of the present application involve personal information, the product applying the technical solutions of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solutions of the present application involve sensitive personal information, the product applying the technical solutions of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that it has entered the personal information collection range and will collect personal information. If the individual voluntarily enters the collection range, it is considered to agree to collect personal information. Or on the device for processing personal information, through the pop-up information or by asking the individual to upload his personal information, the individual's authorization is obtained under the condition that the device uses obvious mark / information to inform the individual of the personal information processing rules. The personal information processing rules can include personal information processor, personal information processing purpose, processing method and personal information type, etc.

[0147] The above has described various embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical application or improvement of technology in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A speech recognition method, characterized in that, The method includes: Obtain the transcribed text, which is obtained by transcribing the speech to be processed based on language prediction labels; The large language model is invoked to perform semantic judgment on the transcribed text, and a first judgment result is obtained; Based on the confirmation result of the language prediction label based on the first judgment result, the transcription result of the speech to be processed is output, and the transcription result of the speech to be processed includes at least the transcribed text.

2. The method according to claim 1, characterized in that, Based on the first judgment result, the confirmation result of the language prediction label is obtained, including: If the first judgment result indicates that the transcribed text is semantically valid, the language prediction label is confirmed as the target label of the speech to be processed; If the first judgment result indicates that the transcribed text is semantically invalid, the language prediction label is discarded and the target label of the speech to be processed is marked as invalid.

3. The method according to claim 1, characterized in that, The process of obtaining the transcribed text includes: A pre-trained language recognition model is invoked to perform preliminary language classification prediction on the speech to be processed, and the language prediction label is obtained. The speech recognition model corresponding to the language prediction label is invoked to perform speech-to-text transcription on the speech to be processed, and the transcribed text is obtained. Based on the confirmation result of the language prediction label according to the first judgment result, the transcription result of the speech to be processed is output, including: The language prediction label, the transcribed text, the first judgment result, and the confirmation result are output as the transcription result of the speech to be processed.

4. The method according to claim 3, characterized in that, The speech recognition model corresponding to the language prediction label is invoked to perform a speech-to-text operation on the speech to be processed, resulting in the transcribed text, including: Based on the language prediction label and the pre-configured mapping table, find the speech recognition model corresponding to the language prediction label; The speech to be processed is input into the speech recognition model corresponding to the language prediction label to generate the transcribed text of the speech to be processed.

5. The method according to claim 3, characterized in that, A pre-trained language recognition model is invoked to perform preliminary language classification prediction on the speech to be processed, and the language prediction label is obtained, including: The speech to be processed is first segmented to obtain at least one speech segment; The pre-trained language recognition model is invoked to perform language classification prediction for each speech segment, and the language prediction label for each speech segment is obtained.

6. The method according to claim 5, characterized in that, If the first judgment result indicates that the transcribed text is semantically valid, the language prediction label is confirmed as the target label of the speech to be processed, including: If the first judgment result indicates that the transcribed text semantics corresponding to each speech segment in the speech to be processed are valid, the language prediction label is used as the target label of the speech segment; The target labels of different speech segments in the same speech to be processed are summarized to obtain the target labels of the speech to be processed.

7. The method according to claim 6, characterized in that, The target labels of different speech segments in the same speech to be processed are summarized to obtain the target labels of the speech to be processed, including: When the target labels of different speech segments in the same speech to be processed are inconsistent, obtain the target label of each speech sub-segment in the first speech segment with inconsistent target labels; The target labels of different speech segments and / or different speech sub-segments in the same speech to be processed are summarized to obtain the target labels of the speech to be processed.

8. The method according to claim 7, characterized in that, When the target labels of different speech segments in the same speech to be processed are inconsistent, the target labels of each speech sub-segment in the first speech segment with inconsistent target labels are obtained, including: When the target labels of different speech segments in the same speech to be processed are inconsistent, the first speech segment with inconsistent target labels is subjected to second segmentation processing to obtain at least one speech sub-segment. The pre-trained language recognition model is invoked to perform language classification prediction on each speech sub-segment in the first speech segment, and the language prediction sub-label of each speech sub-segment is obtained. The speech recognition model corresponding to the language prediction sub-label is invoked to perform speech transcription operation on each speech sub-segment in the first speech segment, so as to obtain the transcribed text of each speech sub-segment; The large language model is invoked to perform semantic judgment on the transcribed text of each speech sub-segment, and a second judgment result is obtained; If the second judgment result indicates that the transcribed text semantics of the speech sub-segment are valid, the language prediction sub-label is confirmed as the target label of the speech sub-segment.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes acquiring the speech to be processed, wherein the acquisition method for the speech to be processed includes: The acquired raw audio is preprocessed to obtain the preprocessed first audio. Speech activity detection is performed on the first audio to obtain the speech to be processed corresponding to each voice interval in the first audio.

10. The method according to claim 9, characterized in that, The acquired raw audio is preprocessed to obtain the preprocessed first audio, including: The original audio is subjected to sampling rate normalization to obtain a second audio with the target sampling rate; The second audio is subjected to volume normalization processing to obtain a third audio with the target volume; The third audio is subjected to noise reduction processing to obtain the noise-reduced fourth audio; The fourth audio signal is subjected to channel unification processing to obtain the fifth audio signal after channel unification. The fifth audio file is converted to obtain the first audio file with the target format.

11. A voice recognition device, characterized in that, include: The acquisition module is used to acquire the transcribed text, which is obtained by transcribing the speech to be processed based on language prediction labels; The calling module is used to call the large language model to perform semantic judgment on the transcribed text and obtain the first judgment result; The output module is used to output the transcription result of the speech to be processed based on the confirmation result of the language prediction label of the first judgment result, wherein the transcription result of the speech to be processed includes at least the transcribed text.

12. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 10.

Images