Speech transcription method and system based on multi-feature fusion and scenario-based semantic association

Abstract

The embodiment of the application discloses a speech transcription method and system based on multi-feature fusion and scenario semantic association. The method comprises the following steps: obtaining audio data to be recognized and a trained speech recognition model; using a feature extraction module to extract multi-dimensional core features, multi-dimensional low-frequency part features and short-time energy entropy features of the audio data to be recognized and to fuse the features to obtain multi-fusion acoustic features; and using a language model layer with a scenario semantic association model to perform piecewise transcription on the multi-fusion acoustic features to obtain text of the audio data to be recognized. The embodiment of the application solves the technical problem of how to improve the speech transcription accuracy in a target scenario.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech transcription method and system based on multi-feature fusion and contextual semantic association. Background Technology

[0002] With the development of speech recognition technology, speech-to-text technology has made breakthrough progress, and real-time speech-to-text has been successfully applied in many scenarios. For example, meetings are an important application scenario for speech-to-text. Through online or offline speech-to-text technology, speeches in meetings are converted into text to form detailed meeting records, which facilitates the storage, retrieval, searching, and dissemination of meeting content.

[0003] Existing speech recognition and transcription technologies rely on a general ASR (Automatic Speech Recognition) model and have not been designed and adjusted for the audio characteristics of specific scenarios, resulting in limited transcription accuracy in some specific scenarios. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a speech transcription method and system based on multi-feature fusion and contextualized semantic association to solve the technical problem of how to improve speech transcription accuracy in target scenarios.

[0005] In a first aspect, this application provides a speech-to-text method based on multi-feature fusion and contextual semantic association, the method comprising:

[0006] The audio data to be recognized is acquired, and a trained speech recognition model is obtained; wherein, the speech recognition model is obtained by dynamic adaptive weighted training based on a training set constructed from general spoken language corpus and professional language corpus of various industries; the speech recognition model includes a feature extraction module and a language model layer;

[0007] The feature extraction module is used to extract multidimensional core features, multidimensional low-frequency features, and short-time energy entropy features from the audio data to be identified and then fused to obtain multi-fused acoustic features.

[0008] The language model layer with a contextualized semantic association model is used to segment and transcribe the multi-fusion acoustic features to obtain the text of the audio data to be identified; wherein, the contextualized semantic association model adopts a multi-head attention mechanism and is pre-trained based on annotated corpus with multiple semantic association rules of the target scene.

[0009] In at least some embodiments of this application,

[0010] The training set, constructed based on general spoken language corpora and industry-specific professional corpora, undergoes dynamic adaptive weighted training, including:

[0011] The general spoken language corpus and industry-specific professional corpus are subjected to layered mixing preprocessing. The preprocessing includes: selecting daily dialogues and impromptu speech fragments of target scenarios from the general spoken language corpus, and retaining the real acoustic features; classifying the industry-specific corpus by industry and retaining industry-specific terms, professional expressions and scenario-based sentence structures; and randomly splicing the corpus in the mixing process and labeling them with scenario tags.

[0012] In the early stages of training the speech recognition model, the weight of general spoken language corpus is set to be greater than the weight of industry-specific professional language corpus.

[0013] During the training phase of the speech recognition model, the weights of the general spoken language corpus are set to be equal to the weights of the industry-specific professional language corpus.

[0014] In the later stages of training the speech recognition model, a validation set is set up, and the weights of the industry-specific corpora are dynamically fine-tuned based on the accuracy of the validation set.

[0015] In at least some embodiments of this application,

[0016] The feature extraction module extracts multidimensional core features, multidimensional low-frequency features, and short-time energy entropy features from the audio data to be identified and fuses them to obtain multi-fused acoustic features, including:

[0017] The audio data to be identified is standardized to remove DC components and extreme outliers from the audio data.

[0018] The processed audio data is divided into frames according to preset frame length and preset frame shift, and Hanning window is used to suppress spectral leakage.

[0019] For each frame of audio data after framing, multi-dimensional core features are extracted based on the MFCC algorithm; wherein, the core features include the core frequency band of human voice and the spectral envelope features;

[0020] Multidimensional low-frequency features are extracted based on the LPCC algorithm;

[0021] One-dimensional features are calculated based on the short-time energy entropy algorithm to quantify the energy distribution of each frame of audio data, distinguish effective speech segments from environmental noise segments, and obtain short-time energy entropy features.

[0022] The extracted multidimensional core features, multidimensional low-frequency features, and short-time energy entropy features are fused through an attention mechanism to obtain multi-fused acoustic features. During fusion, the weight proportions of the multidimensional core features, multidimensional low-frequency features, and short-time energy entropy features decrease sequentially. The attention mechanism identifies the feature importance of the current frame audio data in real time and dynamically adjusts the weight allocation.

[0023] In at least some embodiments of this application,

[0024] The method further includes:

[0025] A temporal attention mechanism is introduced into the feature extraction module to perform correlation calculation on the fusion features of audio data with consecutive preset frame lengths, thereby capturing the temporal correlation of features between frames.

[0026] A noise robustness training branch is set in the feature extraction module, and the multi-fusion acoustic features are input into the speech recognition model for training. The extraction parameters of the feature extraction module are iteratively optimized based on the gradient descent algorithm so that the feature extraction module can adapt to the audio characteristics of the target scene.

[0027] In at least some embodiments of this application,

[0028] During the process of transcribing the multi-fused acoustic features by the language model layer, the contextualized semantic association model performs at least one of the following:

[0029] The system receives the multi-fused acoustic features and synchronously reads the corresponding timestamp information, then sorts the multi-fused acoustic features in chronological order.

[0030] Based on the multi-head attention mechanism, the semantic features of the current transcribed segment are cross-correlated with the semantic features of adjacent transcribed segments to extract semantic association points between segments; based on the semantic association points, the semantic association between segments is determined to decide whether to splice them.

[0031] Based on the pre-trained rules of the target scene, the spliced ​​speech is verified, and splicing results that do not conform to the semantic logic of the target scene are filtered out.

[0032] In at least some embodiments of this application,

[0033] The language model layer also has a terminology enhancement layer; the terminology enhancement layer is configured to convert professional terms in the dictionary into multi-dimensional vectors based on the terminology embedding algorithm, embed them into the vocabulary of the language model layer, and learn the collocation rules of terms in different sentence patterns by adjusting and optimizing the terminology context adaptation logic.

[0034] The method further includes: optimizing the loss function of the language model layer by introducing a joint loss function of cross-entropy loss and term matching loss; wherein the weight of the cross-entropy loss is greater than the weight of the term matching loss; the cross-entropy loss is used to optimize the transcription accuracy of the speech recognition model, and the term matching loss is used to enhance the term transcription accuracy of the speech recognition model.

[0035] The language model layer is linked with the acoustic feature extraction module, and the text annotation data of the target scene is used for training and adjustment. The process is iterated until the term transcription accuracy and semantic coherence of the speech recognition model reach the preset values.

[0036] In at least some embodiments of this application,

[0037] The method further includes: performing semantic verification on the splicing result output by the contextualized semantic association model; if an error is found in the splicing result, it is fed back to the contextualized semantic association model, which recalculates the semantic association degree of adjacent transcribed segments and adjusts the splicing boundary, and combines the user-corrected transcribed data to reverse-optimize its own semantic association rules.

[0038] In at least some embodiments of this application,

[0039] The method further includes: correcting errors in the text of the obtained audio data to be recognized; the error correction process is performed according to at least one of the following:

[0040] Based on the audio characteristics of the audio data to be identified and the confidence level of each transcribed text output by the speech recognition model, the text to be prioritized for error correction is selected.

[0041] Based on the training results of the general spoken language corpus during the training of the speech recognition model and the Chinese grammar rule base, grammatical errors in the text are corrected;

[0042] Based on the contextualized semantic association model and the semantic rules of the target scene, the semantic association between the current transcribed segment and the adjacent transcribed segments is compared, and the timestamp information in the audio data to be identified is used to correct semantic errors in the text.

[0043] Based on a multi-industry terminology dictionary, suspected terms in the text are identified, the rationality of term collocations is verified, and terminology expressions are standardized; and terminology dictionaries corresponding to the industry of the target scenario are called first.

[0044] Based on the timestamp information and audio energy features in the audio data to be identified, the actual pause duration of the speech is identified, and the punctuation of the sentence breaks in the text is corrected accordingly, and the semantic deviation caused by the misjudgment of pauses is corrected.

[0045] A noise mis-transcription feature library is established based on the noise type and intensity fed back by the device that picks up the audio data to be identified. Meaningless noise mis-transcription content is automatically identified and deleted. For text defects caused by noise, the missing text is completed by combining the context semantics and terminology dictionary in the scenario-based semantic association model.

[0046] The speech recognition model is iteratively optimized based on the user-corrected transcription data.

[0047] In at least some embodiments of this application,

[0048] The method further includes: after correcting the text of the obtained audio data to be identified, calling the contextualized semantic association model to verify the corrected text, and judging the semantic coherence, grammatical consistency and terminology uniformity of the text with the preceding and following segments.

[0049] If there are contradictory error correction results, the error correction result that conforms to semantic logic shall be selected first, taking into account the audio features and transcription confidence of the audio data to be identified.

[0050] If the verification is successful, the corrected transcribed text will be output, and the error correction data generated during the error correction process will be fed back to the speech recognition model for iterative upgrades.

[0051] In a second aspect, this application provides a speech-to-text system, including a server; the server is configured to perform the method as described in any one of the first aspects.

[0052] In a third aspect, this application provides an electronic device including a memory, one or more processors, and one or more application programs, wherein the one or more application programs are stored in the memory and configured to, when invoked by the one or more processors, cause the one or more processors to perform the methods as described in any of the first aspects.

[0053] In a fourth aspect, this application provides a computer-readable storage medium storing a plurality of program codes adapted to be loaded and run by a processor to perform the method of any of the first aspects.

[0054] The above-described technical solutions of this application have at least one or more of the following beneficial effects:

[0055] In implementing the technical solution of this application, by integrating general spoken language corpora with multi-industry professional corpora and adopting a dynamic adaptive weighted training mechanism, the model can cover both daily expressions and understand professional terms and industry language, thus enabling the model to maintain high recognition accuracy in various practical application scenarios. The feature extraction module not only extracts multi-dimensional core features but also integrates multi-dimensional low-frequency features and short-time energy entropy features to comprehensively capture detailed information and dynamic changes in speech signals, enhancing the model's robustness to complex speech environments and providing richer and more discriminative acoustic representations for subsequent recognition. The language model layer introduces a scenario-based semantic association model, which adopts a multi-head attention mechanism and is pre-trained based on multiple annotated corpora with semantic association rules for the target scenario. This provides targeted semantic associations for the target scenario, strengthens semantic coherence, and improves the applicability of the target scenario and the accuracy of splicing. In summary, this application constructs an end-to-end high-precision speech recognition process by linking four key technologies: multi-feature fusion extraction, dynamic adaptive training, segmented semantic association analysis, and scenario-based semantic verification. This not only improves the accuracy of audio-to-text transcription but also significantly enhances the practicality and reliability of the system in different industries and application scenarios.

[0056] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0057] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Furthermore, similar numbers in the drawings are used to denote similar components, wherein:

[0058] Figure 1 This is a schematic diagram of the main steps of a speech transcription method based on multi-feature fusion and contextual semantic association according to one embodiment of this application. Detailed Implementation

[0059] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0060] See appendix Figure 1 , Figure 1 This is a schematic diagram illustrating the main steps of a speech-to-text method based on multi-feature fusion and contextual semantic association, according to an embodiment of this application. Figure 1As shown, a speech-to-text method based on multi-feature fusion and contextual semantic association in this application embodiment mainly includes the following steps S101-S103:

[0061] Step S101: Obtain the audio data to be recognized and obtain the trained speech recognition model; wherein, the speech recognition model is obtained by dynamically adaptive weighted training based on the training set constructed from general spoken language corpus and professional language corpus of various industries; the speech recognition model includes a feature extraction module and a language model layer;

[0062] Step S102: Use the feature extraction module to extract multi-dimensional core features, multi-dimensional low-frequency features, and short-time energy entropy features from the audio data to be identified, and fuse them to obtain multi-fused acoustic features;

[0063] Step S103: Based on the language model layer with a contextualized semantic association model, the multi-fusion acoustic features are segmented and transcribed to obtain the text of the audio data to be identified; wherein, the contextualized semantic association model adopts a multi-head attention mechanism and is pre-trained based on annotated corpus with multiple semantic association rules of the target scene.

[0064] Among them, the contextualized semantic association model adopts a multi-layer encoder structure and a multi-head attention mechanism; and is pre-trained based on multiple labeled corpora of the target scene to learn the semantic association rules between sentences and paragraphs in the target scene and enhance semantic coherence.

[0065] Specifically, this can be understood as inserting a contextualized semantic association model between the original encoder and decoder of the speech recognition model. It adopts a lightweight design to avoid the running delay caused by too many model parameters and ensures that it is compatible with the low latency requirements of front-end sound pickup and transmission. The contextualized semantic association model introduces a lightweight Transformer encoder, which adopts a multi-layer encoder structure and a multi-head attention mechanism, such as a 6-layer encoder structure and an 8-head attention mechanism. It is pre-trained on more than 100,000 labeled corpora of target scenarios such as meetings and interviews to learn the semantic association rules between sentences and paragraphs in the scenario (such as topic connection in meeting scenarios and question-and-answer logic in interview scenarios), thereby strengthening semantic coherence.

[0066] This application employs a segmented streaming transcription architecture, combined with a contextualized semantic association model to achieve seamless splicing of segmented results, ensuring end-to-end transcription latency of <500ms. This streaming transcription algorithm can collaborate with various front-end audio pickup algorithms, and its segmentation rhythm (e.g., 200ms) is precisely matched with the transmission rate, ensuring that audio data transmission and segmented transcription are synchronized. At the same time, the contextualized semantic association model's contextual semantic association algorithm can correct local transcription deviations caused by front-end noise residue and minor transmission distortion, achieving seamless collaboration between "low-latency transmission in the middle and real-time transcription in the back end," ensuring transcription real-time performance and continuity.

[0067] Based on the steps S101-S103 above, the speech-to-text method based on multi-feature fusion and contextualized semantic association in this embodiment integrates general spoken language corpora with multi-industry professional corpora and adopts a dynamic adaptive weighted training mechanism. This enables the model to cover both everyday expressions and professional terminology and industry language, thus maintaining high recognition accuracy in various practical application scenarios. The feature extraction module not only extracts multi-dimensional core features but also integrates multi-dimensional low-frequency features and short-time energy entropy features to comprehensively capture detailed information and dynamic changes in the speech signal, enhancing the model's robustness to complex speech environments and providing richer and more discriminative acoustic representations for subsequent recognition. The language model layer introduces a contextualized semantic association model, which adopts a multi-head attention mechanism and is pre-trained based on multiple annotated corpora with semantic association rules for the target scene. This provides targeted semantic associations for the target scene, strengthens semantic coherence, and improves the applicability of the target scene and the accuracy of splicing. In summary, this application constructs an end-to-end high-precision speech recognition process by linking four key technologies: multi-feature fusion extraction, dynamic adaptive training, segmented semantic association analysis, and scenario-based semantic verification. This not only improves the accuracy of audio-to-text transcription but also significantly enhances the practicality and reliability of the system in different industries and application scenarios.

[0068] In one embodiment, dynamic adaptive weighted training is performed on a training set constructed based on general spoken language corpora and industry-specific professional corpora, including:

[0069] The general spoken language corpus and industry-specific professional corpus are preprocessed by layering and mixing. The preprocessing includes: screening daily dialogues and impromptu speech fragments of target scenarios from the general spoken language corpus, preserving the real acoustic features; classifying industry-specific corpus by industry and preserving industry-specific terms, professional expressions and scenario-based sentence structures; and randomly splicing the corpus in the mixing process and labeling them with scenario tags.

[0070] In the early stages of training the speech recognition model, the weight of general spoken language corpus is set to be greater than the weight of professional language corpus from various industries.

[0071] During the training phase of the speech recognition model, the weights of general spoken language corpora are set to be equal to the weights of industry-specific professional language corpora.

[0072] In the later stages of training the speech recognition model, a validation set is set up, and the weights of the industry-specific corpora are dynamically fine-tuned based on the accuracy of the validation set.

[0073] It is understandable that the speech recognition model in this application adopts a "layered hybrid + dynamic weighting" training strategy to ensure that the corpus adapts to the actual needs of the target sound pickup scenario. First, the general spoken language corpus and the professional corpus of various industries are preprocessed in layers. The general spoken language corpus (accounting for about 67%) is selected to cover daily dialogues and impromptu speech fragments covering target scenarios such as meetings and interviews, retaining the real acoustic features such as colloquial interjections, pauses, and repetitions; the professional corpus of various industries (accounting for about 33%) is organized according to industry classification (such as business meetings, academic interviews, etc.), focusing on retaining industry-specific terminology, professional expressions, and scenario-based sentence structures. The weighting strategy employs dynamic adaptive weighting. In the initial training phase, the weight of general spoken language corpora is set higher than that of industry-specific corpora (e.g., 2:1), focusing on improving the model's basic adaptability to spoken language. In the middle training phase, the weights are gradually adjusted until the weights of general spoken language corpora are equal to those of industry-specific corpora (e.g., 1:1), strengthening the integration and adaptation of professional terminology with spoken expressions. In the later training phase, the model is dynamically fine-tuned based on the validation set accuracy (error ≤ 5%). For industries with low terminology transcription accuracy, for example, the weight of the corresponding professional corpora is temporarily increased to 1.2:1, ensuring that the model meets the transcription needs of multiple industries while also considering spoken language adaptation, ultimately achieving an industry terminology transcription accuracy ≥ 95%. During corpus mixing, a randomized corpus concatenation + scene labeling method is used to avoid model overfitting caused by clustering of similar corpora, ensuring the diversity and balance of training data.

[0074] In one embodiment, a feature extraction module is used to extract multi-dimensional core features, multi-dimensional low-frequency features, and short-time energy entropy features from the audio data to be identified, and then fuses them to obtain multi-fused acoustic features, including:

[0075] The audio data to be identified is standardized to remove DC components and extreme outliers.

[0076] The processed audio data is divided into frames according to preset frame length and preset frame shift, and Hanning window is used to suppress spectral leakage.

[0077] For each frame of audio data after framing, multi-dimensional core features are extracted based on the MFCC algorithm; among them, the core features include the core frequency band of human voice and the spectral envelope features;

[0078] Multidimensional low-frequency features are extracted based on the LPCC algorithm;

[0079] One-dimensional features are calculated based on the short-time energy entropy algorithm to quantify the energy distribution of each frame of audio data, distinguish effective speech segments from environmental noise segments, and obtain short-time energy entropy features.

[0080] The extracted multidimensional core features, multidimensional low-frequency features, and short-time energy entropy features are fused through an attention mechanism to obtain multi-fused acoustic features. During fusion, the weight proportions of the multidimensional core features, multidimensional low-frequency features, and short-time energy entropy features decrease sequentially. The attention mechanism identifies the feature importance of the current frame audio data in real time and dynamically adjusts the weight allocation.

[0081] This can be understood as the feature extraction module abandoning the traditional fixed-dimensional feature extraction method and adopting a multi-feature fusion extraction technology of "Mel frequency cepstral coefficients (MFCC) + linear predictive cepstral coefficients (LPCC) + short-time energy entropy". The specific implementation process is as follows:

[0082] The first step is audio preprocessing, which standardizes the clean audio with a signal-to-noise ratio of ≥42dB and the residual analog noise audio (signal-to-noise ratio of 20-30dB) output from the front-end pickup module, with a uniform sampling rate of 16kHz and a bit depth of 16bit, and removes DC components and extreme outliers from the audio.

[0083] The second step is to frame and window the audio. The preprocessed audio is divided into frames with a frame length of 25ms and a frame shift of 10ms. Hanning window is used to suppress spectral leakage and ensure the stability of the audio features of each frame.

[0084] The third step involves extracting multiple features separately. For each audio frame, the MFCC algorithm extracts 13 core features, focusing on the core frequency band of human voice (300Hz-3kHz) to capture the spectral envelope features of speech. The LPCC algorithm extracts 12 supplementary features, focusing on capturing low-frequency features of the audio to compensate for the insufficient recognition of MFCC in low-frequency noise residual scenarios. The short-time energy entropy algorithm calculates 1-dimensional features to quantify the energy distribution of each audio frame, accurately distinguishing effective speech segments (high energy entropy) from environmental noise segments (low energy entropy).

[0085] The fourth step is feature fusion and optimization. The extracted MFCC, LPCC, and short-time energy entropy features are fused through an attention mechanism, with weights set at 50%, 30%, and 20%, respectively. The attention mechanism can identify the feature importance of the current frame audio in real time, dynamically adjust the weight allocation, strengthen effective speech features, and suppress noise features.

[0086] Furthermore, a temporal attention mechanism is introduced to perform correlation calculations on the fusion features of 10-15 consecutive audio frames, capturing the temporal correlation of features between frames, reducing feature loss caused by sound source switching and short pauses, and further improving the accuracy of acoustic feature extraction.

[0087] Finally, a noise robustness training branch is added, and the extracted fusion features are input into the model for training. The feature extraction parameters are iteratively optimized through the gradient descent algorithm, so that the feature extraction module can adapt to the audio characteristics of the front-end sound pickup scenario, has strong anti-interference ability, and ensures accurate matching with the audio requirement of ≥42dB signal-to-noise ratio output by the front-end sound pickup module.

[0088] In one embodiment, during the transcription of multi-fused acoustic features by the language model layer, the contextualized semantic association model performs at least one of the following:

[0089] Receive multiple fused acoustic features and synchronously read the corresponding timestamp information, and sort the multiple fused acoustic features in chronological order;

[0090] A multi-head attention mechanism is used to cross-correlate the semantic features of the current transcribed segment with those of adjacent transcribed segments to extract semantic association points between segments; based on these semantic association points, the semantic association between segments is determined to decide whether to splice them.

[0091] Based on pre-trained rules for the target scene, the spliced ​​speech is validated, and splicing results that do not conform to the semantic logic of the target scene are filtered out.

[0092] During the transcription process, timestamp sorting ensures temporal consistency, and a multi-head attention mechanism is used to perform semantic association analysis on adjacent transcribed segments, effectively identifying and splicing logically coherent speech segments to avoid semantic fragmentation and incorrect splicing caused by simple segmentation. Based on pre-trained rules for the target scene, the splicing results are verified, and outputs that do not conform to the semantic logic of the scene are filtered out, further eliminating unreasonable content in the transcription, ensuring that the final text is not only accurate at the lexical level, but also conforms to the expression habits of the specific scene at the syntactic and semantic levels.

[0093] Specifically, the contextualized semantic association model can achieve contextual association, avoid erroneous splicing, and coordinate with the entire process of front-end sound pickup and back-end transcription. Specifically:

[0094] Temporal feature association: Each single-person audio data after front-end sound pickup separation has a timestamp accurate to 10ms. When the contextual semantic association model receives the feature sequence output by the acoustic feature extraction module, it synchronously reads the corresponding timestamp information, sorts the feature sequence according to the time sequence, and establishes a three-dimensional association of "time-feature-semantics" to ensure the temporal consistency of the context and avoid semantic order disorder caused by sound source switching, thereby reducing erroneous splicing from the source.

[0095] Precise matching through attention mechanism: The scenario-based semantic association model adopts a multi-head attention mechanism to cross-reference the semantic features of the current transcription segment (200ms segments, synchronized with the segment rhythm of mid-end transmission and streaming transcription, with the mid-end forwarding audio data once every 200ms segment) with the semantic features of the previous 3 and the next 2 adjacent segments. It extracts semantic association points between segments (such as keywords of the same topic, sentence connection words, person reference relationships, etc.) and judges the semantic coherence between segments. If the semantic association degree between adjacent segments is ≥85%, seamless splicing is performed; if the association degree is <85%, it is judged as a topic switch or sound source switch, and the splicing boundary is marked to avoid semantic errors caused by forced splicing and ensure precise coordination with the low-latency transmission of front-end audio data and the rhythm of back-end streaming transcription.

[0096] Contextualized rule constraints: The contextualized semantic association model is based on pre-trained rules for target scenarios such as meetings and interviews. It verifies the semantics of the spliced ​​parts and filters splicing results that do not conform to the semantic logic of the scenario (e.g., in a meeting scenario, avoids incorrectly splicing fragments such as "project progress" and "daily trivial matters", and in an interview scenario, avoids incorrectly splicing "question" and "irrelevant answer"). At the same time, combined with the sound source separation label of the front-end sound pickup module, if adjacent segments come from different speakers, the speaker is automatically marked to switch, and semantic splicing is not forced, further avoiding incorrect splicing.

[0097] In one specific implementation, the language model layer also has a terminology enhancement layer; the terminology enhancement layer is configured to convert professional terms in the dictionary into multi-dimensional vectors based on a terminology embedding algorithm, embed them into the vocabulary of the language model layer, and learn the collocation rules of terms in different sentence patterns by adjusting and optimizing the terminology context adaptation logic.

[0098] The method also includes: optimizing the loss function of the language model layer by introducing a joint loss function of cross-entropy loss and term matching loss; wherein the weight of cross-entropy loss is greater than the weight of term matching loss; cross-entropy loss is used to optimize the transcription accuracy of the speech recognition model, and term matching loss is used to enhance the term transcription accuracy of the speech recognition model.

[0099] The language model layer is linked with the acoustic feature extraction module, and the text-annotated data of the target scene is used for training and adjustment. The process is iterated until the term transcription accuracy and semantic coherence of the speech recognition model reach the preset values.

[0100] Specifically, the terminology enhancement layer is also placed between the original encoder and decoder of the speech recognition model, with a lightweight design. The terminology enhancement layer first constructs a multi-industry terminology dictionary (covering mainstream industries such as business, academia, and medicine, containing 50,000+ core terms and common collocations). It adopts term embedding technology to convert the professional terms in the dictionary into 128-dimensional vectors and embed them into the vocabulary of the language model layer to improve the model's priority for term recognition. At the same time, it learns the collocation rules of terms in different sentence patterns by fine-tuning and optimizing the terminology context adaptation logic, thus avoiding semantic deviations caused by isolated transcription of terms.

[0101] Loss function optimization and overall fine-tuning: The loss function of the language model layer is optimized by introducing a joint loss function of cross-entropy loss and term matching loss, for example, with a weight ratio of 7:3. Cross-entropy loss is used to optimize the overall transcription accuracy, while term matching loss is used to enhance the accuracy of term transcription. Subsequently, the language model layer is linked with the optimized acoustic feature extraction module, and overall fine-tuning is performed using audio-text annotation data (100,000+ records) from the front-end sound pickup scenario. The process is iterated for more than 100 rounds until the term transcription accuracy of the model is ≥95% and the semantic coherence rate is ≥98%. This completes the optimization of the language model layer and ensures that it meets the back-end transcription accuracy requirements (industry term transcription accuracy ≥95%, Chinese transcription error rate (CER) <4%).

[0102] In one embodiment, the contextualized semantic association model employs bidirectional feedback correction: semantic verification is performed on the splicing results output by the contextualized semantic association model; if errors are found in the splicing results (such as semantic contradictions or logical breaks), feedback is sent to the contextualized semantic association model, which recalculates the semantic relevance of adjacent transcribed segments and adjusts the splicing boundaries. Combined with the user-corrected transcribed data, it reversely optimizes its own semantic association rules, continuously improving the accuracy of contextual association and reducing the occurrence of erroneous splicing.

[0103] Based on the above embodiments and implementation methods, this application can achieve accurate separate transcription of multiple people speaking alternately. The specific process is as follows:

[0104] First, a front-end layered sound source separation algorithm can be set to complete basic speech separation of the audio data to be recognized, extract the speech features of each speaker (such as fundamental frequency, speech rate, and timbre), complete the initial sound source separation, output multiple independent single-person speech signals, and mark the timestamp information of each voice (accurate to 10ms) to avoid sound source aliasing.

[0105] The speech recognition model in this application is adapted and optimized for the separated single-speaker speech signals. The acoustic feature extraction module of the model can accurately identify the unique features of each speech signal. Combined with the semantic association logic of the language model layer, it distinguishes the language habits and tone characteristics of different speakers, realizes independent transcription of each single-speaker speech signal, and generates transcribed text with speaker tags (the tagging method is "speaker 1 / 2 / ... + transcribed content").

[0106] Finally, a time-series synchronization calibration algorithm is introduced to accurately align the timestamps of each voice source after front-end separation with the timestamps of the back-end transcription results. For scenarios such as cross-pauses and interruptions in multi-person alternating speech, the transcription order is corrected through a scenario-based semantic association model to ensure that the temporal sequence of the transcribed text is completely consistent with the actual speaking order. At the same time, combined with a semantic error correction algorithm, a small number of cross-speech transcription errors caused by incomplete separation are corrected, further improving the transcription accuracy in multi-person alternating speech scenarios. This ensures that the Chinese transcription error rate (CER) is <4% and the transcription accuracy of industry terms is ≥95%, solving the technical pain points of traditional models in multi-sound source scenarios where transcription is chaotic and speakers cannot be distinguished. This achieves the algorithmic linkage of "front-end sound source separation - back-end accurate transcription".

[0107] The core algorithm of the speech recognition model in this application embodiment is deeply integrated with the layered sound source separation algorithm of the front-end sound pickup. The model can accurately extract acoustic features from the separated single-person speech signal, adapt to the signal switching rhythm of multiple people speaking alternately, effectively solve the pain point that the traditional general model cannot adapt to the switching of multiple sound sources, realize the algorithm linkage of "front-end sound source separation - back-end accurate feature extraction", and lay the foundation for subsequent streaming transcription and accuracy optimization.

[0108] In one embodiment, the method further includes: correcting errors in the text of the obtained audio data to be recognized; the error correction process is performed according to at least one of the following:

[0109] Based on the audio characteristics of the audio data to be identified and the confidence score of each transcribed text output by the speech recognition model, texts that are prioritized for error correction are selected. Specifically, the system receives the segmented transcribed text output from the backend streaming transcription (200ms / segment, synchronized with the mid-end transmission and streaming transcription rhythm), and synchronously calls the auxiliary information fed back by the frontend sound pickup module (sound source separation marker, audio signal-to-noise ratio, noise type) and the confidence score output by the backend model (confidence of each transcribed text segment, with a threshold set to 0.7). Transcribed segments with a confidence score <0.7 and those with noise residue or sound source switching reported by the frontend are selected for priority error correction, reducing invalid calculations and improving error correction efficiency.

[0110] Based on the training results of general spoken language corpus during the training of the speech recognition model and the Chinese grammar rule library, correct the grammar errors in the text; specifically, accurately correct the grammar errors in the transcribed text, focusing on the colloquial speech disorders and spelling mistakes caused by front-end noise interference and unclear sound sources. Specifically include: correct typos (such as correcting "project progress" to "project schedule"), correct disordered word order (such as correcting "The meeting will be held tomorrow for us" to "We will hold the meeting tomorrow"), supplement missing punctuation (adapting to the scenario of no punctuation in spoken language transcription), correct collocation errors (such as correcting "achieve the completion of the goal" to "achieve the goal"); during the error correction process, rely on the training results of the spoken language corpus during model fine-tuning, retain reasonable colloquial expressions (such as the modal particles "ne, o"), avoid over-correction resulting in a rigid text, and at the same time combine the Chinese grammar rule library to ensure that the corrected text conforms to grammar norms, and the error correction accuracy rate ≥ 98%;

[0111] Based on the scene-based semantic association model and the semantic rules of the target scene, compare the semantic relevance of the current transcribed segment with the adjacent transcribed segments, and联动 the timestamp information in the audio data to be recognized to correct the semantic errors in the text; specifically, on the basis of grammar error correction, correct the semantic deviations, focusing on solving the semantic contradictions and logical breaks caused by incomplete front-end sound source separation and minor transmission distortion. Specifically include: correct semantic ambiguity (such as correcting "This plan is not feasible" to "The implementation details of this plan are not feasible" in combination with the context), make up for semantic missing (such as missing "the end of the month" in "The project deadline is at the end of the month" due to noise, and complete it in combination with the context semantics before and after), correct logical contradictions (such as correcting "The project has been completed and not started" to "The project has been started and is in progress"); during the error correction process, call the association model of the scene-based semantic association layer, combine the semantic rules of the target scenes such as meetings and interviews, compare the semantic relevance of the current transcribed segment with the adjacent segments, and at the same time联动 the timestamp information of front-end sound source separation to ensure that semantic error correction conforms to the actual speech logic and does not deviate from the original meaning;

[0112] Identify suspected terms in the text based on a multi-industry term dictionary, verify the rationality of term collocations, and standardize term expressions; and preferentially call the term dictionary corresponding to the target scenario's industry. As a special error correction link, focus on ensuring the accuracy of industry term transcription. Relying on the multi-industry term dictionary (more than 50,000 core terms and collocations) constructed during model fine-tuning, it specifically includes: identifying suspected terms in the transcribed text (e.g., miswriting "commercial concrete" as "commercial connection"), verifying the rationality of term collocations (e.g., miswriting "blockchain technology" as "blockchain tool" and correcting it based on industry corpus), and standardizing term expressions (e.g., unifying "AI" and "artificial intelligence" into a preset expression in the same text). At the same time, for multi-industry adaptation requirements, according to the industry label of the front-end voice pickup scenario (target scenario) (users can select the industry type in the APP), preferentially call the corresponding industry's term dictionary to enhance the pertinence of term verification and ensure that the accuracy rate of industry term transcription is ≥95%;

[0113] Identify the actual speaking pause duration based on the timestamp information and audio energy characteristics in the audio data to be recognized, and correspondingly correct the punctuation of the text's sentence breaks and correct the semantic deviation caused by misjudging pauses. Specifically, for the pauses, interruptions in multi-person alternating speech and natural pauses in single-person speech in the front-end target voice pickup scenario, solve the problems of "misjudging pauses as sentence breaks" and "misjudging interruptions as semantic breaks" in the transcribed text. Specific implementation: Relying on the timestamp information (accurate to 10ms) and audio energy characteristics output by the front-end hierarchical sound source separation algorithm, identify the actual speaking pause duration (natural pause 0.5 - 1s, interruption pause <0.3s, end pause ≥1.5s), and correspondingly correct the punctuation of the sentence breaks in the transcribed text: add a comma for natural pauses, retain the coherent expression (do not add punctuation) for interruption pauses, and add a period for end pauses. At the same time, correct the semantic deviation caused by misjudging pauses (e.g., miswriting "I think (pause 0.8s) this plan is feasible" as "I think this plan is feasible" and correcting it to "I think, this plan is feasible"), adapt to the scenarios of multi-person alternating speech and single-person natural speech, and further reduce semantic errors;

[0114] A noise mis-transcription feature library is established based on the noise type and intensity feedback from the device picking up the audio data to be identified. Meaningless noise mis-transcribed content is automatically identified and deleted. For text incompleteness caused by noise, contextual semantics and a terminology dictionary from a scenario-based semantic association model are used to complete the incomplete text. For noise that is not fully suppressed by the front-end multimodal noise reduction algorithm (such as slight air conditioner noise or distant background noise), resulting in "meaningless filler words" and "noise mis-transcribed into text" in the transcribed text, the specific implementation relies on the noise feedback from the front-end audio pickup module. Based on the type (steady-state / non-steady-state noise) and noise intensity, a noise mis-transcription feature library is established (such as redundant interjections like "um" and "oh" easily caused by steady-state noise, and mis-transcribed texts like "ah" and "ya" easily caused by non-steady-state noise). Meaningless noise mis-transcribed content is automatically identified and deleted, while meaningful interjections are retained. At the same time, for text incompleteness caused by noise (such as "project progress (noise interference)"), the missing text is supplemented by combining contextual semantics and terminology dictionary, further making up for the minor shortcomings of front-end noise reduction, especially suitable for speech-to-text transcription scenarios in complex environments.

[0115] Based on the above error correction implementation methods, the final result is a Chinese transcription error rate (CER) of <4% and an industry terminology transcription accuracy of ≥95%. The error correction algorithm can be deeply integrated with the front-end sound pickup algorithm and the back-end speech recognition model algorithm. Among them, the grammatical error correction and semantic error correction algorithms can correct transcription errors and semantic deviations caused by incomplete noise reduction of the front-end algorithm and slight deviations in sound source separation. The terminology verification algorithm relies on industry corpus training during model fine-tuning and can correspond with the optimization logic of the front-end sound pickup module for multiple industry scenarios. It can accurately identify and correct industry terminology transcription errors. The synergistic effect of multiple error correction methods can further make up for the shortcomings of front-end audio processing, significantly improve transcription accuracy, and form a full-link algorithm integration system of "front-end sound pickup optimization - back-end transcription + error correction".

[0116] Furthermore, "voice pause adaptation and correction" and "noise residue adaptation and correction" are practical and feasible error correction dimensions that meet the needs of real-world scenarios. Working in conjunction with other error correction dimensions, they further improve transcription accuracy without increasing transcription latency (end-to-end latency remains <500ms). They employ a logical linkage of "pre-screening - layered error correction - collaborative verification." "Noise residue adaptation and correction," as a pre-correction step, is executed before grammatical correction, first deleting meaningless content caused by noise and completing incomplete text, reducing unnecessary calculations for subsequent grammatical and semantic corrections. "Voice pause adaptation and correction" is executed after semantic correction and before collaborative verification, correcting sentence breaks and punctuation issues to ensure text fluency. The five error correction dimensions work together to further optimize the Chinese transcription error rate (CER) to <3.5%, improving the readability and accuracy of transcribed text without changing the original end-to-end algorithm fusion logic, thus adapting to the overall system design requirements.

[0117] Furthermore, after correcting the text of the audio data to be identified, the contextualized semantic association model is called to verify the corrected text and determine the semantic coherence, grammatical consistency and terminology uniformity of the text with the preceding and following segments.

[0118] If there are contradictions in the error correction results (such as contradictions between grammatical error correction and semantic error correction results), the error correction result that conforms to the semantic logic shall be selected first, taking into account the audio features of the audio data to be identified and the transcription confidence.

[0119] If the verification passes, the corrected transcribed text is output, and the error correction data (error type, correction method) generated during the error correction process is fed back to the speech recognition model for iterative upgrades, providing data support for subsequent model optimization and forming a closed loop of "error correction-feedback-optimization".

[0120] In one embodiment, the optimization and iterative upgrade of the speech recognition model involves iteratively optimizing the speech recognition model based on user-corrected transcription data. This iterative algorithm can form a closed-loop fusion with a multi-dimensional error correction system and a front-end sound pickup algorithm, ensuring full traceability and quantifiability, and guaranteeing that the optimization effect aligns with the actual use case of the system. The specific iterative optimization process is as follows:

[0121] Step 1: Data Collection and Classification: A dedicated iterative data collection module is built to receive real-time transcription data corrected by users on mobile devices (such as smartphones, tablets, etc.) via an app, while simultaneously collecting corresponding related data to form a complete iterative dataset. Data is classified according to error source to ensure targeted optimization: Front-end pickup deviation (e.g., transcription errors caused by incomplete front-end noise reduction, unsuitable beamforming angles, or inaccurate source separation, corresponding to data such as audio signal-to-noise ratio, beamwidth parameters, and source separation markers fed back by the front-end pickup module); Back-end transcription deviation (e.g., transcription errors caused by model acoustic feature extraction deviations or insufficient semantic association in the language model layer, corresponding to data such as confidence scores and feature extraction logs output by the back-end model); Error correction omissions (i.e., omissions not corrected by the back-end five-dimensional error correction system of "grammar-semantics-terminology-pauses-noise residue," corresponding to data such as verification logs and error identification thresholds from the error correction module); Simultaneously, the dataset is deduplicated and cleaned, invalid correction data (such as erroneous corrections) is removed, and valid correction samples are retained to ensure dataset purity ≥ 99%.

[0122] Step 2: Data Labeling and Weight Allocation: An "automatic labeling + manual review" approach is used to label the collected valid correction data. Automatic labeling is performed by a labeling algorithm, and the labeling content includes error type (typos, semantic deviations, terminology errors, etc.), error level (minor / moderate / serious), and corresponding front-end / back-end parameters. Manual review is conducted for moderate and serious error samples to ensure labeling accuracy ≥ 98%. After labeling, iterative training weights are allocated based on the frequency of errors and their impact on transcription accuracy: high-frequency serious errors (such as high-frequency terminology errors, errors leading to complete semantic deviation) are weighted at 1.5, medium-frequency moderate errors (such as occasional word order confusion, misjudged pauses) are weighted at 1.0, and low-frequency minor errors (such as individual punctuation omissions) are weighted at 0.5. Simultaneously, for front-end audio pickup deviation data, corresponding scene audio pickup parameters are additionally associated to improve the targeting of the front-end algorithm's reverse optimization.

[0123] Step 3: Layered Iterative Training: Based on the labeled dataset, layered iterative training is conducted to achieve collaborative optimization of the backend model, frontend audio pickup algorithm, and backend error correction algorithm without changing the original end-to-end architecture. Iterative training time is controlled to avoid impacting real-time system operation. Backend speech recognition model iteration: Corrected data and associated audio feature data are integrated into the original model training set (2,000 hours of general spoken language corpus + 1,000 hours of professional corpus). Gradient descent algorithm is used for fine-tuning training, with iteration rounds dynamically adjusted according to error level (150 rounds for severe errors, 80 rounds for moderate errors, and 50 rounds for minor errors). The focus is on optimizing the model's adaptability to frontend audio features (specific noise, sound source switching, speech pauses), while adjusting the semantic association parameters of the language model layer and the term recognition threshold of the term enhancement layer. Backend error correction algorithm iteration: [Further details on backend error correction algorithm iteration are needed for complete translation.] For omission-related data, fine-tuning the core parameters of the five-dimensional error correction system is performed, such as the grammatical error identification threshold for grammatical correction, the relevance judgment threshold for semantic error correction, the terminology matching accuracy for terminology verification, the pause duration judgment range for pause error correction, and the noise feature identification parameters for noise residue error correction, thereby improving the omission error identification capability of the error correction module. For front-end audio pickup algorithm iteration: For front-end audio pickup deviation data, reverse-engineering the parameters of the front-end audio pickup algorithm is used for fine-tuning. For example, based on transcription errors caused by noise, the steady-state / non-steady-state noise suppression threshold of the multimodal noise reduction algorithm is adjusted; based on audio pickup deviations caused by mismatched beamforming angles, the angle adaptive adjustment parameters of the dynamic beamforming algorithm are optimized; based on errors caused by inaccurate sound source separation, the ICA algorithm feature extraction parameters of the layered sound source separation algorithm are fine-tuned to ensure collaborative optimization between the front-end audio pickup algorithm and the back-end model and error correction algorithm.

[0124] Step 4: Iterative Validation and Parameter Consolidation: After each round of hierarchical iterative training, a dedicated validation set is built (100,000+ audio-to-transcription samples are extracted from real-world usage scenarios, covering different noise environments, multi-person speech scenarios (2-6 people), and multi-industry terminology scenarios) to validate the iterative end-to-end algorithm. Validation metrics are clearly quantified, with core validation metrics including: Chinese transcription error rate (CER), industry terminology transcription accuracy, front-end audio signal-to-noise ratio, terminal-to-mobile transmission latency, and error correction omission rate. When the validation results meet the preset optimization targets (e.g., CER reduced by ≥0.2% compared to before iteration, error correction omission rate reduced by ≥1%, and front-end audio signal-to-noise ratio improved by ≥0.5dB), the iteration is stopped, and the model parameters, front-end audio pickup algorithm parameters, and back-end error correction algorithm parameters are consolidated. If the preset targets are not met, the process returns to Step 3, adjusts the data weights and iteration rounds, and restarts iterative training until the targets are met.

[0125] Step 5: Gray-scale Update and Closed-Loop Feedback: The solidified optimization parameters are synchronized to the terminal, mobile APP, and cloud service using a gray-scale update method to avoid system instability caused by a full update. The gray-scale update covers 10%-30% of users, and the system operation log is monitored in real time to collect usage data and correction data from gray-scale users to verify the stability of the iterative optimization effect. If no abnormalities occur during the gray-scale update (such as increased transcription latency, sound pickup stuttering, increased error correction, etc.), the update scope is gradually expanded until a full update is performed. If an abnormality occurs, the parameters before the iteration are immediately rolled back, the problem is investigated, and optimization is performed again. At the same time, the user correction data after the gray-scale update and the full update are reconnected to the data collection module of Step 1 to form a complete closed loop of "data collection - iterative training - verification and solidification - gray-scale update - data re-collection". This achieves continuous and dynamic collaborative optimization of the entire algorithm, steadily improves the overall system performance, and ensures that transcription accuracy, sound pickup quality, and transmission stability continue to improve during long-term use.

[0126] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of this application.

[0127] Furthermore, embodiments of this application provide a speech-to-text system, including a server; the server is used to execute the method described above.

[0128] The embodiments of this application are applied to cloud servers, which have strong computing power, can avoid computing power limitations, and have wide adaptability.

[0129] In one embodiment, the system further includes a front-end pickup module and a transmission device;

[0130] The front-end sound pickup module is configured to acquire audio data in the target scene;

[0131] The transmission device is configured to receive the transmitted audio data and upload the audio data to the cloud server;

[0132] The cloud server transcribes the audio data to obtain the corresponding text.

[0133] In one specific implementation, the front-end sound pickup module can perform speech processing on the audio data, such as noise reduction, to improve the sound pickup effect and speech quality, laying the foundation for the speech transcription accuracy of this application.

[0134] In one specific implementation, the transmission device is further configured to stream audio data to a cloud server via the WebSocket protocol; and to forward audio data in segments every first preset time interval, the first preset time interval being consistent with the transcription segmentation rhythm of the cloud server. The front-end pickup module and the transmission device use Bluetooth protocol for short-range transmission to reduce transmission latency; the overall system uses a collaborative transmission of Bluetooth and WebSocket protocols, forming a two-layer transmission architecture of "short-range direct transmission + long-range streaming forwarding". The front-end pickup module and the transmission device use Bluetooth protocol for short-range direct transmission, responsible for quickly transmitting the processed audio data to the transmission device (e.g., a mobile app), ensuring low latency; after receiving the audio data, the transmission device streams it to the cloud server for transcription via the WebSocket protocol. The WebSocket protocol handles long-range transmission, forwarding audio data in segments every first preset time interval (e.g., 200ms) (synchronized with the streaming transcription segmentation rhythm of the back-end cloud server), achieving synchronization of the front-end pickup device's pickup processing, transmission, and back-end cloud server's transcription rhythm.

[0135] Furthermore, this application also provides an electronic device, including a memory, one or more processors, and one or more application programs, wherein the one or more application programs are stored in the memory, and the one or more application programs are configured to cause the one or more processors to perform the method described in any of the preceding technical solutions when invoked by the one or more processors.

[0136] The electronic device in this application mainly includes a memory and a processor. The memory can be configured to store a program for executing the methods of the above method embodiments, and the processor can be configured to execute the program in the memory. The program includes, but is not limited to, a program for executing the methods of the above method embodiments. For ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application.

[0137] In the embodiments of this application, the electronic device may be a control device comprising various electronic components. In some possible implementations, the electronic device may include multiple storage devices and multiple processors. The program executing the method of the above method embodiments may be divided into multiple subroutines, each subroutine may be loaded and run by a processor to execute different steps of the method of the above method embodiments. Specifically, each subroutine may be stored in different memories, and each processor may be configured to execute programs in one or more memories to jointly implement the method of the above method embodiments, that is, each processor executes different steps of the method of the above method embodiments to jointly implement the method of the above method embodiments.

[0138] The aforementioned multiple processors can be processors deployed on the same device. For example, the aforementioned computer device can be a high-performance device composed of multiple processors, and the aforementioned multiple processors can be processors configured on that high-performance device. Alternatively, the aforementioned multiple processors can also be processors deployed on different devices. For example, the aforementioned computer device can be a server cluster, and the aforementioned multiple processors can be processors on different servers within the server cluster.

[0139] The aforementioned electronic equipment is used for performing Figure 1 The method embodiments shown are similar in technical principle, technical problem solved and technical effect produced. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic device and related descriptions can be referred to the content described in the method embodiments, and will not be repeated here.

[0140] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0141] Furthermore, this application also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the above-described method embodiments, the program of which can be loaded and run by a processor to implement the above-described methods. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0142] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional modules of the device described in this application, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of both. Therefore, the number of modules shown in the figures is merely illustrative. Those skilled in the art will understand that the various modules in the system can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principles of this application; therefore, the technical solutions after splitting or merging will fall within the protection scope of this application.

[0143] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0144] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0145] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0146] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0147] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A speech-to-text method based on multi-feature fusion and contextualized semantic association, characterized in that, The method includes: The audio data to be recognized is acquired, and a trained speech recognition model is obtained; wherein, the speech recognition model is obtained by dynamic adaptive weighted training based on a training set constructed from general spoken language corpus and professional language corpus of various industries; the speech recognition model includes a feature extraction module and a language model layer; The feature extraction module is used to extract multidimensional core features, multidimensional low-frequency features, and short-time energy entropy features from the audio data to be identified and then fused to obtain multi-fused acoustic features. The language model layer, based on a contextualized semantic association model, performs segmented transcription of the multi-fusion acoustic features to obtain the text of the audio data to be identified. The contextualized semantic association model employs a multi-head attention mechanism and is pre-trained on labeled corpora with multiple semantic association rules for the target scene. During the transcription process of the multi-fusion acoustic features by the language model layer, the contextualized semantic association model performs the following steps: The system receives the multi-fused acoustic features and synchronously reads the corresponding timestamp information, then sorts the multi-fused acoustic features in chronological order. Based on the multi-head attention mechanism, the semantic features of the current transcribed segment are cross-correlated with the semantic features of adjacent transcribed segments to extract semantic association points between segments; based on the semantic association points, the semantic association between segments is determined to decide whether to splice them. Based on the pre-trained rules of the target scene, the spliced ​​speech is verified, and splicing results that do not conform to the semantic logic of the target scene are filtered out.

2. The speech-to-text method based on multi-feature fusion and contextual semantic association according to claim 1, characterized in that, The training set, constructed based on general spoken language corpora and industry-specific professional corpora, undergoes dynamic adaptive weighted training, including: The general spoken language corpus and industry-specific professional corpus are subjected to layered mixing preprocessing. The preprocessing includes: selecting daily dialogues and impromptu speech fragments of target scenarios from the general spoken language corpus, and retaining the real acoustic features; classifying the industry-specific corpus by industry and retaining industry-specific terms, professional expressions and scenario-based sentence structures; and randomly splicing the corpus in the mixing process and labeling them with scenario tags. In the early stages of training the speech recognition model, the weight of general spoken language corpus is set to be greater than the weight of industry-specific professional language corpus. During the training phase of the speech recognition model, the weights of the general spoken language corpus are set to be equal to the weights of the industry-specific professional language corpus. In the later stages of training the speech recognition model, a validation set is set up, and the weights of the industry-specific corpora are dynamically fine-tuned based on the accuracy of the validation set.

3. The speech-to-text method based on multi-feature fusion and contextual semantic association according to claim 1, characterized in that, The feature extraction module extracts multidimensional core features, multidimensional low-frequency features, and short-time energy entropy features from the audio data to be identified and fuses them to obtain multi-fused acoustic features, including: The audio data to be identified is standardized to remove DC components and extreme outliers from the audio data. The processed audio data is divided into frames according to preset frame length and preset frame shift, and Hanning window is used to suppress spectral leakage. For each frame of audio data after framing, multi-dimensional core features are extracted based on the MFCC algorithm; wherein, the core features include the core frequency band of human voice and the spectral envelope features; Multidimensional low-frequency features are extracted based on the LPCC algorithm; One-dimensional features are calculated based on the short-time energy entropy algorithm to quantify the energy distribution of each frame of audio data, distinguish effective speech segments from environmental noise segments, and obtain short-time energy entropy features. The extracted multidimensional core features, multidimensional low-frequency features, and short-time energy entropy features are fused through an attention mechanism to obtain multi-fused acoustic features. During fusion, the weight proportions of the multidimensional core features, multidimensional low-frequency features, and short-time energy entropy features decrease sequentially. The attention mechanism identifies the feature importance of the current frame audio data in real time and dynamically adjusts the weight allocation.

4. The speech-to-text method based on multi-feature fusion and contextual semantic association according to claim 3, characterized in that, The method further includes: A temporal attention mechanism is introduced into the feature extraction module to perform correlation calculation on the fusion features of audio data with consecutive preset frame lengths, thereby capturing the temporal correlation of features between frames. A noise robustness training branch is set in the feature extraction module, and the multi-fusion acoustic features are input into the speech recognition model for training. The extraction parameters of the feature extraction module are iteratively optimized based on the gradient descent algorithm so that the feature extraction module can adapt to the audio characteristics of the target scene.

5. The speech-to-text method based on multi-feature fusion and contextual semantic association according to claim 1, characterized in that, The language model layer also has a terminology enhancement layer; the terminology enhancement layer is configured to convert professional terms in the dictionary into multi-dimensional vectors based on the terminology embedding algorithm, embed them into the vocabulary of the language model layer, and learn the collocation rules of terms in different sentence patterns by adjusting and optimizing the terminology context adaptation logic. The method further includes: optimizing the loss function of the language model layer by introducing a joint loss function of cross-entropy loss and term matching loss; wherein the weight of the cross-entropy loss is greater than the weight of the term matching loss; the cross-entropy loss is used to optimize the transcription accuracy of the speech recognition model, and the term matching loss is used to enhance the term transcription accuracy of the speech recognition model. The language model layer is linked with the acoustic feature extraction module, and the text annotation data of the target scene is used for training and adjustment. The process is iterated until the term transcription accuracy and semantic coherence of the speech recognition model reach the preset values.

6. The speech-to-text method based on multi-feature fusion and contextual semantic association according to claim 1, characterized in that, The method further includes: performing semantic verification on the splicing result output by the contextualized semantic association model; if an error is found in the splicing result, it is fed back to the contextualized semantic association model, which recalculates the semantic association degree of adjacent transcribed segments and adjusts the splicing boundary, and combines the user-corrected transcribed data to reverse-optimize its own semantic association rules.

7. The speech-to-text method based on multi-feature fusion and contextual semantic association according to claim 1, characterized in that, The method further includes: correcting errors in the text of the obtained audio data to be recognized; the error correction process is performed according to at least one of the following: Based on the audio characteristics of the audio data to be identified and the confidence level of each transcribed text output by the speech recognition model, the text to be prioritized for error correction is selected. Based on the training results of the general spoken language corpus during the training of the speech recognition model and the Chinese grammar rule base, grammatical errors in the text are corrected; Based on the contextualized semantic association model and the semantic rules of the target scene, the semantic association between the current transcribed segment and the adjacent transcribed segments is compared, and the timestamp information in the audio data to be identified is used to correct semantic errors in the text. Based on a multi-industry terminology dictionary, suspected terms in the text are identified, the rationality of term collocations is verified, and terminology expressions are standardized; and terminology dictionaries corresponding to the industry of the target scenario are called first. Based on the timestamp information and audio energy features in the audio data to be identified, the actual pause duration of the speech is identified, and the punctuation of the sentence breaks in the text is corrected accordingly, and the semantic deviation caused by the misjudgment of pauses is corrected. A noise mis-transcription feature library is established based on the noise type and intensity fed back by the device that picks up the audio data to be identified. Meaningless noise mis-transcription content is automatically identified and deleted. For text defects caused by noise, the missing text is completed by combining the context semantics and terminology dictionary in the scenario-based semantic association model. The speech recognition model is iteratively optimized based on the user-corrected transcription data.

8. The speech-to-text method based on multi-feature fusion and contextual semantic association according to claim 7, characterized in that, The method further includes: after correcting the text of the obtained audio data to be identified, calling the contextualized semantic association model to verify the corrected text, and judging the semantic coherence, grammatical consistency and terminology uniformity of the text with the preceding and following segments. If there are contradictory error correction results, the error correction result that conforms to semantic logic shall be selected first, taking into account the audio features and transcription confidence of the audio data to be identified. If the verification is successful, the corrected transcribed text will be output, and the error correction data generated during the error correction process will be fed back to the speech recognition model for iterative upgrades.

9. A speech-to-text system, characterized in that, Includes a server; the server is used to perform the method as described in any one of claims 1-8.

Images