Multi-scenario speech recognition automatic annotation system for Zhuang language spoken corpus

Abstract

This invention relates to the field of speech signal processing technology, specifically to an automatic speech recognition and annotation system for Zhuang spoken language corpora in multiple scenarios. The system includes a noise separation module, a feature extraction module, a speech recognition module, a multi-dimensional annotation module, and a result generation module. The noise separation module performs multi-level background noise separation on the original Zhuang speech stream to obtain a clean speech waveform and a scene noise fingerprint. The feature extraction module dynamically adjusts parameters using the noise fingerprint to extract scene-adaptive high-dimensional acoustic features from the clean speech. The speech recognition module processes the features using a hybrid acoustic model and a Zhuang language model to generate an initial text sequence at the word level. The multi-dimensional annotation module combines prosodic and sentiment data to annotate the text sequence with prosodic and sentiment biases. The result generation module integrates the noise fingerprint, labeled text, and timestamps to generate multi-dimensional structured annotation results.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to an automatic speech recognition and annotation system for Zhuang spoken language corpora in multiple scenarios. Background Technology

[0002] Current research on speech processing for low-resource languages ​​such as Zhuang faces significant challenges. In the speech recognition front-end, existing technologies typically rely on speech data collected in relatively clean laboratory environments for model training. When applied to real-world, variable scenarios, such as noisy streets or indoor reverberant environments, background noise is severely coupled with the target speech. Traditional single-path noise reduction or speech enhancement methods experience a sharp decline in performance under such complex acoustic conditions. These methods often suppress or filter noise, lacking modeling and utilization of the noise environment itself, leading to speech distortion or residual noise interference after processing, which in turn affects the stability and discriminative power of back-end acoustic feature extraction. In the speech recognition and annotation back-end, mainstream technologies focus on converting speech into text, with annotation output limited to phonemes or word sequences. This pure text transcription loses the rich information-carrying prosodic structure, intonation variations, and speaker emotional states—paralinguistic features—of the speech signal.

[0003] Furthermore, Zhuang, as a complex tonal language with six open tones and three closed tones, is highly susceptible to distortion due to background noise. Common noise suppression methods (such as spectral subtraction) often indiscriminately weaken high-frequency energy, resulting in damage to the fundamental frequency and formant structure that carry tonal information, severely impacting subsequent recognition accuracy. Simultaneously, Zhuang lacks large-scale standard spoken language corpora, making it a typical low-resource language, which limits the training of deep acoustic models. For languages ​​like Zhuang, which are highly colloquial and have distinct prosodic features, the lack of these multi-dimensional annotations severely limits the value of speech corpora in deeper applications such as speech synthesis, affective computing, and linguistic research. Therefore, there is an urgent need for a Zhuang speech processing solution that can effectively overcome noise interference in complex scenarios and simultaneously annotate both text and hypertext information.

[0004] Existing technologies disclose a multi-scene speech recognition method that employs multi-model fusion to improve cross-scene adaptability. However, it fails to model and fingerprint the noisy scenes themselves, relying solely on general noise reduction algorithms to suppress noise. This results in the fundamental frequency and formants of tonal languages ​​like Zhuang being susceptible to over-filtering damage, and a significant decrease in recognition accuracy in low signal-to-noise ratio scenarios. The difference in this invention lies in its first-ever construction of a closed-loop chain of noise separation, fingerprint extraction, and adaptive feature adjustment. This transforms noise from interference into quantifiable scene features, specifically addressing the tonal distortion problem in Zhuang. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing an automatic speech recognition and annotation system for Zhuang spoken language corpora in multiple scenarios. The innovations of this invention are reflected in three aspects: first, constructing a closed-loop structure for noise separation and fingerprint utilization; second, dynamically adjusting feature extraction parameters based on noise fingerprints to adapt to the tonal characteristics of Zhuang language; and third, outputting multi-dimensional structured annotations that integrate prosody and emotion tags.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a multi-scenario speech recognition automatic annotation system for Zhuang language spoken corpora, comprising: The noise separation module acquires the original Zhuang language speech stream containing various real-world scenarios, performs multi-level background noise separation processing on the original Zhuang language speech stream, and extracts clean speech waveform data and corresponding scene noise fingerprint data. The feature extraction module inputs the clean speech waveform data into an improved acoustic feature extraction algorithm, dynamically adjusts parameters based on the scene noise fingerprint, and generates a scene-adaptive high-dimensional acoustic feature vector. The speech recognition module calls a pre-built multi-scenario hybrid acoustic model to perform preliminary decoding on the high-dimensional acoustic feature vector, generating a phoneme-level candidate recognition sequence, and performs word boundary localization and error correction on the phoneme-level candidate recognition sequence based on the Zhuang language model to generate a word-level initial text sequence. A multi-dimensional annotation module, combining prosodic and emotional features extracted from the original speech stream, annotates the initial text with prosodic and emotional tags; The result generation module integrates the scene noise fingerprint data, the intermediate annotation results with prosody and emotion tags, and the timestamp information of the speech stream to construct a structured annotation result with multi-dimensional attributes.

[0007] As a further aspect of the present invention, multi-level background noise separation processing is performed on the original Zhuang language speech stream to extract clean speech waveform data and corresponding scene noise fingerprint data, including: The original Zhuang language speech stream is subjected to a short-time Fourier transform to convert the time-domain speech signal into a time-spectrum graph. In the frequency domain, a noise power spectrum estimation method based on the auditory masking effect is used to estimate the noise floor power spectrum of each frame of the signal. The estimated noise floor power spectrum is subtracted frame by frame from the power spectrum of the original Zhuang language speech stream to obtain the preliminary enhanced power spectrum; Perform an inverse short-time Fourier transform on the initially enhanced power spectrum and convert it back to the time domain to obtain the initial denoised speech waveform; For the initial denoised speech waveform, it is decomposed into a speech component matrix and a noise component matrix using nonnegative matrix factorization. Subtract the time-domain signal corresponding to the noise component matrix from the initial denoised speech waveform to obtain the final clean speech waveform data; Statistical features are extracted from the noise component matrix, including the distribution of noise energy in the frequency band, the time-varying stability coefficient of noise, and the periodic noise pattern of a specific scene. The statistical features are combined to form the scene noise fingerprint data that characterizes the background noise of a specific scene. The nonnegative matrix decomposition process introduces sparsity constraints, with the sparsity of the speech component matrix being higher than that of the noise component matrix, to ensure that the decomposition results conform to the physical interpretability of speech and noise, and to avoid misassigning speech components to the noise component matrix.

[0008] As a further aspect of the present invention, the improved acoustic feature extraction algorithm dynamically adjusts the filtering parameters and energy compensation coefficients of the feature extraction process based on the scene noise fingerprint data, including: Analyze the scene noise fingerprint data to obtain the main noise frequency band distribution, average noise energy, and noise fluctuation period in the current scene; Based on the main noise frequency band distribution, an adaptive band-stop filter bank is dynamically constructed. The center stopband frequency of the adaptive band-stop filter bank corresponds to the frequency band where the noise energy is concentrated, and the stopband width is adjusted proportionally according to the noise energy magnitude. During the extraction of cepstral features, the adaptive band-stop filter bank is used to pre-filter the clean speech waveform data to suppress the energy of the noise frequency band. Based on the average noise energy, an energy compensation coefficient is calculated, which is used to compensate for the energy of speech bands that are excessively attenuated due to noise suppression when calculating the log-Mel spectrum energy. Based on the noise fluctuation period, the size of the time window used to calculate the first and second differences of the Mel frequency cepstral coefficients is dynamically adjusted. When the noise fluctuation period is greater than or equal to the preset stationary noise determination threshold, a first type of fixed-length time window that is positively correlated with the noise fluctuation period is used to obtain more stable dynamic characteristics. When the noise fluctuation period is less than the preset stationary noise determination threshold, a second type of fixed-length time window that is positively correlated with the noise fluctuation period is used to capture temporal changes with higher temporal resolution. The Mel frequency cepstral coefficients, first-order difference, and second-order difference features generated after adaptive filtering, energy compensation, and dynamic window adjustment are combined to form the high-dimensional acoustic feature vector with scene-adaptive characteristics.

[0009] As a further aspect of the present invention, the step of calling a pre-constructed multi-scene hybrid acoustic model to perform preliminary decoding on the high-dimensional acoustic feature vector to generate a phoneme-level candidate recognition sequence includes: The multi-scene hybrid acoustic model is formed by fusing multiple sub-acoustic models through a weight adaptive mechanism. Each sub-acoustic model is specifically trained for Zhuang language speech in a specific real-world scenario. The high-dimensional acoustic feature vector is simultaneously input into all sub-acoustic models of the multi-scene hybrid acoustic model, and each sub-acoustic model independently outputs a phoneme posterior probability sequence. Based on the current dominant scene type identified from the scene noise fingerprint data, a set of dynamic fusion weights is calculated, and the dynamic fusion weights give higher weights to the sub-acoustic models that have a high degree of matching with the dominant scene type. The dynamic fusion weights are used to weight and fuse the phoneme posterior probability sequences output by each sub-acoustic model to obtain a comprehensive phoneme posterior probability sequence. A Viterbi decoding search is performed on the synthesized phoneme posterior probability sequence to find the optimal phoneme state path; Based on the optimal phoneme state path and combined with the phoneme context binding relationship, a candidate recognition sequence at the phoneme level is generated. The candidate recognition sequence includes the symbol identifier, start time, end time and confidence score of each phoneme.

[0010] As a further aspect of the present invention, the step of performing word boundary localization and error correction processing on the phoneme-level candidate recognition sequence based on the Zhuang language model to generate an initial text sequence at the word level includes: Load an N-grammar language model trained on a large-scale Zhuang text corpus and a Zhuang pronunciation dictionary, wherein the pronunciation dictionary establishes a mapping relationship between Zhuang vocabulary and its standard phoneme sequences; Using the phoneme-level candidate recognition sequence as input, and guided by the pronunciation dictionary, a word grid is constructed through a graph search algorithm. The nodes of the word grid correspond to word candidates that conform to the matching rules of the pronunciation dictionary, and the edges correspond to the legitimate connection relationships between adjacent word candidates in the Zhuang pronunciation sequence. Using the word sequence probabilities provided by the N-gram language model, a new Viterbi search is performed on the word grid to calculate the optimal word sequence path from the starting point to the ending point; The optimal word sequence path determines the final word segmentation boundary and corrects insertion, deletion, or replacement errors that occur during the phoneme recognition stage. Output the word sequence corresponding to the optimal word sequence path as the initial text sequence at the word level. The initial text sequence includes each identified Zhuang word, the time boundary of the word in the speech stream, and the word confidence after language model correction.

[0011] As a further aspect of the present invention, prosodic feature data and sentiment intensity data synchronously extracted from the original Zhuang language speech stream are combined to perform prosodic annotation and sentiment tendency annotation on the initial text sequence at the word level, including: The fundamental frequency trajectory, energy envelope, and speech spectrum tilt are extracted from the original Zhuang language speech stream as prosodic feature data. Specific emotion-related acoustic features, including formant variation rate, pronunciation speed and voice jitter, are extracted from the original Zhuang language speech stream and input into a pre-trained emotion classifier to obtain the emotion intensity data, which includes discrete emotion categories and continuous emotion intensity values. Based on the fundamental frequency trajectory and energy envelope, prosodic labels are marked on each word of the initial text sequence at the word level. The prosodic labels include tone type, stress level and pause position. The pre-trained emotion classifier adopts a transfer learning strategy. It is first pre-trained using a public emotion speech database, and then fine-tuned using a self-built Zhuang language emotion speech database. Based on the emotional intensity data and speech spectrum tilt, emotional tendency labels are marked at the lexical level and the sentence level respectively. The lexical level labels reflect the emotional color carried by the words, and the sentence level labels reflect the emotional tone of the whole sentence. The labeled prosodic and sentiment labels are aligned and bound to the words and time boundaries in the initial text sequence at the word level to form the intermediate annotation result with prosodic and sentiment labels.

[0012] As a further aspect of the present invention, the scene noise fingerprint data, the intermediate annotation results with prosodic and emotion tags, and the timestamp information of the speech stream are integrated to construct a structured annotation result with multi-dimensional attributes, including: Establish a structured data container, which includes an audio information layer, a transcribed text layer, a prosody annotation layer, a sentiment annotation layer, and a scene description layer; In the audio information layer, the storage path, sampling rate, duration, and overall signal-to-noise ratio of the clean speech waveform data are recorded; In the transcribed text layer, the initial text sequence at the word level is stored, and each word is associated with its start and end timestamps in the audio. In the prosodic annotation layer, the prosodic tag corresponding to each word is stored, and a link relationship is established with the corresponding word in the transcribed text layer; In the sentiment annotation layer, the sentiment tendency labels at the word level and sentence level are stored, and a link relationship is established with the corresponding words or sentences in the transcribed text layer. In the scene description layer, the scene noise fingerprint data is stored and mapped to a readable scene type description and noise characteristic description; All layers of data are aligned and encapsulated according to a unified timeline to generate a standardized structured data format file, which serves as the final structured annotation result.

[0013] As a further aspect of the present invention, the improved acoustic feature extraction algorithm also includes an online adaptive filtering update mechanism, specifically comprising: While extracting the high-dimensional acoustic feature vector, the residual noise level of the clean speech waveform data after processing by the adaptive band-stop filter bank is continuously monitored. When the residual noise level exceeds a preset threshold, the filter parameter update process is triggered; The filter parameter update process uses the clean speech waveform data and the corresponding scene noise fingerprint data from the latest time period to re-estimate the noise power spectrum and noise frequency band distribution. Based on the re-estimation results, the center stopband frequency and stopband width of the adaptive bandstop filter bank are updated.

[0014] As a further aspect of the present invention, based on the currently dominant scene type identified from the scene noise fingerprint data, a set of dynamic fusion weights is calculated, including: Pattern matching is performed on the scene noise fingerprint data. The scene noise fingerprint data is compared with multiple predefined typical scene noise templates. The scene corresponding to the typical scene noise template with the highest similarity is identified as the current dominant scene type. Based on the identified dominant scene type, a mapping table between predefined scene types and sub-acoustic models is obtained. The mapping table defines one or more scene types that each sub-acoustic model is specifically targeting. For each sub-acoustic model, its matching degree with the current dominant scene type is calculated. The matching degree is calculated based on the mapping relationship table. If the sub-acoustic model is specifically designed for the current dominant scene type, the matching degree is the highest value. If the sub-acoustic model is designed for a scene with similar acoustic characteristics to the current dominant scene type, the matching degree is the medium value. If the sub-acoustic model is designed for a scene that is very different from the current dominant scene type, the matching degree is the lowest value. The matching degrees of all sub-acoustic models are normalized so that the sum of all matching degrees is 1. The resulting set of normalized matching degrees is the dynamic fusion weight.

[0015] As a further aspect of the present invention, the system further includes: a data management module, used for: The structured annotation results are stored in a speech corpus annotation database, and an annotation quality report is generated, specifically including: The structured annotation results of the Extensible Markup Language (Extreme Markup Language) format are parsed, and the audio information layer field, transcribed text layer field, prosody annotation layer field, sentiment annotation layer field, and scene description layer field are mapped to the corresponding data tables in the database for storage. During storage, a globally unique identifier is generated for each annotation record, and an index relationship with the original audio file is established. The structured annotation results stored in the database are subjected to automatic quality measurement, which includes: calculating the word error rate estimate of the speech recognition results, checking the coverage and consistency of prosody and sentiment labels, and verifying the accuracy of timestamp alignment; Based on the results of the automatic quality metric, the annotation quality report is generated, which includes a detailed quality score for each annotation record, a list of potential problems found, and revision suggestions for low-quality annotation records.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: Through multi-level background noise separation processing, not only is a clean speech waveform extracted for recognition, but noise fingerprint data representing the characteristics of the noisy scene is also simultaneously separated. This process transforms noise from an interference factor that needs to be eliminated into a scene feature that can be characterized and utilized. Based on this noise fingerprint data, subsequent acoustic feature extraction algorithms can dynamically adjust their filtering parameters and energy compensation coefficients, making the generated high-dimensional acoustic feature vectors intrinsically adaptable to the specific acoustic environment of the original speech. This scene-adaptive feature extraction mechanism significantly improves the robustness and consistency of acoustic features under different noise conditions, providing higher-quality and more stable input for the backend recognition model, and directly enhancing the accuracy and reliability of speech recognition in complex real-world scenarios. At the same time, the noise fingerprint itself is preserved as an important type of metadata, enriching the dimensions of the final annotation results.

[0017] Building upon the initial word-level text sequence recognition, the system concurrently utilizes prosodic feature data and sentiment intensity data parsed synchronously from the original speech stream. This temporal paralinguistic information is used for deep annotation of the text sequence, including prosodic labels such as prosodic structure boundaries and stress positions, as well as sentiment labels at the sentence or word level. This process achieves the fusion annotation of textual and non-textual information in the speech signal, resulting in a final output that is no longer a flat text string, but a structured annotation result carrying speaking rhythm, intonation, and emotional coloring. This multi-dimensional annotation greatly expands the information capacity and application boundaries of the corpus. The resulting Zhuang spoken language corpus, rich in prosodic and sentiment labels, can be directly used to train expressive speech synthesis systems, support more context-aware dialogue systems, or provide a refined data foundation for prosodic research in linguistics. Attached Figure Description

[0018] Figure 1 This is a timing diagram of the multi-scenario speech recognition automatic annotation system for Zhuang spoken language corpora described in this invention; Figure 2 A flowchart illustrating the workflow of the improved acoustic feature extraction algorithm; Figure 3 This is a flowchart of word boundary localization and error correction processing based on the Zhuang language model. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0021] See Figure 1This invention provides an automatic speech recognition and annotation system for Zhuang spoken language corpora across multiple scenarios. The system includes a noise separation module, a feature extraction module, a speech recognition module, a multi-dimensional annotation module, and a result generation module. The overall implementation scheme is as follows: The noise separation module acquires raw Zhuang speech streams containing various real-world scenarios and performs multi-level background noise separation processing on these streams to extract clean speech waveform data and corresponding scene noise fingerprint data. The feature extraction module inputs the clean speech waveform data into an improved acoustic feature extraction algorithm. This algorithm dynamically adjusts the filtering parameters and energy compensation coefficients of the feature extraction process based on the scene noise fingerprint data, thereby generating a high-dimensional acoustic feature vector with scene-adaptive characteristics. The speech recognition module calls a pre-constructed multi-scenario hybrid acoustic model to perform preliminary decoding of the high-dimensional acoustic feature vector, generating phoneme-level candidate recognition sequences. Based on the Zhuang language model, the module performs word boundary localization and error correction processing on the phoneme-level candidate recognition sequences to generate an initial text sequence at the word level. The multi-dimensional annotation module combines prosodic feature data and sentiment intensity data extracted synchronously from the original Zhuang language speech stream to perform prosodic and sentiment annotation on the initial text sequence at the word level, generating intermediate annotation results with prosodic and sentiment labels. The result generation module integrates scene noise fingerprint data, intermediate annotation results with prosodic and sentiment labels, and timestamp information of the speech stream to construct a structured annotation result with multi-dimensional attributes.

[0022] In one embodiment of the present invention, a short-time Fourier transform is performed on the original Zhuang language speech stream to convert the time-domain speech signal into a time-spectrum. In the frequency domain, a noise power spectrum estimation method based on auditory masking effect is used to estimate the noise floor power spectrum of each frame of the signal. The estimated noise floor power spectrum is subtracted frame by frame from the power spectrum of the original Zhuang language speech stream to obtain a preliminary enhanced power spectrum. An inverse short-time Fourier transform is performed on the preliminary enhanced power spectrum to convert it back to the time domain, resulting in the initial denoised speech waveform. The initial denoised speech waveform is decomposed into a speech component matrix and a noise component matrix using non-negative matrix factorization. The time-domain signal corresponding to the noise component matrix is ​​subtracted from the initial denoised speech waveform to obtain the final clean speech waveform data. Statistical features are extracted from the noise component matrix, including the distribution of noise energy in the frequency band, the time-varying stability coefficient of noise, and the periodic noise pattern of a specific scene. These statistical features are combined to form scene noise fingerprint data characterizing the background noise of a specific scene.

[0023] In practice, multi-level background noise separation processing is performed on the original Zhuang language speech stream containing various real-world scenarios. The first step is to convert the original Zhuang language speech stream in the time domain into a time-spectrum graph using a short-time Fourier transform. In the frequency domain, a noise power spectrum estimation method based on auditory masking effect is used to estimate the noise floor power spectrum of each frame of the signal. In practice, this estimation process is achieved by tracking the frequency band with the lowest energy in the speech signal and smoothing and updating it using an auditory masking threshold. The estimated noise floor power spectrum is subtracted frame by frame from the power spectrum of the original Zhuang language speech stream to obtain a preliminary enhanced power spectrum. In some embodiments, the subtraction process incorporates an over-subtraction factor and a lower spectral limit to prevent the generation of musical noise. An inverse short-time Fourier transform is performed on the preliminary enhanced power spectrum to convert it back to the time domain signal, thereby obtaining the initial denoised speech waveform.

[0024] For the initial denoised speech waveform, a deeper noise separation is performed using nonnegative matrix factorization (NMF). In practice, the amplitude spectrum matrix of the initial denoised speech waveform is used as input. NMF decomposes the input amplitude spectrum matrix into the product of the speech component matrix and the noise component matrix, i.e.: Where: matrix The matrix represents the amplitude spectrum of the initial denoised speech waveform. sum matrix These represent the basis matrix and coefficient matrix of the speech components, respectively. sum matrix These represent the basis matrix and coefficient matrix of the noise component, respectively. The noise component matrix is ​​subtracted from the initial denoised speech waveform. The corresponding time-domain signal is used to obtain the final clean speech waveform data. The speech basis matrix is ​​initialized using the statistical values ​​of the pure Zhuang language speech spectrum, and the noise basis matrix is ​​initialized using the mean of the silence segment spectrum. Sparsity constraints are introduced to ensure that the speech and noise components have physical interpretability. In some embodiments, the phase information of the original speech signal is preserved when reconstructing the clean speech waveform data.

[0025] Statistical features characterizing background noise are extracted from the noise component matrix obtained by nonnegative matrix factorization. Optionally, these statistical features include the noise energy distribution across frequency bands, the time-varying stability coefficient of the noise, and periodic noise patterns specific to a particular scene. The noise energy distribution across frequency bands is obtained by calculating the average energy of the noise component matrix across different Mel frequency bands. The time-varying stability coefficient of the noise is calculated by analyzing the variance of the noise component matrix over time. Periodic noise patterns are identified by detecting peaks and harmonic structures in the spectrum of the noise component matrix. These calculated statistical features are combined to form a multidimensional vector, which is defined as the scene noise fingerprint data characterizing the background noise of a specific scene. Optionally, the scene noise fingerprint data is normalized to facilitate subsequent matching and comparison. This multi-level processing, combining traditional spectral subtraction and model-based nonnegative matrix factorization, can more effectively separate noise and characterize its properties from complex real-world Zhuang language speech.

[0026] In one embodiment of the present invention, see [reference] Figure 2The method analyzes scene noise fingerprint data to obtain the main noise frequency band distribution, average noise energy, and noise fluctuation period in the current scene. Based on the main noise frequency band distribution, an adaptive band-stop filter bank is dynamically constructed. The center stopband frequency of this adaptive band-stop filter bank corresponds to the frequency band where the noise energy is concentrated, and the stopband width is adjusted proportionally according to the noise energy magnitude. During the extraction of cepstral features, the adaptive band-stop filter bank is used to pre-filter the clean speech waveform data to suppress the energy of the noise frequency band. Based on the average noise energy, an energy compensation coefficient is calculated. This energy compensation coefficient is used to compensate for the energy of speech bands that are excessively attenuated due to noise suppression when calculating the log-Mel spectrum energy. Based on the noise fluctuation period, the time window sizes for calculating the first and second differences of the Mel frequency cepstral coefficients are dynamically adjusted. When the noise fluctuation period is greater than or equal to a preset stationary noise judgment threshold, a first type of fixed-length time window positively correlated with the noise fluctuation period is used to obtain more stable dynamic features; when the noise fluctuation period is less than the preset stationary noise judgment threshold, a second type of fixed-length time window positively correlated with the noise fluctuation period is used to capture temporal changes with higher temporal resolution. The Mel-frequency cepstral coefficients, first-order difference, and second-order difference features generated after adaptive filtering, energy compensation, and dynamic window adjustment are combined to form a high-dimensional acoustic feature vector with scene-adaptive characteristics. While extracting the high-dimensional acoustic feature vector, the residual noise level of the clean speech waveform data after processing by the adaptive band-stop filter bank is continuously monitored. When the residual noise level exceeds a preset threshold, the filter parameter update process is triggered. The filter parameter update process uses the clean speech waveform data from the most recent period and the corresponding scene noise fingerprint data to re-estimate the noise power spectrum and noise frequency band distribution. Based on the re-estimated results, the center stopband frequency and stopband width of the adaptive band-stop filter bank are updated.

[0027] In practical implementation, the improved acoustic feature extraction algorithm dynamically adjusts feature extraction parameters based on scene noise fingerprint data, and analyzes the scene noise fingerprint data to obtain the main noise frequency band distribution, average noise energy, and noise fluctuation period in the current scene. An adaptive band-stop filter bank is dynamically constructed based on the main noise frequency band distribution. The center stopband frequency of the adaptive band-stop filter bank precisely corresponds to the frequency band where noise energy is concentrated, and the stopband width is adjusted proportionally according to the noise energy magnitude. During the extraction of cepstral features, the adaptive band-stop filter bank is used to pre-filter the clean speech waveform data, thereby suppressing the energy of noise frequency bands. An energy compensation coefficient is calculated based on the average noise energy. This energy compensation coefficient is used to compensate for the excessive attenuation of speech frequency bands due to noise suppression when calculating the log-Mel spectrum energy. In some embodiments, the energy compensation coefficient... The calculation method is as follows: in: This represents the average noise energy obtained from the scene noise fingerprint data. The average short-time energy represents the pure speech waveform data. This is a preset scaling factor. The time window sizes for calculating the first and second differences of the Mel frequency cepstral coefficients are dynamically adjusted based on the noise fluctuation period. When the noise fluctuation period is greater than or equal to a preset stationary noise threshold, a first-type fixed-length time window positively correlated with the noise fluctuation period is used to obtain more stable dynamic features. When the noise fluctuation period is less than the preset stationary noise threshold, a second-type fixed-length time window positively correlated with the noise fluctuation period is used to capture temporal changes with higher temporal resolution. The Mel frequency cepstral coefficients, first-order differences, and second-order difference features generated after adaptive filtering, energy compensation, and dynamic window adjustment are combined to form a high-dimensional acoustic feature vector with scene-adaptive characteristics. In practice, the threshold for judging stable noise is set to 200ms, k1 is 0.5, and k2 is 0.2. For example, when the noise fluctuation period is 400ms (≥200ms), the length of the first type of window is 200ms; when the noise fluctuation period is 100ms (<200ms), the length of the second type of window is 20ms. This setting has been verified by Zhuang language oral test to improve feature stability by 15% when the noise is stable and improve timing capture accuracy by 12% when the noise changes abruptly.

[0028] The online adaptive filtering update mechanism continuously monitors the residual noise level of the clean speech waveform data after processing by the adaptive band-stop filter bank while extracting high-dimensional acoustic feature vectors. When the residual noise level exceeds a preset threshold, a filter parameter update process is triggered. In some embodiments, the residual noise level is determined by calculating the ratio of the power of the filtered signal to the initial noise estimate during a silent speech segment. The filter parameter update process re-estimates the noise power spectrum and noise frequency band distribution using the clean speech waveform data from the most recent period and the corresponding scene noise fingerprint data. It can be understood that the re-estimation process combines historical noise fingerprint data with current frame information for recursive updates. The center stopband frequency and stopband width of the adaptive band-stop filter bank are updated based on the re-estimation results. Optionally, a smoothing strategy is employed in the update process to avoid abrupt changes in filter parameters interfering with feature extraction. The online adaptive filtering update mechanism enables the feature extraction process to adapt to slow changes in scene noise. This mechanism is beneficial for handling real Zhuang language spoken scenarios with non-stationary background noise.

[0029] In one embodiment of the present invention, the multi-scene hybrid acoustic model is formed by fusing multiple sub-acoustic models through a weight adaptive mechanism. Each sub-acoustic model is specifically trained for Zhuang language speech in a specific real-world scenario. High-dimensional acoustic feature vectors are simultaneously input into all sub-acoustic models of the multi-scene hybrid acoustic model, and each sub-acoustic model independently outputs a phoneme posterior probability sequence. Based on the currently dominant scene type identified from scene noise fingerprint data, a set of dynamic fusion weights is calculated. Pattern matching is performed on the scene noise fingerprint data, comparing its similarity with multiple predefined typical scene noise templates. The scene corresponding to the typical scene noise template with the highest similarity is identified as the currently dominant scene type. Based on the identified currently dominant scene type, a predefined mapping table between scene types and sub-acoustic models is obtained. This mapping table defines one or more scene types specifically targeted by each sub-acoustic model. For each sub-acoustic model, its matching degree with the current dominant scene type is calculated. The matching degree calculation is based on a mapping table. If the sub-acoustic model is specifically designed for the current dominant scene type, the matching degree is the highest. If the sub-acoustic model is designed for a scene with similar acoustic characteristics to the current dominant scene type, the matching degree is medium. If the sub-acoustic model is designed for a scene that is significantly different from the current dominant scene type, the matching degree is the lowest. The matching degrees of all sub-acoustic models are normalized so that the sum of all matching degrees is 1. The resulting set of normalized matching degrees is the dynamic fusion weight. The phoneme posterior probability sequences output by each sub-acoustic model are weighted and fused using the dynamic fusion weight to obtain a comprehensive phoneme posterior probability sequence. Viterbi decoding search is performed on the comprehensive phoneme posterior probability sequence to find the optimal phoneme state path. Based on the optimal phoneme state path and combined with the phoneme context binding relationship, a phoneme-level candidate recognition sequence is generated. This candidate recognition sequence includes the symbol identifier, start time, end time, and confidence score of each phoneme.

[0030] In practical implementation, the multi-scene hybrid acoustic model is formed by fusing multiple sub-acoustic models through a weight adaptive mechanism. Each sub-acoustic model is specifically trained for Zhuang language speech in a particular real-world scenario. High-dimensional acoustic feature vectors are simultaneously input into all sub-acoustic models of the multi-scene hybrid acoustic model, and each sub-acoustic model independently outputs a phoneme posterior probability sequence. Based on the currently dominant scene type identified from the scene noise fingerprint data, a set of dynamic fusion weights is calculated. Pattern matching is performed on the scene noise fingerprint data, comparing its similarity with multiple predefined typical scene noise templates. The scene corresponding to the typical scene noise template with the highest similarity is identified as the currently dominant scene type. In some embodiments, the similarity comparison uses a cosine similarity metric, calculating the best-matching template by comparing the scene noise fingerprint data with the feature vectors of each typical scene noise template.

[0031] In practice, based on the identified dominant scene type, a predefined mapping table between scene types and sub-acoustic models is obtained. This mapping table defines one or more scene types that each sub-acoustic model is specifically designed for. See Table 1 for the mapping relationships.

[0032] Table 1: Mapping Relationship between Scene Type and Subacoustic Model For each sub-acoustic model, the matching degree between the sub-acoustic model and the current dominant scene type is calculated. The matching degree is calculated based on a mapping table. If the sub-acoustic model is specifically designed for the current dominant scene type, the matching degree is the highest value. If the sub-acoustic model is designed for a scene with similar acoustic characteristics to the current dominant scene type, the matching degree is a medium value. If the sub-acoustic model is designed for a scene that differs greatly from the current dominant scene type, the matching degree is the lowest value. In some embodiments, the highest value is set to 1.0, the medium value is set to 0.5, and the lowest value is set to 0.2. For subacoustic models The calculation is based on the weights assigned by the mapping table. The matching degrees of all calculated sub-acoustic models are normalized so that the sum of all matching degrees is 1. The resulting set of normalized matching degrees is the dynamic fusion weight. The normalization formula is: in: Subacoustic model Dynamic fusion weights, Subacoustic model The degree of matching, It is the matching degree of the subacoustic model j. This represents the total number of sub-acoustic models. It can be understood that the dynamic fusion weights reflect the degree to which each sub-acoustic model adapts to the current scene.

[0033] The phoneme posterior probability sequences output by each sub-acoustic model are weighted and fused using dynamic fusion weights to obtain a comprehensive phoneme posterior probability sequence. The weighted fusion process involves multiplying the posterior probability vector output by each sub-acoustic model by its corresponding dynamic fusion weight and then summing the results. Optionally, the weighted fusion is performed in the logarithmic probability domain to improve numerical stability. A Viterbi decoding search is then performed on the comprehensive phoneme posterior probability sequence to find the optimal phoneme state path. The Viterbi decoding search is based on the state transition probabilities and observation probabilities of a Hidden Markov Model. Based on the optimal phoneme state path and the phoneme context binding relationship, a phoneme-level candidate recognition sequence is generated. This phoneme-level candidate recognition sequence includes the symbol identifier, start time, end time, and confidence score for each phoneme. Optionally, the phoneme context binding relationship considers the coarticulation phenomenon in Zhuang speech. It can be understood that by adaptively combining the outputs of multiple sub-acoustic models using dynamic fusion weights, the multi-scenario hybrid acoustic model can better adapt to Zhuang speech recognition tasks under different noise environments.

[0034] In one embodiment of the present invention, see [reference] Figure 3 This study loads an N-gram language model trained on a large-scale Zhuang text corpus and a Zhuang pronunciation dictionary. Specifically, the large-scale Zhuang text corpus was constructed by collecting and cleaning 300,000 sentences of Zhuang online social media text, 200,000 sentences of Zhuang folk tales and novels, and 100,000 sentences of official documents from the Guangxi Zhuang Autonomous Region government website in Zhuang language. After word segmentation and removal of punctuation and special characters, the resulting training corpus consisted of approximately 600,000 sentences. The N-gram model used a trigram (N=3) structure and was trained using the KenLM toolkit. To address the data sparsity issue inherent in Zhuang as a low-resource language, the Kneser-Ney smoothing algorithm was employed to optimize the model, estimating the probability of unseen trigram phrases and improving the language model's generalization ability in open scenarios. The Zhuang pronunciation dictionary was constructed using a combination of rule-based automatic generation and manual verification. First, a phoneme set for standard Zhuang syllables was defined, comprising phoneme markers for 23 initials, 94 finals, and 9 tones (including open and closed tones). Then, based on Zhuang orthography rules, a regular expression parsing script was developed to automatically break down each written Zhuang word into its corresponding phoneme sequence. Finally, considering the polyphonic characters and dialectal pronunciations in Zhuang, three Zhuang linguistics experts manually reviewed and annotated the automatically generated dictionary, resulting in the final Zhuang pronunciation dictionary.

[0035] In practical implementation, word boundary localization and error correction are performed on phoneme-level candidate recognition sequences based on the Zhuang language model. An N-gram language model trained on a large-scale Zhuang text corpus and a Zhuang pronunciation dictionary are loaded, establishing a mapping relationship between Zhuang vocabulary and its standard phoneme sequences. Using phoneme-level candidate recognition sequences as input, a word grid is constructed using a graph search algorithm under the guidance of the Zhuang pronunciation dictionary. Nodes in the word grid correspond to word candidates that conform to the matching rules of the Zhuang pronunciation dictionary, and edges correspond to the legitimate connection relationships between adjacent word candidates in the Zhuang pronunciation sequence. Utilizing the word sequence probabilities provided by the N-gram language model, a new Viterbi search is performed on the word grid to calculate the optimal word sequence path from the starting point to the ending point. The Viterbi search uses dynamic programming to evaluate the cumulative score of all possible paths to find the optimal path. Acoustic score and language model score Weighted combination, i.e.: in: This represents the acoustic likelihood score generated by the acoustic model. This represents the probability score of a word sequence generated by an N-gram language model. It is the interpolation factor that balances the two weights. The optimal word sequence path determines the final word segmentation boundary and corrects insertion, deletion, or replacement errors that occur in the phoneme recognition stage. The output word sequence corresponding to the optimal word sequence path is used as the initial text sequence at the word level. The initial text sequence at the word level includes each recognized Zhuang word, the word's temporal boundary in the speech stream, and the word confidence score corrected by the language model.

[0036] In practice, prosodic feature data and sentiment intensity data extracted synchronously from the original Zhuang language speech stream are used to perform prosodic and sentiment annotation on the initial text sequence at the word level. Fundamental frequency trajectory, energy envelope, and speech spectrum tilt are extracted from the original Zhuang language speech stream as prosodic feature data. Specific emotion-related acoustic features, including formant variation rate, vocal velocity, and vocal jitter, are extracted from the original Zhuang language speech stream and input into a pre-trained sentiment classifier to obtain sentiment intensity data. This sentiment intensity data includes discrete sentiment categories and continuous sentiment intensity values. The continuous emotional intensity value is quantified using two dimensions: valence and activation, forming a two-dimensional emotional vector. The specific mapping method is as follows: Based on the correspondence between emotional acoustic features and emotional tendency labels shown in Table 2, when a significant increase in the formant variation rate, a faster vocal speed, and lower vocal tremor are detected, the valence value is determined to be greater than 0.6 and the activation value greater than 0.7, and the system maps this two-dimensional emotional vector to the discrete emotional category of happiness. When the formant variation rate fluctuates drastically, the vocal speed is faster, and the full-band energy is high, both the valence value and activation value are in the high range, corresponding to the discrete emotional category of anger. When the formant variation rate decreases slowly, the vocal speed is slower, and low-frequency energy is prominent, the valence value is less than 0.4 and the activation value is less than 0.3, corresponding to the discrete emotional category of sadness. When all acoustic feature indicators are in a stable intermediate range, the emotion is determined to be calm. The system outputs the above two-dimensional emotional vector and its mapped discrete category labels as the emotional intensity data.

[0037] In this embodiment, the sentiment classifier adopts a three-layer long short-term memory network structure. The input layer dimension is the same as the dimension of the extracted sentiment acoustic feature vector, the number of hidden layer nodes is 128, the number of output layer nodes is the same as the number of sentiment categories, the input is 39-dimensional acoustic features, the Zhuang language sentiment corpus used for training is 50 hours in size, and the annotation consistency reaches over 95%.

[0038] The classifier is trained using a transfer learning strategy. It is pre-trained using a publicly available Chinese emotional speech database; then fine-tuned using a self-built Zhuang emotional speech database. This database contains approximately 50 hours of Zhuang speech, recorded and annotated by five native Zhuang linguists in a professional recording studio. The corpus covers various scenarios including family, market, and street scenes, with annotation consistency exceeding 95%. Emotional acoustic feature extraction is implemented using an open-source speech analysis library, with a specific computation window length of 20ms and a frame shift of 10ms.

[0039] Based on the fundamental frequency trajectory and energy envelope, prosodic labels are annotated on each word in the initial text sequence at the word level. These prosodic labels include tone type, stress level, and pause position. It is understood that Zhuang is a tonal language, and accurate tone type annotation depends on the recognition of fundamental frequency trajectory patterns. Based on sentiment intensity data and speech spectrum tilt, sentiment tendency labels are annotated at both the word and sentence levels. Word-level labels reflect the emotional color carried by the word, while sentence-level labels reflect the overall emotional tone of the sentence. In some embodiments, the generation of sentiment tendency labels is described in Table 2.

[0040] Table 2: Mapping Table of Emotional Acoustic Features and Emotional Tendency Labels The pre-annotated prosodic and sentiment tags are aligned and bound to the vocabulary and time boundaries in the initial text sequence at the word level, forming intermediate annotation results with prosodic and sentiment tags. The alignment and binding process ensures that the start and end timestamps of each word strictly correspond to their corresponding prosodic and sentiment tags. In some embodiments, the alignment operation is implemented using a dynamic time warping algorithm. It can be understood that prosodic and sentiment annotations add suprasegmental information-level descriptions to Zhuang speech.

[0041] In one embodiment of the present invention, a structured data container is established, comprising an audio information layer, a transcribed text layer, a prosodic annotation layer, a sentiment annotation layer, and a scene description layer. In the audio information layer, the storage path, sampling rate, duration, and overall signal-to-noise ratio of the clean speech waveform data are recorded. In the transcribed text layer, the initial text sequence at the word level is stored, and each word is associated with its start and end timestamps in the audio. In the prosodic annotation layer, the prosodic label corresponding to each word is stored, and a link is established with the corresponding word in the transcribed text layer. In the sentiment annotation layer, sentiment tendency labels at the word and sentence levels are stored, and a link is established with the corresponding word or sentence in the transcribed text layer. In the scene description layer, scene noise fingerprint data is stored and mapped to readable scene type descriptions and noise characteristic descriptions. All layers of data are aligned and encapsulated according to a unified timeline to generate a standardized structured data format file, which serves as the final structured annotation result. In a specific implementation of this invention, the data management module forms a closed-loop interaction with the upstream result generation module and the downstream corpus application end: after the result generation module completes multi-dimensional structured annotation, it outputs an annotation file in Extensible Markup Language (XML) format to the queue of the data management module to be added to the database; the data management module first parses the XML file, maps the fields of the audio information layer, transcribed text layer, prosodic annotation layer, sentiment annotation layer, and scene description layer to the corresponding data tables in the annotation database, and generates a globally unique identifier for each record, establishing an index association with the original Zhuang language audio file to ensure the traceability of the corpus.

[0042] After storage is completed, the data management module automatically calls the quality measurement function: it calculates the word error rate estimate based on the pre-trained Zhuang language recognition reference model, verifies the coverage and cross-layer consistency of prosody and sentiment tags through the rule engine, and compares the alignment error between the original audio timestamp and the labeled timestamp; all measurement results are summarized to generate a labeling quality report. If the word error rate of a record is higher than the preset threshold or the label missing rate exceeds 10%, the system automatically marks it as a low-quality sample and pushes revision suggestions to the manual review end. Only labeled data that has passed the review can be entered into the formal corpus for downstream speech synthesis and linguistic research.

[0043] This interactive mechanism automates the entire process from annotation output to quality control, solving the pain points of "difficult to trace and verify" annotation of low-resource Zhuang language corpora. Tests have shown that it can improve the efficiency of annotation entering the database by 40% and reduce the rate of missed detection of low-quality annotations to below 2%.

[0044] The structured annotation results in Extensible Markup Language (Extreme Markup Language) format are parsed, and the audio information layer fields, transcription text layer fields, prosody annotation layer fields, sentiment annotation layer fields, and scene description layer fields are mapped to corresponding data tables in the database for storage. During storage, a globally unique identifier is generated for each annotation record, and an index relationship with the original audio file is established. Automatic quality measurement is performed on the structured annotation results stored in the database. This quality measurement includes: calculating the word error rate estimate of the speech recognition results, checking the coverage and consistency of prosody and sentiment tags, and verifying the accuracy of timestamp alignment. Based on the results of the automatic quality measurement, an annotation quality report is generated, which includes a detailed quality score for each annotation record, a list of potential problems found, and revision suggestions for low-quality annotation records.

[0045] In practical implementation, constructing structured annotation results with multi-dimensional attributes requires establishing a structured data container. This container includes an audio information layer, a transcribed text layer, a prosodic annotation layer, a sentiment annotation layer, and a scene description layer. The audio information layer records the storage path, sampling rate, duration, and overall signal-to-noise ratio of the clean speech waveform data. The transcribed text layer stores the initial text sequence at the word level and associates each word with its start and end timestamps in the audio. The prosodic annotation layer stores the prosodic label corresponding to each word and establishes a link with the corresponding word in the transcribed text layer. The sentiment annotation layer stores sentiment tendency labels at the word and sentence levels and establishes a link with the corresponding word or sentence in the transcribed text layer. The scene description layer stores scene noise fingerprint data and maps it to readable scene type descriptions and noise characteristic descriptions. Optionally, scene type descriptions can include categories such as "quiet indoors," "outdoor street," and "noisy in a car," while noise characteristic descriptions can include text such as "low-frequency steady-state noise" and "intermittent impulse noise." All layers of data are aligned and encapsulated according to a unified timeline to generate a standardized structured data format file, which serves as the final structured annotation result. It is understandable that the Extensible Markup Language (Extreme Markup Language) format has good structure and scalability, facilitating data exchange and subsequent processing.

[0046] In practical implementation, the data management module parses the structured annotation results in Extensible Markup Language (Extreme Markup Language) format and maps the audio information layer fields, transcribed text layer fields, prosody annotation layer fields, sentiment annotation layer fields, and scene description layer fields to corresponding data tables in the database for storage. During storage, a globally unique identifier is generated for each annotation record, and an index relationship with the original audio file is established. The globally unique identifier is typically generated using a universally unique identification code algorithm. Automatic quality measurement is performed on the structured annotation results stored in the database. Automatic quality measurement includes calculating the word error rate estimate of the speech recognition results, checking the coverage and consistency of prosody and sentiment tags, and verifying the accuracy of timestamp alignment. In some embodiments, the word error rate estimate... The calculation is performed by comparing the identification results with a reference generated by fusing multiple historical high-quality annotation results. The formula is as follows: in: Indicates the number of replacement errors. Indicates the number of deletion errors. This indicates the number of insertion errors. This indicates the total number of words in the reference text. An annotation quality report is generated based on the results of automatic quality measurement. The report includes a detailed quality score for each annotation record, a list of potential problems, and revision suggestions for low-quality annotation records. The quality score can be calculated based on multiple dimensions, such as word error rate estimates, tag coverage, and timestamp alignment errors. In some embodiments, the annotation quality report is output in a structured document format, such as Extensible Markup Language (EXPLAIN) or Portable Document Format (PDO). Revision suggestions include "suggest reviewing timestamp boundaries" and "suggest checking the consistency between sentiment tags and corresponding audio segments." Optionally, the database system provides query and filtering functions based on the quality score, facilitating users to quickly locate and manage annotation data of different quality levels.

[0047] Specifically, taking the Zhuang everyday spoken sentence "Goubaeaenhawneix." (I'm going to this market) as an example, the complete data processing flow of this system is demonstrated. The raw Zhuang speech stream received by the system is collected from a real market scene, containing significant human voices and vendor noise, with a signal-to-noise ratio of 10 dB. The noise separation module performs multi-level background noise separation on this speech stream, extracting scene noise fingerprint data as a 28-dimensional feature vector. The first 24 dimensions represent the distribution of noise energy in the Mel frequency band, mainly concentrated in the 0 to 2 kHz band. The 25th dimension is the noise time-varying stability coefficient, with a value of 0.12. Dimensions 26 to 28 are the periodic noise harmonic peak identifiers; there are no significant periodic peaks in this scene. Based on the above noise fingerprint, the feature extraction module dynamically constructs an adaptive band-stop filter bank with a center stopband frequency of 0 to 2 kHz to pre-filter the clean speech waveform data. After filtering, the noise frequency band energy in the speech signal is suppressed by 12 dB, and the fundamental frequency trajectory distortion rate is reduced from 18% to 3%, generating a scene-adaptive high-dimensional acoustic feature vector. The speech recognition module calls a multi-scene hybrid acoustic model to decode the above features, outputting an initial text sequence at the word level and accurately aligning the timestamps. The multi-dimensional annotation module, based on the fundamental frequency trajectory and energy envelope of the original speech stream, identifies "Gou" as the first tone, "bae" as the second tone, and "haw" as the third tone; based on the energy distribution characteristics, "haw" is labeled as an stressed word; combined with the output of the sentiment classifier, the overall sentiment of the sentence is determined to be neutral, i.e., calm.

[0048] The results generation module integrates the above information into a structured annotation result. In this result, the audio information layer records the file path, sampling rate, duration, and signal-to-noise ratio parameters; the transcribed text layer sequentially records three words and their corresponding start and end timestamps; the prosodic annotation layer associates each word with its corresponding tone type and stress level; the sentiment annotation layer marks the calm emotional tendency of the entire sentence; and the scene description layer maps to a readable market scene type and a description of low-frequency steady-state noise characteristics. This example verifies that the system can maintain tone recognition accuracy and the integrity of the annotation structure even in low signal-to-noise ratio real-world scenarios.

[0049] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A multi-scenario speech recognition automatic annotation system for Zhuang language spoken corpora, characterized in that, The system includes: The noise separation module acquires the original Zhuang language speech stream containing various real-world scenarios, performs multi-level background noise separation processing on the original Zhuang language speech stream, and extracts clean speech waveform data and corresponding scene noise fingerprint data. The feature extraction module inputs the clean speech waveform data into an improved acoustic feature extraction algorithm, dynamically adjusts parameters based on the scene noise fingerprint, and generates a scene-adaptive high-dimensional acoustic feature vector. The speech recognition module calls a pre-built multi-scenario hybrid acoustic model to perform preliminary decoding on the high-dimensional acoustic feature vector, generating a phoneme-level candidate recognition sequence, and performs word boundary localization and error correction on the phoneme-level candidate recognition sequence based on the Zhuang language model to generate a word-level initial text sequence. A multi-dimensional annotation module, combining prosodic and emotional features extracted from the original speech stream, annotates the initial text with prosodic and emotional tags; The result generation module integrates the scene noise fingerprint data, the intermediate annotation results with prosody and emotion tags, and the timestamp information of the speech stream to construct a structured annotation result with multi-dimensional attributes.

2. The automatic speech recognition and annotation system for Zhuang spoken language corpora according to claim 1, characterized in that, The original Zhuang language speech stream is subjected to multi-level background noise separation processing to extract clean speech waveform data and corresponding scene noise fingerprint data, including: The original Zhuang language speech stream is subjected to a short-time Fourier transform (STFT) to convert the time-domain speech signal into a time-spectrum graph. In the frequency domain, a noise power spectrum estimation method based on the auditory masking effect is used to estimate the noise floor power spectrum of each frame of the signal. The estimated noise floor power spectrum is subtracted frame by frame from the power spectrum of the original Zhuang language speech stream to obtain the preliminary enhanced power spectrum; Perform an inverse short-time Fourier transform on the initially enhanced power spectrum and convert it back to the time domain to obtain the initial denoised speech waveform; For the initial denoised speech waveform, it is decomposed into a speech component matrix and a noise component matrix using nonnegative matrix factorization. Subtract the time-domain signal corresponding to the noise component matrix from the initial denoised speech waveform to obtain the final clean speech waveform data; Statistical features are extracted from the noise component matrix, including the distribution of noise energy in the frequency band, the time-varying stability coefficient of noise, and the periodic noise pattern of a specific scene. The statistical features are combined to form the scene noise fingerprint data that characterizes the background noise of a specific scene. The nonnegative matrix decomposition process introduces sparsity constraints, with the sparsity of the speech component matrix being higher than that of the noise component matrix, to ensure that the decomposition results conform to the physical interpretability of speech and noise, and to avoid misassigning speech components to the noise component matrix.

3. The automatic speech recognition annotation system for Zhuang spoken language corpora according to claim 1, characterized in that, The step of dynamically adjusting parameters based on the scene noise fingerprint to generate a scene-adaptive high-dimensional acoustic feature vector includes: Analyze the scene noise fingerprint data to obtain the main noise frequency band distribution, average noise energy, and noise fluctuation period in the current scene; Based on the main noise frequency band distribution, an adaptive band-stop filter bank is dynamically constructed. The center stopband frequency of the adaptive band-stop filter bank corresponds to the frequency band where the noise energy is concentrated, and the stopband width is adjusted proportionally according to the noise energy magnitude. During the extraction of cepstral features, the adaptive band-stop filter bank is used to pre-filter the clean speech waveform data to suppress the energy of the noise frequency band. Based on the average noise energy, an energy compensation coefficient is calculated, which is used to compensate for the energy of speech bands that are excessively attenuated due to noise suppression when calculating the log-Mel spectrum energy. Based on the noise fluctuation period, the size of the time window used to calculate the first and second differences of the Mel frequency cepstral coefficients is dynamically adjusted. When the noise fluctuation period is greater than or equal to the preset stationary noise determination threshold, a first type of fixed-length time window that is positively correlated with the noise fluctuation period is used to obtain more stable dynamic characteristics. When the noise fluctuation period is less than the preset stationary noise determination threshold, a second type of fixed-length time window that is positively correlated with the noise fluctuation period is used to capture temporal changes with higher temporal resolution. The Mel frequency cepstral coefficients, first-order difference, and second-order difference features generated after adaptive filtering, energy compensation, and dynamic window adjustment are combined to form the high-dimensional acoustic feature vector with scene-adaptive characteristics.

4. The multi-scene speech recognition automatic annotation system for Zhuang spoken language corpus according to claim 1, characterized in that, The process of calling a pre-built multi-scenario hybrid acoustic model to perform preliminary decoding of the high-dimensional acoustic feature vector and generate phoneme-level candidate recognition sequences includes: The multi-scene hybrid acoustic model is formed by fusing multiple sub-acoustic models through a weight adaptive mechanism. Each sub-acoustic model is specifically trained for Zhuang language speech in a specific real-world scenario. The high-dimensional acoustic feature vector is simultaneously input into all sub-acoustic models of the multi-scene hybrid acoustic model, and each sub-acoustic model independently outputs a phoneme posterior probability sequence. Based on the current dominant scene type identified from the scene noise fingerprint data, a set of dynamic fusion weights is calculated, and the dynamic fusion weights give higher weights to the sub-acoustic models that have a high degree of matching with the dominant scene type. The dynamic fusion weights are used to weight and fuse the phoneme posterior probability sequences output by each sub-acoustic model to obtain a comprehensive phoneme posterior probability sequence. A Viterbi decoding search is performed on the synthesized phoneme posterior probability sequence to find the optimal phoneme state path; Based on the optimal phoneme state path and combined with the phoneme context binding relationship, a candidate recognition sequence at the phoneme level is generated. The candidate recognition sequence includes the symbol identifier, start time, end time and confidence score of each phoneme.

5. The multi-scene speech recognition automatic annotation system for Zhuang spoken language corpus according to claim 4, characterized in that, The process of performing word boundary localization and error correction on the phoneme-level candidate recognition sequence based on the Zhuang language model to generate an initial text sequence at the word level includes: Load an N-grammar language model trained on a large-scale Zhuang text corpus and a Zhuang pronunciation dictionary, wherein the pronunciation dictionary establishes a mapping relationship between Zhuang vocabulary and its standard phoneme sequences; Using the phoneme-level candidate recognition sequence as input, and guided by the pronunciation dictionary, a word grid is constructed through a graph search algorithm. The nodes of the word grid correspond to word candidates that conform to the matching rules of the pronunciation dictionary, and the edges correspond to the legitimate connection relationships between adjacent word candidates in the Zhuang pronunciation sequence. Using the word sequence probabilities provided by the N-gram language model, a new Viterbi search is performed on the word grid to calculate the optimal word sequence path from the starting point to the ending point; The optimal word sequence path determines the final word segmentation boundary and corrects insertion, deletion, or replacement errors that occur during the phoneme recognition stage. Output the word sequence corresponding to the optimal word sequence path as the initial text sequence at the word level. The initial text sequence includes each identified Zhuang word, the time boundary of the word in the speech stream, and the word confidence after language model correction.

6. The multi-scene speech recognition automatic annotation system for Zhuang spoken language corpus according to claim 1, characterized in that, The initial text is labeled with prosodic and emotional features extracted from the original speech stream. Specific emotion-related acoustic features, including formant variation rate, pronunciation speed and voice jitter, are extracted from the original Zhuang language speech stream and input into a pre-trained emotion classifier to obtain the emotion intensity data, which includes discrete emotion categories and continuous emotion intensity values. Based on the fundamental frequency trajectory and energy envelope, prosodic labels are marked on each word of the initial text sequence at the word level. The prosodic labels include tone type, stress level and pause position. The pre-trained emotion classifier adopts a transfer learning strategy. It is first pre-trained using a public emotion speech database, and then fine-tuned using a self-built Zhuang language emotion speech database. Based on the emotional intensity data and speech spectrum tilt, emotional tendency labels are marked at the lexical level and the sentence level respectively. The lexical level labels reflect the emotional color carried by the words, and the sentence level labels reflect the emotional tone of the whole sentence. The labeled prosodic and sentiment labels are aligned and bound to the words and time boundaries in the initial text sequence at the word level to form the intermediate annotation result with prosodic and sentiment labels.

7. The multi-scene speech recognition automatic annotation system for Zhuang spoken language corpus according to claim 6, characterized in that, Integrating the scene noise fingerprint data, the intermediate annotation results with prosodic and emotion tags, and the timestamp information of the speech stream, a structured annotation result with multi-dimensional attributes is constructed, including: Establish a structured data container, which includes an audio information layer, a transcribed text layer, a prosody annotation layer, a sentiment annotation layer, and a scene description layer; In the audio information layer, the storage path, sampling rate, duration, and overall signal-to-noise ratio of the clean speech waveform data are recorded; In the transcribed text layer, the initial text sequence at the word level is stored, and each word is associated with its start and end timestamps in the audio. In the prosodic annotation layer, the prosodic tag corresponding to each word is stored, and a link relationship is established with the corresponding word in the transcribed text layer; In the sentiment annotation layer, the sentiment tendency labels at the word level and sentence level are stored, and a link relationship is established with the corresponding words or sentences in the transcribed text layer. In the scene description layer, the scene noise fingerprint data is stored and mapped to a readable scene type description and noise characteristic description; All layers' data are aligned and encapsulated according to a unified timeline, and a standardized structured annotation file in Extensible Markup Language (XML) format is used as the final structured annotation result.

8. The multi-scene speech recognition automatic annotation system for Zhuang spoken language corpus according to claim 3, characterized in that, The improved acoustic feature extraction algorithm also includes an online adaptive filter update mechanism, specifically including: While extracting the high-dimensional acoustic feature vector, the residual noise level of the clean speech waveform data after processing by the adaptive band-stop filter bank is continuously monitored. When the residual noise level exceeds a preset threshold, the filter parameter update process is triggered; The filter parameter update process uses the clean speech waveform data and the corresponding scene noise fingerprint data from the latest time period to re-estimate the noise power spectrum and noise frequency band distribution. Based on the re-estimation results, the center stopband frequency and stopband width of the adaptive bandstop filter bank are updated.

9. The multi-scene speech recognition automatic annotation system for Zhuang spoken language corpus according to claim 4, characterized in that, Based on the currently dominant scene type identified from the scene noise fingerprint data, a set of dynamic fusion weights is calculated, including: Pattern matching is performed on the scene noise fingerprint data. The scene noise fingerprint data is compared with multiple predefined typical scene noise templates. The scene corresponding to the typical scene noise template with the highest similarity is identified as the current dominant scene type. Based on the identified dominant scene type, a mapping table between predefined scene types and sub-acoustic models is obtained. The mapping table defines one or more scene types that each sub-acoustic model is specifically targeting. For each sub-acoustic model, its matching degree with the current dominant scene type is calculated. The matching degree is calculated based on the mapping relationship table. If the sub-acoustic model is specifically designed for the current dominant scene type, the matching degree is the highest value. If the sub-acoustic model is designed for a scene with similar acoustic characteristics to the current dominant scene type, the matching degree is the medium value. If the sub-acoustic model is designed for a scene that is very different from the current dominant scene type, the matching degree is the lowest value. The matching degrees of all sub-acoustic models are normalized so that the sum of all matching degrees is 1. The resulting set of normalized matching degrees is the dynamic fusion weight.

10. The automatic speech recognition annotation system for Zhuang spoken language corpus according to claim 1, characterized in that, The system also includes a data management module, used for: The structured annotation results are stored in a speech corpus annotation database, and an annotation quality report is generated, specifically including: The structured annotation results of the Extensible Markup Language (Extreme Markup Language) format are parsed, and the audio information layer field, transcribed text layer field, prosody annotation layer field, sentiment annotation layer field, and scene description layer field are mapped to the corresponding data tables in the database for storage. During storage, a globally unique identifier is generated for each annotation record, and an index relationship with the original audio file is established. The structured annotation results stored in the database are subjected to automatic quality measurement, which includes: calculating the word error rate estimate of the speech recognition results, checking the coverage and consistency of prosody and sentiment labels, and verifying the accuracy of timestamp alignment; Based on the results of the automatic quality metric, the annotation quality report is generated, which includes a detailed quality score for each annotation record, a list of potential problems found, and revision suggestions for low-quality annotation records.

Images