A speech recognition system and method fusing voiceprint and environment double-factor perception

Abstract

The application belongs to the technical field of speech recognition, and particularly relates to a speech recognition system and method fusing voiceprint and environment double-factor perception, which comprises the following steps: processing multi-speaker speech signals through an audio acquisition and pre-normalization module; adopting a speaker segmentation module to segment the multi-speaker speech signals into multiple speech segments; inputting each speech segment into a voiceprint verification model and an environment perception model respectively to obtain a speaker similarity and a noise level value; if the speaker similarity is lower than a threshold value, performing amplitude attenuation processing on the speech segment, otherwise inputting the speaker similarity and the noise level value into a fusion controller to perform enhancement processing on the speech segment; a speech recognition engine receives all processed speech segments output by the fusion controller, splices them in time sequence, and then performs automatic speech recognition to output a structured recognition result containing text content, a time stamp and a speaker identification; the application can significantly reduce the ASR word error rate while ensuring the intelligibility of target speech.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, specifically relating to a speech recognition system and method that integrates voiceprint and environmental dual-factor perception. Background Technology

[0002] Although speech recognition technology has matured in single-speaker and silent environments, in real-world scenarios with multiple speakers (such as meetings and home environments), interference from non-target speakers often causes the system to misjudge invalid information as valid input, resulting in a significant increase in word error rate (WER).

[0003] Current mainstream solutions typically build their front-end processing flow based on Voice Activity Detection (VAD) and fixed-parameter enhancement models (such as RNNoise and DeepFilterNet). However, such solutions have inherent limitations: VAD can only determine the activity of the audio signal and cannot distinguish the speaker's identity, making it prone to enhancing interfering speech as well, which in turn exacerbates the confusion in the Automatic Speech Recognition (ASR) system; although recent studies have introduced speaker recognition technology to achieve speaker filtering, most of them adopt a static binary decision mechanism (Target / Non-target), lacking dynamic consideration of the real-time environmental noise state, which leads to over-suppression of target speech at low signal-to-noise ratio (SNR) or under-enhancing of weak speaker signals at high SNR.

[0004] Furthermore, in existing architectures, voiceprint extraction and noise estimation often rely on separate preprocessing modules, which not only introduces computational redundancy but also easily leads to inconsistencies in front-end feature representation. Therefore, there is an urgent need to build a speech recognition system with environmental adaptability. This system should be able to collaboratively perceive the confidence level of the speaker's identity and the dynamic distribution of environmental noise, and adjust the front-end enhancement strategy in real time accordingly, thereby achieving an optimal balance between recognition accuracy and robustness in complex scenarios. Summary of the Invention

[0005] To effectively address the problems in the background technology, such as significant environmental noise interference, difficulty in distinguishing between target and non-target speakers by traditional speech recognition systems, and the tendency of fixed noise reduction strategies to lead to distortion of target speech or noise residue, this invention proposes a speech recognition system and method that integrates voiceprint and environmental dual-factor perception.

[0006] In a first aspect, the present invention provides a speech recognition system that integrates voiceprint and environmental dual-factor perception, comprising:

[0007] The audio acquisition and pre-normalization module is used to perform audio signal acquisition and format normalization processing; the audio signals include two types: single-speaker speech and multi-speaker mixed speech;

[0008] A registered voiceprint database is used to store the registered voiceprint embedding vector of the target speaker; the registered voiceprint embedding vector is obtained by performing format standardization processing and voiceprint feature extraction on the single speaker's speech provided by the target speaker during the registration stage;

[0009] The speaker segmentation module is used to segment multi-speaker aliased speech after format standardization to obtain multiple speech segments; each speech segment is marked with a start and end timestamp.

[0010] The voiceprint verification module is used to extract voiceprint features from each speech segment to generate corresponding voiceprint embedding vectors. For each speech segment, the cosine similarity between its voiceprint embedding vector and each registered voiceprint embedding vector in the registered voiceprint database is calculated. The maximum cosine similarity is selected as the speaker similarity of the current speech segment, and the target speaker to which the registered voiceprint embedding vector corresponding to the maximum cosine similarity belongs is marked as the speaker label of the current speech segment.

[0011] The environment perception module is used to extract acoustic features from each speech segment to obtain the corresponding noise level value;

[0012] A fusion controller is used to enhance or attenuate each speech segment based on speaker similarity and noise level values;

[0013] The speech recognition engine receives all processed speech segments output by the fusion controller, splices them together in chronological order, performs automatic speech recognition, and outputs a structured recognition result containing text content, timestamps, and speaker identifiers.

[0014] In a second aspect, based on the system mentioned in the first aspect, the present invention also provides a speech recognition method that integrates voiceprint and environmental dual-factor perception, comprising the following steps:

[0015] D1. Acquire and process multi-speaker aliased speech signals through an audio acquisition and pre-normalization module;

[0016] D2. The processed multi-speaker aliased speech signal is segmented into multiple speech segments using a speaker segmentation module;

[0017] D3. For each speech segment, input it into the speaker verification model and the environment perception model respectively to obtain speaker similarity and noise level values;

[0018] D4. For each speech segment, its speaker similarity and noise level values ​​are input into the fusion controller for enhancement processing;

[0019] D5. The speech recognition engine receives all processed speech segments output by the fusion controller, splices them together in chronological order, performs automatic speech recognition, and outputs a structured recognition result containing text content, timestamps, and speaker identifiers.

[0020] The beneficial effects of this invention are:

[0021] By synchronously acquiring voiceprint similarity and environmental noise level through a shared acoustic feature front end, a two-factor fusion control mechanism is constructed to achieve selective enhancement of the target speaker's speech and effective suppression of non-target speech; it avoids the erroneous enhancement of interfering speakers by the traditional VAD+fixed enhancement scheme; it significantly reduces the ASR word error rate while ensuring the intelligibility of the target speech; the system architecture is compact and suitable for deployment on edge devices. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention;

[0023] Figure 2 This is a structural block diagram of the acoustic feature extraction unit in an embodiment of the present invention;

[0024] Figure 3 This is a structural block diagram of the speaker segmentation module in this invention;

[0025] Figure 4 This is a structural block diagram of the environmental perception module in this invention;

[0026] Figure 5 This is a structural block diagram of the deep neural network speech enhancement model in this invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figures 1 to 5 This invention provides a speech recognition system that integrates voiceprint and environmental dual-factor perception, comprising:

[0029] The audio acquisition and pre-normalization module is used to perform audio signal acquisition and format normalization processing; the audio signals include two types: single-speaker speech and multi-speaker mixed speech;

[0030] A registered voiceprint database is used to store the registered voiceprint embedding vector of the target speaker; the registered voiceprint embedding vector is obtained by performing format standardization processing and voiceprint feature extraction on the single speaker's speech provided by the target speaker during the registration stage;

[0031] The speaker segmentation module is used to segment multi-speaker aliased speech after format standardization to obtain multiple speech segments; each speech segment is marked with a start and end timestamp.

[0032] The voiceprint verification module is used to extract voiceprint features from each speech segment to generate corresponding voiceprint embedding vectors. For each speech segment, the cosine similarity between its voiceprint embedding vector and each registered voiceprint embedding vector in the registered voiceprint database is calculated. The maximum cosine similarity is selected as the speaker similarity of the current speech segment, and the target speaker to which the registered voiceprint embedding vector corresponding to the maximum cosine similarity belongs is marked as the speaker label of the current speech segment.

[0033] The environment perception module is used to extract acoustic features from each speech segment to obtain the corresponding noise level value;

[0034] A fusion controller is used to enhance or attenuate each speech segment based on speaker similarity and noise level values;

[0035] The speech recognition engine receives all processed speech segments output by the fusion controller, splices them together in chronological order, performs automatic speech recognition, and outputs a structured recognition result containing text content, timestamps, and speaker identifiers.

[0036] In some embodiments, to address the problem of poor system robustness and decreased recognition accuracy in voiceprint recognition and speech processing systems caused by inconsistencies in the format, sampling rate, bit depth, and number of channels of the input audio, this invention proposes a front-end audio acquisition and pre-normalization module. This module performs uniform format standardization processing on all input audio signals (including single-speaker speech during registration and multi-speaker aliased speech during testing) at the system front end to eliminate interference from differences in audio attributes on subsequent processing flows, thereby ensuring stable operation and high performance of all system modules.

[0037] The audio acquisition and pre-normalization module is responsible for performing normalization processing on the input audio signal, and its process is as follows:

[0038] S11. Resampling Processing: The sampling rate of the input audio signal is uniformly resampled to 16kHz to generate a resampled audio signal to adapt to the input requirements of the voiceprint verification module and the speech recognition engine.

[0039] S12. Channel Format Detection and Conversion: Detect the channel attributes of the resampled audio signal. If the channel attribute is stereo or multi-channel format, convert the resampled audio signal to mono format and proceed to step S13; if the channel attribute is mono format, proceed directly to step S13.

[0040] Specifically, mono format conversion is achieved by averaging or extracting the main channel.

[0041] S13. Quantization and Bit Depth Normalization: The resampled audio signal is uniformly quantized into a 16-bit integer format or a 32-bit floating-point format (range [−1.0,+1.0]) to avoid numerical anomalies caused by differences in quantization precision.

[0042] S14. Silent trimming and length verification:

[0043] If the audio signal is a single speaker's voice, the system will automatically locate and trim the silence segments at the beginning and end of the quantized resampled audio signal, ensuring that the effective duration after trimming is no less than 3 seconds.

[0044] In some embodiments, this system implements a speaker segmentation module based on the open-source framework PyAnnote Audio, such as... Figure 3 As shown, its underlying algorithm employs a CNN encoder + Transformer decoder, capable of identifying speaker switching boundaries with a minimum granularity of 0.5 seconds. This module boasts fully automated processing capabilities, eliminating the need for manual annotation to complete speech segmentation and initial speaker label assignment, thereby significantly reducing the complexity and cost of system deployment.

[0045] In practical applications, even when faced with complex situations such as overlapping speech or short speeches (duration <1 second), this module can still maintain the stability of the segmentation results through a sliding window mechanism and post-processing strategies (such as merging adjacent segments with the same label), providing high-quality input units for subsequent speaker recognition tasks. In particular, when a speech segment is detected to be too short in duration (e.g., <0.3 seconds) and has low confidence, the system will merge it with the preceding and following speech segments before sending it to the speaker verification module, avoiding instability of the embedding vector due to the shortness of the speech segment.

[0046] In some embodiments, the registered voiceprint library and the voiceprint verification module share the same acoustic feature extraction unit for voiceprint feature extraction. This design not only reduces redundant computation overhead (saving approximately 30% of CPU resources) but also ensures that the two modules operate in the same feature space, avoiding decision conflicts caused by feature inconsistencies and improving the overall consistency and stability of the system. This acoustic feature extraction unit is implemented in C++ or Rust, supports real-time streaming processing, and has a latency of less than 5ms.

[0047] In this embodiment of the invention, the acoustic feature extraction unit selects the ECAPA-TDNN model to perform the voiceprint feature extraction operation, such as... Figure 2 As shown, it includes:

[0048] S21. Convert the input speech waveform into a Mel spectrogram;

[0049] S22. Input the Mel spectrogram into a one-dimensional convolutional layer with a kernel size of 5 and a stride of 1 to obtain local features;

[0050] S23. The local features are processed through multiple cascaded SE-Res2Blocks, and the outputs of all SE-Res2Blocks are concatenated and fused to obtain multi-scale features;

[0051] S24. Input the multi-scale features into the attention statistical pooling layer to obtain the key features, and pass the key features through the fully connected layer to obtain the 192-dimensional voiceprint embedding vector.

[0052] The ECAPA-TDNN neural network possesses powerful temporal modeling and feature representation capabilities, effectively capturing long-term dependencies and speaker-specific patterns in speech, maintaining high recognition accuracy across channels, devices, and languages. Notably, the ECAPA-TDNN neural network's multi-scale feature aggregation mechanism makes it robust to short speech segments (<1 second), which perfectly matches the minimum 0.5-second speech fragment output by the speaker segmentation module, ensuring reliable operation even at fine-grained segmentation.

[0053] In some embodiments, the environment perception module extracts acoustic features from any speech segment using a lightweight convolutional neural network, such as... Figure 4 As shown, it includes:

[0054] S31. Convert the speech segment into a Mel spectrogram;

[0055] S32. Input the Mel spectrogram into three parallel deep convolutional layers to obtain three deep convolutional features; the kernel sizes of the three deep convolutional layers are 7×1, 5×1, and 3×1, respectively;

[0056] S33. After concatenating the three deep convolutional features, pass them through a 1×1 convolutional layer to obtain the fused features;

[0057] S34. Pass the fused features sequentially through a global average pooling layer, a first fully connected layer, and a second fully connected layer to obtain the noise level value.

[0058] Specifically, the noise level value ranges from [0,1], with a larger value indicating a noisier environment.

[0059] In some embodiments, the processing of any speech segment by the fusion controller includes:

[0060] S41. Determine whether the speaker similarity of the speech segment is less than the threshold θ. If yes, proceed to step S44; otherwise, proceed to step S42.

[0061] S42. Calculate the speech enhancement constraint parameter atten_lim_db based on speaker similarity and noise level, expressed as:

[0062] ,

[0063] ,

[0064] ,

[0065] In the formula, f() represents nonlinear change, g() represents inverse activation function; n represents the noise level value of the speech segment, and s i The similarity between speakers in a speech segment is represented by ReLU(), which is the ReLU activation function. σ represents the Sigmoid function, used to smoothly and non-linearly normalize the noise level n to avoid abrupt changes in the output. k represents the noise sensitivity scaling factor, used to control the strength of the influence of the noise level n on f(n). b represents the noise bias term, used to adjust the center point of the Sigmoid function, thereby determining the noise level threshold at which f(n) significantly increases; w s This represents the inverse sensitivity weight of similarity, controlling the speaker similarity s. i The direction and magnitude of the effect on the suppression intensity. Due to the use of 1−s... i The lower the similarity (the less like the target speaker), the better g(s) i The larger the value of g(s), the greater the value of g(s) will be, thus increasing the value of atten_lim_db. c represents the similarity activation bias, which determines at what similarity threshold g(s) begins to activate. i w1 and w2 represent adjustable weighting coefficients, where:

[0066] ,

[0067] .

[0068] S43. Using the enhancement constraint parameter atten_lim_db as the suppression strength parameter, the speech segment is enhanced through a deep neural network speech enhancement model to obtain the enhanced speech segment.

[0069] In some embodiments, such as Figure 5 As shown, the deep neural network speech enhancement model uses DeepFilterNet, and the processing includes:

[0070] The noisy speech segments are converted into STFT spectra using a convolutional encoder;

[0071] The time-frequency characteristic h is obtained by passing the STFT spectrum through LSTM;

[0072] Adaptive modulation of the time-frequency feature h is performed based on the enhancement constraint parameter atten_lim_db to obtain the modulation parameters. , represented as:

[0073] ,

[0074] ,

[0075] ,

[0076] In the formula, This represents the learnable scaling weight parameters used to map atten_lim_db to linear transformation coefficients of the scaling factor γ. This represents the learnable offset weight parameter used to map atten_lim_db to offset factors. The linear transformation coefficients, This represents the scaling bias term, which, together with W1, determines the scaling factor. The baseline, This represents the offset term, which, together with W2, determines the offset factor. The baseline value, This represents element-wise multiplication, that is, multiplying each element of the time-frequency feature h by the corresponding... value (if) (for vectors) or uniform scaling (if) (for scalars) Indicates the scaling factor. This represents the offset factor.

[0077] modulation parameters The complex mask is obtained by the mask generator. The complex mask is then multiplied element-wise with the STFT spectrum and restored by the inverse short-time Fourier transform (ISTFT) to reconstruct the enhanced speech segment.

[0078] S44. Perform amplitude attenuation processing on the speech segment to obtain the attenuated speech segment; wherein, the amplitude attenuation processing includes:

[0079] The attenuation factor α is calculated based on speaker similarity. i = max(0.1, 1−3s) i );

[0080] Multiply the temporal amplitude of the speech segment by the attenuation factor α. i .

[0081] Specifically, to avoid ineffective or even harmful enhancement of non-target speaker speech and to prevent the introduction of artifacts that interfere with the Automatic Speech Recognition (ASR) system, this embodiment of the invention directly performs amplitude attenuation processing on speech segments with speaker similarity less than a threshold. This ensures that extremely low confidence segments (such as s...) are attenuated. i <0.3) was significantly suppressed (decayed to below 10%), while high-confidence fragments (such as s) were significantly suppressed. i If the value is greater than 0.6, then it will remain basically unchanged (α). i ≈1), thereby achieving active suppression of unknown speaker speech and preventing it from interfering with ASR recognition results. This strategy is particularly important in scenarios such as meeting minutes and court hearings where it is necessary to strictly distinguish the speaker's identity. The attenuation operation is performed on the time-domain waveform, i.e., y(t)=α i x(t) is used to maintain phase consistency and avoid introducing additional distortion in the frequency domain.

[0082] In some embodiments, the threshold θ ranges from 0.3 to 0.5, preferably 0.35, and is used to distinguish between high-confidence target speaker speech and low-confidence non-target or unknown speaker speech. This threshold is optimized based on cross-validation on VoxCeleb, LibriSpeech, and a self-built multi-speaker conference dataset, ensuring a target speaker recall rate >95% while controlling the non-target false recognition rate to <5%, achieving a good balance between security and usability. The system supports dynamically adjusting θ according to the application scenario: 0.45 for high-security scenarios (such as financial authentication) and 0.3 for ordinary interaction scenarios (such as smart homes).

[0083] In some embodiments, the speech recognition engine employs the Whisper end-to-end speech recognition model released by OpenAI or an optimized variant thereof. Whisper is a large-scale multilingual ASR model based on the Transformer architecture, which possesses strong robustness and generalization ability through self-supervised pre-training and supervised fine-tuning on 680,000 hours of multilingual, multi-domain, noisy speech data.

[0084] In this system, speech segments enhanced or attenuated by the adaptive front-end processing module are fed into the Whisper model (preferably medium or small size to balance accuracy and inference efficiency). The model directly outputs the corresponding text and language identifiers. Whisper's encoder-decoder architecture inherently supports contextual modeling and anti-interference capabilities, maintaining high recognition accuracy even in scenarios with low signal-to-noise ratios or residual noise. Furthermore, for Mandarin Chinese applications, a Whisper Chinese Enhanced Version (e.g., "Whisper-large-v3-chinese") with further fine-tuned Chinese data can be selected, significantly improving the recognition accuracy of Chinese proper nouns, numbers, and punctuation marks.

[0085] Real-world testing shows that, with clean target speech input provided by this invention, the Whisper model can reduce the character error rate (CER) to below 5.7% in conference recording scenarios, far superior to the results of directly processing raw mixed speech. This model supports deployment on multiple CPU / GPU platforms, and combined with the intelligent front-end processing of this invention, it can significantly improve the overall transcription quality and speaker attribution accuracy of the system without increasing the complexity of the ASR model.

[0086] The entire system of this invention adopts a pipelined parallel architecture, in which the speaker segmentation module, voiceprint verification module, and environment awareness module can process different speech segments in parallel; the enhancement processing in the fusion controller is triggered on demand; and finally, the speech recognition engine receives the processed audio stream in a streaming manner. This architecture supports multi-channel concurrency and is suitable for conference scenarios with up to 8 people. On resource-constrained devices, the system can dynamically disable the environment awareness module, using only a fixed atten_lim_db (e.g., 6dB), degrading to a single-factor mode to ensure the availability of basic functions.

[0087] In addition, the system has a built-in exception handling mechanism:

[0088] If the voiceprint verification module returns a NaN or Inf value, it is automatically considered as s. i =0 and perform decay;

[0089] If the environment perception module does not respond within a timeout period, the default value is n=0.5.

[0090] If the speech energy is abnormally attenuated (<-40dBFS) after DeepFilterNet enhancement, then revert to the original segment.

[0091] These mechanisms ensure the stability of the system under extreme conditions.

[0092] In some embodiments, the present invention also provides a speech recognition method that integrates voiceprint and environmental dual-factor perception, comprising the following steps:

[0093] D1. Acquire and process multi-speaker aliased speech signals through an audio acquisition and pre-normalization module;

[0094] D2. The processed multi-speaker speech signal is segmented into multiple speech segments using a speaker segmentation module;

[0095] D3. For each speech segment, input it into the speaker verification model and the environment perception model respectively to obtain speaker similarity and noise level values;

[0096] D4. For each speech segment, its speaker similarity and noise level values ​​are input into the fusion controller for enhancement processing;

[0097] D5. The speech recognition engine receives all processed speech segments output by the fusion controller, splices them together in chronological order, performs automatic speech recognition, and outputs a structured recognition result containing text content, timestamps, and speaker identifiers.

[0098] In some embodiments, a registration process precedes step D1, including:

[0099] Collect at least 30 seconds of speech samples from the target speaker, perform format standardization and voiceprint feature extraction on the speech samples to obtain the registered voiceprint embedding vector, and store it in the registered voiceprint database.

[0100] In summary, this invention achieves intelligent processing of mixed speech from multiple speakers by constructing a complete pipeline of "speaker segmentation → two-factor perception → adaptive enhancement → ASR recognition". The system fully utilizes both speaker and environmental information, employing ECAPA-TDNN for high-precision speaker matching, a lightweight CNN for real-time noise environment perception, DeepFilterNet for dynamic target speech enhancement, and a fusion controller for intelligent decision-making, ultimately outputting high-precision transcription results with speaker tags. This method not only improves speech recognition performance in complex scenarios but also possesses excellent real-time performance and deployability, making it suitable for practical applications such as smart speakers, remote conferencing systems, judicial recording analysis, and smart home security monitoring, demonstrating broad application prospects and promotional value.

[0101] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A speech recognition system integrating voiceprint and environmental dual-factor perception, characterized in that, include: The audio acquisition and pre-normalization module is used to perform audio signal acquisition and format normalization processing; the audio signals include two types: single-speaker speech and multi-speaker mixed speech; A registered voiceprint database is used to store the registered voiceprint embedding vector of the target speaker; the registered voiceprint embedding vector is obtained by performing format standardization processing and voiceprint feature extraction on the single speaker's speech provided by the target speaker during the registration stage; The speaker segmentation module is used to segment multi-speaker aliased speech after format standardization to obtain multiple speech segments; each speech segment is marked with a start and end timestamp. The voiceprint verification module is used to extract voiceprint features from each speech segment to generate corresponding voiceprint embedding vectors. For each speech segment, the cosine similarity between its voiceprint embedding vector and each registered voiceprint embedding vector in the registered voiceprint database is calculated. The maximum cosine similarity is selected as the speaker similarity of the current speech segment, and the target speaker to which the registered voiceprint embedding vector corresponding to the maximum cosine similarity belongs is marked as the speaker label of the current speech segment. The environment perception module is used to extract acoustic features from each speech segment to obtain the corresponding noise level value; A fusion controller is used to enhance or attenuate each speech segment based on speaker similarity and noise level values; The speech recognition engine receives all processed speech segments output by the fusion controller, splices them together in chronological order, performs automatic speech recognition, and outputs a structured recognition result containing text content, timestamps, and speaker identifiers.

2. The speech recognition system integrating voiceprint and environmental dual-factor perception according to claim 1, characterized in that, The audio acquisition and pre-normalization module is responsible for performing normalization processing on the input audio signal, and its process is as follows: S11. Resampling Processing: The sampling rate of the input audio signal is uniformly resampled to 16kHz to generate a resampled audio signal; S12. Channel Format Detection and Conversion: Detect the channel attributes of the resampled audio signal. If the channel attribute is stereo or multi-channel format, convert the resampled audio signal to mono format and proceed to step S13; if the channel attribute is mono format, proceed directly to step S13. S13. Quantization and Bit Depth Normalization: Quantize the resampled audio signal into a 16-bit integer format or a 32-bit floating-point format; S14. Silent trimming and length verification: If the audio signal is a single speaker's voice, the system will automatically locate and trim the silence segments at the beginning and end of the quantized resampled audio signal, and ensure that the effective duration after trimming is not less than 3 seconds. If the audio signal is a mixture of multiple speakers' speech, then no silence segments will be trimmed; only filling or segmentation will be performed.

3. The speech recognition system integrating voiceprint and environmental dual-factor perception according to claim 1, characterized in that, The ECAPA-TDNN model is used to perform voiceprint feature extraction, including: S21. Convert the input speech waveform into a Mel spectrogram; S22. Input the Mel spectrogram into a one-dimensional convolutional layer with a kernel size of 5 and a stride of 1 to obtain local features; S23. The local features are processed through multiple cascaded SE-Res2Blocks, and the outputs of all SE-Res2Blocks are concatenated and fused to obtain multi-scale features; S24. Input the multi-scale features into the attention statistical pooling layer to obtain the key features, and then pass the key features through a fully connected layer to obtain the voiceprint embedding vector.

4. The speech recognition system integrating voiceprint and environmental dual-factor perception according to claim 1, characterized in that, The environment perception module extracts acoustic features from any speech segment using a lightweight convolutional neural network, including: S31. Convert the speech segment into a Mel spectrogram; S32. Input the Mel spectrogram into three parallel deep convolutional layers to obtain three deep convolutional features; the kernel sizes of the three deep convolutional layers are 7×1, 5×1, and 3×1, respectively; S33. After concatenating the three deep convolutional features, pass them through a 1×1 convolutional layer to obtain the fused features; S34. Pass the fused features sequentially through a global average pooling layer, a first fully connected layer, and a second fully connected layer to obtain the noise level value.

5. A speech recognition system integrating voiceprint and environmental dual-factor perception according to claim 1, characterized in that, The fusion controller's processing of any speech segment includes: S41. Determine whether the speaker similarity of the speech segment is less than the threshold. If yes, proceed to step S44; otherwise, proceed to step S42. S42. Calculate the speech enhancement constraint parameter atten_lim_db based on speaker similarity and noise level, expressed as: ， ， ， In the formula, w1 and w2 represent adjustable weight coefficients, f() represents nonlinear change, g() represents the inverse activation function; n represents the noise level value of the speech segment, and s i Represents the speaker similarity of a speech segment; ReLU() represents the ReLU activation function; σ represents the Sigmoid function; k represents the noise sensitivity scaling factor; b represents the noise bias term; w s represents the similarity inverse sensitivity weight, and c represents the similarity activation bias; S43. Using the enhancement constraint parameter atten_lim_db as the suppression strength parameter, the speech segment is enhanced through a deep neural network speech enhancement model to obtain the enhanced speech segment; S44. Perform amplitude attenuation processing on the speech segment to obtain the attenuated speech segment; wherein, the amplitude attenuation processing includes: The attenuation factor α is calculated based on speaker similarity. i = max(0.1, 1−3s) i ); Multiply the temporal amplitude of the speech segment by the attenuation factor α. i .

6. A speech recognition system integrating voiceprint and environmental dual-factor perception according to claim 5, characterized in that, The deep neural network speech enhancement model uses DeepFilterNet, and the processing includes: The noisy speech segments are converted into STFT spectra using a convolutional encoder; The time-frequency characteristic h is obtained by passing the STFT spectrum through LSTM; Adaptive modulation of the time-frequency feature h is performed based on the enhancement constraint parameter atten_lim_db to obtain the modulation parameters. , is represented as: ， ， ， In the formula, This represents the learnable scaling weight parameters. This represents the learnable offset weight parameters. This indicates the scaling bias term. Indicates the offset term. This represents element-wise multiplication. Indicates the scaling factor. Indicates the offset factor; modulation parameters The complex mask is obtained by the mask generator. The complex mask is then multiplied element-wise with the STFT spectrum and restored by inverse short-time Fourier transform to reconstruct the enhanced speech segment.

7. A speech recognition system integrating voiceprint and environmental dual-factor perception according to claim 5, characterized in that, The threshold value ranges from 0.3 to 0.

5.

8. A speech recognition method integrating voiceprint and environmental dual-factor perception, characterized in that, A speech recognition system that integrates voiceprint and environmental dual-factor perception as described in any one of claims 1 to 7 includes the following steps: D1. Acquire and process multi-speaker aliased speech signals through an audio acquisition and pre-normalization module; D2. The processed multi-speaker aliased speech signal is segmented into multiple speech segments using a speaker segmentation module; D3. For each speech segment, input it into the speaker verification model and the environment perception model respectively to obtain speaker similarity and noise level values; D4. For each speech segment, its speaker similarity and noise level values ​​are input into the fusion controller for enhancement processing; D5. The speech recognition engine receives all processed speech segments output by the fusion controller, splices them together in chronological order, performs automatic speech recognition, and outputs a structured recognition result containing text content, timestamps, and speaker identifiers.

9. The speech recognition method integrating voiceprint and environmental dual-factor perception according to claim 8, characterized in that, There is a registration process before step D1, including: Collect at least 30 seconds of speech samples from the target speaker, perform format standardization and voiceprint feature extraction on the speech samples to obtain the registered voiceprint embedding vector, and store it in the registered voiceprint database.

Images