Voice enhancement method based on dynamic voiceprint sample pool

Abstract

The application discloses a voice enhancement method based on a dynamic voiceprint sample pool and specifically relates to the technical field of voice processing, and comprises the following steps: maintaining a dynamic voiceprint sample pool for each speaker, associating audio segments with speech recognition text and comprehensive quality scores; collecting clone reference candidates from the sample pool, taking a segment with a text length within a preset interval as a candidate, and performing greedy merging segmentation on an overlong segment based on punctuation perception; sorting all candidates in descending order of text length and quality score, and selecting an optimal candidate as original reference material; when there are two candidates with similar quality and good length, generating a fusion audio through waveform weighted fusion; and performing neural network voice enhancement processing on the selected reference audio to obtain an enhanced clone reference audio. The application realizes automatic selection and continuous evolution of the clone reference without recording, and effectively improves the quality of timbre cloning and user experience.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and more specifically, to a speech enhancement method based on a dynamic speaker sample pool. Background Technology

[0002] Zero-sample voice cloning technology refers to the technique of synthesizing speech with a similar timbre to the target speaker using only a small amount of reference audio. It has been widely used in the field of intelligent voice interaction in recent years. Existing voice cloning systems typically require users to pre-record a reference audio clip, or for system administrators to pre-configure fixed reference samples. This reference audio is then input into the cloning model for voice feature extraction and transfer during speech synthesis. This approach has significant drawbacks in natural interaction scenarios on smart terminals (such as smart speakers, in-vehicle assistants, and virtual digital humans): users need to interrupt normal conversations and actively complete the specific reference audio recording operation, increasing the barrier to entry and preventing truly intervention-free, adaptive cloning. Furthermore, existing systems typically maintain reference audio statically; once selected, it is permanently used, failing to continuously optimize the cloning effect based on the accumulation of daily user interactions.

[0003] Current technologies lack sufficient consideration of the relationship between the text length of reference audio and cloning quality. Research shows that zero-shot cloning models have an optimal working range for the text length of reference audio, typically a few to a dozen characters. Text that is too short leads to insufficient extraction of timbre features, while text that is too long may introduce prosodic bias, affecting the naturalness of the synthesized speech. However, existing systems often ignore the crucial factor of text length when selecting reference audio, failing to automatically filter text lengths for routinely collected audio or intelligently segment excessively long texts. When segmentation is necessary, existing solutions often use fixed-character truncation or simple punctuation segmentation, easily breaking down semantically complete short sentences, disrupting the semantic coherence and prosodic naturalness of the reference segment, and thus affecting the fluency of the cloned speech. Furthermore, audio collected in daily interactions inevitably contains environmental noise, far-field reverberation, and other interference factors; directly using it as a cloning reference will cause the synthesized speech to reproduce these noise features, severely impacting the listening experience and user experience.

[0004] In summary, existing voice cloning systems have shortcomings in terms of reference audio acquisition methods, text length optimization, semantic integrity segmentation, and noise interference elimination, failing to meet the needs of smart terminals for automatic selection, dynamic updating, and quality enhancement of clone references in continuous interaction scenarios. Particularly in the reference audio selection stage, there is a lack of an automated mechanism that comprehensively considers text length, semantic integrity, and acoustic quality; in the reference audio updating stage, there is a lack of adaptive evolution capabilities linked to a dynamic voiceprint sample pool, resulting in the cloning effect failing to continuously improve with the accumulation of user interaction data. Therefore, how to automatically mine and enhance the optimal clone reference audio from daily conversations without relying on manual recording has become a pressing technical problem to be solved in this field. Summary of the Invention

[0005] To overcome the above-mentioned deficiencies of the prior art, embodiments of the present invention provide a speech enhancement method based on a dynamic voiceprint sample pool.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The speech enhancement method based on dynamic speaker pooling includes the following steps: S1. Maintain a dynamic voiceprint sample pool for each speaker, the sample pool continuously collects audio segments generated by the speaker in daily interactions, and each audio segment is associated with its speech recognition text and comprehensive quality score; S2. Collect clone reference candidates from the sample pool. For segments whose text length is within the preset optimal length range, the entire segment is used as a candidate. For segments whose text length exceeds the preset maximum length, perform greedy merging and segmentation with punctuation awareness, and use the effective segments generated by the segmentation as candidates. S3. Sort all candidates in descending order of text length and then in descending order of overall quality score, and select the candidate ranked first as the original reference material; S4. Perform neural network speech enhancement processing on the selected original reference material to obtain enhanced cloned reference audio, which is used as reference input for the speech synthesis engine for timbre transfer.

[0007] Specifically, the comprehensive quality score is calculated by weighted fusion of the speech recognition confidence score, the signal-to-noise ratio estimate score, and the audio loudness score; Among them, the speech recognition confidence score directly adopts the posterior probability value output by the speech recognition model, the signal-to-noise ratio estimation score is obtained by calculating the energy ratio of the speech segment and the silent segment after detecting speech activity in the audio segment and normalizing it by the Sigmoid function, and the audio loudness score is obtained by calculating the root mean square energy of the audio segment and normalizing it to the optimal loudness range. The weighting coefficients of the weighted fusion are dynamically adjusted according to the application scenario.

[0008] Specifically, in S2: When the difference in the overall quality score between the top-ranked candidate and the second-ranked candidate is less than a preset fusion threshold, and the text lengths of both candidates are within a preset optimal length range, a multi-candidate fusion reference strategy is triggered; specifically including: After aligning the durations of the two candidate original audio files, they are linearly weighted and superimposed using the overall quality score as the weight to generate the fused audio. The merged text prompt is obtained by concatenating two candidate texts; if the total length after concatenation exceeds the preset maximum length, only the candidate text ranked first is retained.

[0009] Specifically, the greedy merging and segmentation based on punctuation awareness includes: The text is divided into several clauses using Chinese and English punctuation marks as separators, with each punctuation mark belonging to the clause preceding it. Starting from the first clause, greedily merge subsequent consecutive clauses in sequence, with the merging condition being that the total number of characters after merging does not exceed the preset maximum length; Based on the character position ratio, the corresponding audio segments are mapped from the original audio, and adaptive padding is added at the boundaries.

[0010] Specifically, after the greedy merging and grouping is completed, a dynamic segmentation optimization based on prosodic boundaries is introduced, as follows: For each finalized text group, a pre-trained prosodic boundary prediction model is used to predict the prosodic level of each clause boundary within it, and strong prosodic boundaries are used as candidate segmentation points. When the length of a text group is less than the preset maximum length and there is a strong rhythmic boundary, try to split it at the boundary. After splitting, the character length of each subgroup must not be less than the preset minimum length. Splitting is only allowed when the sum of the text lengths on both sides of a strong rhythm boundary exceeds the product of the splitting threshold and the preset maximum length.

[0011] Specifically, the neural network speech enhancement processing adopts the Conv-TasNet architecture based on a temporal convolutional network, which includes: The encoder maps the input audio waveform into a high-dimensional feature representation through a one-dimensional convolutional layer; The separation module consists of multiple repeated TCN blocks stacked together, each TCN block containing depthwise separable convolutions, gated linear unit activation functions, and residual connections; The decoder reconstructs the enhanced audio waveform from the separated features using transposed convolution; The neural network speech enhancement processing includes noise reduction, dereverberation, and bandwidth expansion; if the enhancement service is unavailable, the original audio is used as a fallback.

[0012] Specifically, this also includes a pool-based update mechanism for cloned references, the process of which is as follows: The enhanced clone reference audio is persistently stored, and a linkage update mechanism is established between the sample pool and the clone reference. When a new audio segment is added to the sample pool, a low-quality segment is eliminated according to the preset elimination rules, or the user triggers a cloning target switch via voice command, S2 to S4 are automatically re-executed to update the cloning reference. The sample pool self-cleaning mechanism includes timed triggering, incremental triggering, and capacity triggering. The elimination rules include comprehensive quality score elimination, redundancy elimination, and timeliness elimination. Currently used cloned reference fragments are permanently retained during the self-cleaning process.

[0013] Specifically, this also includes a safety net strategy: When the candidate list is empty after traversing the entire sample pool, select the segment with the highest overall quality score in the sample pool, and extract an audio segment of the corresponding duration from the beginning of the segment's audio as reference material according to the ratio of the preset maximum text length to the total number of characters in the original text. If the extracted duration is less than the preset minimum duration, the preset minimum duration will be used directly as the reference material.

[0014] The technical effects and advantages of this invention are as follows: This invention achieves automatic acquisition and continuous optimization of clone reference audio without recording. By constructing a dynamic speaker sample pool and jointly evaluating text length and comprehensive quality scores, it can automatically select the audio segments with the optimal text length and highest quality from daily conversations as clone references, completely eliminating the need for users to actively record reference audio and significantly lowering the barrier to entry. Simultaneously, a multi-candidate fusion reference strategy is introduced. When two segments with similar quality and good length exist, waveform-weighted fusion is used to generate richer reference audio. This allows the clone reference to maintain a high signal-to-noise ratio while possessing more comprehensive speaker timbre features, effectively improving the fidelity and stability of synthesized speech.

[0015] This invention ensures the semantic integrity and natural rhythm of the segmented audio clips through punctuation-aware greedy merging and segmentation combined with prosodic boundary-based dynamic segmentation optimization. This satisfies the optimal working range requirements of the cloning model for text length while avoiding the damage to the naturalness of the reference audio caused by abrupt truncation. Furthermore, by combining neural network speech enhancement processing with a pool-based update mechanism, the selected reference audio is denoised, dedeverped, and bandwidth-extended to eliminate the interference of environmental noise on the cloning quality. The optimal reference is automatically reselected when new samples are added or removed from the sample pool, achieving automatic evolution and continuous optimization of the cloning reference without any manual intervention, providing users with an experience improvement that increases with the number of interactions. Attached Figure Description

[0016] Figure 1This is a flowchart of the method of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, the steps of the speech enhancement method based on dynamic speaker sample pooling are as follows: Step 1: Intelligent selection and fusion. Audio segments are selected from the dynamic speaker sample pool, and the text length and quality scores are comprehensively evaluated. The segments with the best length and highest quality are prioritized as candidates. When two segments with similar quality and good length exist, waveform-weighted fusion is used to generate richer reference audio, providing high-quality original materials for subsequent enhancement. The specific steps are as follows: A dynamic voiceprint sample pool is constructed: a dynamic voiceprint sample pool is maintained for each speaker. This sample pool continuously collects audio clips generated by the speaker in daily interactions. When each clip is added to the database, it is associated with its corresponding speech recognition text (ASR text) and a comprehensive quality score. The comprehensive quality score is calculated by fusing multiple dimensions such as speech recognition confidence, signal-to-noise ratio estimation, and audio loudness, and is used to initially filter out low-quality clips.

[0019] Specifically, the overall quality score Q is calculated in the following way: First, calculate the sub-ratings for each of the three dimensions, including: Speech recognition confidence score : Directly uses the posterior probability value output by the speech recognition model, with a value range of [0,1]; Signal-to-noise ratio estimate score After performing speech activity detection on the audio segment, the energy ratio of the speech segment to the silence segment is calculated and normalized to the [0,1] interval using the Sigmoid function. The formula is as follows: ;in To estimate the signal-to-noise ratio (in dB). The preset reference threshold is set to 15dB. This is the slope adjustment factor (taken as 0.3). Score the signal-to-noise ratio estimate; Audio loudness score : Calculate the root mean square energy of the audio segment and normalize it to the score range corresponding to the optimal loudness interval [−26,−18] LUFS. Scores outside this interval decrease. Ultimately, the overall quality score Q is calculated using a weighted fusion method: ;in For the overall quality score, This refers to the confidence weighting coefficient for speech recognition. The weighting coefficients for the signal-to-noise ratio estimate are as follows: For audio loudness weighting coefficients, satisfying Typical value =0.5、 =0.3、 =0.2. The weighting coefficient is dynamically adjusted according to the application scenario; Cloning reference candidate collection: Traverse all audio segments in the speaker sample pool and perform the following filtering and preprocessing logic: Direct Candidates: Obtain the ASR text length of a segment. If its length is within the preset optimal length range (e.g., minimum length m is 5 characters, maximum length M is 15 characters), then add the entire segment to the candidate list.

[0020] Candidate segmentation: If the text length of a segment exceeds the preset maximum length M, a punctuation-aware greedy merging and segmentation process is initiated to segment it into one or more semantically complete and appropriately sized valid sub-segments. Each valid sub-segment generated by the segmentation is also added to the candidate list as an independent candidate.

[0021] Joint ranking and optimal selection: All candidates (including direct candidates and segmented candidates) are ranked according to their priority (descending order of text length, descending order of overall quality score). The candidate ranked first is selected as the highest quality original material for cloning. This strategy prioritizes maximizing text length to provide richer timbre information to the cloning model, and only selects candidates with higher quality scores when the lengths are the same.

[0022] To improve the timbre richness and synthesis stability of the cloned reference, a multi-candidate fusion reference strategy is introduced when there are multiple high-quality segments with similar quality and text lengths within the optimal range in the candidate list; the specific process is as follows: Fusion condition judgment: Let the overall quality score of the candidate ranked first be... The second-ranked candidate has an overall quality score of Define the fusion threshold (Value range 0≤) ≤0.1, typically 0.05). If the following conditions are met: And the text lengths of the two candidates All are within the preset optimal range If it occurs inside, the fusion mechanism will be triggered.

[0023] Audio waveform fusion: combining two candidate original audio waveforms and Linear weighted superposition is performed at the waveform level. To ensure consistent audio length after fusion, the two audio clips are first time-aligned: the shorter clip is used as the baseline, and the longer clip is trimmed (retaining the continuous segment with the highest energy). The fusion formula is: ;in The overall quality score for the top candidate in the ranking. The overall quality score for ranking the second candidate. The original audio file is the first candidate for sorting. The original audio for the second candidate in the ranking. The resulting audio is fused; the weighting coefficients are dynamically calculated based on the overall quality score to ensure that high-quality candidates dominate the fusion process.

[0024] Post-fusion text processing: The text prompt corresponding to the fused audio is obtained by concatenating two candidate texts, separated by a period, to ensure semantic coherence. If the total length after concatenation exceeds the maximum length M, only the first-ranked candidate text is retained as the prompt to avoid excessively long text interfering with the clone model.

[0025] Post-processing: This involves processing the merged audio... As a new candidate, it replaces the original top-ranked candidate and is fed into the subsequent neural network speech enhancement processing (step three). The fusion process preserves the complementarity of the timbre features of multiple candidates, enabling the cloned reference to have richer speaker feature details while maintaining a high signal-to-noise ratio.

[0026] If the text length or quality score of two candidates does not meet the fusion criteria, the candidate ranked first will still be selected directly according to the original mechanism.

[0027] Fallback Strategy: If the candidate list is empty after traversing the entire sample pool (e.g., all text fragments are shorter than the minimum length m), then a fallback strategy is adopted: select the fragment with the highest overall quality score in the sample pool, and extract an audio segment of the corresponding duration from the beginning of the audio of that fragment according to the ratio of its text length to the maximum length M, as reference material; specifically, the formula for calculating the audio segment duration is: ;in To extract the audio duration, To preset the maximum text length, This represents the total number of characters in the original text of the segment. This represents the total original audio duration of the segment. If the calculation result... Less than the preset minimum duration (e.g., 1.5 seconds), then directly extract the previous time. Duration is used as a reference material to ensure that the cloned model has enough audio information for timbre feature extraction.

[0028] Step Two: Semantic Segmentation and Optimization. For extremely long texts, a punctuation-aware greedy merging and segmentation process is used to ensure that the segmented fragments are semantically complete and their length meets the model requirements. Based on this, prosodic boundary detection is introduced to dynamically adjust the segmentation results, making the audio fragments more closely resemble pauses in natural speech flow and improving the prosodic naturalness of the cloned reference. Specific steps are as follows: Punctuation Location and Clause Segmentation: When the length of the text to be processed exceeds the maximum length M, the text is first segmented using punctuation awareness. All Chinese and English punctuation marks (such as ., !, ?, ;, :, etc.) are identified in the text, and these punctuation marks are used as delimiters to segment the text into multiple clauses. The segmentation rule is: each punctuation mark belongs to the clause preceding it, that is, the clause contains its ending punctuation.

[0029] Greedy merging and grouping: Starting with the first clause, it sequentially attempts to merge subsequent consecutive clauses into the current group. The merging condition is whether the total number of characters in the current group (including all punctuation) still does not exceed the maximum length M. If it does not exceed the maximum length M, it continues to merge the next clause; if it does exceed the maximum length M, it immediately stops merging, the current group is defined as a complete candidate segment, and the greedy merging process is repeated from the next clause until all clauses have been processed.

[0030] To improve the prosodic naturalness of the segmented audio clips, a dynamic segmentation optimization based on prosodic boundaries is introduced after the greedy merging and grouping process; the specific process is as follows: Prosodic boundary detection: For each defined text group, a pre-trained prosodic boundary prediction model (such as a BERT-based prosodic annotation model) is used to predict the prosodic level of the boundaries of each clause within it. The predicted strong prosodic boundaries (such as intonation phrase boundaries and breath group boundaries) are used as candidate segmentation points. Dynamic boundary adjustment: Let the character length of the current text group be... The default maximum length is M. If If the length of the text group is less than M and there is at least one strong prosodic boundary, then an attempt is made to split the text group into two subgroups at that boundary. After splitting, the character length of each subgroup must not be less than the minimum length m. If the length of any subgroup after splitting is less than m, then the splitting is not performed. This adjustment makes the audio clips more in line with the pauses in natural speech flow, avoiding the forced merging of semantic units that should be separated.

[0031] Audio mapping adjustment: Recalculate the audio mapping positions based on the adjusted text group boundaries. Since the adjustment may introduce new segmentation points, adaptive padding (the padding calculation formula is the same as the original text) should also be applied at the new segmentation points to ensure that no noise is introduced at the boundaries.

[0032] Balancing merging and splitting: To avoid excessive splitting leading to overly short fragments, a splitting threshold is set. (Value range 0.6≤) ≤0.8, typical value 0.7). Only when the sum of the text lengths on both sides of the strong prosodic boundary exceeds... Splitting is only allowed when a certain threshold is reached. This threshold ensures that each segment still has sufficient timbre information after splitting.

[0033] If the current text group does not have strong prosodic boundaries or the conditions are not met after splitting, the original greedy merging result is retained, and subsequent audio segment mapping is continued.

[0034] Audio segment mapping: Based on the start and end positions of each character in the original text for each finalized text group, calculate its corresponding time interval in the original audio. At the boundaries of the time intervals, add a very short padding amount (e.g., 20-50 milliseconds before and after) to avoid audio truncation or popping due to segmentation boundaries. Then, extract the corresponding audio segments from the original audio based on the calculated time intervals. Specifically, the padding amount is calculated as follows: ,in This refers to the boundary fill amount. This represents the total duration of the original audio. The total number of characters in the original text represents the total number of characters. The ratio reflects the average duration of each character. 50 and 100 are the baseline padding coefficient and the maximum padding limit, respectively. This adaptive padding strategy ensures sufficient buffering at the boundaries to avoid truncation of the first and last phonemes due to segmentation errors, while also preventing excessive padding from introducing irrelevant noise.

[0035] Validity Filtering: Each extracted audio segment and its corresponding text are filtered. If the text length is less than the minimum length m, or the audio duration is less than the preset minimum length threshold (e.g., 1 second), the segment is discarded and not considered as a candidate. Finally, all filtered segments are output as the result of punctuation-aware greedy merging and segmentation.

[0036] Step 3: Enhancement and Dynamic Update. Neural network speech enhancement processing is performed on the selected reference audio, including noise reduction, dereverberation, and bandwidth expansion to eliminate the impact of environmental noise on cloning quality. Simultaneously, a linkage update mechanism is established between the sample pool and the cloning reference. When changes occur in the sample pool, such as additions or removals, the optimal reference is automatically reselected, achieving continuous evolution of the cloning reference. Specific steps are as follows: Neural network speech enhancement processing: After obtaining the optimal reference source material (audio segment) through steps one and two, neural network speech enhancement processing is forcibly performed on the audio segment before saving it to the cloned reference library; this processing flow includes: Noise Reduction and De-reverberation: The audio is sent to a pre-built neural network speech enhancement service. This service uses a deep neural network model to process the input audio in real time, effectively filtering out ambient noise and eliminating room reverberation, while preserving and restoring the speaker's original timbre and pronunciation details to the greatest extent possible.

[0037] Specifically, the neural network speech enhancement service adopts the Conv-TasNet architecture based on temporal convolutional networks, which includes: Model Structure: Encoder: Maps the input audio waveform to a high-dimensional feature representation through a one-dimensional convolutional layer, with a kernel size of 16 and a stride of 8; Separation Module: Consists of multiple repeated TCN blocks stacked together, each TCN block containing a depthwise separable convolution, a gated linear unit activation function, and a residual connection; The number of TCN blocks is set to 8, and the dilation factor increases exponentially (1, 2, 4, ..., 128) to expand the receptive field; Decoder: Reconstructs the enhanced audio waveform from the separated features through transposed convolution.

[0038] Input / Output: Input: Original audio clip, 16kHz sampling rate, mono; Output: Enhanced audio clip, sampling rate selectable at 16kHz or upsampled to 24kHz / 48kHz after bandwidth expansion.

[0039] Training data: Clean speech data uses open-source datasets such as LibriTTS or AISHELL-3; noise data uses the MUSAN noise library, which includes various types of environmental noise and human voice noise; reverberation data is generated by simulating room impulse response, with reverberation time T60 randomly sampled between 0.1s and 0.8s.

[0040] During training, clean speech is superimposed with noise and reverberation at random signal-to-noise ratios (-5dB to 20dB) to form a noisy input, with clean speech as the target output, and scale-invariant signal-to-noise ratio is used as the loss function for training.

[0041] Inference Deployment: The model is deployed at the edge or in the cloud after being optimized by TensorRT or ONNX; streaming chunking is used during inference, with a chunk length of 32ms and 50% overlap to achieve low latency enhancement.

[0042] Bandwidth expansion: For bandwidth-limited audio (such as telephone voice), further bandwidth expansion (super-resolution) processing is performed to increase the audio sampling rate to the high-quality specifications required by the synthesis engine (such as 24kHz or 48kHz), restore high-frequency details, and make the synthesized speech more transparent and natural.

[0043] Degradation handling: If the enhanced service becomes unavailable due to network or other reasons, the system will automatically degrade and use the original audio as a reference to ensure basic functionality availability.

[0044] The clone reference is updated in-pool as needed: The speech-enhanced audio and its corresponding text are persistently stored as the clone reference for the current speaker. Simultaneously, a linked update mechanism is established between the sample pool and the clone reference. This mechanism monitors all change events in the sample pool, including: Newly acquired audio clips are added to the pool; specifically, the sample pool self-purification mechanism is implemented according to the following rules: Triggering conditions: Timed trigger: Self-cleaning is triggered every 24 hours; Incremental trigger: Triggered when the number of newly added audio segments in the sample pool reaches a preset threshold (e.g., 50 segments); Capacity trigger: Triggered when the total number of segments in the sample pool exceeds the preset maximum capacity (e.g., 1000 segments).

[0045] Elimination Rules: Overall Quality Score Elimination: Calculate the overall quality score Q for each segment, and eliminate segments with scores below a preset threshold. Fragments with a value of 0.3 or higher are marked as pending elimination.

[0046] Redundancy elimination: For the same speaker, if there are multiple segments with highly similar text content (text similarity > 0.9), only the one with the highest overall quality score is retained, and the rest are eliminated.

[0047] Time-sensitive elimination: Fragments that have been stored for longer than the preset validity period (e.g., 90 days) and have never been selected as clone references will be eliminated first.

[0048] Retention Strategy: Currently used cloned reference segments are permanently retained during the self-cleaning process and are not subject to elimination. Each speaker retains at least one segment with the highest overall quality score to ensure the sample pool is not empty.

[0049] The sample pool self-cleaning mechanism eliminates low-quality fragments based on the new quality score; users can trigger clone target switching via voice commands. Whenever the sample pool undergoes the above changes and is persisted, a complete clone reference reselection process is automatically triggered; this mechanism ensures that the clone reference is always the currently optimal fragment in the sample pool, achieving automatic, real-time, and continuous optimization of the clone reference without any manual intervention.

[0050] The above formulas are all dimensionless calculations. Dimensionless calculations can be performed using various methods such as standardization, which will not be elaborated here. The formulas are derived from software simulations based on a large amount of collected data, and the preset parameters in the formulas can be set by those skilled in the art according to the actual situation.

[0051] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, ATA hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state ATA hard disk.

[0052] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0053] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0054] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0055] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0056] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0057] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable ATA hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech enhancement method based on a dynamic speaker sample pool, characterized in that, Includes the following steps: S1. Maintain a dynamic voiceprint sample pool for each speaker, the sample pool continuously collects audio segments generated by the speaker in daily interactions, and each audio segment is associated with its speech recognition text and comprehensive quality score; S2. Collect clone reference candidates from the sample pool. For segments whose text length is within the preset optimal length range, the entire segment is used as a candidate. For segments whose text length exceeds the preset maximum length, perform greedy merging and segmentation with punctuation awareness, and use the effective segments generated by the segmentation as candidates. S3. Sort all candidates in descending order of text length and then in descending order of overall quality score, and select the candidate ranked first as the original reference material; S4. Perform neural network speech enhancement processing on the selected original reference material to obtain enhanced cloned reference audio, which is used as reference input for the speech synthesis engine for timbre transfer.

2. The speech enhancement method based on dynamic speaker sample pooling according to claim 1, characterized in that, The overall quality score is calculated by weighted fusion of speech recognition confidence score, signal-to-noise ratio estimate score, and audio loudness score. Among them, the speech recognition confidence score directly adopts the posterior probability value output by the speech recognition model, the signal-to-noise ratio estimation score is obtained by calculating the energy ratio of the speech segment and the silent segment after detecting speech activity in the audio segment and normalizing it by the Sigmoid function, and the audio loudness score is obtained by calculating the root mean square energy of the audio segment and normalizing it to the optimal loudness range. The weighting coefficients of the weighted fusion are dynamically adjusted according to the application scenario.

3. The speech enhancement method based on dynamic speaker sample pooling according to claim 1, characterized in that, In S2: When the difference in the overall quality score between the first-ranked candidate and the second-ranked candidate is less than the preset fusion threshold, and the text lengths of both candidates are within the preset optimal length range, the multi-candidate fusion reference strategy is triggered. Specifically, it includes: After aligning the durations of the two candidate original audio files, they are linearly weighted and superimposed using the overall quality score as the weight to generate the fused audio. The merged text prompt is obtained by concatenating two candidate texts; if the total length after concatenation exceeds the preset maximum length, only the candidate text ranked first is retained.

4. The speech enhancement method based on dynamic speaker sample pooling according to claim 1, characterized in that, The greedy merging and segmentation based on punctuation awareness includes: The text is divided into several clauses using Chinese and English punctuation marks as separators, with each punctuation mark belonging to the clause preceding it. Starting from the first clause, greedily merge subsequent consecutive clauses in sequence, with the merging condition being that the total number of characters after merging does not exceed the preset maximum length; Based on the character position ratio, the corresponding audio segments are mapped from the original audio, and adaptive padding is added at the boundaries.

5. The speech enhancement method based on dynamic speaker sample pooling according to claim 4, characterized in that, After the greedy merging and grouping is completed, a dynamic segmentation optimization based on prosodic boundaries is introduced, as follows: For each finalized text group, a pre-trained prosodic boundary prediction model is used to predict the prosodic level of each clause boundary within it, and strong prosodic boundaries are used as candidate segmentation points. When the length of a text group is less than the preset maximum length and there is a strong rhythmic boundary, try to split it at the boundary. After splitting, the character length of each subgroup must not be less than the preset minimum length. Splitting is only allowed when the sum of the text lengths on both sides of a strong rhythm boundary exceeds the product of the splitting threshold and the preset maximum length.

6. The speech enhancement method based on dynamic speaker sample pooling according to claim 1, characterized in that, The neural network speech enhancement processing adopts the Conv-TasNet architecture based on a temporal convolutional network, specifically including: The encoder maps the input audio waveform into a high-dimensional feature representation through a one-dimensional convolutional layer; The separation module consists of multiple repeated TCN blocks stacked together, each TCN block containing depthwise separable convolutions, gated linear unit activation functions, and residual connections; The decoder reconstructs the enhanced audio waveform from the separated features using transposed convolution; The neural network speech enhancement processing includes noise reduction, dereverberation, and bandwidth expansion; if the enhancement service is unavailable, the original audio is used as a fallback.

7. The speech enhancement method based on dynamic speaker sample pooling according to claim 1, characterized in that, It also includes a pool-based update mechanism for cloned references, the process of which is as follows: The enhanced clone reference audio is persistently stored, and a linkage update mechanism is established between the sample pool and the clone reference. When a new audio segment is added to the sample pool, a low-quality segment is eliminated according to the preset elimination rules, or the user triggers a cloning target switch via voice command, S2 to S4 are automatically re-executed to update the cloning reference. The sample pool self-cleaning mechanism includes timed triggering, incremental triggering, and capacity triggering. The elimination rules include comprehensive quality score elimination, redundancy elimination, and timeliness elimination. Currently used cloned reference fragments are permanently retained during the self-cleaning process.

8. The speech enhancement method based on dynamic speaker sample pooling according to claim 1, characterized in that, It also includes a fallback strategy: When the candidate list is empty after traversing the entire sample pool, select the segment with the highest overall quality score in the sample pool, and extract an audio segment of the corresponding duration from the beginning of the segment's audio as reference material according to the ratio of the preset maximum text length to the total number of characters in the original text. If the extracted duration is less than the preset minimum duration, the preset minimum duration will be used directly as the reference material.

Images