A method, system and device for constructing and optimizing a mouth shape driving model data based on a cooperative pronunciation coverage

Abstract

The application discloses a mouth shape driving model data construction and optimization method, system and equipment based on cooperative pronunciation coverage, which is applied to the technical field of data processing, and carries out mouth shape driving data set construction around cooperative pronunciation. First, coverage features are extracted through association modeling, mode classification, and emotion-voice speed joint representation, and initial data sets are constructed by incorporating constraint, balance and complement rules. Then, the feature association is established through multi-dimensional mapping, clustering and distribution learning, and the coverage mode graph is generated by optimization strategy. Then, feature modeling is completed through completion, detection and prediction, and a gap mode vector is generated. Finally, self-adaptive generation and reinforcement learning complement denoising are relied on, feedback iterative optimization is combined, and a high-quality mouth shape driving data set is formed.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system, and device for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage. Background Technology

[0002] Current solutions for creating digital lip-syncing materials mainly employ the following methods: Real people select scripts based on experience for filming or recording, and then extract lip-sync frames or training data from these; TTS (Text-to-Speech) is used to generate audio of several common sentences, and then lip-syncing models are used to generate materials in batches; Frames are extracted from existing videos and labeled and organized according to phonemes or time intervals. These solutions generally share a common characteristic: the script design and sample selection lack clear phonetic coverage standards, mostly ensuring only "sufficient sentence quantity" rather than "adequate coverage of pronunciation combinations."

[0003] The existing technology has the following objective drawbacks: 1. There are too many high-frequency phoneme samples and insufficient low-frequency phoneme and long-tailed cophonic samples; 2. Although there is a lot of text, the coverage of pronunciation combinations that truly contribute to mouth shape is still incomplete; 3. In the materials constructed for lip-syncing models, complex co-pronunciation segments are often the most unstable, but the existing copy does not provide sufficient samples to address these challenges; 4. Variables such as emotion, speech rate, and pauses are often designed separately from pronunciation coverage, leading to an imbalance in distribution during subsequent codebook construction; 5. The long-tail problem of samples is prone to manifest as occasional jitter, incorrect switching of lip movements, and a sudden drop in confidence during the inference stage. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A method for constructing and optimizing lip-shape-driven model data based on cophonic coverage includes: extracting coverage features through cophonic association modeling, pronunciation pattern classification learning, and emotion-speed joint representation; incorporating coverage constraints, sample balancing rules, and long-tail completion strategies into the machine learning training paradigm to construct a lip-shape-driven pattern learning dataset; constructing feature association representations based on multi-dimensional feature space mapping, association pattern clustering, and coverage distribution learning; forming a cophonic coverage pattern graph through combinatorial optimization learning, greedy strategy decision-making, and multi-objective constraint optimization; completing feature modeling through a sparse feature completion model, abnormal pattern detection, and long-tail prediction learning; generating coverage gap pattern vectors through high-dimensional embedding representation and feature vector learning; inputting the pattern vectors into an adaptive generation learning model and a reinforcement learning decision optimization engine; completing sample completion and noise reduction through multi-constraint pattern coverage and context association learning to construct a lip-shape-driven model dataset based on coverage pattern learning; and completing adaptive optimization of the dataset by combining driving effect feedback learning, model generalization optimization, and coverage index regression iteration, outputting information on the construction and optimization of lip-shape-driven model data based on cophonic coverage.

[0005] A system for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage is provided. The system is used to execute executable instructions to perform the aforementioned method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage.

[0006] Its beneficial effects are as follows: First, coverage features are extracted through co-pronunciation association modeling, pronunciation pattern classification, and emotion-rate joint representation. A pattern learning dataset is then constructed by combining coverage constraints and balance rules. Next, a co-pronunciation coverage pattern map is generated through multi-dimensional feature mapping and association clustering. Gap pattern vectors are generated through sparse completion, anomaly detection, and long-tail prediction. These vectors are then input into an adaptive generation model to complete sample completion and noise reduction, constructing a lip-sync driven dataset. Finally, adaptive updates of the dataset are achieved through feedback learning and iterative optimization. This invention optimizes data construction across the entire co-pronunciation coverage process, covering multiple dimensions including phonemes, co-pronunciation, emotion, and rate of speech. It addresses the problems of incomplete coverage, long-tail jitter, and disjointed co-pronunciation in existing materials, and is suitable for real-time driven scenarios for digital humans.

[0007] This application aims to address the issues of high-frequency redundancy and insufficient long-tail speech by filling gaps in low-frequency sounds, rare coordination, and emotional speech rate. It significantly reduces lip-sync jitter and incorrect switching, standardizes the coordination rules of syllables, words, and sentences, ensuring natural and seamless lip-sync transitions that align with authentic pronunciation habits. Anomaly removal and noise reduction optimizations ensure physical compliance and consistent emotional rhythm, driving improvements in stability and naturalness. This application is suitable for broadcasting, dialogue, and emotional expression scenarios, requiring minimal manual parameter tuning and offering high deployment efficiency. Attached Figure Description

[0008] Figure 1A flowchart illustrating a method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a lip-shape driven model data construction and optimization system based on co-pronunciation coverage, provided in an embodiment of the present invention. Detailed Implementation

[0009] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Figure 1 This paper describes a method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage according to an exemplary embodiment of this application.

[0010] In this application embodiment, a method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage is provided, such as... Figure 1 As shown: S101 completes the extraction of coverage features through collaborative pronunciation association modeling, pronunciation pattern classification learning, and emotion-speed joint representation. It incorporates coverage constraints, sample balancing rules, and long-tail completion strategies into the machine learning training paradigm to construct a lip-shape-driven pattern learning dataset.

[0011] In one implementation, based on co-pronunciation association modeling, pronunciation pattern classification learning, and emotion-rate joint representation, a comprehensive feature extraction process is conducted to generate multi-dimensional comprehensive feature data covering phoneme sequences, co-pronunciation types, emotion tags, and rate tags. The associated data used in this process includes target scene corpus, phoneme frequency statistics, co-pronunciation statistics, and emotion and rate configuration data, all of which are the basic raw data for material production and truly reflect the characteristics of industry speech and pronunciation distribution patterns.

[0012] From the target scene corpus, phoneme frequency statistics, coarticulation statistics, and sentiment and speech rate configuration data, the corpus is first converted into standard phoneme sequences using text transcription rules. Then, coarticulation types are classified according to the relationship between preceding and following phonemes. Finally, preset sentiment and speech rate labels are matched. Relying on coarticulation association modeling, pronunciation pattern classification learning, and sentiment-speech rate joint representation, complete multi-dimensional feature data is extracted. The data fully includes all phoneme sequences, seven coarticulation types, clear sentiment labels, and graded speech rate labels. The feature dimensions are not reduced or simplified, and the pronunciation association and representation details are fully preserved, ensuring that the data is authentic, traceable, and reusable.

[0013] For example, when a customer service scenario sentence is input, "When will my order be shipped?", the system first breaks down the phoneme sequences q, i, n, g, d, a, n, h, e, s, h, i, f, a according to Chinese phoneme transcription rules. Then, based on the association between preceding and following phonemes, it identifies coprophonic types such as "qi" and "da" and matches them with predetermined categories such as consonant-vowel and vowel-consonant. Finally, it matches neutral sentiment labels and normal speech rate labels to generate complete multidimensional feature data, clearly labeling phoneme sequences, coprophonic categories, sentiment types, and speech rate levels, fully preserving the pronunciation associations and representation details of the sentence, with no missing data or format simplifications.

[0014] Coverage constraint determination is performed on multi-dimensional coverage feature data to generate six types of text batches: high-frequency core phonemes, cophony-specific features, long-tail phonemes, emotional expression, speech rate variation, and scenario-based FAQs. Four coverage targets are clearly defined: phoneme coverage, cophony coverage, emotional coverage, and speech rate coverage. Coverage constraint determination is also performed on multi-dimensional coverage feature data composed of phoneme sequences, cophony types, emotional tags, and speech rate tags. First, according to preset statistical criteria, the sample size and proportion of each type of pronunciation unit are calculated, and gaps with sample sizes below a set threshold are identified. Then, considering the degree of impact on pronunciation stability, gap priorities are assigned, ultimately generating six types of standard text batches. Four quantitative coverage targets are clearly defined, ensuring complete text types, clear and unambiguous coverage targets, and full traceability and reproducibility.

[0015] In the process of determining coverage constraints, statistics and verification are carried out based on multi-dimensional coverage feature data: the number of samples for each basic phoneme is counted, the frequency of each of the seven types of coarse articulations is counted, and the sample distribution of neutral, happy, serious, and surprised emotions and slow, normal, and fast speech rates is cross-statistically counted; a minimum sample threshold of 50 samples / class and a low-frequency proportion threshold of 10% are set, and phonemes, coarse articulations, emotions, and speech rates with sample sizes below the threshold are determined as coverage gaps. The gap levels are then divided according to the following priority: nasal-vowel, plosive-vowel, fricative / affricate-coarse articulations are the first priority, core vowels and consonants are the second priority, and rare long tails are the third priority, to ensure that the determination logic conforms to the requirements of pronunciation stability.

[0016] Based on the gap determination results, six types of standard copy batches are generated: high-frequency core phoneme copy, used to fill in high-frequency units such as a, o, e, b, p, d, t; collaborative pronunciation-specific copy, generated specifically for 7 types of collaborative gaps; long-tail phoneme completion copy, covering rare phonemes and low-frequency combinations; emotional expression copy, filling in low-frequency emotional gaps such as seriousness and surprise; speech rate variation copy, improving speech rate samples such as slow and fast; and scenario FAQ copy, tailored to high-frequency question scenarios in customer service, broadcasting, and other business operations.

[0017] Four quantitative coverage targets are clearly defined: basic phoneme coverage ≥ 95%, 7 types of collaborative pronunciation coverage = 100%, emotional coverage ≥ 95%, and speech rate coverage ≥ 95%. The coverage rate calculation formula is: Coverage Rate = (Number of covered units ÷ Total number of units) × 100%. The minimum sample requirements are: core phonemes ≥ 50, collaborative types ≥ 50, and emotional speech rate cross-sections ≥ 30. The target values ​​are specific, the judgment criteria are clear, and there are no vague expressions. For example, in the statistical analysis of multi-dimensional coverage feature data in customer service scenarios, there are only 28 samples of nasal-vowel collaborative pronunciation, which is lower than the minimum threshold of 50 and is judged as the first priority gap. Based on this, collaborative special copy such as "Good night," "Peace of mind," and "Safety" are generated. At the same time, clear targets of basic phoneme coverage ≥ 95% and collaborative pronunciation coverage = 100% are set. The gap judgment criteria, copy generation logic, and target quantitative standards are fully preserved to ensure traceability throughout the entire process.

[0018] Based on coverage targets and text batches, a material generation and quality screening model is constructed. This model generates three core coverage indicators: basic phoneme coverage, co-pronunciation type coverage, and the proportion of high-frequency / mid-frequency / low-frequency samples. Simultaneously, it verifies the clarity of pronunciation, accuracy of boundary annotation, and consistency of identity, matching the high-coverage production requirements of digital lip-syncing materials. Based on the defined four coverage targets (phonemes, co-pronunciation, emotion, and speech rate) and six text batches, a standardized material generation and quality screening model is constructed. The model consists of a material generation unit and a quality screening unit connected in series. Inputs include coverage target parameters, six text types, a unified benchmark portrait frame, and multi-emotion, multi-speed TTS audio. Outputs are compliant materials and the three core coverage indicators. The entire process is executed according to unified standards, and the results are quantifiable and traceable.

[0019] The material generation unit receives six types of text: high-frequency core phonemes, collaborative pronunciation features, long-tailed phonemes, emotional expressions, speech rate variations, and scenario-based FAQs. Combining different emotional and speech rate configurations, it calls the TTS module to generate corresponding audio. The audio is then bound to a lip-sync video clip with the same identity and the same baseline portrait frame. The generation process strictly adheres to input standards: the baseline portrait resolution is uniformly 512×512, and the format is RGB image; the TTS audio sampling rate is 16kHz, bit depth is 16bit, and it is mono, ensuring the consistency and standardization of the generated materials.

[0020] The quality screening unit performs multi-dimensional verification on the generated lip-sync videos and audios. All verification standards are quantified and executable: pronunciation clarity verification requires an audio signal-to-noise ratio ≥20dB, syllable duration controlled between 60 and 800ms, and no swallowing or blurring issues; boundary annotation accuracy verification requires phoneme and syllable boundary alignment confidence ≥0.6, with no obvious misalignment; identity consistency verification requires the generated face and the baseline portrait feature similarity ≥0.75, the average Euclidean distance of key points ≤15 pixels, and removes identity-drifting materials. All materials that fail verification are directly rejected, and qualified materials enter the indicator statistics stage.

[0021] After the material generation and screening are completed, three core coverage indicators are automatically calculated. The calculation methods and judgment criteria for these indicators are clearly defined: Basic phoneme coverage rate = (number of covered phoneme types ÷ total number of phoneme types) × 100%, with a target value of ≥95%; Coarse articulation type coverage rate = (number of covered 7 coarse articulation types ÷ 7) × 100%, with a target value of 100%; The proportion of high-frequency / mid-frequency / low-frequency samples is divided according to statistical intervals, with a low-frequency sample proportion of ≥10% considered acceptable. All indicators are comprehensive in dimensions and quantifiable in value, accurately matching the high coverage and high-quality production requirements of digital phonological materials. For example, the text "Goodnight, rest assured" was selected for collaborative pronunciation, and a neutral emotional tone and normal speaking speed TTS audio was generated with a sampling rate of 16kHz and a duration of about 2 seconds. A lip-sync video was generated by binding a 512×512 baseline portrait frame. The audio signal-to-noise ratio was verified to be 25dB, the boundary confidence score was 0.8, and the identity similarity was 0.82, all of which met the standards. The statistics showed that the basic phoneme coverage rate was 96%, the collaborative pronunciation coverage rate was 100%, and the low-frequency sample ratio was 12%, all of which met the standards. The materials were compliant and valid, and the entire process of generation, verification, and indicator statistics was fully preserved.

[0022] Integrating coverage features, copywriting batches, and coverage indicator data, a coverage matrix is ​​generated to support the closed-loop iterative completion of the digital phonology material library. The coverage matrix is ​​constructed from existing multi-dimensional coverage features, six types of copywriting batches, three types of core coverage indicators, and compliant material data. The coverage matrix uses phonemes, co-pronunciation, emotion, and speech rate as its core dimensions. It horizontally lists various pronunciation units, emotion types, and speech rate levels, and vertically labels the number of samples, coverage rate, and compliance status. The matrix is ​​comprehensive in dimensions, clearly hierarchical, and the data is authentic and unaltered, fully presenting the full-dimensional coverage status and sample distribution, providing accurate data support for closed-loop iterative completion.

[0023] The construction process strictly follows data integration rules: First, basic phoneme data is compiled, the number of valid samples for each phoneme is counted, and the coverage rate is calculated using the formula: Phoneme Coverage Rate = (Number of phoneme types present ÷ Total number of target phonemes) × 100%. ≥95% is marked as meeting the standard, and <95% is marked as a gap. Second, seven types of co-pronunciation data are compiled, the number of samples for each type is counted, and the coverage rate is calculated using the formula: Co-pronunciation Coverage Rate = (Number of covered types ÷ 7) × 100%. 100% is marked as meeting the standard, and insufficient coverage is marked as a gap. Third, neutral, happy, serious, and surprised emotional data, as well as slow, medium, and fast speech rates, are compiled, the number of cross-samples is counted, and the emotional speech rate coverage rate is calculated using the formula: Emotional Speech Rate Coverage Rate = (Number of filled cells ÷ Total number of cross-samples) × 100%. 100% is marked as meeting the standard. Fourth, high-frequency, medium-frequency, and low-frequency sample proportion data are integrated. Low-frequency proportion = Low-frequency sample count ÷ Total sample count × 100%. ≥10% is considered acceptable. Finally, all data is filled into a matrix according to dimensions to ensure data traceability and verifiability.

[0024] For example, when compiling compliance material data for customer service scenarios, the basic phonemes show 32 samples for "an" (92% coverage, a gap), 68 samples for "a" (98% coverage, meeting the standard), 28 samples for nasal-vowel co-pronunciation (71% coverage, a gap), 15 samples for serious-slow cross-pronunciation (83% coverage, a gap), and low-frequency components accounting for 8% (a gap). By filling these data into a matrix, clearly labeling the gap items, sample values, coverage, and compliance status, and fully preserving the data source, statistical methods, and matrix construction details, the matrix accurately presents shortcomings such as low-frequency sounds and easily jittered co-pronunciation, providing a clear direction for subsequent copywriting completion and material regeneration.

[0025] S102 constructs a feature association representation based on multi-dimensional feature space mapping, association pattern clustering, and coverage distribution learning. It then forms a collaborative pronunciation coverage pattern graph through combinatorial optimization learning, greedy strategy decision-making, and multi-objective constraint optimization.

[0026] In one implementation, combining the requirements of co-pronunciation coverage analysis with the logic of constructing multi-dimensional feature associations, feature space mapping, association pattern clustering, and coverage distribution learning mechanisms are introduced to determine standardized processing schemes for feature mapping, pattern clustering, coverage analysis, and pattern graph generation. Based on the requirements of co-pronunciation coverage analysis using digital lip-sync materials, and the logic of constructing multi-dimensional feature associations for phonemes, co-pronunciation, emotion, and speech rate, three core processing mechanisms—feature space mapping, association pattern clustering, and coverage distribution learning—are introduced. Standardized processing schemes for the four stages—feature mapping, pattern clustering, coverage analysis, and pattern graph generation—are clarified, unifying process steps, judgment rules, and parameter standards to ensure that the entire process is reproducible, traceable, and quantifiable.

[0027] In the feature space mapping stage, a structured high-dimensional feature space is constructed using the original phoneme sequence, seven types of coarse articulation, emotion tags, and speech rate tags as input. The mapping rule follows the principle of "one unique feature vector corresponding to each articulation unit," mapping each basic phoneme to a 128-dimensional feature vector, each type of coarse articulation to a 256-dimensional feature vector, and a single emotion / speech rate tag to a 64-dimensional feature vector. This completes the dimensional alignment and association binding of multi-dimensional features, avoiding feature dimensional confusion and broken associations.

[0028] In the association pattern clustering stage, an unsupervised clustering algorithm is used to group the mapped high-dimensional features. The core clustering rule is "pronunciation pattern similarity ≥ 0.8 belongs to the same class," dividing the features into four categories: high-frequency core pronunciation, low-frequency long-tail pronunciation, emotional special pronunciation, and speech rate special pronunciation. The number of clusters is fixed at 8, covering all pronunciation patterns, with each class having ≥ 30 samples to ensure balanced clustering results and no extremely sparse classes.

[0029] In the coverage analysis phase, the sample proportion and gap situation of various pronunciation modes are quantified according to preset criteria. The core calculation formula is: Mode coverage rate = (Number of effective samples for this mode ÷ Minimum required sample number for this mode) × 100%. The minimum required sample number is set as follows: High frequency mode ≥ 100 samples, Low frequency mode ≥ 50 samples, Emotion / Speed ​​mode ≥ 30 samples. A coverage rate < 100% is identified as a gap, accurately pinpointing the shortcomings.

[0030] In the pattern diagram generation stage, feature mapping results, clustering grouping information, and coverage gap data are integrated. Greedy decision-making and multi-objective constraint optimization are adopted to prioritize the labeling of high-impact gaps such as nasal-vowel and plosive-vowel. The pattern diagram is presented in layers according to phonemes, coarticulation, emotion, and speech rate to generate a standardized coarticulation coverage pattern diagram. This diagram intuitively displays the pronunciation distribution, gap location, and priority, providing clear guidance for subsequent completion. For example, the creation of digital phonograph materials for customer service scenarios uses phonemes, 7 types of co-pronunciation, neutral / happy / serious emotions, and slow / normal / speech speed as core dimensions, and is executed according to a standardized scheme: First, the phonemes "an" and "ang" are mapped to 128-dimensional vectors, the nasal-vowel co-pronunciation is mapped to 256-dimensional vectors, and the neutral emotion is mapped to 64-dimensional vectors; Second, clustering is performed to obtain 8 groups of pronunciation patterns, such as the high-frequency "a / o / e" group and the low-frequency "ang / ing" group; Third, the low-frequency pattern coverage rate is calculated to be 65%, which is below 100%, and is judged as a gap; Fourth, a pattern diagram is generated, marking the nasal-vowel gap as the highest priority, fully presenting the pronunciation distribution and gap details, with standardized process and quantifiable results.

[0031] Perform spatial mapping on the multi-dimensional features of phonemes, co-articulation, emotion, and speech rate to construct a high-dimensional feature space representation. Combining the actual requirements of co-articulation coverage analysis in the digital human mouth shape material production scenario, as well as the intrinsic association construction logic of the four types of multi-dimensional features including phonemes, co-articulation, emotion, and speech rate, the system introduces three core processing mechanisms: feature space mapping, association pattern clustering, and coverage distribution learning, clarifies the standardized processing schemes for four key links: feature mapping, pattern clustering, coverage analysis, and pattern graph generation, unifies the process steps, judgment rules, calculation calibers and parameter thresholds, ensures the entire process is reproducible, traceable, and quantifiable, and supports the accurate construction of co-articulation coverage patterns.

[0032] Feature space mapping takes the original phoneme sequence, 7 types of co-articulation, emotion labels, and speech rate labels as basic inputs, establishes conversion rules from original features to high-dimensional structured features, and realizes feature dimension unification and association binding. Mapping rules: basic phonemes are mapped to 128-dimensional feature vectors, 7 types of co-articulation (vowel-vowel, vowel-consonant, consonant-vowel, nasal-vowel, plosive-vowel, fricative / affricate-vowel, lateral / approximant-vowel) are respectively mapped to 256-dimensional feature vectors, the four types of emotions of neutral, happy, serious, and surprised, and the three levels of speech rate of slow, normal, and fast are each mapped to 64-dimensional feature vectors, so that the mapped feature dimensions are aligned and the association relationships are completely retained. For example, for the phoneme "n" of the character "nin" (you) in the input customer service corpus, the "n-i" nasal-vowel co-articulation, neutral emotion, and normal speech rate are respectively mapped to 128-dimensional, 256-dimensional, 64-dimensional, and 64-dimensional feature vectors according to the rules, completing the conversion from original features to the high-dimensional space, with standardized feature dimensions and no information loss.

[0033] Association pattern clustering adopts an unsupervised clustering algorithm to perform grouping processing on the mapped high-dimensional features, divides association patterns according to pronunciation similarity, and accurately distinguishes the distribution characteristics of different pronunciation combinations. The clustering rules are set as follows: the number of clusters is 8, the pronunciation similarity threshold is ≥ 0.8, the similarity is calculated by the cosine distance of feature vectors, and the formula is: similarity = 1 - (cosine value of the included angle of the feature vectors), and a similarity ≥ 0.8 is judged as the same association pattern; after clustering, the number of samples in each category shall be ≥ 30 to avoid extremely sparse categories affecting subsequent analysis. For example, by clustering the mapped high-dimensional features, 8 patterns are divided: high-frequency core pronunciation, low-frequency long-tail pronunciation, nasal-vowel special pronunciation, plosive-vowel special pronunciation, fricative / affricate-vowel special pronunciation, lateral / approximant-vowel special pronunciation, emotion special pronunciation, and speech rate special pronunciation, each with a sample size of ≥ 30, the clustering results are balanced and conform to the pronunciation distribution law.

[0034] Coverage distribution learning, based on clustering results, quantifies the sample proportion and coverage gap of various pronunciation patterns, and uses standardized methods to calculate coverage rate, accurately locating shortcomings. The coverage calculation formulas are: Phoneme coverage rate = (Number of covered phoneme types ÷ Total target phonemes) × 100%, with a threshold of ≥95%; Cophony coverage rate = (Number of covered types ÷ 7) × 100%, with a threshold of 100%; Emotion / Speed ​​coverage rate = (Number of filled cross cells ÷ 12) × 100%, with a threshold of ≥95%; Low-frequency proportion = (Number of low-frequency samples ÷ Total number of samples) × 100%, with a threshold of ≥10%. For example, after clustering, the nasal-vowel pattern has 28 samples and a minimum requirement of 50, resulting in a coverage rate of (28 ÷ 50) × 100% = 56% < 100%, which is identified as a high-priority coverage gap; the low-frequency pronunciation sample proportion is 8% < 10%, which is identified as a long-tail gap. The gap location is accurate, and the data is quantifiable.

[0035] By integrating feature mapping results, clustering information, and coverage gap data, a greedy strategy and multi-objective constraints are used to optimize and generate a co-pronunciation coverage pattern diagram, intuitively presenting the relationship between pronunciation distribution and gaps. Generation rules: Prioritize labeling high-impact gaps such as nasal-vowel, plosive-vowel, and fricative / affricate consonants, labeling them hierarchically according to phoneme, co-pronunciation, emotion, and speech rate. Gap priority is sorted by its impact on mouth shape stability, with high-priority gaps displayed at the top. The pattern diagram must label four core pieces of information: gap type, sample quantity, coverage rate, and priority, ensuring data accuracy and clear relationships. For example, by summarizing feature mapping, clustering, and coverage data, priority is given to labeling nasal-vowel gaps (coverage rate 56%, priority 1) and plosive-vowel gaps (coverage rate 62%, priority 2), presenting the coverage status hierarchically by phoneme, co-pronunciation, emotion, and speech rate, generating a fully labeled and clearly related co-pronunciation coverage pattern diagram, providing precise guidance for subsequent text completion.

[0036] For digital speech mapping content creation scenarios, this approach focuses on phonemes, co-pronunciation, emotion, and speech rate as core dimensions. It strictly adheres to the complete process of "feature mapping - clustering - coverage analysis - pattern diagram generation," with unified processing standards, calculation methods, and parameter thresholds at each stage to avoid confusion and inconsistent standards. This ensures compliance and reliability of the entire co-pronunciation coverage analysis process.

[0037] Based on the association pattern clustering algorithm, the mapped high-dimensional features are grouped and processed. Combined with the coverage distribution learning method, the sample proportion and coverage gap of different pronunciation patterns are quantified, achieving accurate classification and gap location of pronunciation patterns, providing a clear basis for subsequent supplementation. Association pattern clustering uses similarity matching rules, with pronunciation feature consistency as the core, setting a similarity threshold ≥0.8 to divide the mapped high-dimensional features into different pronunciation pattern groups. During clustering, priority is given to matching co-pronunciation association features, while also considering the correlation of phonemes, emotion, and speech rate dimensions, ensuring that the pronunciation patterns in the same group have consistent features and are closely related. After clustering, each group must have ≥30 samples to avoid sample sparsity affecting subsequent statistical accuracy.

[0038] Coverage distribution learning follows standardized criteria, statistically analyzing the sample size, percentage, and coverage rate for each pronunciation pattern. The coverage rate is calculated using the formula: Pattern Coverage Rate = (Number of valid samples for that pattern ÷ Minimum required sample size for that pattern) × 100%. Minimum required sample size is set as follows: High-frequency patterns ≥ 100, Low-frequency patterns ≥ 50, and Emotion / Speed ​​patterns ≥ 30. Coverage rates < 100% are considered coverage gaps, and the severity of these gaps is graded according to the number of samples lacking. For example, high-dimensional feature clustering after mapping customer service scenarios resulted in eight patterns: high-frequency core pronunciation, low-frequency long-tail pronunciation, nasal-vowel special pronunciation, plosive-vowel special pronunciation, fricative / affricate-vowel special pronunciation, lateral / vowel-vowel special pronunciation, emotional special pronunciation, and speech speed special pronunciation. Statistical sample proportions: high-frequency patterns accounted for 65%, low-frequency patterns accounted for 8%, and there were 28 samples of the nasal-vowel pattern. Calculated coverage rates: low-frequency pattern coverage = (28 ÷ 50) × 100% = 56%, and serious-slow emotional speech speed pattern coverage = (12 ÷ 30) × 100% = 40%. The nasal-vowel coordination and serious emotional speech speed were accurately identified as high-priority coverage gaps, with clear data gaps and distinct hierarchical classifications.

[0039] By integrating feature mapping results, clustering patterns, and coverage data, a co-pronunciation coverage pattern map is generated through combinatorial optimization learning, greedy strategy decision-making, and multi-objective constraint optimization. After completing feature space mapping, association pattern clustering, and coverage quantification, the three types of core data are integrated, and a strategy combining combinatorial optimization, greedy decision-making, and multi-objective constraint optimization is adopted to generate a complete co-pronunciation coverage pattern map. This map intuitively presents the full-dimensional pronunciation distribution, sample size, coverage rate, and gap associations, providing clear guidance for subsequent fill-in-the-blank exercises.

[0040] The integrated data includes: First, four types of feature mapping results: phonemes, coarticulation, emotion, and speech rate, i.e., high-dimensional feature vectors corresponding to each articulation unit; Second, clustering information for eight types of articulation patterns, including high-frequency core, low-frequency long tail, nasal-vowel, plosive-vowel, fricative / affricate-vowel, lateral / vowel-vowel, emotion-specific, and speech rate-specific patterns; Third, coverage gap data for each type of pattern, including sample size, coverage rate, gap value, and priority ranking.

[0041] Pattern generation follows standardized rules: First, a priority rule, sorting by impact on lip shape stability, with nasal-vowel, plosive-vowel, and fricative / affricate-vowel having the highest priority and being marked and displayed first; second, a greedy strategy, prioritizing the completion of combinations that can cover 2-4 gaps in a single sentence to maximize completion efficiency; third, multi-objective constraints, simultaneously balancing four coverage objectives: phonemes, coordination, emotion, and speech rate, avoiding over-optimization of a single dimension; and fourth, standardized annotation, with the pattern diagram horizontally listing 7 coordination types, emotion, and speech rate dimensions, and vertically annotating the number of samples, coverage rate, gap value, and priority, ensuring data authenticity and traceability.

[0042] Coverage calculation formula: Pattern coverage = (Number of valid samples for this pattern ÷ Number of minimum required samples for this pattern) × 100%. Minimum requirements are ≥100 for high-frequency patterns, ≥50 for low-frequency patterns, and ≥30 for emotional / speed patterns. Coverage < 100% is marked as a gap, with the gap value = minimum requirement - number of valid samples. For example, summarizing customer service scenario data: High-frequency core pronunciation pattern samples: 120, coverage 100% (meets the standard); Nasal-vowel pattern samples: 28, minimum requirement 50, coverage = (28 ÷ 50) × 100% = 56%, gap 22, priority 1; Plosive-vowel pattern samples: 35, coverage 70%, gap 15, priority 2; Serious-slow emotional speed pattern samples: 12, coverage 40%, gap 18, priority 3. Prioritize and label nasal-vowel and plosive-vowel gaps, balance phoneme and emotional coverage targets, generate pattern diagrams, horizontally label 7 types of coordination, 4 types of emotion, and 3 speech rates, and vertically label sample number, coverage rate, gap value, and priority, fully presenting pronunciation distribution, gap location and correlation, providing accurate basis for copywriting completion.

[0043] S103 completes feature modeling through sparse feature completion model, abnormal pattern detection, and long-tail prediction learning, and generates a covering gap pattern vector through high-dimensional embedding representation and feature vector learning.

[0044] In one implementation, a feature modeling process is performed on multi-dimensional features of phonemes, coarticulation, emotion, and speech rate, relying on a technical chain of sparse feature completion, abnormal pattern detection, and long-tail prediction learning. This generates feature modeling data that is feature-complete, pattern-standardized, and dimension-matched. By leveraging this complete technical chain, feature modeling is performed on four basic data types—phonemes, coarticulation, emotion, and speech rate—to generate feature modeling data that is feature-complete, pattern-standardized, and dimension-matched. This ensures the reliability and usability of the basic features and provides compliant input for subsequent gap pattern vector generation.

[0045] Before modeling, the input data standards are standardized: phoneme data covers all basic units of Chinese, co-pronunciation data includes 7 core types: vowel-vowel, vowel-consonant, consonant-vowel, nasal-vowel, plosive-vowel, fricative / affricate-vowel, lateral / vowel-vowel, emotion data covers 4 categories: neutral, happy, serious, and surprised, and speech rate data is divided into 3 levels: slow, normal, and fast, to ensure that the input dimensions are complete and the standards are consistent.

[0046] The modeling process is executed in three steps: The first step is sparse feature completion, which involves completing low-frequency phonemes and long-tailed coprolalia based on phonetic syllable combination rules for missing samples. The completion threshold is set at a minimum of 30 samples for a single phoneme and a minimum of 25 samples for a single coprolalia type. Completion is automatically triggered when the number of samples falls below the threshold. The second step is abnormal pattern detection, which removes invalid features such as pronunciation errors, boundary misalignments, and emotional-pace mismatches. The abnormality criteria are phoneme alignment confidence < 0.6 and emotional-pace matching < 0.7. Unqualified data are directly removed. The third step is long-tail prediction learning, which predicts reasonable representations of low-frequency and rare pronunciation features based on the distribution patterns of high-frequency features, ensuring that long-tail features are not missing and that the representations are standardized. For example, the input customer service scenario corpus data includes 7 categories of co-pronunciation, neutral / serious / happy emotions, and slow / medium / fast speech rates. Among them, there are only 12 samples of the "eng" phoneme and 18 samples of nasal-vowel co-pronunciation, which are below the completion threshold. After sparse completion, the "eng" phoneme is increased to 35 samples and the nasal-vowel co-pronunciation is increased to 28 samples. Abnormal data with an alignment confidence of 0.5 and an emotion-speech rate matching degree of 0.6 are removed. Then, the rare "iong" phoneme features are supplemented through long-tail prediction. Finally, the feature modeling data with no missing parts, uniform format, and dimension alignment is output to ensure the accuracy of subsequent gap analysis.

[0047] Integrating five core elements—sparse completion, anomaly detection, long-tail prediction, high-dimensional embedding, and feature vectors—this approach aims to achieve feature integrity, pattern discriminability, and dimensional adaptability. It constructs standardized feature modeling constraints, clarifies modeling rules, judgment criteria, and parameter thresholds, and ensures that the modeling process is compliant and the results are standardized, providing a unified standard for subsequent gap pattern vector generation.

[0048] To address the feature integrity constraint, the modeling data must cover all basic phonemes in Chinese, 7 types of co-pronunciation, 4 types of emotion, and 3 levels of speech speed, with no missing dimensions; the minimum number of samples for a single phoneme, single co-pronunciation type, and single emotion speech speed combination must be no less than 30. If the number of samples is less than the threshold, it is considered incomplete and triggers the completion process.

[0049] To address the pattern discrimination constraint, a similarity threshold of <0.8 is set for pronunciation patterns. Similarity is calculated using the cosine distance of feature vectors. Patterns with a similarity ≥0.8 are classified as similar to other patterns, while those <0.8 are classified as dissimilar patterns. This ensures that high-frequency, low-frequency, emotional, and speech rate patterns can be clearly distinguished. Dissimilar patterns have a feature difference value ≥0.2. Patterns with a feature difference value below this value are considered to have insufficient discrimination and require feature optimization.

[0050] To address dimensionality constraints, the phoneme feature dimensions are unified to 128, the collaborative feature dimensions to 256, and the emotion / speed feature dimensions to 64, with a dimensional alignment error of <5%. After multi-feature fusion, there are no dimensional conflicts and the format is unified, ensuring that subsequent embedding and learning can be performed stably.

[0051] To address anomaly detection constraints, clear criteria for judging abnormal pronunciation are established. Phoneme alignment confidence < 0.6, syllable duration < 60ms or > 800ms, and emotional speech rate matching < 0.7 are all considered abnormal patterns and are directly removed to avoid invalid data interfering with modeling. For example, when constructing constraints for customer service scenario feature modeling, the following rules are set: feature completeness must cover all phonemes and 7 types of collaborations, with no fewer than 30 single nasal-vowel collaboration samples; pattern discrimination similarity threshold of 0.8, with a difference between high-frequency and low-frequency patterns ≥ 0.2; dimension adaptation is unified with 128 phoneme dimensions and 256 collaboration dimensions; anomaly detection removes invalid data with alignment confidence 0.5 and duration 900ms, forming complete constraints, standardizing the entire modeling process, and ensuring that the output data is compliant and usable.

[0052] Based on a sparse feature completion model, anomaly detection mechanism, long-tail prediction learning strategy, high-dimensional embedding representation method, and feature vector learning algorithm, this system completes sparse feature completion, anomaly pattern removal, long-tail trend capture, high-dimensional feature mapping, and feature vector learning to form highly reliable feature optimization information. The sparse feature completion model uses low-frequency phonemes and long-tail co-pronunciation as completion targets. According to phonetics syllable combination rules, it sets a minimum sample threshold of 30 for a single phoneme and 25 for a single co-pronunciation type. Completion is triggered when the sample count falls below the threshold. The completion method uses interpolation of high-frequency related features from the same scene to ensure that the completed features conform to pronunciation rules and are naturally readable.

[0053] The abnormal pattern detection mechanism verifies the features from three dimensions: pronunciation, boundary, and matching. Pronunciation anomalies are determined by an audio signal-to-noise ratio <20dB and syllable duration <60ms or >800. Boundary anomalies are determined by phoneme alignment confidence <0.6. Matching anomalies are determined by emotional speech rate matching <0.7. Any feature meeting any of these conditions is directly removed, ensuring that features are authentic, effective, and free from invalid interference. The long-tail prediction learning strategy is based on the distribution patterns of high-frequency pronunciation features, constructing a probabilistic prediction model to predict reasonable representations of low-frequency, rare pronunciation features. A long-tail feature similarity threshold of ≥0.75 is set to ensure that the predicted features are consistent with the actual pronunciation without bias or distortion.

[0054] The high-dimensional embedding representation method maps according to fixed dimensions. Specifically, the basic phoneme mapping is 128 dimensions, the 7 types of coarticulation mapping is 256 dimensions, and the emotion / rate mapping is 64 dimensions, with a dimension alignment error of <5%, ensuring that the feature dimensions are consistent and can be matched. The feature vector learning algorithm adopts supervised learning training, using pronunciation category as label and feature similarity as loss function, with ≥50 training iterations and a convergence threshold of <0.01, outputting standardized feature vectors to ensure high feature discrimination and adaptability to model input.

[0055] For example, in the input customer service scenario modeling data, there were 12 "eng" phoneme samples and 18 nasal-vowel co-consonant samples, which were below the completion threshold. After sparse completion, the number of samples increased to 35 and 28 respectively. Abnormal data with a signal-to-noise ratio of 18dB, alignment confidence of 0.5, and sentiment matching of 0.6 were removed. Rare pronunciation features of "iong" were supplemented through long-tail prediction. The phoneme was mapped to 128 dimensions, the co-consonant to 256 dimensions, and the sentiment to 64 dimensions. After 50 rounds of training and convergence, a standardized feature vector was output, resulting in optimized features that are free of missing features, abnormal features, dimensionally adapted, and highly discriminative. The feature quality meets the requirements for subsequent modeling.

[0056] By integrating the full-chain processing results of sparse feature completion, abnormal pattern detection, long-tail prediction learning, high-dimensional embedding representation, and feature vector learning, a standardized feature fusion and vector generation process is constructed. The final output is a coverage gap pattern vector adapted to digital lip-shape materials, which accurately and completely presents the type, location, quantity, and priority of various pronunciation gaps, providing a clear basis for subsequent accurate text completion and targeted material generation.

[0057] First, the system aggregates complete phoneme features after sparse completion, clean co-articulation features after anomaly removal, and rare articulation features after long-tail prediction. Then, it performs dimensional alignment through high-dimensional embedding (128 dimensions for phonemes, 256 dimensions for co-articulation, and 64 dimensions for sentiment / rate). Finally, it performs pattern differentiation through feature vector learning, ensuring that the fused features are conflict-free, closely related, and highly discriminative. The pattern vectors are constructed according to a three-layer structure of gap type, sample gap quantity, and priority, with dimensions that perfectly match the front-end modeling data, allowing direct input to the completion module.

[0058] The gap amount is calculated using a unified standard: gap amount = minimum sample threshold - number of existing samples. The minimum threshold for single phoneme / single co-pronunciation is 30 samples, and for emotional speech rate cross-pronunciation, it is 25 samples. Priorities are ranked according to their impact on lip shape stability: nasal-vowel and plosive-vowel are level 1 (highest), core phonemes are level 2, and emotional speech rate long tails are level 3, ensuring accurate gap labeling and clear grading. For example, after integrating optimized features from customer service scenarios, the basic sound "eng" has 12 existing samples, a gap of 30-12=18 samples, and a priority of level 2; nasal-vowel co-pronunciation has 18 existing samples, a gap of 30-18=12 samples, and a priority of level 1; and serious-slow cross-pronunciation has 10 samples, a gap of 25-10=15 samples, and a priority of level 3. By integrating the above data to generate gap pattern vectors, the types, quantities, and priorities of the three types of gaps are clearly marked, and the distribution details of low-frequency sounds, easily jittery collaborative pronunciation, and emotional speech rate long tails are fully presented. These vectors can be directly used for subsequent targeted copywriting generation and material supplementation, and the data across the entire chain is traceable and reproducible.

[0059] S104 inputs the pattern vector into the adaptive generative learning model and the reinforcement learning decision optimization engine. It completes sample completion and noise reduction through multi-constraint pattern coverage and contextual association learning, and constructs a lip-shape driven model dataset based on coverage pattern learning.

[0060] In one implementation, nine types of data are used as core inputs: target industry corpus, phoneme sequences, co-pronunciation types, phoneme frequencies, sentiment annotation, speech rate annotation, coverage targets, gap lists, and baseline portraits. Following a standardized business process, the entire chain of processing is carried out sequentially, including phoneme statistics, co-pronunciation classification, coverage target definition, gap identification, text generation, TTS synthesis, portrait frame matching, alignment annotation, and quality screening. This results in a complete, compliant, and directly usable lip-sync driven dataset for model training. The entire data flow is traceable, the processing rules are reproducible, and the output results are verifiable.

[0061] The target industry corpus is limited to standard scripts used in customer service, broadcasting, and other similar fields, with each sentence containing 5-15 characters and excluding rare or polyphonic characters. Phoneme sequences are derived from standard Chinese phoneme decomposition, covering all basic vowels and consonants with unambiguous decomposition. Coarticulation types strictly correspond to seven standard coarticulation categories: vowel-vowel, vowel-consonant, consonant-vowel, nasal-vowel, plosive-vowel, fricative / affricate-vowel, and lateral / vowel-vowel. Sentiment labeling is fixed as neutral and cheerful. Four standard emotional categories: serious, surprised, etc.; speech rate is marked as follows: slow 120 words / min, normal 180 words / min, fast 240 words / min, etc.; basic phoneme coverage ≥95%, coordination coverage = 100%, emotion / speech rate coverage ≥95%; phonemes, coordination, and emotion speech rate items with a sample size lower than the minimum threshold of 50 items per class are marked; the baseline portrait is a 512×512 resolution RGB standard image, and the same batch has the same identity, lighting, and posture.

[0062] Based on the basic Chinese phoneme list, the frequency of each phoneme was analyzed sentence by sentence and character by character. The statistical standard was that the same phoneme was counted only once within a single sentence to avoid duplicate counting. A phoneme frequency statistics table was generated, clearly specifying the sample quantity and proportion of each phoneme, providing basic data for coverage determination. A binary judgment rule for front and back phonemes was adopted, with priority as follows: nasal > plosive > fricative > lateral > vowel > others, dividing the phoneme sequence into 7 standard coarticulations. A coarticulation classification table was generated, indicating the sample quantity and proportion of each category to ensure no overlap or omission in classification.

[0063] The basic phoneme coverage rate is calculated as follows: Coverage rate = (Number of covered phoneme types ÷ Total number of target phonemes) × 100%, with a threshold of ≥ 95%; the collaborative coverage rate is calculated as follows: Coverage rate = (Number of covered types ÷ 7) × 100%, with a threshold of 100%; the emotion / speed coverage rate is calculated as follows: Coverage rate = (Number of filled cross cells ÷ 12) × 100%, with a threshold of ≥ 95%; the three coverage rate standards are clearly defined, and the coverage qualification boundary is quantified.

[0064] Comparing statistical results with the minimum threshold of 50 samples for each category, those below the threshold are marked as gaps. Gap priority is ranked as follows: nasal-vowel and plosive-vowel are level 1, core phonemes are level 2, and emotional speech rate is level 3. A gap priority list is generated, marking the gap type, gap quantity, and priority to accurately pinpoint weaknesses. Four fixed templates are used: phrase, short sentence, FAQ, and continuous coverage, with each sentence covering 2-4 gaps. Adhering to hard constraints on pronunciation, length, emotional speech rate, and volume, compliant supplementary text is generated to ensure suitability for the supplementary needs. The audio format is WAV, with a sampling rate of 16kHz, a bit depth of 16bit, and mono. Matching the text with corresponding emotion and speech rate generates natural and readable audio without punctuation errors, pronunciation mistakes, or abnormal connected speech, ensuring audio quality. A unified 512×512 baseline portrait is used to ensure consistent facial structure, lighting, and posture when generating lip-sync videos. The portrait source is unified to avoid identity drift and style confusion, ensuring consistent lip-sync. Generate phoneme-level and syllable-level time boundaries; ensure alignment confidence ≥ 0.6 and boundary error ≤ 20ms to guarantee audio-visual synchronization, accurate annotation, and no misalignment or offset. Pronunciation clarity is verified as an audio signal-to-noise ratio ≥ 20dB and syllable duration 60–800ms; boundary accuracy is verified as a phoneme alignment confidence ≥ 0.6; identity consistency is verified as facial feature similarity ≥ 0.75 and keypoint displacement ≤ 15 pixels; materials meeting any of these criteria are directly discarded to ensure compliance and usability.

[0065] For example, inputting 1000 customer service industry corpora, corresponding standard phoneme sequences, 7 collaborative classifications, neutral / happy emotions, slow / normal speaking speed, a coverage target of ≥95%, a list of nasal-vowel gaps, and a 512×512 unified benchmark portrait; sequentially counting the "a" phoneme 680 times and the "eng" phoneme 28 times, classifying into 7 collaborative categories, setting a coverage threshold, identifying the nasal-vowel level 1 gap, generating the copy "Goodnight, I love you", synthesizing neutral normal speaking speed audio, binding to the benchmark portrait, marking phoneme syllable boundaries, removing abnormal materials with a signal-to-noise ratio of 18dB / similarity of 0.7, and finally outputting compliant lip-sync material, with clear rules, specific values, and a closed-loop process throughout.

[0066] The input data is categorized, organized, and matched according to five dimensions: phoneme coverage, coprophony coverage, emotion coverage, speech rate coverage, and long-tail completion. This application categorizes all basic vowel and consonant-related data into the phoneme coverage dimension for statistical analysis of phoneme completeness and frequency distribution. Phonemes are divided into three categories—high-frequency, mid-frequency, and low-frequency—based on their frequency of occurrence in sentences: high-frequency (100 times or more), mid-frequency (30 to 99 times), and low-frequency (less than 30 times). The number of high-frequency phoneme samples is no less than 50, mid-frequency no less than 30, and low-frequency no less than 20, used to match phoneme coverage rates with minimum sample requirements. For example, "a, o, e" appearing 120 times is categorized as high-frequency; "eng" appearing 28 times is categorized as low-frequency.

[0067] The present application classifies 7 types of standard co-articulation data into the co-articulation coverage dimension, and divides priorities according to the degree of influence on lip-sync stability. The high priority includes nasal-vowel, plosive-vowel, fricative / affricate-vowel; the medium priority includes vowel-vowel, consonant-vowel; the low priority includes vowel-consonant, lateral / approximant-vowel. The number of high-priority samples is no less than 50, the medium-priority samples no less than 30, and the low-priority samples no less than 20, which are used to meet the requirements of co-articulation coverage and gap priority. For example, 28 nasal-vowel co-articulation samples belong to the high priority; 35 lateral-vowel samples belong to the low priority.

[0068] The present application classifies four types of emotion annotation data, namely neutral, happy, serious and surprised, into the emotion coverage dimension, and divides them into high-frequency and low-frequency categories according to the occurrence frequency of service sentences. The high-frequency category includes neutral and happy; the low-frequency category includes serious and surprised. The number of high-frequency emotion samples is no less than 30, and the low-frequency samples no less than 20, which are used to meet the requirements of emotion coverage and the intersection of emotion and speech rate. For example, 42 neutral emotion samples and 38 happy samples are classified as high-frequency; 15 serious samples are classified as low-frequency. The present application classifies three levels of speech rate annotation data, namely slow, normal and fast, into the speech rate coverage dimension, and divides them into stable and sub-stable categories according to the daily use frequency. The normal speech rate of 180 characters per minute is the stable level; the slow speech rate of 120 characters per minute and the fast speech rate of 240 characters per minute are the sub-stable levels. The number of samples for each speech rate level is no less than 30, which are used to meet the requirements of speech rate coverage and the intersection of speech rate and emotion. For example, 45 normal speech rate samples and 22 slow speech rate samples are classified as the stable level and the sub-stable level respectively.

[0069] The present application classifies gap data related to low-frequency sounds, low-frequency co-articulation, low-frequency emotion and low-frequency speech rate into the long-tail completion dimension, and divides them into four categories according to the causes of gaps: phoneme long-tail, co-articulation long-tail, emotion long-tail, and speech rate long-tail. The proportion of low-frequency samples is no less than 10%, which is used to meet the requirements of low-frequency distribution and iterative completion. For example, the low-frequency sound "eng" and the low-frequency co-articulation nasal-vowel are both classified into the long-tail completion dimension. The present application adopts the method of unique dimension for single data and precise cross-dimension binding to systematically classify and associate and match the data of the five dimensions. Each original corpus corresponds to one main dimension, and is bound with relevant dimension information at the same time, ensuring that each sentence corresponds to 1 phoneme, 1 to 2 co-articulation labels, 1 emotion label, and 1 speech rate label, so as to avoid cross-dimension misalignment and data confusion. For example, the sentence "nin fang xin" (rest assured) corresponds to high-frequency sound, nasal-vowel co-articulation, neutral emotion, and normal speech rate, with a complete and accurate association relationship.

[0070] Throughout the entire data processing and model building process, nine key parameters are incorporated, including basic phoneme coverage, co-pronunciation coverage, emotion / speed coverage, high and low frequency ratio, minimum sample size, gap priority, iteration rounds, pronunciation legality, and identity consistency. The definitions, value ranges, and judgment rules of each parameter are clearly defined, providing a unified and executable quantitative standard for data screening, copywriting generation, material production, and iterative optimization. This ensures that the entire process is consistent, the results are reproducible, and compliance is achieved.

[0071] Basic phoneme coverage refers to the proportion of basic phoneme types actually covered out of the total number of standard phoneme types. The formula is: Basic Phoneme Coverage = (Number of Phoneme Types Actually Covered ÷ Total Number of Standard Phoneme Types) × 100%. The threshold for achieving this is set at ≥95%, used to determine the completeness of basic articulation unit coverage. For example, if there are 42 standard basic phoneme types and 40 types are actually covered, the basic phoneme coverage rate is (40 ÷ 42) × 100% ≈ 95.2%, which is considered satisfactory. Coarse articulation coverage refers to the proportion of the 7 standard coarse articulation types actually covered out of all coarse articulation types. The formula is: Coarse Articulation Coverage = (Number of Coarse Articulation Types Actually Covered ÷ 7) × 100%. The threshold for achieving this is set at 100%, ensuring that all key coarse articulations are included. For example, if all 7 coarse articulation types are covered, the coarse articulation coverage rate is (7 ÷ 7) × 100% = 100%, which is considered satisfactory. Emotion / Speed ​​Coverage Rate refers to the proportion of actual emotion-speed cross-combinations covered out of the total number of combinations. The calculation formula is: Emotion-Speed ​​Coverage Rate = (Actual Coverage of Cross-Combinations ÷ 12) × 100%. The threshold is set at ≥95% to ensure complete matching of emotion and speed. For example, if 11 out of 12 groups are covered, the Emotion-Speed ​​Coverage Rate = (11 ÷ 12) × 100% ≈ 91.7%, which is considered insufficient and needs to be supplemented.

[0072] The high-low frequency ratio refers to the proportion of low-frequency pronunciation samples to the total number of samples. The calculation formula is: Low-frequency ratio = (Number of low-frequency samples ÷ Total number of samples) × 100%. The threshold is set at ≥10% to avoid insufficient long-tail pronunciation samples. For example, with a total of 1000 samples and 120 low-frequency samples, the low-frequency ratio = (120 ÷ 100) × 100% = 12%, which is considered acceptable. The minimum sample size refers to the minimum number of samples required for a single pronunciation, emotion, or speech rate combination: high-frequency sounds / coordination ≥50 samples, low-frequency sounds / coordination ≥30 samples, and emotion-speech rate combinations ≥25 samples, ensuring sufficient and balanced data samples for each type. For example, the minimum sample size for nasal-vowel coordination is 50 samples, but currently there are only 28, which is considered a gap and needs to be filled.

[0073] Prioritization is divided into three levels based on the impact on lip-pronunciation stability: Level 1 prioritizes nasal-vowel, plosive-vowel, and fricative / affricate-vowel coordination; Level 2 prioritizes core vowels and high-frequency consonants; and Level 3 prioritizes low-frequency emotional nuances and slow speech rate combinations. High-priority gaps are addressed first. For example, a nasal-vowel gap is Level 1 and will be prioritized for generating specific supplementary text. The maximum number of closed-loop iterations is set at 3-5 rounds. Each round completes the entire process of statistics, supplementation, and verification. The process can terminate early if all indicators are met, avoiding ineffective loops. For example, in the current second iteration, if the criteria are not met, execution continues until the fifth iteration, at which point it is forcibly stopped.

[0074] The audio quality and pronunciation accuracy are verified. The audio signal-to-noise ratio (SNR) must be ≥20dB, syllable duration 60-800ms, and there should be no elision / linking abnormalities to ensure clear and readable pronunciation. For example, an audio SNR of 25dB and syllable duration of 200ms are considered valid. Using a benchmark portrait as a reference, facial feature similarity must be ≥0.75, and the average Euclidean distance of key points ≤15 pixels to avoid identity drift and style confusion. For example, a feature similarity of 0.82 and a key point distance of 12 pixels are considered consistent. For example, the following parameters are configured: basic phoneme coverage ≥95%, collaborative coverage 100%, low-frequency proportion ≥10%, minimum of 50 high-frequency collaborative samples, 30 low-frequency collaborative samples, priority given to filling the gap between primary nasal and vowel sounds, 3-5 iterations, pronunciation SNR ≥20dB, and identity similarity ≥0.75. These unified quantitative standards provide clear execution guidelines for data screening, copywriting generation, material production, and iterative optimization, ensuring compliance and achieving the desired results throughout the entire process.

[0075] This dataset integrates phoneme frequency statistics, seven categories of coarse articulation classification data, multi-dimensional coverage targets, a gap list, six types of text batches, multi-emotional and multi-speed TTS audio, baseline portrait frames, phoneme / syllable boundary annotations, quality-screened materials, a coverage matrix, and iteration records to form a digital phonology-driven model dataset. The phoneme frequency statistics include the frequency occurrence and percentage of all basic vowels and consonants in Chinese, clearly defining high-frequency, mid-frequency, and low-frequency phoneme divisions, and counting each phoneme only once within a single sentence to avoid duplicate counting. The seven categories of coarse articulation classification data include the sample quantity and distribution percentage for vowel-vowel, vowel-consonant, consonant-vowel, nasal-vowel, plosive-vowel, fricative / affricate, and lateral / vowel-vowel combinations, sorted by priority.

[0076] The multi-dimensional coverage targets include basic phoneme coverage ≥95%, co-pronunciation coverage 100%, emotional coverage ≥95%, speech rate coverage ≥95%, low-frequency proportion ≥10%, and quantitative indicators such as the minimum sample threshold. A gap list records the unmet phonemes, co-pronunciation, emotional, and speech rate items, labeling the gap type, gap size, and priority. The gap size is calculated as minimum sample size - current sample size. Six batches of copywriting are included, covering high-frequency core texts, co-pronunciation specific texts, long-tail completion texts, emotional expression texts, speech rate variation texts, and scenario-based FAQ texts. Each sentence consists of 5-15 characters, covering 2-4 gaps, and conforming to scenario and phonics rules.

[0077] This is a multi-emotional, multi-speed TTS audio recording, including neutral, happy, serious, and surprised emotions. Speech speeds are 120 words / minute (slow), 180 words / minute (normal), and 240 words / minute (fast). The sampling rate is 16kHz, 16-bit mono, and pronunciation is error-free. A baseline portrait frame is used, with a uniform 512×512 RGB image, consistent across the same identity and batch, and consistent lighting and posture to avoid stylistic inconsistencies. Phoneme and syllable boundary annotations are provided, including phoneme-level and syllable-level timestamps, with alignment confidence ≥0.6 and boundary error ≤20ms, ensuring accurate and misaligned annotations.

[0078] The materials undergo quality screening, removing anomalous samples with a signal-to-noise ratio <20dB, syllable duration <60ms or >800ms, alignment confidence <0.6, and identity similarity <0.75. All samples are considered compliant and valid. A coverage matrix is ​​created, with horizontal columns for phonemes, coordination, emotion, and speech rate, and vertical columns for sample count, coverage rate, gap value, and priority. The data is authentic and unaltered. Iteration logs record the completion, verification, and update processes for 3-5 iterations, including iteration rounds, coverage changes, and gap updates. For example, the dataset includes a summary of customer service scenario phoneme frequency tables (680 times for the 'a' sound and 28 times for the 'eng' sound), 7 types of coordination details (28 nasal-vowel sounds), coverage target files, gap lists, six types of text, multi-emotional speech rate audio, unified baseline portraits, annotation files, compliant materials, a coverage matrix, and iteration logs. These are merged and organized in a unified format to form a standardized lip-sync driven dataset. The data is complete, consistent in definition, and directly usable for model training and codebook construction.

[0079] S105 combines driving effect feedback learning, model generalization optimization, and coverage index regression iteration to complete dataset adaptive optimization, and outputs information on the construction and optimization of mouth shape driven model data based on co-pronunciation coverage.

[0080] In one implementation, combining the needs of coverage metric regression iteration, material closed-loop completion, and coverage matrix verification, a mechanism is introduced that includes coverage statistics, gap identification, text completion, material updates, and iteration termination. This mechanism defines coverage targets, gap priorities, completion rules, and iteration boundaries. By combining the needs of coverage metric regression iteration, material closed-loop completion, and coverage matrix verification, five core mechanisms are introduced: coverage statistics, gap identification, text completion, material updates, and iteration termination. This clarifies coverage targets, gap priorities, completion rules, and iteration boundaries, achieving closed-loop control throughout the entire material production process and ensuring coverage targets are met and data is balanced.

[0081] Regarding the coverage statistics mechanism, coverage data for four categories—basic phonemes, seven types of co-pronunciation, emotion, and speech rate—are aggregated round by round. Coverage rates are calculated according to standard criteria, generating a coverage report for each round, which serves as the basis for gap identification and iterative decision-making. The calculation formulas are: Basic phoneme coverage rate = (Number of covered phonemes ÷ Total number of phonemes) × 100%; Co-pronunciation coverage rate = (Number of covered types ÷ 7) × 100%; Emotion and speech rate coverage rate = (Number of filled cross-cells ÷ 12) × 100%. By comparing the coverage rate of the current round with the target threshold, items that fail to meet the standards are identified. Based on their impact on lip-sync stability, gaps are prioritized into three levels to accurately pinpoint weaknesses. The judgment rules are as follows: Level 1 priority: nasal-vowel, plosive-vowel, fricative / affricate coordination (samples < 50); Level 2 priority: core high-frequency sounds (samples < 50); Level 3 priority: low-frequency emotion, slow speech speed combination (samples < 30). Following the rules of gap priority, speech legality, scenario adaptation, brevity and efficiency, and unambiguity, a multi-gap fusion strategy is used to generate copy, with each sentence covering 2-4 gaps to maximize completion efficiency. Constraints are as follows: 5-15 characters per sentence, 3-8 syllables; suitable for customer service / broadcast scenarios; conforms to Chinese spelling rules, and contains no uncommon characters.

[0082] After generating the text, multi-emotion, multi-speed TTS audio is batch-synthesized, and lip-sync materials are generated by binding them to a unified benchmark portrait. Quality screening is then performed to remove materials with abnormal pronunciation, boundaries, or identities, and compliant data is added to the material library. The screening criteria are as follows: audio signal-to-noise ratio ≥20dB, syllable duration 60-800ms; phoneme alignment confidence ≥0.6; identity similarity ≥0.75.

[0083] Set dual termination conditions to balance coverage achievement and iteration efficiency. Termination rules are as follows: Automatic termination: Basic sound coverage ≥ 95%, collaborative coverage = 100%, emotional speech speed coverage ≥ 95%, low-frequency proportion ≥ 10%, all indicators meet the criteria. Forced termination: Maximum iteration rounds 3-5 to avoid infinite loops. For example, in the first iteration of a customer service scenario, the statistics show a basic sound coverage of 92%, nasal-vowel collaborative coverage of 56%, serious-slow speech speed coverage of 83%, and low-frequency proportion of 8%; nasal-vowel is identified as a primary gap, and basic sound as a secondary gap; multiple gap texts such as "Goodnight," "Peace of mind," and "I love you" are generated; neutral / serious and normal / slow speech audio is synthesized and materials are generated, with anomalies removed and supplemented; if the criteria are not met, the second round begins, and iteration terminates after 3-5 rounds or when all indicators meet the criteria.

[0084] Extracting phoneme, coprolalia, emotion, and speech rate coverage data, and combining this with material optimization needs, a gap-filling, coverage verification, and closed-loop iteration strategy is introduced to determine the gap-filling direction, verification dimensions, and iteration scope. Coverage data extraction refers to obtaining four types of coverage information: phonemes, coprolalia, emotion, and speech rate. Specifically, this includes: the number of samples for each type of pronunciation, coverage rate, high and low frequency distribution, and the proportion of cross-combinations. For example, 42 basic Chinese sounds, 7 types of coprolalia, 4 types of emotion, and 3 speech rate levels can be extracted, such as 28 nasal-vowel samples with a coverage rate of 56%. Material optimization needs to focus on three core needs: stable mouth shape, balanced coverage, and long-tail gap filling. Priority is given to filling easily jittery coprolalia, balancing high and low frequency distribution, and verifying full-dimensional coverage. For example, priority is given to filling high-risk coprolalia such as nasal-vowel and plosive-vowel; setting a low-frequency sample ratio of ≥10% as an optimization target. Gap filling adopts a hierarchical priority strategy. Priority is given to filling easily jittery coprolalia, then core sounds, and finally long-tail emotion / speech rate data. For example, Level 1 includes nasal-vowel, plosive-vowel, and fricative / affricate sounds (≥50 samples); Level 2 includes high-frequency vowels and consonants (≥50 samples); and Level 3 includes low-frequency emotion and speech rate (≥30 samples).

[0085] Coverage verification must include four dimensions: phonemes, coordination, emotion, and speech rate. Each dimension—coverage rate, sample size, and distribution balance—must be verified individually. Phoneme coverage ≥ 95%, coordination coverage = 100%, emotion / speech rate coverage ≥ 95%, and low-frequency proportion ≥ 10%. A closed-loop iteration with clearly defined boundaries is required. The number of iterations should be controlled to 3-5 rounds. Each round completes statistics, supplementation, and verification; the process terminates when all indicators meet the standards. For example, if the first round has a low-frequency proportion of 8% and a nasal tone coverage rate of 56%, after supplementation, the second round begins, stopping when the standard is met or after 5 rounds.

[0086] Based on the goal of balanced pronunciation coverage, five processing intervals are defined: coverage statistics, gap identification, text completion, material generation, and iterative optimization. The coverage statistics interval is used to summarize coverage data for four categories: phonemes, co-pronunciation, emotion, and speech rate, forming a complete coverage table. The statistics include the number of samples for each pronunciation category, coverage rate, and the proportion of high and low frequencies. The coverage rate is calculated using the formula: Coverage Rate = (Number of valid samples ÷ Minimum sample threshold) × 100%. For example, the statistics yielded 28 nasal-vowel co-pronunciation samples, with a coverage rate of 56% and a low-frequency sample proportion of 8%.

[0087] The gap identification interval is used to compare statistical results with preset coverage targets, identify items that fail to meet the standards, and prioritize them. The judgment criteria are: if the coverage rate does not meet the standard or the sample is insufficient, it is judged as a gap. Gap priorities are divided into three levels: Level 1 is nasal-vowel and plosive-vowel; Level 2 is core sound; and Level 3 is emotional / speech rate combination. For example, if the nasal-vowel coverage rate is 56%, it does not reach the 100% standard and is judged as a Level 1 gap. The copy completion interval is used to generate compliant complete copy according to the gap priority. The copy must meet the following requirements: cover 2-4 gaps per sentence, have 5-15 words, and fit the business scenario. Copy templates include phrase type, short sentence type, FAQ type, and continuous coverage type. For example, for the nasal-vowel gap, generate short sentences such as "Good night" and "Rest assured".

[0088] The material generation interval is used to generate lip-sync materials with multiple emotions and speaking speeds. Generation conditions include: a unified baseline portrait, an audio sampling rate of 16kHz, and labeled phonemes and syllable boundaries. Quality standards include: audio signal-to-noise ratio ≥20dB and identity similarity ≥0.75. For example, generating a neutral, normal-speed speech video requires an audio signal-to-noise ratio of 25dB and an identity similarity of 0.82. The iterative optimization interval is used to update the coverage status in a closed loop until all dimensions meet the standards. The iteration rule is: a maximum of 3-5 iterations, ending when all indicators meet the standards. Updated content includes: coverage matrix, gap list, and text batches. For example, after the first round of supplementation, the nasal sound coverage rate increases to 92%, entering the second iteration.

[0089] Four types of completion logic are defined: phoneme, co-pronunciation, emotion, and speech rate, clarifying the mapping relationship between gaps and text. The phoneme completion logic mainly generates corresponding text for low-frequency phoneme gaps, prioritizing the coverage of single-phoneme units with insufficient samples. The specific rule is: for low-frequency phonemes with fewer than 30 samples, short sentences composed of commonly used characters are generated, with each sentence focusing on 1-2 target low-frequency phonemes, and the sentence length controlled to 5-8 characters to ensure natural readability. For example, if there are only 22 samples of the low-frequency sound "eng", the short sentence "The wind is light, the lights are bright" is generated to focus on completing the "eng" sound gap.

[0090] The coarticulation completion logic focuses on connecting multiple missing coarticulation combinations, prioritizing the coverage of high-priority coarticulations that are prone to jitter. The specific rules are: for coarticulations such as nasal-vowel and stop-vowel with less than 50 samples, generate sentences containing 2 to 3 missing coarticulations, with a sentence length of 5 to 15 characters, to ensure natural contextual connection. For example, with two missing coarticulations "n-i, ang-ao", the sentence "nin an hao, xiang yong bao" (meaning "you are well, want to hug") is generated, which covers both coarticulation gaps in one sentence.

[0091] The emotion completion logic is used to fill in the blank texts of low-frequency emotions and match the corresponding emotional tones. The specific rules are: for low-frequency emotions such as serious and surprise (with less than 20 samples), generate sentences that fit the emotional expression, with a sentence length of 6 to 12 characters, to adapt to the emotional semantics. For example, there are 15 serious emotion samples, and the texts "please confirm the order, do not operate in violation of regulations" are generated to fill the gap of the serious emotion.

[0092] The speech rate completion logic is used to improve the special materials for slow speed and fast speed, and match the corresponding speech rate rhythm. The specific rules are: the slow speed (120 characters per minute) generates soothing short sentences, and the fast speed (240 characters per minute) generates compact short sentences, with no less than 30 samples for each type of speech rate. For example, there are 25 slow speed samples, and the short sentence "speak slowly, don't worry" is generated; there are 28 fast speed samples, and the short sentence "please handle it as soon as possible, reply in time" is generated.

[0093] The mapping between gaps and texts follows the rule of "one type of gap corresponds to one type of template, and multiple gaps can be integrated". Specifically: a single phoneme gap matches a phrase template; a single coarticulation gap matches a short sentence template; an emotion / speech rate gap matches a scenario template; multiple gaps match a continuous coverage template, so as to ensure that the gap type and the text format correspond accurately. For example, when there are three gaps of the "eng" phoneme, nasal-vowel, and slow speed at the same time, the sentence "evening breeze is light, wait slowly" is generated, which maps and fills the three gaps at one time.

[0094] According to the coverage compliance constraint, three types of processing rules are set: gap priority, sample lower limit, and iterative loop, to limit the optimization scope. To ensure that the material optimization process is compliant, controllable and implementable, strictly follow the coverage compliance constraint, and clarify the three processing rules of gap priority, sample lower limit, and iterative loop, so as to limit the overall optimization scope, avoid problems such as disorderly expansion, invalid iteration or insufficient samples, and ensure that the final material set reaches the standard stably and is balanced and usable.

[0095] The gap priority rule prioritizes gaps based on their impact on lip-sync stability, addressing high-impact gaps first and avoiding prioritizing resources for less important gaps. Specifically, gaps are categorized into three priority levels: first priority gaps are nasal-vowel, plosive-vowel, and fricative / affricate consonant co-occurrence gaps; second priority gaps are core high-frequency sounds; and third priority gaps are low-frequency emotional and slow-speed speech gaps. For example, 28 nasal-vowel samples, below the 50-sample threshold, are classified as first priority and prioritized for generating complete text; 15 samples of serious emotional content, below the 20-sample threshold, are classified as third priority and processed later.

[0096] The sample lower limit rule specifies the minimum number of samples required for each combination of pronunciation, emotion, and speech rate to avoid insufficient coverage due to sparse samples. Specifically, high-frequency co-occurrence and core sound samples must be ≥50; low-frequency co-occurrence and normal sound samples must be ≥30; and emotion-speech rate cross-samples must be ≥25. For example, 35 plosive-vowel samples, below the 50-sample lower limit, are considered insufficient; while 32 neutral-normal speech rate samples, meeting the 25-sample lower limit, are considered sufficient.

[0097] The iterative loop rule refers to setting start and stop conditions for iteration. It terminates when the criteria are met and loops to complete the iteration if the criteria are not met, avoiding infinite iteration or premature termination. Specifically, the iteration terminates when all criteria are met: basic sound coverage ≥95%, collaborative coverage = 100%, and low-frequency proportion ≥10%. The maximum number of iterations is limited to 3-5 rounds; it continues if the criteria are not met, and stops when a full round is completed. For example, if the first round has a low-frequency proportion of 8% and a nasal sound coverage of 56%, it proceeds to the second round; if all criteria are met in the third round, the iteration automatically terminates.

[0098] Based on a closed-loop coverage logic, statistics, supplementation, and verification are performed round by round to generate compliant lip-syncing material data. The closed-loop coverage logic refers to using a coverage matrix as a benchmark, conducting statistics, supplementation, and verification round by round according to a fixed process, iterating until full-dimensional coverage is achieved, and outputting compliant lip-syncing material. For the round-by-round statistics stage, each round first summarizes the phoneme, coordination, emotion, and speech rate data of the current material, and calculates the coverage rate and gap according to standard criteria. The calculation formulas are: Coverage rate = (Number of effective samples ÷ Minimum sample threshold) × 100%; Gap = Minimum sample threshold - Current number of samples. For example, in the first round of statistics, there are 28 nasal-vowel samples, the minimum threshold is 50, the coverage rate is 56%, and there is a gap of 22.

[0099] For the round-by-round completion phase, supplementary text and materials are generated based on the priority of the gaps. Priority: Level 1 (nasal-vowel, plosive-vowel), Level 2 (core sound), Level 3 (emotion / speech rate). For example, the first round prioritizes completing the nasal-vowel gap, generating text such as "Goodnight, rest assured," synthesizing neutral, normal-speed audio, and binding it to a baseline portrait to generate materials.

[0100] For each round of verification, checks are conducted on two aspects: coverage metrics and material quality. Coverage verification: basic sounds ≥95%, coordination = 100%, emotional speech rate ≥95%, low frequency ≥10%; Quality verification: signal-to-noise ratio ≥20dB, alignment confidence ≥0.6, identity similarity ≥0.75. For example, if the verification material has a signal-to-noise ratio of 25dB and a coverage rate of 92%, it fails to meet the standards and proceeds to the next round. Regarding the iteration termination rule, a maximum of 3-5 rounds are set, and the iteration stops once all metrics are met. For example, in the third round, if the nasal sound coverage rate is 100% and the low frequency percentage is 12%, all metrics are met, the iteration terminates, and compliant material is output.

[0101] like Figure 2 As shown, a lip-shape-driven model data construction and optimization system based on co-pronunciation coverage includes: The coverage feature extraction module 201 is used to extract multi-dimensional coverage features through collaborative pronunciation association modeling, pronunciation pattern classification learning, and emotion-speed joint representation; and incorporates coverage constraints, sample balancing rules, and long-tail completion strategies to construct a lip-shaped driven pattern learning dataset. The feature association construction module 202 is used to construct feature association representations based on multi-dimensional feature space mapping, association pattern clustering, and coverage distribution learning; and to generate a co-pronunciation coverage pattern map through combinatorial optimization learning, greedy policy decision-making, and multi-objective constraint optimization. The gap feature modeling module 203 is used to complete feature modeling through sparse feature completion, abnormal pattern detection, and long-tail prediction learning; and to generate a gap pattern vector through high-dimensional embedding representation and feature vector learning. The sample completion and denoising module 204 is used to input the pattern vector into the adaptive generation model and reinforcement learning engine; relying on multi-constraint pattern coverage and context association learning, it completes sample completion and denoising, and constructs a covered pattern learning dataset. The dataset adaptive optimization module 205 is used to combine driving effect feedback, model generalization optimization, and coverage index regression iteration to complete the dynamic optimization of the dataset; it outputs information on the construction and optimization of the mouth shape driven model based on co-pronunciation coverage.

[0102] A computing device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute any of the methods for constructing and optimizing lip-synchrotron model data based on coarse articulation coverage.

[0103] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0104] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0105] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application.

Claims

1. A method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage, characterized in that, include: Coverage feature extraction is completed through collaborative pronunciation association modeling, pronunciation pattern classification learning, and emotion-speed joint representation. Coverage constraints, sample balancing rules, and long-tail completion strategies are incorporated into the machine learning training paradigm to construct a lip-shape-driven pattern learning dataset. Based on multidimensional feature space mapping, association pattern clustering, and coverage distribution learning, feature association representation is constructed. Through combinatorial optimization learning, greedy strategy decision-making, and multi-objective constraint optimization, a collaborative pronunciation coverage pattern map is formed. Feature modeling is completed through sparse feature completion model, abnormal pattern detection, and long-tail prediction learning. Coverage gap pattern vectors containing gap type, sample gap quantity and priority are generated through high-dimensional embedding representation and feature vector learning. The pattern vectors are input into the adaptive generative learning model and the reinforcement learning decision optimization engine. The lip shape driven sample filling and noise reduction corresponding to the coverage gap are completed by multi-constraint pattern coverage and context association learning, and a lip shape driven model dataset based on coverage pattern learning is constructed. By combining driving effect feedback learning, model generalization optimization, and coverage index regression iteration, the dataset is adaptively optimized, and the data construction and optimization information of the mouth shape driven model based on co-pronunciation coverage is output.

2. The method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage according to claim 1, characterized in that, Coverage feature extraction is achieved through collaborative pronunciation association modeling, pronunciation pattern classification learning, and emotion-rate joint representation. Coverage constraints, sample balancing rules, and long-tail completion strategies are incorporated into the machine learning training paradigm to construct a lip-shape-driven pattern learning dataset, including: Based on co-pronunciation association modeling, pronunciation pattern classification learning, and emotion-speed joint representation, the coverage feature extraction is completed, generating multi-dimensional coverage feature data covering phoneme sequences, co-pronunciation types, emotion labels, and speed labels. The associated data includes target scene corpus, phoneme frequency statistics, co-pronunciation statistics, and emotion and speed configuration data. Coverage constraint determination is carried out on multi-dimensional coverage feature data to generate six types of copywriting batches: high-frequency core phonemes, co-pronunciation specific, long-tail phonemes, emotional expression, speech rate change, and scenario FAQ, and to clarify four types of coverage objectives: phoneme coverage, co-pronunciation coverage, emotional coverage, and speech rate coverage. Based on coverage targets and copy batches, a material generation and quality screening model is constructed to generate three core coverage indicators: basic phoneme coverage, co-pronunciation type coverage, and high / mid / low frequency sample ratio. Simultaneously, the material pronunciation clarity, boundary labeling accuracy, and identity consistency are verified to match the high coverage production needs of digital phonology materials. Integrate coverage features, copy batches, and coverage indicator data to generate a coverage matrix, supporting the closed-loop iterative completion of the digital demographic material library.

3. The method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage according to claim 1, characterized in that, Feature association representations are constructed based on multidimensional feature space mapping, association pattern clustering, and coverage distribution learning. A collaborative pronunciation coverage pattern graph is formed through combinatorial optimization learning, greedy policy decision-making, and multi-objective constraint optimization, including: Combining the requirements of collaborative pronunciation coverage analysis with the logic of multidimensional feature association construction, feature space mapping, association pattern clustering, and coverage distribution learning mechanisms are introduced to determine standardized processing schemes for feature mapping, pattern clustering, coverage analysis, and pattern graph generation. Spatial mapping is performed on multi-dimensional features such as phonemes, coarticulation, emotion, and speech rate to construct a high-dimensional feature space representation; Based on the association pattern clustering algorithm, the mapping features are grouped, and the coverage distribution is combined to learn and quantify the proportion and gap of various pronunciation patterns. By integrating feature mapping results, clustering patterns, and coverage data, a collaborative pronunciation coverage pattern diagram is generated through combinatorial optimization learning, greedy strategy decision-making, and multi-objective constraint optimization.

4. The method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage according to claim 1, characterized in that, Feature modeling is achieved through sparse feature completion, abnormal pattern detection, and long-tail prediction learning. Covering gap pattern vectors are generated via high-dimensional embedding representation and feature vector learning, including: Based on the technology chain of sparse feature completion, abnormal pattern detection, and long-tail prediction learning, feature modeling is carried out on multi-dimensional features such as phonemes, co-pronunciation, emotion, and speech rate to generate feature modeling data with complete features, standardized patterns, and matching dimensions. By integrating core elements such as sparse completion, anomaly detection, long-tail prediction, high-dimensional embedding, and feature vectors, and matching the goals of feature integrity, pattern discriminability, and dimensionality adaptability, feature modeling constraint information is formed. Based on the sparse feature completion model, the abnormal pattern detection mechanism, the long-tail prediction learning strategy, the high-dimensional embedding representation method, and the feature vector learning algorithm, sparse feature completion, abnormal pattern removal, long-tail trend capture, high-dimensional feature mapping, and feature vector learning are completed to form highly reliable feature optimization information. By integrating information from completion, detection, prediction, embedding, and vectors, a streamlined process is implemented to complete sparse features, detect abnormal patterns, learn long-tail predictions, perform high-dimensional embedding representations, and learn feature vectors, generating coverage gap pattern vectors adapted to digital population data.

5. The method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage according to claim 1, characterized in that, The pattern vectors are input into the adaptive generative learning model and the reinforcement learning decision optimization engine. Sample completion and noise reduction are achieved through multi-constraint pattern coverage and contextual association learning. This constructs a lip-sync driven model dataset based on pattern coverage learning, including: Based on target industry corpus, phoneme sequence, co-pronunciation type, phoneme frequency, sentiment annotation, speech rate annotation, coverage target, gap list recording non-compliant pronunciation items and gap type, gap quantity priority, and nine core input data of benchmark human image, the system performs phoneme statistics, co-pronunciation classification, coverage target definition, gap recognition, text generation, TTS synthesis, human image frame matching, alignment annotation, and quality screening processing. The input data is categorized, organized, and matched according to five dimensions: phoneme coverage, coprophony coverage, emotion coverage, speech rate coverage, and long-tail completion. The parameters to be included are: basic phoneme coverage, coprophony coverage, emotion coverage, speech rate coverage, high and low frequency ratio, minimum number of samples, gap priority, iteration rounds, pronunciation legality, and identity consistency. By integrating phoneme frequency statistics, 7 types of co-pronunciation classification data, multi-dimensional coverage targets, gap lists, 6 types of text batches, multi-emotion and multi-speed TTS audio, benchmark portrait frames, phoneme / syllable boundary annotations, quality-screened materials, coverage matrices, and iterative records, a digital morphology-driven model dataset is formed.

6. The method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage according to claim 5, characterized in that, By combining driving effect feedback learning, model generalization optimization, and coverage metric regression iteration, adaptive optimization of the dataset is completed, outputting data construction and optimization information for the lip shape-driven model based on co-pronunciation coverage, including: Combining the needs of coverage metric regression iteration, material closed-loop completion, and coverage matrix verification, we introduce coverage statistics, gap identification, text completion, material update, and iteration termination mechanisms to determine coverage targets, gap priorities, completion rules, and iteration boundaries. Extract phoneme coverage data, co-pronunciation coverage data, emotion coverage data, and speech rate coverage data. Combine these with material optimization requirements and introduce gap filling, coverage verification, and closed-loop iteration strategies to determine the filling direction, verification dimensions, and iteration scope. Based on the goal of balanced pronunciation coverage, five processing intervals are defined: coverage statistics, gap identification, text completion based on gap priority, material generation, and iterative optimization. Define four categories of fill-in logic: phonemes, coprophony, emotion, and speech rate, and clarify the mapping relationship between gaps and copywriting; Based on the coverage achievement constraint, three types of processing rules are set: gap priority, sample lower limit, and iterative loop, to limit the optimization scope; Based on the closed-loop logic of coverage, statistics are collected, supplemented, and verified round by round to generate material data that meets the coverage standards and can be used for lip-syncing.

7. A system for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage, characterized in that, The system is used to execute executable instructions to perform the method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, include: First processor; And a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the method for constructing and optimizing lip-shape driven model data based on co-pronunciation coverage as described in any one of claims 1 to 6 by executing the executable instructions.

9. A computing device, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein, When the computer program instructions are executed by the processor, the device is triggered to execute the lip-shape driving model data construction and optimization method based on co-pronunciation coverage as described in any one of claims 1 to 6.

Images